Energy efficiency (EE) is a critical performance indicator for the device-to-device (D2D) communication underlaying cellular networks due to limited battery capacity and serious interference between user equipment. In this study, we proposed a power control and channel allocation scheme for the EE maximization of the D2D pairs, while jointly reusing uplink–downlink resources and guaranteeing the cellular users’ (CUs) quality of service (QoS). The formulated problem was a mixed-integer nonlinear programming (MINLP) problem, which is generally an unsolved non-deterministic polynomial-time hardness (NP-hard) problem within polynomial time. To make it tractable to solve, the original problem was divided into two sub-problems: power control and channel allocation. A power control algorithm based on the Lambert W function was proposed to maximize the EE of the individual D2D pair. Assigning either an uplink or downlink resource to reuse, the EE of each D2D pair was calculated using the power control results. A channel allocation scheme based on the Kuhn–Munkres algorithm utilized the EE weights to optimize the overall EE of the D2D pairs. The simulation results verified the theoretical analysis and proved that the proposed algorithm could remarkably improve the EE of D2D pairs while guaranteeing the QoS of the CUs.