Device-to-Device (D2D) communication has become a promising and new paradigm for enhancing network performance in cellular networks. D2D communication enables users to communicate directly without passing through the base station, thereby improving spectral efficiency and reducing communication delay. However, due to the intertwined interference environment, the shared spectrum and reused frequency may limit the network performance. In this paper, We propose a Proximal Policy Optimisation (PPO) algorithm based on Markov Decision Process (MDP) to optimise resource allocation and improve energy efficiency. Resource allocation and power control are jointly considered with the aim of maximising the overall throughput of the network while guaranteeing the minimum requirement of Quality of Service (QoS). Extensive simulation experiments are conducted to validate the efficacy of our proposed scheme. The results demonstrate that our method outperforms the traditional method in terms of energy efficiency and training time.