Device-to-device communication is envisioning next-generation wireless communication. The utility of the device-to-device communication model encourages emerging communication systems such as 5G and the Internet of Things. The allocation of resources and channel interference are major challenges in device-to-device communication. This paper proposes machine-learning-based algorithms for resource allocation and optimization of device-to-device communication. The proposed machine learning algorithm is a cascaded support vector machine. The cascaded support vector machine mapped the parameters of CUEs and DUEs. We create an iterative algorithm to achieve low-power, energy-efficient resource allocation with mode selection by formulating a novel optimization problem to maximize energy efficiency using the subtractive form method to solve a fractional objective function. We obtain data samples from a suboptimal algorithm to train the cascaded algorithm and verify the trained algorithm. Our numerical results show that the proposed cascaded machine learning-based transmission algorithm's accuracy reaches about 88%–95% despite its simple structure due to the limitation in computing power. The analysis of results suggests that the proposed algorithm is more efficient than SVM, DSVM, and the reinforcement learning (RL) algorithm.