Optimal Power Control Problem Research Articles

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.

Secrecy Rate Maximization in THz-Aided Heterogeneous Networks: A Deep Reinforcement Learning Approach

Densely deployed femtocells in heterogeneous networks (HetNets) improve the quality-of-service (QoS) and quality-of-experience (QoE) for both next-generation networks and end users, respectively. However, users near the femtocell edge, i.e., femto edge users (FEUs), which are far away from the Femto base station (FBS) may receive a low signal-to-interference-plus-noise ratio (SINR). Moreover, due to the distributed and dynamic nature of HetNets, femtocells may not resist various eavesdropping attacks from different attackers. Thus, it is essential to ensure the wireless channel's secrecy through which FEUs communicate with FBS since they may face eavesdropping attacks and may have low SINR and achievable data rates. Aiming to enhance the secrecy rate of femtocells, in this article, we formulate a joint optimization problem of beamforming vectors, artificial noise (AN), and power control for terahertz (THz)-enabled femtocells. Addressing this non-convex optimization problem is challenging due to the system's complexity and dynamic nature of FEUs. Thus, we translate the optimization problem into a multi-agent reinforcement learning (MARL) problem using the Markov decision process (MDP). Then, to solve the MDP with continuous action space, two deep reinforcement learning (DRL) based schemes, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Deterministic Policy Gradient-based Secrecy Maximization (DDPG-SM)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Asynchronous-Advantage-Actor-Critic based Secrecy Maximization (A3C-SM)</i> are proposed to maximize the secrecy rate of femtocells. The secrecy rate performance of the proposed schemes is compared and analyzed with Advantage-Actor-Critic (A2C) and state-of-the-art Perfect Secrecy Rate Maximization (PSRM) schemes. Simulation results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> schemes achieve 37.73% and 22.64% better average secrecy rate in comparison to the PSRM scheme. Also, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> scheme outperforms all other <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> , A2C, and PSRM schemes in terms of average secrecy rate, received SINR, and energy-efficiency of femtocells.

Epsilon-Greedy-Based MQTT QoS Mode Selection and Power Control Algorithm for Power Distribution IoT

Employing message queuing telemetry transport (MQTT) in the power distribution internet of things (PD-IoT) can meet the demands of reliable data transmission while significantly reducing energy consumption through the dynamic and flexible selection of three different quality of service (QoS) modes and power control. However, there are still some challenges, including incomplete information, coupling of optimization variables, and dynamic tradeoff between packet-loss ratio and energy consumption. In this paper, the authors propose a joint optimization algorithm named EMMA for MQTT QoS mode selection and power control based on the epsilon-greedy algorithm. Firstly, the joint optimization problem of MQTT QoS mode selection and power control is modeled as a multi-armed bandit (MAB) problem. Secondly, the authors leverage the online learning capability of the epsilon-greedy algorithm to achieve joint optimization of MQTT QoS mode selection and power control. Finally, they verify the superior performance of the proposed algorithm through simulations.

SecBoost: Secrecy-Aware Deep Reinforcement Learning Based Energy-Efficient Scheme for 5G HetNets

In this paper, we propose a secrecy-aware energy-efficient scheme for a two-tier heterogeneous network (HetNet), consisting of a sub-6 GHz macrocell and multiple millimeter wave (mmWave) picocells. Each picocell is assumed to have several users and an eavesdropper (Eve) which intercepts the signal of the picocell users. In the proposed scheme, firstly, to maximize the secrecy energy-efficiency (SEE) of picocell users, a joint optimization problem of power control, channel allocation, and beamforming is formulated by considering the minimum secrecy rate and signal-to-interference-plus-noise ratio (SINR) constraints. Due to the non-convex nature of the aforementioned optimization problem in a highly dynamic HetNet environment, we transform it into a reinforcement learning (RL) problem using the Markov decision process (MDP). Then, a multi-agent reinforcement learning (MARL) technique is used to obtain the maximum long-term reward. Moreover, we propose a multi-agent cooperative deep reinforcement learning (DRL) scheme known as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SecBoost</i> to solve the MDP with large number of action and state spaces. It uses the dueling and double-Q architecture of dueling double deep Q-network (D3QN) to optimize power control, channel allocation, and beamforming vectors to maximize the SEE of picocells. Also, prioritized experience replay is used to increase the sampling efficiency of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SecBoost</i> . The SEE performance of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SecBoost</i> is compared with MARL, multi-agent deep Q-network (MA-DQN), state-of-the-art joint beamforming based secrecy energy efficiency maximization (JBF-SEEM) scheme, and one-time pad based encrypted data transmission (O-EDT). Simulation results demonstrated that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SecBoost</i> scheme achieves 14.7%, 8.33%, 30%, and 69% better average SEE in comparison to MARL, MA-DQN, JBF-SEEM, and O-EDT schemes, respectively, which reveals its effectiveness in improving SEE of picocells.

Concurrent Transmission and Multiuser Detection of LoRa Signals

This article investigates a new model to improve the scalability of low-power long-range (LoRa) networks by allowing a group of multiple end devices (EDs) to communicate with multiple multi-antenna gateways simultaneously (i.e., in the same time slot) on the same frequency band and using the same spreading factor. The maximum-likelihood (ML) decision rule is first derived for noncoherent detection of information bits transmitted by multiple devices in a group. To overcome the high complexity of the ML detection, we propose a suboptimal two-stage detection algorithm to balance the computational complexity and error performance. In the first stage, we identify transmitted chirps (without knowing which EDs transmit them). In the second stage, we determine the EDs that transmit the specific chirps identified from the first stage. To improve the detection performance in the second stage, we also optimize the transmit powers of EDs to minimize the similarity, measured by the Jaccard coefficient, between the received powers of any pair of EDs in the same group. As the power control optimization problem is nonconvex, we use concepts from successive convex approximation to transform it to an approximate convex optimization problem that can be solved iteratively and guaranteed to reach a suboptimal solution. Simulation results demonstrate and justify the tradeoff between transmit power penalties and network scalability of the proposed LoRa network model. In particular, by grouping two or three EDs in each group for concurrent transmission, the uplink capacity of the proposed network can be doubled or tripled over that of a conventional LoRa network, albeit at the expense of additional 3.0 or 4.7 dB transmit power.

Distributed Data Offloading in Ultra-Dense LEO Satellite Networks: A Stackelberg Mean-Field Game Approach

The low earth orbit (LEO) satellite network is regarded as a promising technology for delivering seamless service to remote areas such as rural areas. In this paper, we consider a dynamic data offloading problem in ultra-dense LEO satellites networks with large-scale ground users, where each ground user makes the distributed offloading decision based on its state information, the influence from other ground users, and the fees paid to satellites. To investigate the interaction problem between ground users and satellites, we formulate the problem as a Stackelberg game. Specifically, the satellites serve as the leaders, who decide the data service price at each time slot. The ground users, on the contrary, are the followers who decide the power control based on their states, the influence from other ground users and the fees paid to satellites. Since the influence is difficult to estimate due to the large number of ground users, we employ the mean field game algorithm to transform the influence from others and satellites into the mean field term, and reformulate the optimization problem as a Stackelberg mean field game (SMFG). Each ground user makes the data offloading decision by learning the future impact of the whole network distributively. For ground users, we solve the power control optimization problem by utilizing the G-prox primal-dual hybrid gradient (PDHG) algorithm, where the Fokker-Planck-Kolmogorov (FPK) equation is converted into a linear form via the Taylor expansion. For satellites, we address the service pricing optimization problem by using the adjoint algorithm. Finally, the numerical results demonstrate the effectiveness of the proposed algorithm.

Uplink power control scheme for spectral efficiency maximization in NOMA systems

Non-orthogonal multiple access (NOMA) has recently gained much research attention as a possible multiple access solution for fifth-generation (5G) wireless communication. The NOMA can further enhance spectral performance compared with orthogonal multiple access (OMA). This paper focuses on the joint optimization problem of power control and user grouping by considering the uplink NOMA system. The power control issues are solved using a linear programming approach and three alternative algorithms solve user grouping: Randomly, 2-Opt, and the Hybrid. The numerical results show that the proposed solution for power control and user-grouping achieves significant spectral performance compared to OMA and existing schemes reported in the literature with lower system complexity for uplink NOMA systems.

Robust Design of Multicell D2D Communication Under Partial CSI

We consider device-to-device (D2D) communication underlaid in a cellular network to share the uplink resource of cellular users (CUs). It is a key the emerging Internet of Things to support vehicle-to-everything communication networks. In a multicell scenario, both D2D pairs and CUs may cause significant significant intercell interference (ICI) to the neighboring cells. Furthermore, due to substantial signaling overhead, we assume only partial channel state information (CSI) of D2D links at the base station. We consider joint power control, beamforming, and CU-D2D matching problem, assuming partial CSI from D2D pairs under the general Nakagami fading model. We formulate a joint receive beamforming and robust power control optimization problem for a CU-D2D pair to expected sum rate under the power budget, while meeting the minimum SINR requirements and worst case ICI limits at neighboring cells in a probabilistic sense. We propose an efficient algorithm that combines an iterative D2D feasibility check and a ratio-of-expectation approximation. A performance upper bound is also developed for benchmarking. For multiple CUs and D2D pairs, due to orthogonal channelization within each cell, we first focus on the problem of joint power control and beamforming for a CU-D2D pair and show how our proposed solution can be leveraged to find a solution for this general problem. The complexity analysis of the proposed approach is also provided. Simulation results show that the proposed algorithm gives performance close to the upper bound.

Joint Channel Allocation and Power Control for Uplink NOMA-Assisted Multi-UAV Networks

The explosive growth of data leads to that the traditional wireless networks cannot enable various quality of service (QoS) communication for cellular-connected multi-UAV (unmanned aerial vehicle) networks. To overcome this obstacle, we solve the joint optimization problem of channel allocation and power control for uplink NOMA-assisted multi-UAV networks. Firstly, we design a mixed integer nonlinear programming framework, where the channel gains are characterized with integral form in time interval and sorted in nondescending order as the priority index of the decoded signal. In order to propose a feasible algorithm, the initial power levels of UAVs are obtained and integrated into the original problem which is reduced to integer programming problem. Then, the UAVs whose channel gain differences satisfy the constraints will be divided into a group to share the same channel, while the initial power levels of UAVs are adjusted to get a more satisfactory initial solution for power control. Combining the solution of channel allocation and the initial power levels, we solve power control problem with asynchronous update mechanism until the power levels of UAVs remain unchanged. Finally, we propose a channel allocation algorithm and a power control algorithm with the asynchronous optimization mechanism, respectively. Simulation results show that the proposed algorithms can effectively improve the network performance in terms of the aggregated rate.

Quantum-Driven Energy-Efficiency Optimization for Next-Generation Communications Systems

The advent of deep-learning technology promises major leaps forward in addressing the ever-enduring problems of wireless resource control and optimization, and improving key network performances, such as energy efficiency, spectral efficiency, transmission latency, etc. Therefore, a common understanding for quantum deep-learning algorithms is that they exploit advantages of quantum hardware, enabling massive optimization speed ups, which cannot be achieved by using classical computer hardware. In this respect, this paper investigates the possibility of resolving the energy efficiency problem in wireless communications by developing a quantum neural network (QNN) algorithm of deep-learning that can be tested on a classical computer setting by using any popular numerical simulation tool, such as Python. The computed results show that our QNN algorithm can be indeed trainable and that it can lead to solution convergence during the training phase. We also show that the proposed QNN algorithm exhibits slightly faster convergence speed than its classical ANN counterpart, which was considered in our previous work. Finally, we conclude that our solution can accurately resolve the energy efficiency problem and that it can be extended to optimize other communications problems, such as the global optimal power control problem, with promising trainability and generalization ability.

Joint Placement, Power Control, and Spectrum Allocation for UAV Wireless Backhaul Networks

Unmanned aerial vehicle (UAV) communication is a promising technology to yield great benefits for 5G wireless networks and beyond. In this paper, we investigate the downlink transmission in wireless backhaul networks when a UAV is deployed as the flying base station. A joint optimization problem of UAV's placement, spectrum allocation, and power control is formulated, which is an NP-hard problem. To solve the above-mentioned problem efficiently, we propose decomposing the underlying problem into two subproblems, which are then solved alternatively in an iterative manner. Simulation results under various settings are provided to demonstrate the performance improvement of the proposed algorithm over baseline schemes.

Generalized User Grouping in NOMA: An Overlapping Perspective

Non-orthogonal multiple access (NOMA) is regarded as a promising technology to provide high spectral efficiency and support massive connectivity in 5G systems. Traditionally, NOMA user grouping is non-overlapping, leading to a waste of power resources within each NOMA group. Motivated by this, in this paper we propose a novel generalized user grouping (GuG) concept for NOMA from an overlapping perspective, which allows each user to participate in multiple user groups but subject to individual maximum power constraint. In order to achieve effective GuG and maximize the system sum rate, we formulate a joint power control and GuG optimization problem. Then we further provide a machine learning-based GuG scheme to obtain the optimized feasible GuG and the optimal power control solutions efficiently, in which the established machine learning-based model is exploited to explore the relative relationships of channel gains of users and obtain several fixed grouping patterns via Merge operation. Simulation results verify the efficiency of GuG in NOMA systems and indicate that compared with traditional NOMA user grouping schemes, our proposed GuG scheme achieves significant performance gains in terms of system sum rate.

Deep Reinforcement Learning for Power Controlled Channel Allocation in Wireless Avionics Intra-Communications

Wireless avionics intra-communications (WAIC) can play an important role in alleviate the issue of fuel consumption, stability and maintenance costs over traditional wired avionics systems. However, in order for WAIC system to coexist with other systems in aircraft, the transmitted power level of WAIC is strictly limited to 50 mW. Meanwhile, WAIC require extremely low outage probability for the safety-critical avionics applications. Hence, it is urgently needed to effectively allocate channels and utilize the limited power while ensuring the reliability and real-time performance of the WAIC system. In this paper, a deep reinforcement learning (DRL)-based power controlled channel allocation (DRL-PCCA) scheme for WAIC network is proposed, with physical layer of frequency hopping orthogonal frequency division multiplexing (FH-OFDM). First, we formulate a sub-bands allocation and power control optimization problem whose aim is to minimize the overall transmit power provided that all end nodes achieve their requested data rates and desired bit error rate (BER). However, the problem formulated is non-convex and NP-hard. To tackle this problem, we propose a DRL-based scheme, which can effectively solve the optimization problem of sequence decision making in complex environment by using neural networks to extract spatial correlation features of WAIC. In particular, a reliability framework for WAIC is presented to analyse the performance of the proposed scheme against the system requirements of the flight certification. Simulation results demonstrate that the performance of proposed DRL-based scheme is superior to the traditional power control and channel allocation scheme.

QoS-oriented joint optimization of concurrent scheduling and power control in millimeter wave mesh backhaul network

Millimeter-wave mesh backhaul network (mm-wave MBN) is a key potential technology for massive traffic backhaul in 5G and beyond. However, one of the most urgent challenges is the high capacity and green backhaul solution. To that end, a QoS-oriented joint optimization of concurrent scheduling and power control is investigated, where the QoS demands of the transmission flows are fully considered. Firstly, the QoS-oriented joint optimization problem of concurrent scheduling and power control is formulated as a mixed integer nonlinear programming (MINLP) problem, where the energy consumption is reduced and the number of the successfully transmitted flows is enhanced. Secondly, a heuristics and energy-efficient backhaul mechanism (HEBM) is proposed for the suboptimal solution obtained in polynomial time because that the formulated problem is a NP-hard problem. The HEBM is composed of two parts, i.e., concurrent scheduling algorithm and power control algorithm. In the first algorithm, the concurrent scheduling set is decided with the maximum independent set of the constructed conflict graph and the defined priority of the transmission flow. In the second algorithm, the transmission power of each flow is calculated. Thirdly, the relationship between the key threshold of HEBM and the transmission power is derived and verified. Finally, HEBM is compared with the state-of-the-arts and the superiority of HEBM is demonstrated through extensive simulations under various traffic modes and system parameters.

Deep Reinforcement Learning for Joint Channel Selection and Power Control in D2D Networks

Device-to-device (D2D) technology, which allows direct communications between proximal devices, is widely acknowledged as a promising candidate to alleviate the mobile traffic explosion problem. In this paper, we consider an overlay D2D network, in which multiple D2D pairs coexist on several orthogonal spectrum bands, i.e., channels. Due to spectrum scarcity, the number of D2D pairs is typically more than that of available channels, and thus multiple D2D pairs may use a single channel simultaneously. This may lead to severe co-channel interference and degrade network performance. To deal with this issue, we formulate a joint channel selection and power control optimization problem, with the aim to maximize the weighted-sum-rate (WSR) of the D2D network. Unfortunately, this problem is non-convex and NP-hard. To solve this problem, we first adopt the state-of-art fractional programming (FP) technique and develop an FP-based algorithm to obtain a near-optimal solution. However, the FP-based algorithm requires instantaneous global channel state information (CSI) for centralized processing, resulting in poor scalability and prohibitively high signalling overheads. Therefore, we further propose a distributed deep reinforcement learning (DRL)-based scheme, with which D2D pairs can autonomously optimize channel selection and transmit power by only exploiting local information and outdated nonlocal information. Compared with the FP-based algorithm, the DRL-based scheme can achieve better scalability and reduce signalling overheads significantly. Simulation results demonstrate that even without instantaneous global CSI, the performance of the DRL-based scheme can approach closely to that of the FP-based algorithm.

Generalized User Grouping in NOMA Based on Overlapping Coalition Formation Game

Non-orthogonal multiple access (NOMA) is regarded as a promising technology to provide high spectral efficiency and support massive connectivity in 5G systems. In most existing NOMA user grouping approaches, users are grouped into disjoint groups, which may lead to a waste of power resources within each NOMA group. Motivated by this, in this paper we propose a novel generalized user grouping (GuG) concept for NOMA from an overlapping perspective, which allows each user to participate in multiple groups but subject to individual maximum power constraint. In order to achieve effective GuG and maximize the system sum rate, we formulate a joint power control and GuG optimization problem. Then, we address this problem by exploiting the overlapping coalition formation (OCF) game framework, and we further propose an OCF-based algorithm in which each user can be self-organized into a desirable overlapping coalition structure. Simulation results verify the efficiency of GuG in NOMA systems and indicate that compared with traditional NOMA user grouping schemes, our proposed OCF-based GuG NOMA scheme achieves significant performance gains in terms of system sum rate.

Popular Matching for Security-Enhanced Resource Allocation in Social Internet of Flying Things

As the Internet of Things (IoT) is maturing and acquires its social flavor, the Social IoT enables smart devices to build inter-thing social networks without human intervention. As a new form of smart devices, unmanned aerial vehicles (UAVs) are finding their way into IoT applications. The integrated Social Internet of Flying Things (SIoFT) can provide the social-aware UAV-assisted services. However, the broadcast nature of air-to-ground (A2G) channels makes them vulnerable to being eavesdropped by terrestrial malicious users due to their strong line-of-sight (LoS) links. In this paper, we investigate to ensure the security of A2G communications when the location information of multiple potential eavesdroppers cannot be perfectly estimated. Following the “no pain no gain” principle, the terrestrial users who reuse the UAV cellular spectrum will act as friendly jammers to realize “win-win” situation. Hence, joint trajectory design, power control, and channel allocation optimization problem is formulated to maximize the average secrecy rate of UAVs in worst case. In the first stage, we utilize the block coordinate descent method and successive convex optimization method to solve the trajectory design and power control problems in an iterative manner. In the second stage, we convert the user pairing problem into a popular matching problem with externalities. Two distributed algorithms are proposed to maintain the popular matching under dynamics. Moreover, we conduct detailed analysis of the popularity, convergence, and computational complexity. Simulation results demonstrate the superiority of our proposed method in terms of different performance metrics.

On Power Minimization for IRS-Aided Downlink NOMA Systems

Both intelligent reflecting surface (IRS) and non-orthogonal multiple access (NOMA) are the cutting-edge technologies for future wireless communications. In this letter, we examine the effectiveness of IRS in NOMA system with respect to transmit power consumption. Towards this end, the power minimization problem for IRS-aided downlink NOMA system is formulated, taking into consideration the constraint of each user’s minimum signal-to-interference-ratio (SINR). To solve the joint power control and IRS phase shifts optimization problem, we first explore the relationship between individual user’s transmit power and the variables of phase shifts. Then, we solve the phase shift determination problem via the sequential rotation algorithm. Numerical results show that our designed joint optimization algorithm can significantly reduce the required transmit power when compared with various benchmarking schemes.

Rate-Aware User Matching and Power Control for Full-Duplex-Based Small Cells

This letter investigates the joint uplink-downlink user matching and power control strategy in a small cell where the base station is full-duplex (FD) enabled. The strategy is applicable to some emerging scenarios, e.g., vehicle-to-infrastructure (V2I) communications, in which the transmission rate of each user should be guaranteed with best effort. Specifically, we formulate a user matching combined with power control optimization problem, with the goal of maximizing the number of rate-satisfied user pairs. The problem is then decoupled into two subproblems for ease of handling, wherein the joint user matching and uplink power control subproblem is addressed by graph-based algorithm, while the downlink power control subproblem is addressed via iterating through all the possible value ranges. The two subproblems are performed alternatively till the convergence. Simulation results show that our proposed strategy outperforms the fairness-oriented and sum-rate-oriented strategies.

A deep reinforcement learning for user association and power control in heterogeneous networks

Heterogeneous network (HetNet) is a promising solution to satisfy the unprecedented demand for higher data rate in the next generation mobile networks. Different from the traditional single-layer cellular networks, how to provide the best service to the user equipments (UEs) under the limited resource is an urgent problem to solve. In order to efficiently address the above challenge and strive towards high network energy efficiency, the joint optimization problem of user association and power control in orthogonal frequency division multiple access (OFDMA) based uplink HetNets is studied. Considering the non-convex and non-linear characteristics of the problem, a multi-agent deep Q-learning Network (DQN) method is studied to solve the problem. Different from the traditional methods, such as game theory, fractional programming and convex optimization, which need more and accurate network information in practice, the multi-agent DQN method requires less communication information of the environment. Moreover, for the communication environment dynamics, the maximum long-term overall network utility with a new reward function while ensuring the UE’s quality of service (QoS) requirements is achieved by using the multi-agent DQN method. Then, according to the application scenario, the action space, state space and reward function of the multi-agent DQN based framework are redefined and formulated. Simulation results demonstrate that the multi-agent DQN method has the best performance on convergence and energy efficiency compared with the traditional reinforcement learning (Q-learning).