Q-learning-based power control in small-cell networks

Abstract

Because of the time-varying nature of wireless channels, it is difficult to guarantee the deterministic quality of service (QoS) in wireless networks. In this chapter, by combining information theory with the effective capacity (EC) principle, the energy-efficiency optimization problem with statistical QoS guarantee is formulated in the uplink of a two-tier femtocell network. To solve the problem, we introduce a Q-learning mechanism based on Stackelberg game framework. The macro users act as leaders and know the emission power strategy of all femtocell users (FUS).The femtocell user is the follower and only communicates with the macrocell base station (MBS) without communicating with other femtocell base stations (FBSs). In Stackelberg game studying procedure, the macro user chooses the transmit power level first according to the best response of the femtocell, and the micro users interact directly with the environment, i.e., leader's transmit power strategies, and find their best responses. Then, the optimization problem is modeled as a noncooperative game, and the existence of Nash equilibriums (NEs) is studied. Finally, in order to improve the self-organizing ability of femtocell, we adopt Q-learning framework based on noncooperative game, in which all the FBS are regarded as agents to achieve power allocation. Numerical results show that the algorithm cannot only meet the delay requirements of delay-sensitive traffic but also has good convergence.