A Q-Learning Based Energy Threshold Optimization Algorithm in LAA Networks

Abstract

The energy detection technology is recommended in the licensed assisted access (LAA) scheme by 3GPP because of its simplicity and low cost. However, due to its inherent limitation, there may exist imperfect channel detection, which can lead to the decrease of the channel utilization efficiency and the deterioration of fairness. The imperfect detection can generally be represented by the detection probability and false alarm probability, which depend on detection time, signal to noise ration (SNR), sampling rate and energy threshold. However, among the parameters, only the energy threshold can be dominated by LAA small base stations (SBSs) in the LAA scheme. Therefore, the energy threshold should be dynamically adjusted in the changeable channel environment such that the detection accuracy can improved as high as possible. Consider the fact that the optimization theory cannot be used to optimize the energy threshold since the expressions of performance indexes about the energy threshold are extremely complex, a Q-learning based energy threshold optimization algorithm (QLET) is thus proposed in the paper, where LAA SBSs act as the agent, the energy threshold is defined as the agent action, the different combinations of fairness and throughput are defined as the agent states, and the fairness and the reward function are also redefined. In order to ensure the smooth implementation of the proposed QLET algorithm, the information exchange mechanism, where the sending-confirmation mechanism and 1- persistent CSMA are used, is also proposed. Based on the proposed QL framework, the agent can learn the optimal energy threshold by repeatedly interacting with the environment, which enables the coexistence system to obtain the best coexistence performance. A large number of simulation results show that the proposed QLET is superior to the traditional fixed energy threshold scheme (FET) in terms of the fairness, WiFi collision probability and transmission delay, and that QLET is almost the same as FET in term of throughput.