Intelligent Spectrum Management Based on Transfer Actor-Critic Learning for Rateless Transmissions in Cognitive Radio Networks

Abstract

This paper presents an intelligent spectrum mobility management scheme for cognitive radio networks. The spectrum mobility could involve spectrum handoff (i.e., the user switches to a new channel) or stay-and-wait (i.e., the user pauses the transmission for a while until the channel quality improves again). An optimal spectrum mobility management scheme needs to consider its long-term impact on the network performance, such as throughput and delay, instead of optimizing only the short-term performance. We use a machine learning scheme, called the Transfer Actor-Critic Learning (TACT), for the spectrum mobility management. The proposed scheme uses a comprehensive reward function that considers the channel utilization factor (CUF), packet error rate (PER), packet dropping rate (PDR), and flow throughput. Here, the CUF is determined by the spectrum sensing accuracy and channel holding time. The PDR is calculated from the non-preemptive M/G/1 queueing model, and the flow throughput is estimated from a link-adaptive transmission scheme, which utilizes the rateless (Raptor) codes. The proposed scheme achieves a higher reward, in terms of the mean opinion score, compared to the myopic and Q-learning based spectrum management schemes.