Hybrid UCB-HMM: A Machine Learning Strategy for Cognitive Radio in HF Band

Abstract

Multiple users transmit in the HF band with worldwide coverage but collide with other HF users. New techniques based on cognitive radio principles are discussed to reduce the inefficient use of this band. In this paper, we show the feasibility of the Upper Confidence Bound (UCB) algorithm, based on reinforcement learning, for an opportunistic access to the HF band. The exploration vs. exploitation dilemma is evaluated in single-channel and multi-channel UCB algorithms in order to obtain their best performance in the HF environment. Furthermore, we propose a new hybrid system, which combines two types of machine learning techniques based on reinforcement learning and learning with Hidden Markov Models. This system can be understood as a metacognitive engine that automatically adapts its data transmission strategy according to HF environment’s behaviour to efficiently use spectrum holes. The proposed hybrid UCB-HMM system increases the duration of data transmission’s slots when conditions are favourable, and is also able to reduce the required signalling transmissions between transmitter and receiver to inform which channels have been selected for data transmission. This reduction can be as high as 61% with respect to the signalling required by multi-channel UCB.