Non-cooperative reinforcement learning based routing in cognitive radio networks

Abstract

Cognitive Radio Networks (CRNs) have emerged to overcome the problem of spectrum scarcity caused by spreading of wireless applications. In CRNs, Secondary Users (SUs) are permitted to access opportunistically the authorized frequency bands owned by Primary Users (PUs). In this paper, we address the problem of routing several flows generated by SUs to a given destination considering the presence of PUs traffic modeled by a more realistic model based on Markov Modulated Poisson Process (MMPP). Each source SU wants to selfishly minimize the end-to-end delay of its flow meanwhile the Quality of Service (QoS) requirements of the PUs would be met. To consider quick adaptation of SUs routing decision to environment changes and non-cooperative interaction of them, we formulate the routing problem as stochastic learning processes featured by non-cooperative games. Then, we propose a distributed reinforcement learning-based scheme for solving the routing problem that can avoid information exchange between the competing SUs. The proposed scheme provably converges and simulation results demonstrate effectiveness of the proposed scheme in decreasing the delay while meeting the QoS requirements of the PUs.