Decentralized Learning for Opportunistic Spectrum Access: Multiuser Restless Multiarmed Bandit Formulation

Abstract

In opportunistic spectrum access, each secondary user selects a channel from a pool of multiple channels based on their local observations. The challenge here is to learn the best channel in terms of availability, as the channel availability statistics are unknown. In order to learn these unknown statistics, a novel decentralized multiuser learning technique termed as DSEE for channel selection in dynamic networks (DSEE-CSDN) has been proposed. DSEE-CSDN allows secondary users to enter the network during different time slots. Thus, the number of secondary users is not known beforehand. Moreover, the availability status of different independent channels is considered to be changing according to the two-state restless Markov chain model, which, in practice, is more realistic as compared to independent and identically distributed channel state model. Thus, the problem is formulated as a stochastic multiuser restless multiarmed bandit. The proposed algorithm achieves system-wide order-optimal performance under self-play. Results indicate that DSEE-CSDN is able to achieve a logarithmic order of regret. Furthermore, collisions and switching cost are just around 5% and 2% of total time slots, respectively. Also, DSEE-CSDN can achieve probabilistic fairness in channel selection without any preagreement among users.