Dynamic Spectrum Access Problem Research Articles

Deep Reinforcement Learning for Simultaneous Sensing and Channel Access in Cognitive Networks

We consider the problem of dynamic spectrum access (DSA) in cognitive wireless networks, consisting of primary users (PUs) and secondary users (SUs), where only partial observations are available at the SUs due to narrowband sensing and transmissions. The network operates in a time-slotted regime, where the traffic patterns of the PUs are modeled as finite-memory Markov chains, that are unknown to the SUs. Since observations are partial, then both channel sensing and access actions affect the throughput. Focusing on the case in which there is a single SU, our objective is to maximize the SU’s long-term throughput. To that aim, we develop a novel algorithm that learns <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">both</i> access and sensing policies via deep Q-learning, dubbed Double Deep Q-network for Sensing and Access (DDQSA). To the best of our knowledge, this is the first work that jointly optimizes both sensing and access policies for DSA via deep Q-learning. Next, we consider wireless networks with access policy which implements a fixed channel hopping dynamics, for which we analytically determine the optimal SU sensing and access policy and its associated throughput. Then, we demonstrate that indeed, the proposed DDQSA algorithm can achieve near-optimal performance for the considered network. Our results show that the proposed DDQSA algorithm learns a policy that implements both sensing and channel access, which significantly outperforms existing approaches, and can achieve the optimal performance in certain scenarios.

Cooperative Multi-Agent Reinforcement-Learning-Based Distributed Dynamic Spectrum Access in Cognitive Radio Networks

With the development of wireless communication and Internet of Things (IoT), there are massive wireless devices that need to share the limited spectrum resources. Dynamic spectrum access (DSA) is a promising paradigm to remedy the problem of inefficient spectrum utilization brought upon by the historical command-and-control approach to spectrum allocation. In this article, we investigate the distributed DSA problem for multiusers in a typical multichannel cognitive radio network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP), and we propose a centralized off-line training and distributed online execution framework based on cooperative multi-agent reinforcement learning (MARL). We employ the deep recurrent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DRQN) to address the partial observability of the state for each cognitive user. The ultimate goal is to learn a cooperative strategy which maximizes the sum throughput of a cognitive radio network in a distributed fashion without information exchange between cognitive users. Finally, we validate the proposed algorithm in various settings through extensive experiments. The experimental results show that the proposed CoMARL-DSA algorithm outperforms the state-of-the-art deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning for spectrum access (DQSA) in terms of successful access rate and collision rate by at least 14% and 12%, respectively.

Deep Reinforcement Learning for Dynamic Multichannel Access in Multi-Cognitive Radio Networks

We propose a reinforcement learning framework based on deep recurrent learning to solve the dynamic spectrum access problem in the scenario where multiple cognitive networks coexist. In this scenario, the shared spectrum is divided into multiple channels, and the channel occupation by the primary user is modeled as a Markov model. The observation of the channel status by secondary users in this area obeys the partially observed Markov process, that is, each user can only observe the status of one channel in each time slot, and cannot obtain global information. To avoid collision in different cognitive networks, we adopt the distributed multi-agent deep reinforcement learning method. Without the interaction of different cognitive networks, based on the current partial observations, a suitable channel access policy can be obtained by a neural network. Comparing with Slotted-aloha and DQN-DSA Algorithm, the result indicates DRQN-DSA proposed in this paper performs better.

Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access

We consider the problem of dynamic spectrum access for network utility maximization in multichannel wireless networks. The shared bandwidth is divided into $K$ orthogonal channels. In the beginning of each time slot, each user selects a channel and transmits a packet with a certain transmission probability. After each time slot, each user that has transmitted a packet receives a local observation indicating whether its packet was successfully delivered or not (i.e., ACK signal). The objective is a multi-user strategy for accessing the spectrum that maximizes a certain network utility in a distributed manner without online coordination or message exchanges between users. Obtaining an optimal solution for the spectrum access problem is computationally expensive, in general, due to the large-state space and partial observability of the states. To tackle this problem, we develop a novel distributed dynamic spectrum access algorithm based on deep multi-user reinforcement leaning. Specifically, at each time slot, each user maps its current state to the spectrum access actions based on a trained deep-Q network used to maximize the objective function. Game theoretic analysis of the system dynamics is developed for establishing design principles for the implementation of the algorithm. The experimental results demonstrate the strong performance of the algorithm.

Dynamic Spectrum Access in Time-Varying Environment: Distributed Learning Beyond Expectation Optimization

This paper investigates the problem of dynamic spectrum access for canonical wireless networks, in which the channel states are time-varying. In the most existing work, the commonly used optimization objective is to maximize the expectation of a certain metric (e.g., throughput or achievable rate). However, it is realized that expectation alone is not enough since some applications are sensitive to fluctuations. Effective capacity is a promising metric for time-varying service process since it characterizes the packet delay violating probability (regarded as an important statistical quality-of-service index), by taking into account not only the expectation but also other high-order statistic. Therefore, we formulate the interactions among the users in the time-varying environment as a non-cooperative game, in which the utility function is defined as the achieved effective capacity. We prove that it is an ordinal potential game which has at least one pure strategy Nash equilibrium. Based on an approximated utility function, we propose a multi-agent learning algorithm which is proved to achieve stable solutions with dynamic and incomplete information constraints. The convergence of the proposed learning algorithm is verified by simulation results. Also, it is shown that the proposed multi-agent learning algorithm achieves satisfactory performance.

Load-Aware Dynamic Spectrum Access for Small-Cell Networks: A Graphical Game Approach

In this letter, we investigate the problem of dynamic spectrum access for small cell networks, using a graphical game approach. Compared with existing studies, we take the features of different cell loads and local interference relationship into account. It is proved that the formulated spectrum access game is an exact potential game with the aggregate interference level as the potential function, and Nash equilibrium (NE) of the game corresponds to the global or local optima of the original optimization problem. A lower bound of the achievable aggregate interference level is rigorously derived. Finally, we propose an autonomous best response learning algorithm to converge towards its NE. It is shown that the proposed game-theoretic solution converges rapidly and its achievable performance is close to the optimum solution.

Editorial for Crowncom 2013 Special Issue

This special issue features six selected papers with high quality from the conference CrownCom 2013, which was held in Washington DC, USA, July 8–10, 2013. The first article titled “Coopetive Spectrum Trading – Creating Endogenous Spectrum Holes” by Liu Cui, Taieb Znati, and Martin B.H. Weiss, discusses the problem of dynamic spectrum access. The authors argue that spectrum holes should be endogenous, enabled by incentives to incumbent users to increase spectrum availability while mitigating risks. In the proposed framework, the primary users’ spectrum is organized in three bands, namely exclusive usage band, usage right band, and management right band. Based on this structure, spectrum is traded as financial options to increase trading flexibility and capture the value primary and secondary users attach to spectrum access and usage. Spectrum trading between primary and secondary users is formalized as a utility maximization problem. Approximate solutions to this problem are derived and their performance is analyzed. The results show that spectrum trading, which combines revocable and exclusive leasing, provides higher utilities for both primary and secondary users. The second article, “Performance of Secondary Wireless Networks With Contention Control in TV White Spaces”, is co-authored by Byungjin Cho, Konstantinos Koufos, and Riku Jantti to study the spectrum coexistence of secondary wireless networks with contention control and TV services. In the proposed approach, a carrier sensing threshold is used as a parameter to control the density of secondary transmitters and accordingly the generated secondary interference to the TV system. In order to obtain the calculated threshold, a simple method to assess the data rate of the secondary system is developed. Using this method, one can study whether the secondary performance is limited due to the TV protection requirements. The method can also be used to investigate the trade-off between the secondary performance and the TV protection requirements. The third article, “Multi-channel Energy Detection under Phase Noise: Analysis and Mitigation” from Ahmet Gokceoglu, Yaning Zou, Mikko Valkama and Paschalis C. Sofotasios, investigates the impact of oscillator phase noise on energy detection (ED) based spectrum sensing in multi-channel direct-conversion receiver scenario. Based on complex Gaussian PU signal models, the authors first derive the detection and false alarm probabilities in closed-form expressions. They also show that the false alarm probability of multi-channel energy detection increases significantly, compared to the ideal RF receiver case. In order to prevent such performance degradation in spectrum identification, an enhanced energy detection technique is proposed. This technique calculates the leakage power from neighboring channels for each channel and improves the sample energy statistics by subtracting this leakage power from the raw value. An analytical expression is derived for the leakage power, which is shown to be a function of power spectral levels of neighboring channels and 3-dB bandwidth of phase noise process. Practical schemes for estimating these two quantities are discussed. Extensive computer simulations show that the proposed enhanced detection yields false alarm rates that are very close to those of an ideal RF receiver and hence clearly outperforms classical energy detection. The fourth article, “Efficient Spectrum Sensing with Minimum Transmission Delay in Cognitive Radio Networks”, is co-authored by Hang Hu, Hang Zhang, and Hong Yu. This article investigates the problem of designing the sensing time to minimize the SU transmission delay under the condition of sufficient protection to PUs. A novel cooperative spectrum X. Cheng (*) Department of Computer Science, The George Washington University, Washington, DC, USA e-mail: cheng@gwu.edu

Adaptive mechanism design and game theoretic analysis of auction-driven dynamic spectrum access in cognitive radio networks

This paper studies the auction-driven dynamic spectrum access in cognitive radio networks with heterogeneous secondary users, who have different risk attitudes. First, a game theoretic framework is established for auction-driven dynamic spectrum access in cognitive radio networks. The utility functions and bidding strategies of heterogeneous secondary users are defined, and the parameterized auction mechanisms of primary user are also introduced. Then, we formulate the auction-driven dynamic spectrum access problem as a finite discrete game with a mixed- or pure-strategy Nash equilibrium solution. We study the existence and uniqueness properties of the pure-strategy Nash equilibrium in the defined game. Next, we propose a distributed learning automata algorithm (DLA) to attain the Nash equilibrium of the defined game with limited feedback. The adaptive mechanism design is realized in the updating procedure of our DLA algorithm. We further prove that our DLA algorithm converges to a Nash equilibrium of the defined game. Finally, simulation results show that our DLA algorithm is efficient and outperforms the dynamic spectrum access schemes with fixed auction mechanism.

Two Dimension Spectrum Allocation for Cognitive Radio Networks

In this paper, we develop a truthful and efficient combinatorial auction scheme under a novel spectrum allocation model that can achieve a worst-case approximation ratio \sqrt{m} in social welfare. We propose to tackle the dynamic spectrum access problem in cognitive radio (CR) networks with time-frequency flexibility requirements. We model the spectrum opportunity in a time-frequency division manner and the spectrum allocation as a combinatorial auction. Then we design an auction mechanism to reach the upper bound in polynomial time and propose a combined approach to improve the bound in the cost of increasing computational complexity. A truthful payment that gives incentive to the SUs for revealing the truthful valuation of the desirable bundle of slots is presented. In order to reduce the complexity, we simplify the general model to a modified model that only allows frequency flexibility, and then present a truthful, optimal and computationally efficient auction mechanism. Extensive simulation results of the social welfare and spectrum ratio show that the performance of the combined approximation algorithm is better than the sorting based greedy algorithm.

Spatio-temporal spectrum modeling: Taxonomy and economic evaluation of context acquisition

Much of the research in dynamic spectrum access (DSA) has focused on opportunistic access in the temporal domain. While this has been quite useful in establishing the technical feasibility of DSA systems, it has missed large sections of the overall DSA problem space. This paper argues that the spatio-temporal operating context of specific environments matters to the selection of the appropriate technology for learning context information. It identifies twelve potential operating environments and compares four context awareness approaches (on-board sensing, databases, sensor network, and cooperative sharing) for these environments. In order for operators, regulators, and users to be interested in deploying DSA based networks, the expected costs should be in proportion to what the users are realistically willing to pay for services. Consequently, it is important to conduct cost analysis for different DSA approaches in parallel with the technical research. Regulators and major stakeholders should pay attention to the operating environment of DSA systems when determining which approaches to context learning to encourage.The incremental capital costs; over a basic software radio; have been compared for the four context awareness approaches after an estimation analysis for each cost component. Since DSA is still a relatively new research field, there is a lot of uncertainty associated with incremental cost analyses. As a result, the cost analysis is parameterized to allow for explicit reasoning about the bounds of cost components.