Multi-armed Bandit Approach Research Articles

QoS Driven Channel Selection Algorithm for Cognitive Radio Network: Multi-User Multi-Armed Bandit Approach

In this paper, we deal with the problem of opportunistic spectrum access in infrastructure-less cognitive networks. Each secondary user (SU) Tx is allowed to select one frequency channel at each transmission trial. We assume that there is no information exchange between SUs, and they have no knowledge of channel quality, availability, and other SUs actions, hence, each SU selfishly tries to select the best band to transmit. This particular problem is designed as a multi-user restless Markov multi-armed bandit problem, in which multiple SUs collect a priori unknown reward by selecting a channel. The main contribution of the paper is to propose an online learning policy for distributed SUs, that takes into account not only the availability criterion of a band but also a quality metric linked to the interference power from the neighboring cells experienced on the sensed band. We also prove that the policy, named distributed restless QoS-UCB, achieves at most logarithmic order regret, for a single-user in a first time and then for multi-user in a second time. Moreover, studies on the achievable throughput, average bit error rate obtained with the proposed policy are conducted and compared to well-known reinforcement learning algorithms.

Dynamic Privacy Pricing: A Multi-Armed Bandit Approach With Time-Variant Rewards

Recently, the conflict between exploiting the value of personal data and protecting individuals’ privacy has attracted much attention. Personal data market provides a promising solution to this conflict, while determining the price of privacy is a tough issue. In this paper, we study the pricing problem in a setting where a data collector sequentially buys data from multiple data owners whose valuations of privacy are randomly drawn from an unknown distribution. To maximize the total payoff, the collector needs to dynamically adjust the prices offered to owners. We model the sequential decision-making problem of the collector as a multi-armed bandit problem with each arm representing a candidate price. Specifically, the privacy protection technique adopted by the collector is taken into account. Protecting privacy generally causes a negative effect on the value of data, and this effect is embodied by the time-variant distributions of the rewards associated with arms. Based on the classic upper confidence bound policy, we propose two learning policies for the bandit problem. The first policy estimates the expected reward of a price by counting how many times the price has been accepted by data owners. The second policy treats the time-variant data value as a context and uses ridge regression to estimate the rewards in different contexts. Simulation results on real-world data demonstrate that by applying the proposed policies, the collector can get a payoff which is close to that he can get by setting a fixed price, which is the best in hindsight, for all data owners.

Relay selection in cooperative power line communication: A multi-armed bandit approach

Power line communication (PLC) exploits the existence of installed infrastructure of power delivery system, in order to trans- mit data over power lines. In PLC networks, different nodes of the network are interconnected via power delivery transmission lines, and the data signal is flowing between them. However, the attenu- ation and the harsh environment of the power line communication channels, makes it difficult to establish a reliable communication between two nodes of the network which are separated by a long distance. Relaying and cooperative communication has been used to overcome this problem. In this paper a two-hop cooperative PLC has been studied, where the data is communicated between a trans- mitter and a receiver node, through a single array node which has to be selected from a set of available arrays. The relay selection problemcan be solved by having channel state information (CSI) at transmitter and selecting the relay which results in the best perfor- mance. However, acquiring the channel state information at trans- mitter increases the complexity of the communication system and introduces undesired overhead to the system. We propose a class of machine learning schemes, namely multi-armed bandit (MAB), to solve the relay selection problem without the knowledge of the channel at the transmitter. Furthermore, we develop a new MAB algorithm which exploits the periodicity of the synchronous impul- sive noise of the PLC channel, in order to improve the relay selec- tion algorithm.

Opportunistic channel access with repetition time diversity and switching cost: a block multi-armed bandit approach

In this paper, we investigate the channel access problem considering switching cost in the block fading channels with unknown information of channel occupation and quality. We formulate this problem as a multi-armed bandit (MAB) problem with the goal of minimizing the outage rate and avoiding frequent channel switching. To achieve this goal, a block based multi-armed bandit (BMAB) learning algorithm is proposed. Furthermore, the BMAB algorithm is extended to cope with the short-term deep channel fading, by exploiting the repetition time diversity (RTD). The regrets of the proposed two algorithms are proved to be logarithmic in time. Performance analysis and simulation results show that the proposed algorithms outperform standard SMAB algorithm in average system outage rate, switching cost and throughput. In addition, the repetition time diversity multi-armed bandit (RTDMAB) algorithm is better than BMAB algorithm in the presence of deep channel fading at the cost of receiving complexity .

Distributed User Association in Energy Harvesting Dense Small Cell Networks: A Mean-Field Multi-Armed Bandit Approach

The emerging Internet of Things (IoT)-driven ultra-dense small cell networks (UD-SCNs) will need to combat a variety of challenges. On one hand, massive number of devices sharing the limited wireless resources will render centralized control mechanisms infeasible due to the excessive cost of information acquisition and computations. On the other hand, to reduce energy consumption from fixed power grid and/or battery, many IoT devices may need to depend on the energy harvested from the ambient environment (e.g., from RF transmissions, environmental sources). However, due to the opportunistic nature of energy harvesting, this will introduce uncertainty in the network operation. In this article, we study the distributed cell association problem for energy harvesting IoT devices in UD-SCNs. After reviewing the state-of-the-art research on the cell association problem in small cell networks, we outline the major challenges for distributed cell association in IoT-driven UD-SCNs where the IoT devices will need to perform cell association in a distributed manner in presence of uncertainty (e.g., limited knowledge on channel/network) and limited computational capabilities. To this end, we propose an approach based on mean-field multi-armed bandit games to solve the uplink cell association problem for energy harvesting IoT devices in a UD-SCN. This approach is particularly suitable to analyze large multi-agent systems under uncertainty and lack of information. We provide some theoretical results as well as preliminary performance evaluation results for the proposed approach.

CBRS Spectrum Sharing between LTE-U and WiFi: A Multiarmed Bandit Approach

The surge of mobile devices such as smartphone and tablets requires additional capacity. To achieve ubiquitous and high data rate Internet connectivity, effective spectrum sharing and utilization of the wireless spectrum carry critical importance. In this paper, we consider the use of unlicensed LTE (LTE-U) technology in the 3.5 GHz Citizens Broadband Radio Service (CBRS) band and develop a multiarmed bandit (MAB) based spectrum sharing technique for a smooth coexistence with WiFi. In particular, we consider LTE-U to operate as a General Authorized Access (GAA) user; hereby MAB is used to adaptively optimize the transmission duty cycle of LTE-U transmissions. Additionally, we incorporate downlink power control which yields a high energy efficiency and interference suppression. Simulation results demonstrate a significant improvement in the aggregate capacity (approximately 33%) and cell-edge throughput of coexisting LTE-U and WiFi networks for different base station densities and user densities.

A Two-Dimensional Multiarmed Bandit Approach to Secondary Users with Network Coding in Cognitive Radio Networks

We study how to utilize network coding to improve the throughput of secondary users (SUs) in cognitive radio networks (CRNs) when the channel quality is unavailable at SUs. We use a two-dimensional multiarmed bandit (MAB) approach to solve the problem of SUs with network coding under unknown channel quality in CRNs. We analytically prove the asymptotical-throughput optimality of the proposed two-dimensional MAB algorithm. Simulation results show that our proposed algorithm achieves comparable throughput performance, compared to both the theoretical upper bound and the scheme assuming known channel quality information.

Learning State Selection for Reconfigurable Antennas: A Multi-Armed Bandit Approach

Reconfigurable antennas are capable of dynamically re-shaping their radiation patterns in response to the needs of a wireless link or a network. In order to utilize the benefits of reconfigurable antennas, selecting an optimal antenna state for communication is essential and depends on the availability of full channel state information for all the available antenna states. We consider the problem of reconfigurable antenna state selection in a single user MIMO system. We first formulate the state selection as a multi-armed bandit problem that aims to optimize arbitrary link quality metrics. We then show that by using online learning under a multi-armed bandit framework, a sequential decision policy can be employed to learn optimal antenna states without instantaneous full CSI and without a priori knowledge of wireless channel statistics. Our objective is to devise an adaptive state selection technique when the channels corresponding to all the states are not directly observable and compare our results against the case of a known model or genie with full information. We evaluate the performance of the proposed antenna state selection technique by identifying key link quality metrics and using measured channels in a 2 × 2 MIMO OFDM system. We show that the proposed technique maximizes long term link performance with reduced channel training frequency.

Adaptive Adversarial Multi-Armed Bandit Approach to Two-Person Zero-Sum Markov Games

This technical note presents a recursive sampling-based algorithm for finite horizon two-person zero-sum Markov games (MGs) based on the Exp3 algorithm developed by Auer et al <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">.</i> for adaptive adversarial multi-armed bandit problems. We provide a finite-iteration bound to the equilibrium value of the induced ¿sample average approximation game¿ of a given MG and prove asymptotic convergence to the equilibrium value of the given MG. The time and space complexities of the algorithm are independent of the state space of the game.

On training with feedback in wideband channels

Transmitter knowledge of channel state has a great impact on wideband fading channel capacity. However, in the low SNR regime, power per dimension does not suffice to provide an accurate measurement of the channel over the entire spectrum. In the presence of feedback, we may collect information at the transmitter about some aspects of the channel quality over a certain portion of the spectrum. In this work, we investigate the effect of such information. We consider channel testing with a finite amount of energy over a block-fading channel in both time and frequency. We consider a transmission scheme in which the wideband channel is decomposed into many parallel narrowband subchannels, each used with a binary modulation scheme. The quality of each subchannel corresponds to the crossover probability of a binary symmetric channel. We use a multi-armed bandit approach to consider the relative costs and benefits of allotting energy for testing versus transmission, and for repeated testing a single subchannel versus testing different subchannels. We give both upper and lower bounds on the number of subchannels that should be probed for throughput maximization under the scheme we have chosen. Our bounds are in terms of available transmission energy, available bandwidth and fading characteristics of the channel. Moreover, in our numerical results, the two bounds are close.