Multi-armed Bandit Framework Research Articles

Spatial and Temporal Contextual Multi-Armed Bandit Handovers in Ultra-Dense mmWave Cellular Networks

Although millimeter wave (mmWave) is a promising technology in 5G communication, its severe path attenuation and susceptibility to line-of-sight (LOS) blockage result in much more unpredictable outages than traditional technologies. This special propagation property raises a significant challenge to the mobility management in mmWave cellular networks. Since conventional handover policies purely rely on the measurement of signal strength, they would cause a large number of unnecessary handovers due to the frequent short-term LOS blockage by obstacles, imposing high signaling and energy overhead. In this paper, we propose two novel handover mechanisms to reduce unnecessary handovers by carefully deciding the next base station (BS) a user should handover to, so that the new user-BS connection after the handover can last as long as possible. Without prior knowledge of user’s mobility and environment, the proposed handover mechanisms exploit the empirical distribution of user’s post-handover trajectory and LOS blockage, learned online through a multi-armed bandit (MAB) framework. Depending on the contexts extracted from RSS information, two different MAB problems for handover are formulated, which focus on spatial and space-time contexts, respectively, The results of numerical simulations demonstrate that the proposed contextual handover mechanisms significantly outperform existing counterparts on reducing handovers in all simulated scenarios.

Low-Complexity Learning for Dynamic Spectrum Access in Multi-User Multi-Channel Networks

In cognitive radio networks (CRNs), dynamic spectrum access allows (unlicensed) users to identify and access unused channels opportunistically, thus improves spectrum utilization. In this paper, we address the user-channel allocation problem in multi-user multi-channel CRNs without a prior knowledge of channel statistics. The result of channel access is stochastic with unknown distribution, and statistically different for each user. In deciding the channel for access, a user needs to either explore a channel to learn its statistics, or exploit the channel with the highest expected reward based on the information collected so far. Further, a channel should be accessed exclusively by one user at a time to avoid collision. Using multi-armed bandit framework, we develop two rate-optimal algorithms with low computational complexities of <inline-formula><tex-math notation="LaTeX">$O(N)$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$O(NK)$</tex-math></inline-formula> , respectively, where <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> denotes the number of users and <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> denotes the number of channels. Further, we extend the results and develop an algorithm that is amenable to implement in a distributed fashion.

Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible. In this work, we focus on learning to coordinate. Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We provide a regret bound that is sublinear in time and low-order polynomial in the highest number of actions of a single agent for sparse coordination graphs. Additionally, we empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks, and a novel benchmark with Poisson distributions. An example of a loosely-coupled multi-agent system is a wind farm. Coordination within the wind farm is necessary to maximize power production. As upstream wind turbines only affect nearby downstream turbines, we can use MATS to efficiently learn the optimal control mechanism for the farm. To demonstrate the benefits of our method toward applications we apply MATS to a realistic wind farm control task. In this task, wind turbines must coordinate their alignments with respect to the incoming wind vector in order to optimize power production. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in practical applications with sparse neighbourhood structures.

A Restless MAB-Based Index Policy for UL Pilot Allocation in Massive MIMO Over Gauss–Markov Fading Channels

In a time-division duplex (TDD) massive multiple-input multiple-output (MIMO) system, the number of available orthogonal pilot sequences in each cell is limited. In this paper, we consider a massive MIMO system where the number of available orthogonal pilot sequences is less than the number of users. In order to optimize the long-term performance in estimating the uplink (UL) channels, the base station (BS) needs to judiciously decide the allocation of the available pilot sequences to different users in each training phase. In this paper, the pilot allocation problem is modeled as a partially observable Markov decision process with an exploitation-exploration tradeoff under the Gauss-Markov fading channels. We then investigate this problem within the restless multi-armed bandit (RMAB) framework and put forward a low-complexity method to derive the Whittle's index approximately. A simple index policy is then proposed to allocate the orthogonal pilots in a massive MIMO system. Numerical results demonstrate the superiority of the proposed index policy.

Learning-Based Beamforming for Multi-User Vehicular Communications: A Combinatorial Multi-Armed Bandit Approach

There is an urgent need to support high data rate among connected vehicles for both safety and infotainment purposes. However, high rate for vehicular communications is impacted by several challenges such as optimal beam tracking for multiple simultaneous-transmitting vehicles in highly-dynamic environments. In this paper, we aim to maximize the overall network throughput for multi-vehicular communications. We propose a reinforcement learning (RL) approach called combinatorial multi-armed bandit (CMAB) framework, where multiple arms (i.e., actions) form a super arm and can be played together, to handle the beam selection problem in a vehicular network. More specifically, we propose an adaptive combinatorial Thompson sampling algorithm, namely adaptive CTS, that CMAB embodies for appropriate selection of simultaneous beams in a high-mobility vehicular environment. The proposed approach applies the smart exploration-exploitation trade-off for the fast selection of beams. As the proposed adaptive CTS scheme produces a higher complexity over the search space which increases exponentially with the number of user, we also propose a sequential Thompson sampling (TS) algorithm where beam selection is performed in a one-by-one manner. We analyze the regret bounds for the proposed beam tracking algorithms. The performance of the proposed strategies are evaluated for multi-vehicle millimeter-wave (mmWave) simulation environments. Our results suggest that the proposed sequential approach performs almost similar to the simultaneous adaptive CTS scheme for tracking optimal beams in a multi-vehicular network with much reduced complexity. Simulation results also show that both of our proposed strategies approach the optimal achievable rate achieved by the genie-aided solution.

One-Bit Feedback Exponential Learning for Beam Alignment in Mobile mmWave

Efficient beam alignment in wireless networks capable of supporting device mobility is currently one of the major challenges in mmWave communications. In this context, we formulate the beam-alignment problem via the adversarial multi-armed bandit (MAB) framework, which copes with arbitrary network dynamics including non-stationary or adversarial components. Building on the well known exponential weights algorithm (EXP3) and by exploiting the structure and sparsity of the mmWave channel, we propose a modified (MEXP3) policy that requires solely one-bit of feedback information (reducing the amount of exchanged data during the beam-alignment process). Our MEXP3 comes with optimal theoretical guarantees in terms of asymptotic regret. Moreover, for finite horizons, our regret upper-bound is tighter than that of the original EXP3 suggesting better performance in practice. We then introduce an additional modification that accounts for the temporal correlation between successive beams and propose another beam-alignment policy. Our numerical results demonstrate that our beam-alignment policies outperform existing ones with respect to the regret but also to the outage, throughput and delay in typical mobile mmWave settings.

Transfer restless multi-armed bandit policy for energy-efficient heterogeneous cellular network

This paper proposes a learning policy to improve the energy efficiency (EE) of heterogeneous cellular networks. The combination of active and inactive base stations (BS) that allows for maximizing EE is identified as a combinatorial learning problem and requires high computational complexity as well as a large signaling overhead. This paper aims at presenting a learning policy that dynamically switches a BS to ON or OFF status in order to follow the traffic load variation during the day. The network traffic load is represented as a Markov decision process, and we propose a modified upper confidence bound algorithm based on restless Markov multi-armed bandit framework for the BS switching operation. Moreover, to cope with initial reward loss and to speed up the convergence of the learning algorithm, the transfer learning concept is adapted to our algorithm in order to benefit from the transferred knowledge observed in historical periods from the same region. Based on our previous work, a convergence theorem is provided for the proposed policy. Extensive simulations demonstrate that the proposed algorithms follow the traffic load variation during the day and contribute to a performance jump-start in EE improvement under various practical traffic load profiles. It also demonstrates that proposed schemes can significantly reduce the total energy consumption of cellular network, e.g., up to 70% potential energy savings based on a real traffic profile.

Power Control and Frequency Band Selection Policies for Underlay MIMO Cognitive Radio

We study power control and frequency band selection policies for multi-band underlay MIMO cognitive radio with the objective of maximizing the rate of a secondary user (SU) link while limiting the interference leakage toward primary users (PUs) below a threshold. The goal of the SU in each policy is to select one frequency band in each time slot and determine the transmit power. To limit the interference toward PU in time-varying channels, we propose fixed and dynamic transmit power control schemes which depend on PU traffic and the temporal correlation of channels between the SU and the PU. We study the performance of frequency band selection policies that use fixed or dynamic power control. We show that dynamic frequency band selection policies, e.g., policies based on multi-armed bandit framework, wherein SU selects a different frequency band in each slot, result in higher interference toward PU as compared to the fixed band policy wherein SU stays on one band. We also provide an expression for the gap between the rate achieved by SU under a clairvoyant policy and the fixed band policy. It is observed that this gap reduces with increased temporal correlation and with increased number of SU antennas.

Online Learning-Based Downlink Transmission Coordination in Ultra-Dense Millimeter Wave Heterogeneous Networks

In heterogeneous ultra-dense networks with millimeter wave macro cells and small cells, base stations (BSs) and mobile user equipments (UEs) perform beamforming operations to establish highly directional links. In spite of the spatial diversity achieved through directional links, as a number of BSs are densely deployed, inter-cell interference caused by concurrent directional transmissions of adjacent BSs becomes severe, resulting in downlink performance degradation in the network. However, it is very difficult to manage inter-cell interference because of the nature of the time-varying wireless fading environment, the dynamic changes in beam propagation directivity, and unpredictable UEs’ locations. In this paper, we propose an online learning-based transmission coordination algorithm based on the framework of multi-armed bandits to learn the unknown characteristics of inter-BS interference and exploit learned data to derive an optimal policy for maximizing the number of successful downlink transmissions. Through the numerical simulations, we verify the effectiveness of the proposed online learning-based inter-BS interference management scheme.

Dynamic Multi-Armed Bandit Algorithm for the Cyclic Bandwidth Sum Problem

Memetic algorithms (MAs) are a powerful resource when dealing with optimization problems, combining the diversification of the population-based approaches with the intensification of local search. However, their success depends on the combination of operators and their ability to cope with the intrinsic difficulties of a problem. Choosing the most suitable combination of operators that better suits a given problem (or a set of instances of a problem) has proved to be a defiant and time-consuming task. An approach to this task is the adaptive operator selection (AOS), based on the idea of choosing operators during execution time based on some reward system related to their performance. In this paper, we continue our previous work on studying the effectiveness of several operators of an MA to solve the cyclic bandwidth sum problem (CBSP), now extending the operator set and incorporating the dynamic multi-armed bandit (DMAB) framework to adaptively adjust the MA's operators. The resulting technique, named DMAB+MA, is compared to the independent MA versions in a full factorial experiment and with respect to two reference algorithms of the literature. It was found that the quality of the solutions achieved by DMAB+MA significantly improved the best-known results provided by the state-of-the-art algorithms while keeping the competitive execution times with respect to the independent MA versions. Moreover, DMAB+MA was able to provide optimal/best-known solutions for the 40 tested graphs (with different topologies) and to establish new better upper bounds for 12 of them. INDEX TERMS Cyclic bandwidth sum problem, dynamic multi-armed bandit, adaptive operator selection, memetic algorithms.

Online Learning for Position-Aided Millimeter Wave Beam Training

Accurate beam alignment is essential for the beam-based millimeter wave communications. The conventional beam sweeping solutions often have large overhead, which is unacceptable for mobile applications, such as a vehicle to everything. The learning-based solutions that leverage the sensor data (e.g., position) to identify the good beam directions are one approach to reduce the overhead. Most existing solutions, though, are supervised learning, where the training data are collected beforehand. In this paper, we use a multi-armed bandit framework to develop the online learning algorithms for beam pair selection and refinement. The beam pair selection algorithm learns coarse beam directions in some predefined beam codebook, e.g., in discrete angles, separated by the 3 dB beamwidths. The beam refinement fine-tunes the identified directions to match the peak of the power angular spectrum at that position. The beam pair selection uses the upper confidence bound with a newly proposed risk-aware feature, while the beam refinement uses a modified optimistic optimization algorithm. The proposed algorithms learn to recommend the good beam pairs quickly. When using $16\times 16$ arrays at both transmitter and receiver, it can achieve, on average, 1-dB gain over the exhaustive search (over $271\times 271$ beam pairs) on the unrefined codebook within 100 time steps with a training budget of only 30 beam pairs.

Optimal Query Selection Using Multi-Armed Bandits.

Query selection for latent variable estimation is conventionally performed by opting for observations with low noise or optimizing information theoretic objectives related to reducing the level of estimated uncertainty based on the current best estimate. In these approaches, typically the system makes a decision by leveraging the current available information about the state. However, trusting the current best estimate results in poor query selection when truth is far from the current estimate, and this negatively impacts the speed and accuracy of the latent variable estimation procedure. We introduce a novel sequential adaptive action value function for query selection using the multi-armed bandit (MAB) framework which allows us to find a tractable solution. For this adaptive-sequential query selection method, we analytically show: (i) performance improvement in the query selection for a dynamical system, (ii) the conditions where the model outperforms competitors. We also present favorable empirical assessments of the performance for this method, compared to alternative methods, both using Monte Carlo simulations and human-in-the-loop experiments with a brain computer interface (BCI) typing system where the language model provides the prior information.

Fully Adaptive Designs for Clinical Trials: Simultaneous Learning from Multiple Patients

Clinical trials have traditionally followed a fixed design, in which randomization probabilities of patients to various treatments remains fixed throughout the trial and specified in the protocol. The primary goal of this static design is to learn about the efficacy of treatments. Response-adaptive designs, on the other hand, allow clinicians to use the learning about treatment effectiveness to dynamically adjust randomization probabilities of patients to various treatments as the trial progresses. An ideal adaptive design is one where patients are treated as effectively as possible without sacrificing the potential learning or compromising the integrity of the trial. We propose such a design, termed Jointly Adaptive, that uses forward-looking algorithms to fully exploit learning from multiple patients simultaneously. Compared to the best existing implementable adaptive design that employs a multiarmed bandit framework in a setting where multiple patients arrive sequentially, we show that our proposed design improves health outcomes of patients in the trial by up to 8.6%, in expectation, under a set of considered scenarios. Further, we demonstrate our design's effectiveness using data from a recently conducted stent trial. This paper also adds to the general understanding of such models by showing the value and nature of improvements over heuristic solutions for problems with short delays in observing patient outcomes. We do this by showing the relative performance of these schemes for maximum expected patient health and maximum expected learning objectives, and by demonstrating the value of a restricted-optimal-policy approximation in a practical example.

Crowdsourcing Exploration

Motivated by the proliferation of online platforms that collect and disseminate consumers’ experiences with alternative substitutable products/services, we investigate the problem of optimal information provision when the goal is to maximize aggregate consumer surplus. We develop a decentralized multiarmed bandit framework where a forward-looking principal (the platform designer) commits up front to a policy that dynamically discloses information regarding the history of outcomes to a series of short-lived rational agents (the consumers). We demonstrate that consumer surplus is nonmonotone in the accuracy of the designer’s information-provision policy. Because consumers are constantly in “exploitation” mode, policies that disclose accurate information on past outcomes suffer from inadequate “exploration.” We illustrate how the designer can (partially) alleviate this inefficiency by employing a policy that strategically obfuscates the information in the platform’s possession; interestingly, such a policy is beneficial despite the fact that consumers are aware of both the designer’s objective and the precise way by which information is being disclosed to them. More generally, we show that the optimal information-provision policy can be obtained as the solution of a large-scale linear program. Noting that such a solution is typically intractable, we use our structural findings to design an intuitive heuristic that underscores the value of information obfuscation in decentralized learning. We further highlight that obfuscation remains beneficial even if the designer can directly incentivize consumers to explore through monetary payments. This paper was accepted by Serguei Netessine, operations management.

Observational and reinforcement pattern-learning: An exploratory study

Understanding how individuals learn in an unknown environment is an important problem in economics. We model and examine experimentally behavior in a very simple multi-armed bandit framework in which participants do not know the inter-temporal payoff structure. We propose a baseline reinforcement learning model that allows for pattern-recognition and change in the strategy space. We also analyse three augmented versions that accommodate observational learning from the actions and/or payoffs of another player. The models successfully reproduce the distributional properties of observed discovery times and total payoffs. Our study further shows that when one of the pair discovers the hidden pattern, observing another’s actions and/or payoffs improves discovery time compared to the baseline case.

Generalized Global Bandit and Its Application in Cellular Coverage Optimization

Motivated by the engineering problem of cellular coverage optimization, we propose a novel multiarmed bandit model called generalized global bandit. We develop a series of greedy algorithms that have the capability to handle nonmonotonic but decomposable reward functions, multidimensional global parameters, and switching costs. The proposed algorithms are rigorously analyzed under the multiarmed bandit framework, where we show that they achieve bounded regret, and hence, they are guaranteed to converge to the optimal arm in finite time. The algorithms are then applied to the cellular coverage optimization problem to achieve the optimal tradeoff between sufficient small cell coverage and limited macroleakage without prior knowledge of the deployment environment. The performance advantage of the new algorithms over existing bandits solutions is revealed analytically and further confirmed via numerical simulations. The key element behind the performance improvement is a more efficient “trial and error” mechanism, in which any trial will help improve the knowledge of all candidate power levels.

Relay Selection for Underwater Acoustic Sensor Networks: A Multi-User Multi-Armed Bandit Formulation

Multi-user cooperative transmission is an attractive architecture for underwater acoustic sensor networks (UASNs). Cooperative transmission depends on careful allocations of resources such as relay selection, but traditional relay selection requires precise measurements of channel state information, which is infeasible for multi-user cooperative transmission due to the unique features and hardware restrictions of UASNs. In this paper, we model multi-user relay selection under a multiuser multi-armed bandit (MU-MAB) framework, whereby users are not provided any prior knowledge about underwater acoustic channel conditions. We first exploit a novel MU-MAB algorithm, DSMU-MAB, for relay selection, assuming that the reward distributions are initially unknown but remain constant. Second, we consider an evolving environment in which the reward distributions undergo changes in time, and DSMU-rMAB, a derivative of DSMU-MAB, is proposed, which can be robust to abrupt changes in underwater communication environments. The proposed algorithms not only help sources find the suitable relays to achieve a high quality transmission and avoid collisions among users but also reduce the mass of information exchanged among users. We established the effectiveness of our proposed algorithms using theoretical and numerical analyses.

Spectrum Blind Recovery and Application in Non-uniform Sampling Based Automatic Modulation Classifier

Multi-standard wireless communication radios (MWCRs) capable of digitizing wideband signal to support wide variety of data-intensive services are desired. Limited reconfigurability of the analog front end along with hardware and cost constraints of high-speed analog-to-digital converters have generated significant interest in non-uniform (sub-Nyquist) sampling (NUS) and digital reconstruction-based MWCRs. Existing reconstruction approaches require prior knowledge of sparsity which may not be available in the dynamic spectrum environment. To alleviate this problem, a blind and adaptive reconstruction approach has been proposed in this paper. The proposed approach employs multi-armed Bandit framework to estimate the spectrum occupancy. Simulation results show that the average normalized mean square error of the proposed approach is 10–20% lower than other reconstruction approaches. Next, cumulant and machine learning-based automatic modulation classifier (AMC) is designed to validate the usefulness of the proposed approach in practical applications. Simulation results show that the classification accuracy of NUS-based AMC approaches, uniform sampling-based AMC with increase in signal-to-noise ratio and proposed approach is superior to others. The simulation results are further verified on the proposed universal software radio peripheral testbed in real radio environment. Experimental results demonstrate the close resemblance with simulation results.

Optimizing Infill Drilling Decisions Using Multi-Armed Bandits: Application in a Long-Term, Multi-Element Stockpile

Mining operations face a decision regarding additional drilling several times during their lifetime. The two questions that always arise upon making this decision are whether more drilling is required and, if so, where the additional drill holes should be located. The method presented in this paper addresses both of these questions through an optimization in a multi-armed bandit (MAB) framework. The MAB optimizes the best infill drilling pattern while taking geological uncertainty into account by using multiple conditional simulations for the deposit under consideration. The proposed method is applied to a long-term, multi-element stockpile, which is a part of a gold mining complex. The stockpiles in this mining complex are of particular interest due to difficult-to-meet blending requirements. In several mining periods grade targets of deleterious elements at the processing plant can only be met by using high amounts of stockpiled material. The best pattern is defined in terms of causing the most material type changes for the blocks in the stockpile. Material type changes are the driver for changes in the extraction sequence, which ultimately defines the value of a mining operation. The results of the proposed method demonstrate its practical aspects and its effectiveness towards the optimization of infill drilling schemes.

Decentralized Blind Spectrum Selection in Cognitive Radio Networks Considering Handoff Cost

Due to the spectrum varying nature of cognitive radio networks, secondary users are required to perform spectrum handoffs when the spectrum is occupied by primary users, which will lead to a handoff delay. In this paper, based on the multi-armed bandit framework of medium access in decentralized cognitive radio networks, we investigate blind spectrum selection problem of secondary users whose sensing ability of cognitive radio is limited and the channel statistics are a priori unknown, taking the handoff delay as a fixed handoff cost into consideration. In this scenario, secondary users have to make the choice of either staying foregoing spectrum with low availability or handing off to another spectrum with higher availability. We model the problem and investigate the performance of three representative policies, i.e., ρPRE, SL(K), kth-UCB1. The simulation results show that, despite the inclusion of the fixed handoff cost, these policies achieve the same asymptotic performance as that without handoff cost. Moreover, through comparison of these policies, we found the kth-UCB1 policy has better overall performance.