Combinatorial Multi-armed Bandit Research Articles

A Multi-Armed Bandit Approach to Wireless Information and Power Transfer

Multiple wireless transmitters communicate with their information receivers while radiating RF energy to multiple nearby energy harvesters. The channels from the transmitters to the receivers and to the energy harvesters are time-varying and unknown. The transmitters jointly determine their multi-antenna transmissions to fairly charge all the energy harvesters while maintaining the signal-to-interference-and-noise ratio at the receivers. In this letter, this task is formulated as a combinatorial multi-armed bandit problem and is solved with the upper confidence bound algorithm. Numerical results show that the algorithm can quickly converge to the optimal strategy with moderate signaling overhead. It has superior performance in fair energy harvesting while maintaining communication quality.

Smart vehicular communication via 5G mmWaves

Millimeter-Wave (mmWave) communication is conceived as a viable approach for 5G vehicular communication systems, where vehicles are equipped with more sensors that generate Gbps data for future autonomous driving. However, such directional mmWave communication relies on accurate beam alignment and is sensitive to blockage. Dense deployment of mmWave base stations (mmBSs) and high mobility of vehicles also lead to frequent handovers and complex beam alignment calculation. 5G mmWave vehicular communication calls for a smart and stable solution. To this end, we propose an online learning scheme to address the problem of beam selection with blockage-free guarantee in 5G mmWave vehicular networks. We first model this problem as a contextual combinatorial multi-armed bandit (MAB) problem with QoS constraints and delayed feedback. Next, we propose an online learning algorithm, BPG, to predict beam directions, with provable sub-linear regret and blockage-free bounds. BPG exploits the context space and learns the expected weight of each beam from arrived vehicles’ contexts and the delayed feedback. To validate the efficiency of BPG, we also conduct trace-driven simulations based on real-world traffic patterns. Simulation results show that BPG achieves close-to-optimal throughput with low violation and outperforms other benchmark algorithms.

Learning-Based Beamforming for Multi-User Vehicular Communications: A Combinatorial Multi-Armed Bandit Approach

There is an urgent need to support high data rate among connected vehicles for both safety and infotainment purposes. However, high rate for vehicular communications is impacted by several challenges such as optimal beam tracking for multiple simultaneous-transmitting vehicles in highly-dynamic environments. In this paper, we aim to maximize the overall network throughput for multi-vehicular communications. We propose a reinforcement learning (RL) approach called combinatorial multi-armed bandit (CMAB) framework, where multiple arms (i.e., actions) form a super arm and can be played together, to handle the beam selection problem in a vehicular network. More specifically, we propose an adaptive combinatorial Thompson sampling algorithm, namely adaptive CTS, that CMAB embodies for appropriate selection of simultaneous beams in a high-mobility vehicular environment. The proposed approach applies the smart exploration-exploitation trade-off for the fast selection of beams. As the proposed adaptive CTS scheme produces a higher complexity over the search space which increases exponentially with the number of user, we also propose a sequential Thompson sampling (TS) algorithm where beam selection is performed in a one-by-one manner. We analyze the regret bounds for the proposed beam tracking algorithms. The performance of the proposed strategies are evaluated for multi-vehicle millimeter-wave (mmWave) simulation environments. Our results suggest that the proposed sequential approach performs almost similar to the simultaneous adaptive CTS scheme for tracking optimal beams in a multi-vehicular network with much reduced complexity. Simulation results also show that both of our proposed strategies approach the optimal achievable rate achieved by the genie-aided solution.

Adaptive policies for perimeter surveillance problems

We consider the problem of sequentially choosing observation regions along a line, with an aim of maximising the detection of events of interest. Such a problem may arise when monitoring the movements of endangered or migratory species, detecting crossings of a border, policing activities at sea, and in many other settings. In each case, the key operational challenge is to learn an allocation of surveillance resources which maximises successful detection of events of interest. We present a combinatorial multi-armed bandit model with Poisson rewards and a novel filtered feedback mechanism - arising from the failure to detect certain intrusions - where reward distributions are dependent on the actions selected. Our solution method is an upper confidence bound approach and we derive upper and lower bounds on its expected performance. We prove that the gap between these bounds is of constant order, and demonstrate empirically that our approach is more reliable in simulated problems than competing algorithms.

Learning-Based Task Offloading for Delay-Sensitive Applications in Dynamic Fog Networks

Fog computing has the potential to liberate the computation-intensive mobile devices by task offloading. In this paper, we propose an online learning based task offloading algorithm for delay-sensitive applications in dynamic fog networks, which combines with the Combinatorial Multi-Armed Bandits (CMAB) framework. First, the proposed algorithm learns the sharing computing resources of fog nodes at a negligible computational cost. Then, we aim to minimize the task's offloading latency by jointly optimizing the task allocation decision and the spectrum scheduling. Finally, simulation results show that the proposed algorithm achieves much better delay performance than the traditional Upper Confidence Bound (UCB) algorithm and maintains ultra-low offloading delay in dynamic system state.

CBMoS: Combinatorial Bandit Learning for Mode Selection and Resource Allocation in D2D Systems

The complexity of the mode selection and resource allocation (MS&RA) problem has hampered the commercialization progress of Device-to-Device (D2D) communication in 5G networks. Furthermore, the combinatorial nature of MS&RA has forced the majority of existing proposals to focus on constrained scenarios or offline solutions to contain the size of the problem. Given the real-time constraints in actual deployments, a reduction in computational complexity is necessary. Adaptability is another key requirement for mobile networks that are exposed to constant changes such as channel quality fluctuations and mobility. In this article, we propose an online learning technique (i.e., CBMoS) which leverages combinatorial multi-armed bandits (CMAB) to tackle the combinatorial nature of MS&RA. Furthermore, our two-stage CMAB design results in a tight model, which eliminates the theoretically feasible but practicality invalid options from the solution space. We prototype the first SDR-based D2D testbed to verify the performance of CBMoS under real-world conditions. The simulations confirm that the fast learning speed of CBMoS leads to outperforming the benchmark schemes by up to 132%. In experiments, CBMoS exhibits even higher performance (up to 142%) than in the simulations. This stems from the adaptability/fast learning speed of CBMoS in presence of high channel dynamics which cannot be captured via statistical channel models used in the simulators.

Budget-Constrained Edge Service Provisioning With Demand Estimation via Bandit Learning

Shared edge computing platforms, which enable Application Service Providers (ASPs) to deploy applications in close proximity to mobile users are providing ultra-low latency and location-awareness to a rich portfolio of services. Though ubiquitous edge service provisioning, i.e., deploying the application at all possible edge sites, is always preferable, it is impractical due to often limited operational budget of ASPs. In this case, an ASP has to cautiously decide where to deploy the edge service and how much budget it is willing to use. A central issue here is that the service demand received by each edge site, which is the key factor of deploying benefit, is unknown to ASPs a priori. What's more complicated is that this demand pattern varies temporally and spatially across geographically distributed edge sites. In this paper, we investigate an edge resource rental problem where the ASP learns service demand patterns for individual edge sites while renting computation resource at these sites to host its applications for edge service provisioning. An online algorithm, called Context-aware Online Edge Resource Rental (COERR), is proposed based on the framework of Contextual Combinatorial Multi-armed Bandit (CC-MAB). COERR observes side-information (context) to learn the demand patterns of edge sites and decides rental decisions (including where to rent the computation resource and how much to rent) to maximize ASP's utility given a limited budget. COERR provides a provable performance achieving sublinear regret compared to an Oracle algorithm that knows exactly the expected service demand of edge sites. Experiments are carried out on a real-world dataset and the results show that COERR significantly outperforms other benchmarks.

IEEE 802.15.4.e TSCH-Based Scheduling for Throughput Optimization: A Combinatorial Multi-Armed Bandit Approach

In TSCH, which is a MAC mechanism set of the IEEE 802.15.4e amendment, calculation, construction, and maintenance of the packet transmission schedules are not defined. Moreover, to ensure optimal throughput, most of the existing scheduling methods are based on the assumption that instantaneous and accurate Channel State Information (CSI) is available. However, due to the inevitable errors in the channel estimation process, this assumption cannot be materialized in many practical scenarios. In this paper, we propose two alternative and realistic approaches. In our first approach, we assume that only the statistical knowledge of CSI is available a priori. Armed with this knowledge, the average packet rate on each link is computed and then, using the results, the throughput-optimal schedule for the assignment of (slot-frame) cells to links can be formulated as a max-weight bipartite matching problem, which can be solved efficiently using the well-known Hungarian algorithm. In the second approach, we assume that no CSI knowledge (even statistical) is available at the design stage. For this zero-knowledge setting, we introduce a machine learning-based algorithm by formally modeling the scheduling problem in terms of a combinatorial multi-armed bandit (CMAB) process. Our CMAB-based scheme is widely applicable to many real operational environments, thanks to its reduced reliance on design-time knowledge. Simulation results show that the average throughput obtained by the statistical CSI-based method is within the margin of 15% from the theoretical upper bound associated with perfect instantaneous CSI. The aforesaid margin is around 18% for our learning-theoretic solution.

Partially Observable Multi-Sensor Sequential Change Detection: A Combinatorial Multi-Armed Bandit Approach

This paper explores machine learning to address a problem of Partially Observable Multi-sensor Sequential Change Detection (POMSCD), where only a subset of sensors can be observed to monitor a target system for change-point detection at each online learning round. In contrast to traditional Multisensor Sequential Change Detection tasks where all the sensors are observable, POMSCD is much more challenging because the learner not only needs to detect on-the-fly whether a change occurs based on partially observed multi-sensor data streams, but also needs to cleverly choose a subset of informative sensors to be observed in the next learning round, in order to maximize the overall sequential change detection performance. In this paper, we present the first online learning study to tackle POMSCD in a systemic and rigorous way. Our approach has twofold novelties: (i) we attempt to detect changepoints from partial observations effectively by exploiting potential correlations between sensors, and (ii) we formulate the sensor subset selection task as a Multi-Armed Bandit (MAB) problem and develop an effective adaptive sampling strategy using MAB algorithms. We offer theoretical analysis for the proposed online learning solution, and further validate its empirical performance via an extensive set of numerical studies together with a case study on real-world data sets.

Selecting multiple web adverts: A contextual multi-armed bandit with state uncertainty

We present a method to solve the problem of choosing a set of adverts to display to each of a sequence of web users. The objective is to maximise user clicks over time and to do so we must learn about the quality of each advert in an online manner by observing user clicks. We formulate the problem as a novel variant of a contextual combinatorial multi-armed bandit problem. The context takes the form of a probability distribution over the user's latent topic preference, and rewards are a particular nonlinear function of the selected set and the context. These features ensure that optimal sets of adverts are appropriately diverse. We give a flexible solution method which combines submodular optimisation with existing bandit index policies. User state uncertainty creates ambiguity in interpreting user feedback which prohibits exact Bayesian updating, but we give an approximate method that is shown to work well.

Action Abstractions for Combinatorial Multi-Armed Bandit Tree Search

Search algorithms based on combinatorial multi-armed bandits (CMABs) are promising for dealing with state-space sequential decision problems. However, current CMAB-based algorithms do not scale to problem domains with very large actions spaces, such as real-time strategy games played in large maps. In this paper we introduce CMAB-based search algorithms that use action abstraction schemes to reduce the action space considered during search. One of the approaches we introduce use regular action abstractions (A1N), while the other two use asymmetric action abstractions (A2N and A3N). Empirical results on MicroRTS show that A1N, A2N, and A3N are able to outperform an existing CMAB-based algorithm in matches played in large maps, and A3N is able to outperform all state-of-the-art search algorithms tested.

User-centric content sharing via cache-enabled device-to-device communication

Cache-enabled device-to-device (D2D) communication has emerged as a promising paradigm for caching popular contents at user devices and using D2D communication among adjacent users to relieve cellular traffic overload. To maximize traffic offloading, optimal content-selection for caching and helper-receiver matching are two key concerns. In this paper, we deal with the two key issues by leveraging user characteristics, which are crucial to D2D content sharing but often omitted by existing work in this field. We first propose a comprehensive model that profiles user characteristics including user preference, user activity degree, and user relationship. To provide an accurate estimation of user relationship, a new metric named expected correlation coefficient (ECC) is further proposed, which can reveal the potential offloading opportunities behind the contacts between users. We then design a novel user-centric caching policy that integrates the above-mentioned models into content caching policy to optimize the caching efficiency. Furthermore, we propose an online learning algorithm by leveraging the Combinatorial Multi-Armed Bandits (CMAB) method to achieve efficient helper-receiver matching among users. Experiments based on real-world traces and content requesting records were conducted. The results demonstrate that when compared with existing algorithms, our proposed algorithm improves the offloading ratio by nearly 10% and the system utility approximately by 20%.

Learning in Combinatorial Optimization: What and How to Explore

When moving from the traditional to combinatorial multiarmed bandit setting, addressing the classical exploration versus exploitation trade-off is a challenging task. In “Learning in Combinatorial Optimization: What and How to Explore,” Modaresi, Sauré, and Vielma show that the combinatorial setting has salient features that distinguish it from the traditional bandit. In particular, combinatorial structure induces correlation between cost of different solutions, thus raising the questions of what parameters to estimate and how to collect and combine information. The authors answer such questions by developing a novel optimization problem called the lower-bound problem (LBP). They establish a fundamental limit on asymptotic performance of any admissible policy and propose near-optimal LBP-based policies. Because LBP is likely intractable in practice, they propose policies that instead solve a proxy for LBP, which they call the optimality cover problem (OCP). They provide strong evidence of practical tractability of OCP and illustrate the markedly superior performance of OCP-based policies numerically.

Adaptive Energy Storage Management in Green Wireless Networks

Time-varying wireless channel as well as the variability of renewable energy supply and energy prices are practically unknown in advance. To address such dynamic statistics of wireless networks, this letter develops an adaptive strategy inspired by combinatorial multiarmed bandit model for energy storage management and cost-aware coordinated load control at the base stations. The proposed strategy makes online foresighted decisions on the amount of energy to be stored in storage to minimize the average energy cost over long-time horizon. Simulation results validate the superiority of the proposed strategy over a recently proposed storage-free learning-based design.

A Bandit Approach to Price-Aware Energy Management in Cellular Networks

We introduce a reinforcement learning algorithm inspired by the combinatorial multi-armed bandit problem to minimize the time-averaged energy cost at individual base stations (BSs), powered by various energy markets and local renewable energy sources, over a finite-time horizon. The algorithm sustains traffic demands by enabling sparse beamforming to schedule dynamic user-to-BS allocation and proactive energy provisioning at BSs to make ahead-of-time price-aware energy management decisions. Simulation results indicate a superior performance of the proposed algorithm in reducing the overall energy cost, as compared with recently proposed cooperative energy management designs.

Combinatorial Multi-armed Bandits for Real-Time Strategy Games

Games with large branching factors pose a significant challenge for game tree search algorithms. In this paper, we address this problem with a sampling strategy for Monte Carlo Tree Search (MCTS) algorithms called "naive sampling", based on a variant of the Multi-armed Bandit problem called "Combinatorial Multi-armed Bandits" (CMAB). We analyze the theoretical properties of several variants of naive sampling, and empirically compare it against the other existing strategies in the literature for CMABs. We then evaluate these strategies in the context of real-time strategy (RTS) games, a genre of computer games characterized by their very large branching factors. Our results show that as the branching factor grows, naive sampling outperforms the other sampling strategies.

Combinatorial multi-armed bandits for real-time strategy games

Games with large branching factors pose a significant challenge for game tree search algorithms. In this paper, we address this problem with a sampling strategy for Monte Carlo Tree Search (MCTS) a...

Combinatorial multi-armed bandit and its extension to probabilistically triggered arms

We define a general framework for a large class of combinatorial multi-armed bandit (CMAB) problems, where subsets of base arms with unknown distributions form super arms. In each round, a super ar...

Efficient task assignment for spatial crowdsourcing: A combinatorial fractional optimization approach with semi-bandit learning

Spatial crowdsourcing has emerged as a new paradigm for solving problems in the physical world with the help of human workers. A major challenge in spatial crowdsourcing is to assign reliable workers to nearby tasks. The goal of such task assignment process is to maximize the task completion in the face of uncertainty. This process is further complicated when tasks arrivals are dynamic and worker reliability is unknown. Recent research proposals have tried to address the challenge of dynamic task assignment. Yet the majority of the proposals do not consider the dynamism of tasks and workers. They also make the unrealistic assumptions of known deterministic or probabilistic workers’ reliabilities. In this paper, we propose a novel approach for dynamic task assignment in spatial crowdsourcing. The proposed approach combines bi-objective optimization with combinatorial multi-armed bandits. We formulate an online optimization problem to maximize task reliability and minimize travel costs in spatial crowdsourcing. We propose the distance-reliability ratio (DRR) algorithm based on a combinatorial fractional programming approach. The DRR algorithm reduces travel costs by 80% while maximizing reliability when compared to existing algorithms. We extend the DRR algorithm for the scenario when worker reliabilities are unknown. We propose a novel algorithm (DRR-UCB) that uses an interval estimation heuristic to approximate worker reliabilities. Experimental results demonstrate that the DRR-UCB achieves high reliability in the face of uncertainty. The proposed approach is particularly suited for real-life dynamic spatial crowdsourcing scenarios. This approach is generalizable to the similar problems in other areas in expert systems. First, it encompasses online assignment problems when the objective function is a ratio of two linear functions. Second, it considers situations when intelligent and repeated assignment decisions are needed under uncertainty.

Planning Curtailment of Renewable Generation in Power Grids

The increasing penetration of renewable sources like solar energy add new dimensions in planning power grid operations. We study the problem of curtailing a subset of prosumers generating solar power with the twin goals of being close to a target collection and maintaining fairness across prosumers. The problem is complicated by the uncertainty in the amount of energy fed-in by each prosumer and the large problem size in terms of number of prosumers. To meet these challenges, we propose an algorithm based on the Combinatorial Multi-Armed Bandit problem with an approximate Knapsack based oracle. With real-data on solar panel output across multiple prosumers, we are able to demonstrate the effectiveness of the proposed algorithm.