Combinatorial Multi-armed Bandit Research Articles

The Combinatorial Multi-Armed Bandit Problem and Its Application to Real-Time Strategy Games

Game tree search in games with large branching factors is a notoriously hard problem. In this paper, we address this problem with a new sampling strategy for Monte Carlo Tree Search (MCTS) algorithms, called "Naive Sampling", based on a variant of the Multi-armed Bandit problem called the "Combinatorial Multi-armed Bandit" (CMAB) problem. We present a new MCTS algorithm based on Naive Sampling called NaiveMCTS, and evaluate it in the context of real-time strategy (RTS) games. Our results show that as the branching factor grows, NaiveMCTS performs significantly better than other algorithms.

Online Learning Approach for Predictive Real-Time Energy Trading in Cloud-RANs.

Constantly changing electricity demand has made variability and uncertainty inherent characteristics of both electric generation and cellular communication systems. This paper develops an online learning algorithm as a prescheduling mechanism to manage the variability and uncertainty to maintain cost-aware and reliable operation in cloud radio access networks (Cloud-RANs). The proposed algorithm employs a combinatorial multi-armed bandit model and minimizes the long-term energy cost at remote radio heads. The algorithm preschedules a set of cost-efficient energy packages to be purchased from an ancillary energy market for the future time slots by learning both from cooperative energy trading at previous time slots and by exploring new energy scheduling strategies at the current time slot. The simulation results confirm a significant performance gain of the proposed scheme in controlling the available power budgets and minimizing the overall energy cost compared with recently proposed approaches for real-time energy resources and energy trading in Cloud-RANs.

Cooperative content caching and power allocation strategy for fog radio access networks

Fog radio access networks (F-RANs) architecture is regarded as a prominent solution to deploy the caching and computing functions at the edge nodes of the network. However, the ever-increasing data traffic demand and the latency requirements of emerging applications pose new challenges to F-RANs. To minimize content fetching latency, we propose a cooperative content caching and power allocation scheme, which proactively caches the content at the network edge and enables users to dynamically obtain the desired content either from Fog-Access points (F-APs) or proximate user equipments (UEs) through Device-to-Device (D2D) communication. Furthermore, the proposed scheme allocates appropriate transmit power to D2D user equipments (DUEs) so that the transmission rate can be maximized. Specifically, the cooperative content caching issue is initially formulated as a probability-triggered combinatorial multi-armed bandit (CMAB) framework. By considering the user preference and content popularity prediction, an enhanced multi-agent reinforcement learning algorithm is proposed to obtain an optimal caching strategy. Besides, to minimize the content fetching latency and guarantee that each UE can retrieve the desired content, the power allocation problem is then modeled as maximizing the sum data rates of users. Finally, a Q-learning based power allocation strategy is derived. The simulation results based on the dataset from MovieLens reveal that compared with baseline methods, our proposed cooperative caching and power allocation scheme can not only reduce the content fetching latency but also increase the cache hit rate.

CMAB-Based Reverse Auction for Unknown Worker Recruitment in Mobile Crowdsensing

Mobile CrowdSensing (MCS), through which a requester can coordinate a crowd of workers to accomplish some data collection tasks, has been recognized as a promising paradigm for large-scale data acquisition in recent years. Many researches focus on the worker recruitment problem in MCS, but most of them either have the assumption that workers’ qualities are known ahead of time or cannot ensure that workers report costs honestly. In this paper, we propose an incentive mechanism based on Combinatorial Multi-Armed Bandit and reverse Auction, called CMABA, to solve the multiple unknown workers recruitment problem in MCS. Our objective is to determine a recruiting strategy to maximize the total sensing quality under a limited budget, while ensuring truthfulness and individual rationality of sensing workers. We theoretically prove that our CMABA mechanism achieves truthfulness and individual rationality, and then analyze the regret of the mechanism. Based on CMABA, we ulteriorly propose an adaptive incentive mechanism, called ACMABA, to recruit workers via the alternative worker recruitment and quality update, which can achieve a higher total sensing quality and lower regret. Additionally, we also demonstrate significant performances of the CMABA and ACMABA mechanisms through extensive simulations on real-world data traces.

Budgeted Unknown Worker Recruitment for Heterogeneous Crowdsensing Using CMAB

Mobile crowdsensing, through which a requester can coordinate a crowd of workers to complete some sensing tasks, has attracted significant attention recently. In this paper, we focus on the unknown worker recruitment problem in mobile crowdsensing, where workers' sensing qualities are unknown a priori. We consider the scenario of recruiting workers to complete some continuous sensing tasks. In each round, every task may be covered by more than one recruited workers, but its completion quality only depends on these workers' maximum sensing quality. Our objective is to determine a recruiting strategy to maximize the total weighted completion quality under a limited budget. We model such unknown worker recruitment process as a novel combinatorial multi-armed bandit problem, and propose an unknown worker recruitment algorithm based on the modified upper confidence bound. Moreover, we extend the problem to the case where the workers' costs are also unknown and design the corresponding algorithm. We analyze the regret bounds of the two proposed algorithms. In addition, we also study the unknown worker recruitment problem with fairness constraints. For this problem, we devise a fairness-aware unknown worker recruitment algorithm. Finally, we demonstrate the performance of the proposed algorithms through extensive experimental simulations on real-world traces.

Joint Switch–Controller Association and Control Devolution for SDN Systems: An Integrated Online Perspective of Control and Learning

In software-defined networking (SDN) systems, it is a common practice to adopt a multi-controller design and control devolution techniques to improve the performance of the control plane. However, in such systems the decision-making for joint switch-controller association and control devolution often involves various uncertainties, e.g., the temporal variations of controller accessibility, and computation and communication costs of switches. In practice, statistics of such uncertainties are unattainable and need to be learned in an online fashion, calling for an integrated design of learning and control. In this article, we formulate a stochastic network optimization problem that aims to minimize time-average system costs and ensure queue stability. By transforming the problem into a combinatorial multi-armed bandit problem with long-term stability constraints, we adopt bandit learning methods and optimal control techniques to handle the exploration-exploitation tradeoff and long-term stability constraints, respectively. Through an integrated design of online learning and online control, we propose an effective Learning-Aided Switch-Controller Association and Control Devolution ( LASAC ) scheme. Our theoretical analysis and simulation results show that LASAC achieves a tunable tradeoff between queue stability and system cost reduction with a sublinear time-averaged regret bound over a finite time horizon.

Thompson Sampling for Combinatorial Network Optimization in Unknown Environments

Influence maximization, adaptive routing, and dynamic spectrum allocation all require choosing the right action from a large set of alternatives. Thanks to the advances in combinatorial optimization, these and many similar problems can be efficiently solved given an environment with known stochasticity. In this paper, we take this one step further and focus on combinatorial optimization in unknown environments. We consider a very general learning framework called combinatorial multi-armed bandit with probabilistically triggered arms and a very powerful Bayesian algorithm called Combinatorial Thompson Sampling (CTS). Under the semi-bandit feedback model and assuming access to an oracle without knowing the expected base arm outcomes beforehand, we show that when the expected reward is Lipschitz continuous in the expected base arm outcomes CTS achieves O(Σ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i=1</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> logT /(p <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> Δ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> )) regret and a O(max{E[m√TlogT/p*], E[m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /p*]}) Bayesian regret, where m denotes the number of base arms, pi and Δi denote the minimum non-zero triggering probability and the minimum suboptimality gap of base arm i respectively, T denotes the time horizon, and p* denotes the overall minimum non-zero triggering probability. We also show that when the expected continuity, CTS achieves O(max{m√TlogT , m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> }) Bayesian reward satisfies the triggering probability modulated Lipschitz regret, and when triggering probabilities are non-zero for all base arms, CTS achieves O(1/p* log(1/p*)) regret independent of the time horizon. Finally, we numerically compare CTS with algorithms based on upper confidence bounds in several networking problems and show that CTS outperforms these algorithms by at least an order of magnitude in majority of the cases.

Caching by User Preference With Delayed Feedback for Heterogeneous Cellular Networks

The burgeoning network traffic imposes a huge burden on the network backbone. Caching popular files at the wireless network edge is promising to address the problem. In practice, file popularity is very unlikely to know in advance. Online learning algorithms are effective to learn this uncertainty in a sequential way. In each slot, the learning agent generates a caching policy (i.e., the to-be-cached files) and can observe users’ feedback about the caching policy within the same slot. This method implicitly requires that all of the users are able to provide feedback promptly. However, in practice, the availability of each individual user is affected by many factors, e.g., users are moving out of the service area temporarily, or they may still consume files in the previous slots, which may result in the feedback delay. In this paper, we propose a delay-tolerant wireless caching system that takes both the feedback delay and users’ availability into consideration. We frame the content caching problem as a stochastic combinatorial multi-armed bandit problem with delayed feedback and forced- to-sleep arms, and devise an intelligent caching algorithm called CFAUD to solve the problem. Also, we show that CFAUD is effective and efficient both theoretically and practically. Finally, experiments are conducted to compare the performance of the proposed algorithm with other well-known algorithms.

Distributed Task Replication for Vehicular Edge Computing: Performance Analysis and Learning-Based Algorithm

In a vehicular edge computing (VEC) system, vehicles can share their surplus computation resources to provide cloud computing services. The highly dynamic environment of the vehicular network makes it challenging to guarantee the task offloading delay. To this end, we introduce task replication to the VEC system, where the replicas of a task are offloaded to multiple vehicles at the same time, and the task is completed upon the first response among replicas. First, the impact of the number of task replicas on the offloading delay is characterized, and the optimal number of task replicas is approximated in closed-form. Based on the analytical result, we design a learning-based task replication algorithm (LTRA) with combinatorial multi-armed bandit theory, which works in a distributed manner and can automatically adapt itself to the dynamics of the VEC system. A realistic traffic scenario is used to evaluate the delay performance of the proposed algorithm. Results show that, under our simulation settings, LTRA with an optimized number of task replicas can reduce the average offloading delay by over 30% compared to the benchmark without task replication, and at the same time can improve the task completion ratio from 97% to 99.6%.

Cognitive Analysis in Social Networks for Viral Marketing

Viral marketing is the modern version of the old “word-of-mouth” advertising, where companies choose a restricted number of persons, considered “influential,” recommending them products or services that will be in turn iteratively suggested. In this article, we propose cognitive models and algorithms for marketing applications through online social networks, considered as a graph database, and define the concept of influence graph leveraging particular user behavioral patterns, by querying the initial heterogeneous graph network. We also model the diffusion across the network, without any preliminary information, as a combinatorial multiarmed bandit problem, for the selection of most influential users. We have used the YELP social network as a case study for our approach, showing how it is possible to generate an influence graph considering several kinds of relevant paths (mainly considering reviews to the same firms) by which a user can influence other ones. Several experiments have been carried out and discussed, putting into evidence the effectiveness and efficacy of the proposed methods for influence maximization with respect to other approaches of state of the art.

A reliability-aware multi-armed bandit approach to learn and select users in demand response

One challenge in the optimization and control of societal systems is to handle the unknown and uncertain user behavior. This paper focuses on residential demand response (DR) and proposes a closed-loop learning scheme to address these issues. In particular, we consider DR programs where an aggregator calls upon residential users to change their demand so that the total load adjustment is close to a target value. To learn and select the right users, we formulate the DR problem as a combinatorial multi-armed bandit (CMAB) problem with a reliability objective. We propose a learning algorithm: CUCB-Avg (Combinatorial Upper Confidence Bound-Average), which utilizes both upper confidence bounds and sample averages to balance the tradeoff between exploration (learning) and exploitation (selecting). We consider both a fixed time-invariant target and time-varying targets, and show that CUCB-Avg achieves O(logT) and O(Tlog(T)) regrets respectively. Finally, we numerically test our algorithms using synthetic and real data, and demonstrate that our CUCB-Avg performs significantly better than the classic CUCB and also better than Thompson Sampling.

A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments

Continuous Integration (CI) environments have been increasingly adopted in the industry to allow frequent integration of software changes, making software evolution faster and cost-effective. In such environments, Test Case Prioritization (TCP) techniques play an important role to reduce regression testing costs, establishing a test case execution order that usually maximizes early fault detection. Existing works on TCP in CI environments (TCPCI) present some limitations. Few pieces of work consider CI particularities, such as the test case volatility, that is, they do not consider the dynamic environment of the software life-cycle in which new test cases can be added or removed (discontinued), characteristic related to the Exploration versus Exploitation (EvE) dilemma. To solve such a dilemma an approach needs to balance: i) the diversity of test suite; and ii) the quantity of new test cases and test cases that are error-prone or that comprise high fault-detection capabilities. To deal with this, most approaches use, besides the failure-history, other measures that rely on code instrumentation or require additional information, such as testing coverage. However, to maintain the information updated can be difficult and time-consuming, not scalable due to the test budget of CI environments. In this context, and to properly deal with the TCPCI problem, this work presents an approach based on Multi-Armed Bandit (MAB) called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">COLEMAN</monospace> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>C</b>ombinatorial V<b>O</b>lati<b>LE</b> <b>M</b>ulti-Armed B<b>AN</b>dit</i> ). The TCPCI problem falls into the category of volatile and combinatorial MAB, because multiple arms (test cases) need to be selected, and they are added or removed over the cycles. We conducted an evaluation considering three time budgets and eleven systems. The results show the applicability of our approach and that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">COLEMAN</monospace> outperforms the most similar approach from literature in terms of early fault detection and performance.

A Multiarmed Bandit Based Incentive Mechanism for a Subset Selection of Customers for Demand Response in Smart Grids

Demand response is a crucial tool to maintain the stability of the smart grids. With the upcoming research trends in the area of electricity markets, it has become a possibility to design a dynamic pricing system, and consumers are made aware of what they are going to pay. Though the dynamic pricing system (pricing based on the total demand a distributor company is facing) seems to be one possible solution, the current dynamic pricing approaches are either too complex for a consumer to understand or are too naive leading to inefficiencies in the system (either consumer side or distributor side). Due to these limitations, the recent literature is focusing on the approach to provide incentives to the consumers to reduce the electricity, especially in peak hours. For each round, the goal is to select a subset of consumers to whom the distributor should offer incentives so as to minimize the loss which comprises of cost of buying the electricity from the market, uncertainties at consumer end, and cost incurred to the consumers to reduce the electricity which is a private information to the consumers. Due to the uncertainties in the loss function (arising from renewable energy resources as well as consumption needs), traditional auction theory-based incentives face manipulation challenges. Towards this, we propose a novel combinatorial multi-armed bandit (MAB) algorithm, which we refer to as \namemab\ to learn the uncertainties along with an auction to elicit true costs incurred by the consumers. We prove that our mechanism is regret optimal and is incentive compatible. We further demonstrate efficacy of our algorithms via simulations.

A Multi-Armed Bandit Approach to Wireless Information and Power Transfer

Multiple wireless transmitters communicate with their information receivers while radiating RF energy to multiple nearby energy harvesters. The channels from the transmitters to the receivers and to the energy harvesters are time-varying and unknown. The transmitters jointly determine their multi-antenna transmissions to fairly charge all the energy harvesters while maintaining the signal-to-interference-and-noise ratio at the receivers. In this letter, this task is formulated as a combinatorial multi-armed bandit problem and is solved with the upper confidence bound algorithm. Numerical results show that the algorithm can quickly converge to the optimal strategy with moderate signaling overhead. It has superior performance in fair energy harvesting while maintaining communication quality.

Smart vehicular communication via 5G mmWaves

Millimeter-Wave (mmWave) communication is conceived as a viable approach for 5G vehicular communication systems, where vehicles are equipped with more sensors that generate Gbps data for future autonomous driving. However, such directional mmWave communication relies on accurate beam alignment and is sensitive to blockage. Dense deployment of mmWave base stations (mmBSs) and high mobility of vehicles also lead to frequent handovers and complex beam alignment calculation. 5G mmWave vehicular communication calls for a smart and stable solution. To this end, we propose an online learning scheme to address the problem of beam selection with blockage-free guarantee in 5G mmWave vehicular networks. We first model this problem as a contextual combinatorial multi-armed bandit (MAB) problem with QoS constraints and delayed feedback. Next, we propose an online learning algorithm, BPG, to predict beam directions, with provable sub-linear regret and blockage-free bounds. BPG exploits the context space and learns the expected weight of each beam from arrived vehicles’ contexts and the delayed feedback. To validate the efficiency of BPG, we also conduct trace-driven simulations based on real-world traffic patterns. Simulation results show that BPG achieves close-to-optimal throughput with low violation and outperforms other benchmark algorithms.

Learning-Based Beamforming for Multi-User Vehicular Communications: A Combinatorial Multi-Armed Bandit Approach

There is an urgent need to support high data rate among connected vehicles for both safety and infotainment purposes. However, high rate for vehicular communications is impacted by several challenges such as optimal beam tracking for multiple simultaneous-transmitting vehicles in highly-dynamic environments. In this paper, we aim to maximize the overall network throughput for multi-vehicular communications. We propose a reinforcement learning (RL) approach called combinatorial multi-armed bandit (CMAB) framework, where multiple arms (i.e., actions) form a super arm and can be played together, to handle the beam selection problem in a vehicular network. More specifically, we propose an adaptive combinatorial Thompson sampling algorithm, namely adaptive CTS, that CMAB embodies for appropriate selection of simultaneous beams in a high-mobility vehicular environment. The proposed approach applies the smart exploration-exploitation trade-off for the fast selection of beams. As the proposed adaptive CTS scheme produces a higher complexity over the search space which increases exponentially with the number of user, we also propose a sequential Thompson sampling (TS) algorithm where beam selection is performed in a one-by-one manner. We analyze the regret bounds for the proposed beam tracking algorithms. The performance of the proposed strategies are evaluated for multi-vehicle millimeter-wave (mmWave) simulation environments. Our results suggest that the proposed sequential approach performs almost similar to the simultaneous adaptive CTS scheme for tracking optimal beams in a multi-vehicular network with much reduced complexity. Simulation results also show that both of our proposed strategies approach the optimal achievable rate achieved by the genie-aided solution.

Adaptive policies for perimeter surveillance problems

We consider the problem of sequentially choosing observation regions along a line, with an aim of maximising the detection of events of interest. Such a problem may arise when monitoring the movements of endangered or migratory species, detecting crossings of a border, policing activities at sea, and in many other settings. In each case, the key operational challenge is to learn an allocation of surveillance resources which maximises successful detection of events of interest. We present a combinatorial multi-armed bandit model with Poisson rewards and a novel filtered feedback mechanism - arising from the failure to detect certain intrusions - where reward distributions are dependent on the actions selected. Our solution method is an upper confidence bound approach and we derive upper and lower bounds on its expected performance. We prove that the gap between these bounds is of constant order, and demonstrate empirically that our approach is more reliable in simulated problems than competing algorithms.

Learning-Based Task Offloading for Delay-Sensitive Applications in Dynamic Fog Networks

Fog computing has the potential to liberate the computation-intensive mobile devices by task offloading. In this paper, we propose an online learning based task offloading algorithm for delay-sensitive applications in dynamic fog networks, which combines with the Combinatorial Multi-Armed Bandits (CMAB) framework. First, the proposed algorithm learns the sharing computing resources of fog nodes at a negligible computational cost. Then, we aim to minimize the task's offloading latency by jointly optimizing the task allocation decision and the spectrum scheduling. Finally, simulation results show that the proposed algorithm achieves much better delay performance than the traditional Upper Confidence Bound (UCB) algorithm and maintains ultra-low offloading delay in dynamic system state.

CBMoS: Combinatorial Bandit Learning for Mode Selection and Resource Allocation in D2D Systems

The complexity of the mode selection and resource allocation (MS&RA) problem has hampered the commercialization progress of Device-to-Device (D2D) communication in 5G networks. Furthermore, the combinatorial nature of MS&RA has forced the majority of existing proposals to focus on constrained scenarios or offline solutions to contain the size of the problem. Given the real-time constraints in actual deployments, a reduction in computational complexity is necessary. Adaptability is another key requirement for mobile networks that are exposed to constant changes such as channel quality fluctuations and mobility. In this article, we propose an online learning technique (i.e., CBMoS) which leverages combinatorial multi-armed bandits (CMAB) to tackle the combinatorial nature of MS&RA. Furthermore, our two-stage CMAB design results in a tight model, which eliminates the theoretically feasible but practicality invalid options from the solution space. We prototype the first SDR-based D2D testbed to verify the performance of CBMoS under real-world conditions. The simulations confirm that the fast learning speed of CBMoS leads to outperforming the benchmark schemes by up to 132%. In experiments, CBMoS exhibits even higher performance (up to 142%) than in the simulations. This stems from the adaptability/fast learning speed of CBMoS in presence of high channel dynamics which cannot be captured via statistical channel models used in the simulators.

Budget-Constrained Edge Service Provisioning With Demand Estimation via Bandit Learning

Shared edge computing platforms, which enable Application Service Providers (ASPs) to deploy applications in close proximity to mobile users are providing ultra-low latency and location-awareness to a rich portfolio of services. Though ubiquitous edge service provisioning, i.e., deploying the application at all possible edge sites, is always preferable, it is impractical due to often limited operational budget of ASPs. In this case, an ASP has to cautiously decide where to deploy the edge service and how much budget it is willing to use. A central issue here is that the service demand received by each edge site, which is the key factor of deploying benefit, is unknown to ASPs a priori. What's more complicated is that this demand pattern varies temporally and spatially across geographically distributed edge sites. In this paper, we investigate an edge resource rental problem where the ASP learns service demand patterns for individual edge sites while renting computation resource at these sites to host its applications for edge service provisioning. An online algorithm, called Context-aware Online Edge Resource Rental (COERR), is proposed based on the framework of Contextual Combinatorial Multi-armed Bandit (CC-MAB). COERR observes side-information (context) to learn the demand patterns of edge sites and decides rental decisions (including where to rent the computation resource and how much to rent) to maximize ASP's utility given a limited budget. COERR provides a provable performance achieving sublinear regret compared to an Oracle algorithm that knows exactly the expected service demand of edge sites. Experiments are carried out on a real-world dataset and the results show that COERR significantly outperforms other benchmarks.