Combinatorial Multi-armed Bandit Problem Research Articles

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijds.2022.00011 . Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Spectrum Allocation and User Scheduling Based on Combinatorial Multi-Armed Bandit for 5G Massive MIMO.

As a key 5G technology, massive multiple-input multiple-output (MIMO) can effectively improve system capacity and reduce latency. This paper proposes a user scheduling and spectrum allocation method based on combinatorial multi-armed bandit (CMAB) for a massive MIMO system. Compared with traditional methods, the proposed CMAB-based method can avoid channel estimation for all users, significantly reduce pilot overhead, and improve spectral efficiency. Specifically, the proposed method is a two-stage method; in the first stage, we transform the user scheduling problem into a CMAB problem, with each user being referred to as a base arm and the energy of the channel being considered a reward. A linear upper confidence bound (UCB) arm selection algorithm is proposed. It is proved that the proposed user scheduling algorithm experiences logarithmic regret over time. In the second stage, by grouping the statistical channel state information (CSI), such that the statistical CSI of the users in the angular domain in different groups is approximately orthogonal, we are able to select one user in each group and allocate a subcarrier to the selected users, so that the channels of users on each subcarrier are approximately orthogonal, which can reduce the inter-user interference and improve the spectral efficiency. The simulation results validate that the proposed method has a high spectral efficiency.

Dynamic User Recruitment in Edge-Aided Mobile Crowdsensing

Most of the existing user recruitment methods are designed for the two-tier structure. However, the computational consumption of the cloud platform rises rapidly as data volumes increase. In recent years, edge computing has been introduced into mobile crowdsensing. In this paper, we study dynamic user recruitment with edge-aided mobile crowdsensing. Specifically, we consider the scenarios for different sensing tasks. One is the long-duration task, and the other is the short-duration task. For the former, it is time-insensitive, while the latter is time-sensitive. In the scenario with the long-duration task, we consider a simple offline scenario and propose an Edge-Node User Recruitment (ENUR) algorithm to recruit users statically. Then, we consider the online scenario and propose a Budget Re-Distribution algorithm for Edge Nodes User Recruitment (BRD-ENUR). It dynamically redistributes the budget according to the task completion ratio of different edge nodes. In the scenario with the short-duration task, the whole sensing process consists of multiple rounds. In each round, edge nodes need to recruit users between selected users (exploitation) and unselected users (exploration) under the budget. We model such a user recruitment process as a novel combinatorial multi-armed bandit problem. Then, we propose an Edge Nodes User Recruitment algorithm based on combinatorial Multi-armed Bandit (EN-URMB). Extensive experiments show the validity and reliability of our user recruitment algorithms based on the real data set.

Two Stage Beamforming in Massive MIMO: A Combinatorial Multi-Armed Bandit Based Approach

In frequency division duplex (FDD) massive multi-input multi-output (MIMO), the two-stage beamforming (TSB) using channel covariance matrices (CCM) can significantly reduce the downlink training length (DTL) and channel feedback. However, the overhead to estimate the CCM is large. In this paper, a combinatorial multi-armed bandit (CMAB) based TSB scheme is proposed without requirement of CMM. Particularly, the problem of the pre-beamforming matrix design is transformed into a CMAB problem. We consider the pre-beamforming matrix design in each slot as the arm selection in the CMAB, and convert the problem of the arm selection into a 0-1 integer linear programming problem, which can be solved by the branch-and-bound method. During the training process, the maximum likelihood method is used to detect the power of angle spectrum, and the angle range of each user is determined adaptively. We prove that the regret grows logarithmically with time, such that the proposed scheme converges towards the optimal action. Finally, simulation results demonstrate that the proposed scheme can significantly improve the spectral efficiency.

Multiuser Scheduling in Centralized Cognitive Radio Networks: A Multi-Armed Bandit Approach

In wireless communication networks, the network provider serves certain licensed primary users who pay for a dedicated use of the frequency channels. However, not all the channels are occupied by the primary users at all times. For efficient spectrum utilization, in centralized cognitive radio networks (CRNs), a cognitive base station (CBS) dynamically identifies the spectrum holes and allocates the frequency channels to the on-demand unlicensed secondary users known as cognitive radios (CRs). Although existing literature has developed various dynamic spectrum access mechanisms for CBS, there is still a dearth of studies due to the wide range of assumptions made in the solutions. Most of the existing works study the CBS scheduling problem scheme by adopting optimization-based methods and rely on the prior knowledge of the network parameters such as primary users’ activity. Moreover, the impact of channel switching costs on the network throughput has not been well studied. In this paper, we aim to maximize the CRNs total throughput, and we formulate the CBS scheduling problem as a non-stochastic (i.e., adversarial) combinatorial multi-armed bandit problem with semi-bandit feedback and arm switching costs. We propose two novel online learning algorithms for CBS scheduling with and without channel switching costs, where their regret performances are proved sublinear order-optimal in time as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$T^{1/2}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$T^{2/3}$ </tex-math></inline-formula> , respectively, offering throughput-optimal scheduling for CRNs. Experiments on the synthetic and real-world spectrum measurement data complement and validate our theoretical findings.

History-Aware Online Cache Placement in Fog-Assisted IoT Systems: An Integration of Learning and Control

In fog-assisted Internet-of-Things systems, it is a common practice to cache popular content at the network edge to achieve high quality of service. Due to uncertainties, in practice, such as unknown file popularities, the cache placement scheme design is still an open problem with unresolved challenges: 1) how to maintain time-averaged storage costs under budgets; 2) how to incorporate online learning to aid cache placement to minimize performance loss [also known as (a.k.a.) regret]; and 3) how to exploit offline historical information to further reduce regret. In this article, we formulate the cache placement problem with unknown file popularities as a constrained combinatorial multiarmed bandit problem. To solve the problem, we employ virtual queue techniques to manage time-averaged storage cost constraints, and adopt history-aware bandit learning methods to integrate offline historical information into the online learning procedure to handle the exploration–exploitation tradeoff. With an effective combination of online control and history-aware online learning, we devise a cache placement scheme with history-aware bandit learning called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CPHBL</i> . Our theoretical analysis and simulations show that CPHBL achieves a sublinear time-averaged regret bound. Moreover, the simulation results verify CPHBL’s advantage over the deep reinforcement learning-based approach.

The Combinatorial Multi-Armed Bandit Problem and Its Application to Real-Time Strategy Games

Game tree search in games with large branching factors is a notoriously hard problem. In this paper, we address this problem with a new sampling strategy for Monte Carlo Tree Search (MCTS) algorithms, called "Naive Sampling", based on a variant of the Multi-armed Bandit problem called the "Combinatorial Multi-armed Bandit" (CMAB) problem. We present a new MCTS algorithm based on Naive Sampling called NaiveMCTS, and evaluate it in the context of real-time strategy (RTS) games. Our results show that as the branching factor grows, NaiveMCTS performs significantly better than other algorithms.

Budgeted Unknown Worker Recruitment for Heterogeneous Crowdsensing Using CMAB

Mobile crowdsensing, through which a requester can coordinate a crowd of workers to complete some sensing tasks, has attracted significant attention recently. In this paper, we focus on the unknown worker recruitment problem in mobile crowdsensing, where workers' sensing qualities are unknown a priori. We consider the scenario of recruiting workers to complete some continuous sensing tasks. In each round, every task may be covered by more than one recruited workers, but its completion quality only depends on these workers' maximum sensing quality. Our objective is to determine a recruiting strategy to maximize the total weighted completion quality under a limited budget. We model such unknown worker recruitment process as a novel combinatorial multi-armed bandit problem, and propose an unknown worker recruitment algorithm based on the modified upper confidence bound. Moreover, we extend the problem to the case where the workers' costs are also unknown and design the corresponding algorithm. We analyze the regret bounds of the two proposed algorithms. In addition, we also study the unknown worker recruitment problem with fairness constraints. For this problem, we devise a fairness-aware unknown worker recruitment algorithm. Finally, we demonstrate the performance of the proposed algorithms through extensive experimental simulations on real-world traces.

Joint Switch–Controller Association and Control Devolution for SDN Systems: An Integrated Online Perspective of Control and Learning

In software-defined networking (SDN) systems, it is a common practice to adopt a multi-controller design and control devolution techniques to improve the performance of the control plane. However, in such systems the decision-making for joint switch-controller association and control devolution often involves various uncertainties, e.g., the temporal variations of controller accessibility, and computation and communication costs of switches. In practice, statistics of such uncertainties are unattainable and need to be learned in an online fashion, calling for an integrated design of learning and control. In this article, we formulate a stochastic network optimization problem that aims to minimize time-average system costs and ensure queue stability. By transforming the problem into a combinatorial multi-armed bandit problem with long-term stability constraints, we adopt bandit learning methods and optimal control techniques to handle the exploration-exploitation tradeoff and long-term stability constraints, respectively. Through an integrated design of online learning and online control, we propose an effective Learning-Aided Switch-Controller Association and Control Devolution ( LASAC ) scheme. Our theoretical analysis and simulation results show that LASAC achieves a tunable tradeoff between queue stability and system cost reduction with a sublinear time-averaged regret bound over a finite time horizon.

Caching by User Preference With Delayed Feedback for Heterogeneous Cellular Networks

The burgeoning network traffic imposes a huge burden on the network backbone. Caching popular files at the wireless network edge is promising to address the problem. In practice, file popularity is very unlikely to know in advance. Online learning algorithms are effective to learn this uncertainty in a sequential way. In each slot, the learning agent generates a caching policy (i.e., the to-be-cached files) and can observe users’ feedback about the caching policy within the same slot. This method implicitly requires that all of the users are able to provide feedback promptly. However, in practice, the availability of each individual user is affected by many factors, e.g., users are moving out of the service area temporarily, or they may still consume files in the previous slots, which may result in the feedback delay. In this paper, we propose a delay-tolerant wireless caching system that takes both the feedback delay and users’ availability into consideration. We frame the content caching problem as a stochastic combinatorial multi-armed bandit problem with delayed feedback and forced- to-sleep arms, and devise an intelligent caching algorithm called CFAUD to solve the problem. Also, we show that CFAUD is effective and efficient both theoretically and practically. Finally, experiments are conducted to compare the performance of the proposed algorithm with other well-known algorithms.

Cognitive Analysis in Social Networks for Viral Marketing

Viral marketing is the modern version of the old “word-of-mouth” advertising, where companies choose a restricted number of persons, considered “influential,” recommending them products or services that will be in turn iteratively suggested. In this article, we propose cognitive models and algorithms for marketing applications through online social networks, considered as a graph database, and define the concept of influence graph leveraging particular user behavioral patterns, by querying the initial heterogeneous graph network. We also model the diffusion across the network, without any preliminary information, as a combinatorial multiarmed bandit problem, for the selection of most influential users. We have used the YELP social network as a case study for our approach, showing how it is possible to generate an influence graph considering several kinds of relevant paths (mainly considering reviews to the same firms) by which a user can influence other ones. Several experiments have been carried out and discussed, putting into evidence the effectiveness and efficacy of the proposed methods for influence maximization with respect to other approaches of state of the art.

A Multi-Armed Bandit Approach to Wireless Information and Power Transfer

Multiple wireless transmitters communicate with their information receivers while radiating RF energy to multiple nearby energy harvesters. The channels from the transmitters to the receivers and to the energy harvesters are time-varying and unknown. The transmitters jointly determine their multi-antenna transmissions to fairly charge all the energy harvesters while maintaining the signal-to-interference-and-noise ratio at the receivers. In this letter, this task is formulated as a combinatorial multi-armed bandit problem and is solved with the upper confidence bound algorithm. Numerical results show that the algorithm can quickly converge to the optimal strategy with moderate signaling overhead. It has superior performance in fair energy harvesting while maintaining communication quality.

Selecting multiple web adverts: A contextual multi-armed bandit with state uncertainty

We present a method to solve the problem of choosing a set of adverts to display to each of a sequence of web users. The objective is to maximise user clicks over time and to do so we must learn about the quality of each advert in an online manner by observing user clicks. We formulate the problem as a novel variant of a contextual combinatorial multi-armed bandit problem. The context takes the form of a probability distribution over the user's latent topic preference, and rewards are a particular nonlinear function of the selected set and the context. These features ensure that optimal sets of adverts are appropriately diverse. We give a flexible solution method which combines submodular optimisation with existing bandit index policies. User state uncertainty creates ambiguity in interpreting user feedback which prohibits exact Bayesian updating, but we give an approximate method that is shown to work well.

A Bandit Approach to Price-Aware Energy Management in Cellular Networks

We introduce a reinforcement learning algorithm inspired by the combinatorial multi-armed bandit problem to minimize the time-averaged energy cost at individual base stations (BSs), powered by various energy markets and local renewable energy sources, over a finite-time horizon. The algorithm sustains traffic demands by enabling sparse beamforming to schedule dynamic user-to-BS allocation and proactive energy provisioning at BSs to make ahead-of-time price-aware energy management decisions. Simulation results indicate a superior performance of the proposed algorithm in reducing the overall energy cost, as compared with recently proposed cooperative energy management designs.

Planning Curtailment of Renewable Generation in Power Grids

The increasing penetration of renewable sources like solar energy add new dimensions in planning power grid operations. We study the problem of curtailing a subset of prosumers generating solar power with the twin goals of being close to a target collection and maintaining fairness across prosumers. The problem is complicated by the uncertainty in the amount of energy fed-in by each prosumer and the large problem size in terms of number of prosumers. To meet these challenges, we propose an algorithm based on the Combinatorial Multi-Armed Bandit problem with an approximate Knapsack based oracle. With real-data on solar panel output across multiple prosumers, we are able to demonstrate the effectiveness of the proposed algorithm.