Multi-armed Bandit Approach Research Articles

Interpreting pretext tasks for active learning: a reinforcement learning approach

As the amount of labeled data increases, the performance of deep neural networks tends to improve. However, annotating a large volume of data can be expensive. Active learning addresses this challenge by selectively annotating unlabeled data. There have been recent attempts to incorporate self-supervised learning into active learning, but there are issues in utilizing the results of self-supervised learning, i.e., it is uncertain how these should be interpreted in the context of active learning. To address this issue, we propose a multi-armed bandit approach to handle the information provided by self-supervised learning in active learning. Furthermore, we devise a data sampling process so that reinforcement learning can be effectively performed. We evaluate the proposed method on various image classification benchmarks, including CIFAR-10, CIFAR-100, Caltech-101, SVHN, and ImageNet, where the proposed method significantly improves previous approaches.

UAV trajectory planning in NOMA-aided UAV-mounted RIS networks: A budgeted Multi-armed bandit approach

Abstract Reconfigurable Intelligent Surface (RIS) and Non-Orthogonal Multiple Access (NOMA) can be used as supporting technologies for driving the effective development of Unmanned Aerial Vehicles (UAV) communication systems. Herein, both technologies are combined to widen the millimeter-wave Base Stations (mmWave-BS) coverage range. In this work, we consider UAV Trajectory Planning (UTP) problem to enhance the performance of a NOMA-based UAV-mounted RIS communication network. The problem is formulated using two Budgeted Multi-Armed Bandits (BMAB) schemes. BMAB is a variant of the classical MAB algorithm that considers the random cost of selecting an arm. The UAV performs as a bandit player attempting to maximize its achievable rate (i.e., reward) via accessing different grids in its trajectory. This reward is accomplished simultaneously while minimizing the energy/cost of flying the UAV from one grid to another to preserve its battery lifetime. Hence, two variants of the Budgeted upper confidence bound (BUCB), namely the BUCB1 and BUCB2, are investigated, where the simulations show that both algorithms outperform other benchmarks.

Online Causal Inference for Advertising in Real-Time Bidding Auctions

This paper introduces a multiarmed bandit approach to estimate advertising effectiveness and manage experimentation costs in real-time bidding auctions concurrently.

Multi-armed bandit algorithm for sequential experiments of molecular properties with dynamic feature selection.

Sequential optimization is one of the promising approaches in identifying the optimal candidate(s) (molecules, reactants, drugs, etc.) with desired properties (reaction yield, selectivity, efficacy, etc.) from a large set of potential candidates, while minimizing the number of experiments required. However, the high dimensionality of the feature space (e.g., molecular descriptors) makes it often difficult to utilize the relevant features during the process of updating the set of candidates to be examined. In this article, we developed a new sequential optimization algorithm for molecular problems based on reinforcement learning, multi-armed linear bandit framework, and online, dynamic feature selections in which relevant molecular descriptors are updated along with the experiments. We also designed a stopping condition aimed to guarantee the reliability of the chosen candidate from the dataset pool. The developed algorithm was examined by comparing with Bayesian optimization (BO), using two synthetic datasets and two real datasets in which one dataset includes hydration free energy of molecules and another one includes a free energy difference between enantiomer products in chemical reaction. We found that the dynamic feature selection in representing the desired properties along the experiments provides a better performance (e.g., time required to find the best candidate and stop the experiment) as the overall trend and that our multi-armed linear bandit approach with a dynamic feature selection scheme outperforms the standard BO with fixed feature variables. The comparison of our algorithm to BO with dynamic feature selection is also addressed.

A Contextual Multi-Armed Bandit approach for NDN forwarding

Named Data Networking (NDN) is a promising Internet architecture that aims to supersede the current IP-based architecture and shift the host-centric model to a data-centric one. Within NDN, forwarding Interest packets remains a significant challenge and has attracted considerable recent research attention. The momentum behind machine learning techniques, especially reinforcement learning, is steadily growing, offering the potential to deliver intelligent, adaptable, and reliable NDN forwarding algorithms. In this context, this paper proposes efficient NDN forwarding strategies based on Contextual Multi-Armed Bandit (CMAB). Initially, we employ CMAB to address the challenge of forwarding Interest packets and introduce a new CMAB model tailored for NDN, dubbed CMAB4NDN. Subsequently, we construct the CMAB context using information derived from the content name and the network congestion state, which are then fed into the CMAB4NDN learning algorithm to decide on the best forwarding action. Further, we develop three CMAB strategies, namely Lin-ɛ-Greedy, Linear Upper Confidence Bound, and Contextual Thompson Sampling, and deploy them within our proposal. CMAB4NDN was implemented in ndnSIM, thoroughly evaluated, and compared with multiple state-of-the-art NDN forwarding algorithms across various scenarios. The obtained results confirm the relevance and superiority of our approach in terms of delay, throughput, and packet loss.

CA-Live360: Crowd-assisted transcoding and delivery for live 360-degree video streaming

In the burgeoning field of digital media, the popularity of live 360-degree video has witnessed an upward trajectory, underpinned by its immersive experience. In live 360-degree video streaming services, transcoding enormous video tiles and providing low-latency services for global viewers require significantly more computing and communication resources than traditional live services. Inspired by the huge crowdsourcing computational resources available at crowd, this paper designs CA-Live360, a Crowd-Assisted Live 360-degree video streaming solution, in which transcoding and delivery are supported by crowdsourcing devices. Due to the dynamic availability of resources, it is challenging to achieve low-latency live services, while guaranteeing fair transcoding. Therefore, firstly an innovative fairness-guaranteed transcoding task assignment scheme is proposed. This scheme employs a novel Fair Bandit algorithm based on the combinatorial multi-armed bandit approach to achieve low-latency transcoding while considering time-varying available resources and fairness constraints. Secondly, to reduce delivery latency, we design a transcoding-aware delivery scheme, in which a provider–requester matching algorithm is introduced to realize effective requester scheduling. Real-world trace-driven experiments demonstrate the improved performance of the proposed CA-Live360 solution in terms of system delay and fairness.

Multi-armed bandit approach for mean field game-based resource allocation in NOMA networks

Facing the exponential demand for massive connectivity and the scarcity of available resources, next-generation wireless networks have to meet very challenging performance targets. Particularly, the operators have to cope with the continuous prosperity of the Internet of things (IoT) along with the ever-increasing deployment of machine-type devices (MTDs). In this regard, due to its compelling benefits, non-orthogonal multiple access (NOMA) has sparked a significant interest as a sophisticated technology to address the above-mentioned challenges. In this paper, we consider a hybrid NOMA scenario, wherein the MTDs are divided into different groups, each of which is allocated an orthogonal resource block (RB) so that the members of each group share a given RB to simultaneously transmit their signals. Firstly, we model the densely deployed network using a mean field game (MFG) framework while taking into consideration the effect of the collective behavior of devices. Then, in order to reduce the complexity of the proposed technique, we apply the multi-armed bandit (MAB) framework to jointly address the resource allocation and the power control problem. Thereafter, we derive two distributed decision-making algorithms that enable the users to autonomously regulate their transmit power levels and self-organize into coalitions based on brief feedback received from the base station (BS). Simulation results are given to underline the equilibrium properties of the proposed resource allocation algorithms and to reveal the robustness of the proposed learning process.

Pareto Front-Diverse Batch Multi-Objective Bayesian Optimization

We consider the problem of multi-objective optimization (MOO) of expensive black-box functions with the goal of discovering high-quality and diverse Pareto fronts where we are allowed to evaluate a batch of inputs. This problem arises in many real-world applications including penicillin production where diversity of solutions is critical. We solve this problem in the framework of Bayesian optimization (BO) and propose a novel approach referred to as Pareto front-Diverse Batch Multi-Objective BO (PDBO). PDBO tackles two important challenges: 1) How to automatically select the best acquisition function in each BO iteration, and 2) How to select a diverse batch of inputs by considering multiple objectives. We propose principled solutions to address these two challenges. First, PDBO employs a multi-armed bandit approach to select one acquisition function from a given library. We solve a cheap MOO problem by assigning the selected acquisition function for each expensive objective function to obtain a candidate set of inputs for evaluation. Second, it utilizes Determinantal Point Processes (DPPs) to choose a Pareto-front-diverse batch of inputs for evaluation from the candidate set obtained from the first step. The key parameters for the methods behind these two steps are updated after each round of function evaluations. Experiments on multiple MOO benchmarks demonstrate that PDBO outperforms prior methods in terms of both the quality and diversity of Pareto solutions.

Comparative analysis of Sliding Window UCB and Discount Factor UCB in non-stationary environments: A Multi-Armed Bandit approach

The Multi-Armed Bandit (MAB) problem is a well-studied topic within stationary environments, where the reward distributions remain consistent over time. Nevertheless, many real-world applications often fall within non-stationary contexts, where the rewards from each arm can evolve. In light of this, our research focuses on examining and contrasting the effectiveness of two leading algorithms tailored for these shifting environments: the Sliding Window Upper Confidence Bound (SW-UCB) and the Discount Factor UCB (DF-UCB). By harnessing both simulated and real-world datasets, our evaluation encompasses adaptability, computational efficiency, and the potential for regret minimization. Our findings reveal that the SW-UCB is adept at swiftly adjusting to sudden shifts, whereas the DF-UCB emerges as the more resource-efficient option amidst gradual transitions. Notably, when pitted against conventional UCB algorithms within non-stationary contexts, both contenders exhibit substantial advancements. Such insights bear significant relevance to fields like online advertising, healthcare, and finance, where the capacity to nimbly adapt to dynamic environments is paramount.

Survey of dynamic pricing based on Multi-Armed Bandit algorithms

Dynamic pricing seeks to determine the most optimal selling price for a product or service, taking into account factors like limited supply and uncertain demand. This study aims to provide a comprehensive exploration of dynamic pricing using the multi-armed bandit problem framework in various contexts. The investigation highlights the prevalence of Thompson sampling in dynamic pricing scenarios with a Bayesian backdrop, where the seller possesses prior knowledge of demand functions. On the other hand, in non-Bayesian situations, the Upper Confidence Bound (UCB) algorithm family gains traction due to their favorable regret bounds. As markets often exhibit temporal fluctuations, the domain of non-stationary multi-armed bandits within dynamic pricing emerges as crucial. Future research directions include enhancing traditional multi-armed bandit algorithms to suit online learning settings, especially those involving dynamic reward distributions. Additionally, merging prior insights into demand functions with contextual multi-armed bandit approaches holds promise for advancing dynamic pricing strategies. In conclusion, this study sheds light on dynamic pricing through the lens of multi-armed bandit problems, offering insights and pathways for further exploration.

Distributed Data-Driven Learning-Based Optimal Dynamic Resource Allocation for Multi-RIS-Assisted Multi-User Ad-Hoc Network

This study investigates the problem of decentralized dynamic resource allocation optimization for ad-hoc network communication with the support of reconfigurable intelligent surfaces (RIS), leveraging a reinforcement learning framework. In the present context of cellular networks, device-to-device (D2D) communication stands out as a promising technique to enhance the spectrum efficiency. Simultaneously, RIS have gained considerable attention due to their ability to enhance the quality of dynamic wireless networks by maximizing the spectrum efficiency without increasing the power consumption. However, prevalent centralized D2D transmission schemes require global information, leading to a significant signaling overhead. Conversely, existing distributed schemes, while avoiding the need for global information, often demand frequent information exchange among D2D users, falling short of achieving global optimization. This paper introduces a framework comprising an outer loop and inner loop. In the outer loop, decentralized dynamic resource allocation optimization has been developed for self-organizing network communication aided by RIS. This is accomplished through the application of a multi-player multi-armed bandit approach, completing strategies for RIS and resource block selection. Notably, these strategies operate without requiring signal interaction during execution. Meanwhile, in the inner loop, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm has been adopted for cooperative learning with neural networks (NNs) to obtain optimal transmit power control and RIS phase shift control for multiple users, with a specified RIS and resource block selection policy from the outer loop. Through the utilization of optimization theory, distributed optimal resource allocation can be attained as the outer and inner reinforcement learning algorithms converge over time. Finally, a series of numerical simulations are presented to validate and illustrate the effectiveness of the proposed scheme.

UCBEE: A Multi Armed Bandit Approach for Early-Exit in Neural Networks

UCBEE: A Multi Armed Bandit Approach for Early-Exit in Neural Networks

Motion Planning as Online Learning: A Multi-Armed Bandit Approach to Kinodynamic Sampling-Based Planning

Motion Planning as Online Learning: A Multi-Armed Bandit Approach to Kinodynamic Sampling-Based Planning

Adaptive designs for best treatment identification with top-two Thompson sampling and acceleration.

We consider outcome adaptive phase II or phase II/III trials to identify the best treatment for further development. Different from many other multi-arm multi-stage designs, we borrow approaches for the best arm identification in multi-armed bandit (MAB) approaches developed for machine learning and adapt them for clinical trial purposes. The best arm identification in MAB focuses on the error rate of identification at the end of the trial, but we are also interested in the cumulative benefit of trial patients, for example, the frequency of patients treated with the best treatment. In particular, we consider Top-Two Thompson Sampling (TTTS) and propose an acceleration approach for better performance in drug development scenarios in which the sample size is much smaller than that considered in machine learning applications. We also propose a variant of TTTS (TTTS2) which is simpler, easier for implementation, and has comparable performance in small sample settings. An extensive simulation study was conducted to evaluate the performance of the proposed approach in multiple typical scenarios in drug development.

Truthful User Recruitment for Cooperative Crowdsensing Task: A Combinatorial Multi-Armed Bandit Approach

Mobile Crowdsensing (MCS) is a promising paradigm that recruits users to cooperatively perform a sensing task. When recruiting users, existing works mainly focus on users objective abilities, e.g., the probability or frequency of covering the task locations. However, we argue that the task completion effect depends not only on the user's objective ability, but also on their subjective collaboration likelihood with each other. Even though we can find a well-behaved group of users in the single-round scenario, while in the multi-round scenario without enough prior knowledge, we still face the problem of exploiting users or exploring users. Additionally, we consider that users have different costs and may report false costs to gain more benefits. To address these problems, we first convert the single-round user recruitment problem into the min-cut problem and propose a graph theory based algorithm to find the approximate solution. Then, in the multi-round scenario, we propose a multi-round User Recruitment strategy under the budget constraint based on the combinatorial Multi-armed Bandit model (URMB), which achieves a tight regret bound. Next, we propose a graph-based payment strategy to ensure the truthfulness and individual rationality. Finally, experiments based on real-world datasets show that URMB always outperforms the state-of-the-art strategies.

A combinatorial multi-armed bandit approach to correlation clustering

Given a graph whose edges are assigned positive-type and negative-type weights, the problem of correlation clustering aims at grouping the graph vertices so as to minimize (resp. maximize) the sum of negative-type (resp. positive-type) intra-cluster weights plus the sum of positive-type (resp. negative-type) inter-cluster weights. In correlation clustering, it is typically assumed that the weights are readily available. This is a rather strong hypothesis, which is unrealistic in several scenarios. To overcome this limitation, in this work we focus on the setting where edge weights of a correlation-clustering instance are unknown, and they have to be estimated in multiple rounds, while performing the clustering. The clustering solutions produced in the various rounds provide a feedback to properly adjust the weight estimates, and the goal is to maximize the cumulative quality of the clusterings. We tackle this problem by resorting to the reinforcement-learning paradigm, and, specifically, we design for the first time a Combinatorial Multi-Armed Bandit (CMAB) framework for correlation clustering. We provide a variety of contributions, namely (1) formulations of the minimization and maximization variants of correlation clustering in a CMAB setting; (2) adaptation of well-established CMAB algorithms to the correlation-clustering context; (3) regret analyses to theoretically bound the accuracy of these algorithms; (4) design of further (heuristic) algorithms to have the probability constraint satisfied at every round (key condition to soundly adopt efficient yet effective algorithms for correlation clustering as CMAB oracles); (5) extensive experimental comparison among a variety of both CMAB and non-CMAB approaches for correlation clustering.

Exploiting Structure in Regular Expression Queries

Regular expression, or regex, is widely used to extract critical information from a large corpus of formatted text by finding patterns of interest. In tasks like log processing, the speed of regex matching is crucial. Data scientists and developers regularly use regex libraries that implement optimized regular expression matching using modern automata theory. However, computing state transitions in the underlying regex evaluation engine can be inefficient when a regex query contains a multitude of string literals. This inefficiency is further exasperated when analyzing large data volumes. This paper presents BLARE, Blazingly Fast Regular Expression, a regular expression matching framework that is inspired by the mechanisms that are used in database engines, which use a declarative framework to explore multiple equivalent execution plans, all of which produce the correct final result. Similarly, BLARE decomposes a regex into multiple regex and string components and then creates evaluation strategies in which the components can be evaluated in an order that is not strictly a left-to-right translation of the input regex query. Rather than using a cost-based optimization approach, BLARE uses an adaptive runtime strategy based on a multi-armed bandit approach to find an efficient execution plan. BLARE is also modular and can be built on top of any existing regex library. We implemented BLARE on four commonly used regex libraries, RE2, PCRE2, Boost Regex, and ICU Regex, and evaluated it using two production workloads and one open-source workload. BLARE was 1.6× to 3.7× faster than RE2 and 3.4× to 7.9× faster than Boost Regex. PCRE2 did not finish on one of the workloads, but on the remaining two workloads, BLARE improved the performance of PCRE2 by 3.1× to over 100×. For the open-source dataset, BLARE provided a speed up of 61.7× for ICU Regex. BLARE code is publicly available at https://github.com/mush-zhang/Blare.

A Bayesian Multi-Armed Bandit Algorithm for Dynamic End-to-End Routing in SDN-Based Networks with Piecewise-Stationary Rewards

To handle the exponential growth of data-intensive network edge services and automatically solve new challenges in routing management, machine learning is steadily being incorporated into software-defined networking solutions. In this line, the article presents the design of a piecewise-stationary Bayesian multi-armed bandit approach for the online optimum end-to-end dynamic routing of data flows in the context of programmable networking systems. This learning-based approach has been analyzed with simulated and emulated data, showing the proposal’s ability to sequentially and proactively self-discover the end-to-end routing path with minimal delay among a considerable number of alternatives, even when facing abrupt changes in transmission delay distributions due to both variable congestion levels on path network devices and dynamic delays to transmission links.

Spoiled for Choice? Personalized Recommendation for Healthcare Decisions: A Multiarmed Bandit Approach

Choice overload is a common problem in many online settings, including healthcare. Online healthcare platforms tend to provide a large variety of behavior intervention information or programs to help individuals modify their lifestyles to improve wellness. However, having too many options can significantly increase searching cost, prevent users from discovering the truly relevant interventions, and harm users’ long-term healthcare decision-making efficiency. This motivates us to propose a personalized healthcare recommendation system to provide tailored support for individuals’ intervention participation. The proposed framework, a deep-learning and diversity-enhanced multiarmed bandit (DLDE-MAB), integrates several predictive and prescriptive analytics components to combat the unique challenges presented in the healthcare recommendation setting. It leverages online machine learning to provide adaptive and real-time support, a theory-guided diversity promotion scheme to cover multiple healthcare needs, and deep learning to further enhance dynamic context representation. Through extensive experiments, we show that the proposed framework outperforms various competing models in terms of its adaptivity to data dynamics, diversity, and uncertainty. The proposed model and evaluation results provide important implications for business intelligence and personalized, contextualized, and agile healthcare decision making.

Online Learning of Time-Varying Unbalanced Networks in Non-Convex Environments: A Multi-Armed Bandit Approach

This study discusses how agents in a time-varying distributed network can converge to the global minimizer of a time-varying graph network. Each agent knows only the local loss of its observation and must cooperate constructively with other agents to find the global minimizer of the network. Unlike most existing works in the literature that consider a convex loss function, this study assumes a generalized local Lipschitz loss function for each agent, which can be convex or non-convex. We propose a multi-armed bandit algorithm CD EXP3 where each agent does not know its loss function but only observes its losses. Through simulations using two different time-varying graph topologies, we show that the algorithm helps all agents converge to the minimizer of the network. In addition, we discuss the effects of the two different topologies and various simulation parameters on convergence. We obtain an upper bound on the expected regret and compare it with the sublinearity of the regret bounds of well-known online distributed algorithms.