Multi-armed Bandit Research Articles

Enhancing movie recommendations through comparative analysis of UCB algorithm variants

In the digital realm, recommendation systems are pivotal in shaping user experiences on online platforms, tailoring content based on user feedback. A notable algorithm in this domain is the multi-armed bandit algorithm, with the Upper Confidence Bound (UCB) emerging as a classic and effective variant. This paper delves into an array of Upper Confidence Bound algorithm variations, encompassing UCB1, Asymptotically Optimal UCB, UCB-V, and UCB1Tuned. The research harnesses the MovieLens dataset to assess the performance of these algorithms, employing cumulative regret as the primary metric. For l in UCB1 and c in UCB-V, both oversized and undersized parameters will result in negative outcomes. And UCB1Tuned outperforms the other three algorithms in this experiment, since it considers variance and adjusts parameters dynamically. The study demonstrates that setting a appropriate UCB index is crucial for enhancing the performance of the UCB algorithm in recommendation system. It holds significance for both improve recommendation system algorithms and enhance user experience.

Selecting workers like expert for crowdsourcing by integration evaluation of individual and collaborative abilities

Team-based worker selection has been extensively studied for Mobile Crowdsourcing (MCS), in which a set of workers are recruited to form a team to complete complex tasks collaboratively. However, existing studies face two typical challenges: 1) how to dynamically evaluate workers’ individual abilities and collaborative contributions to the team; 2) how to select unknown workers to form a team with high quality at low cost. To tackle the above challenges, this paper proposes an Integration of Individual and Collaborative Abilities based Dynamic Worker Selection (IICA-DWS) algorithm to recruit excellent workers as a team in a high-quality and low-cost style. In the IICA-DWS algorithm, each worker’s individual ability and collaborative contribution to the team are evaluated more accurately using the Approximate Shapley Value (ASV). In addition, a high-quality team formation method is established to complete complex tasks at low cost. This involves the selection of both team leaders and team members. In this process, the Multi-Armed Bandit (MAB) model is adopted to dynamically select excellent workers using exploration and exploitation phases. Lastly, the IICA-DWS algorithm is evaluated through theoretical analysis and experimental results. The results show that the IICA-DWS algorithm can improve the data quality of tasks by 47.3% and reduce the cost by 61.7% on average. Moreover, the IICA-DWS algorithm has a high probability of approximating optimal results, which shows the best performance among the comparative algorithms.

Interactive preference analysis: A reinforcement learning framework

Automated investment managers are increasingly popular in personal wealth management due to their cost effectiveness, objectivity, and accessibility. However, it still suffers from several dilemmas, e.g., cold start, over-specialization, and black boxes. To solve these issues, we develop an online reinforcement learning framework based on the multi-armed bandit algorithm to offer personalized investment advice. We provide a comprehensive theoretical procedure for developing this framework. This framework not only enables us to capture the evolving preferences of investors effectively but also has a strong explainability power to provide more implications regarding why one financial product is preferred. We further evaluate our basic model through a large-scale, real-world data set from a leading wealth management platform. The results show a stronger effectiveness of the proposed framework compared to other well-recognized benchmark models. Furthermore, we extend our basic model to address the potential agency problem between the robo-advisor and the investors. Another extension is also provided through an optimization scheme to account for the investors’ demands for diversification in multiple aspects.

Multi-armed bandit approach for mean field game-based resource allocation in NOMA networks

Facing the exponential demand for massive connectivity and the scarcity of available resources, next-generation wireless networks have to meet very challenging performance targets. Particularly, the operators have to cope with the continuous prosperity of the Internet of things (IoT) along with the ever-increasing deployment of machine-type devices (MTDs). In this regard, due to its compelling benefits, non-orthogonal multiple access (NOMA) has sparked a significant interest as a sophisticated technology to address the above-mentioned challenges. In this paper, we consider a hybrid NOMA scenario, wherein the MTDs are divided into different groups, each of which is allocated an orthogonal resource block (RB) so that the members of each group share a given RB to simultaneously transmit their signals. Firstly, we model the densely deployed network using a mean field game (MFG) framework while taking into consideration the effect of the collective behavior of devices. Then, in order to reduce the complexity of the proposed technique, we apply the multi-armed bandit (MAB) framework to jointly address the resource allocation and the power control problem. Thereafter, we derive two distributed decision-making algorithms that enable the users to autonomously regulate their transmit power levels and self-organize into coalitions based on brief feedback received from the base station (BS). Simulation results are given to underline the equilibrium properties of the proposed resource allocation algorithms and to reveal the robustness of the proposed learning process.

Cache content placement in the presence of fictitious requests in mmWave 5G IAB networks

Caching popular content at the edge of wireless networks leads to backhaul congestion mitigation. To come up with an effective caching policy, content popularity distribution should be taken into account, which is not accurately known in most practical scenarios. Moreover, the mobile users’ (MU) request pattern may not always follow a well-defined distribution since some malicious MUs may deliberately issue their requests incompatible with the content popularity statistics. In this paper, we consider the problem of cache content placement in a 5G mmWave small cell network that relies on integrated access and backhaul (IAB) technology for pushing contents to MUs. We assume that the IAB node is equipped with a cache and has no prior knowledge about the content popularity profiles; instead, it only relies on the observation of the instantaneous demands to shape its caching policy. Also, malicious MUs may exist whose goals are to increase cache miss by issuing fictitious requests. The IAB node decides on which contents to cache and for how long, given that frequently replacing contents incurs administrative costs. We model the content placement problem as an ”adversarial combinatorial multi-armed bandit process with switching costs (ACMAB-SC)” and present an online learning algorithm for shaping the caching policy. We conduct extensive simulation experiments to evaluate the convergence property and assess the performance of our algorithm in terms of backhaul congestion, delay, and cache hit ratio. We also compare against two baseline online learning schemes, including a CMAB-based approach and a generic caching policy based on the ”Follow the Perturbed Leader (FPL)” algorithm.

Exploration–Exploitation Mechanisms in Recurrent Neural Networks and Human Learners in Restless Bandit Problems

A key feature of animal and human decision-making is to balance the exploration of unknown options for information gain (directed exploration) versus selecting known options for immediate reward (exploitation), which is often examined using restless bandit tasks. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning, due to their ability to show meta-learning of task domains. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Computational modeling of behavior first revealed that both human and RNN behavioral data contain signatures of higher-order perseveration, i.e., perseveration beyond the last trial, but this effect was more pronounced in RNNs. In contrast, human learners, but not RNNs, exhibited a positive effect of uncertainty on choice probability (directed exploration). RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. Our results highlight both similarities and differences between exploration behavior as it emerges in meta-learning RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.

Transmission scheduling of P2P real-time communication based on restless multi-armed bandit

Transmission scheduling of P2P real-time communication based on restless multi-armed bandit

Corruption-Robust Exploration in Episodic Reinforcement Learning

We initiate the study of episodic reinforcement learning (RL) under adversarial corruptions in both the rewards and the transition probabilities of the underlying system, extending recent results for the special case of multiarmed bandits. We provide a framework that modifies the aggressive exploration enjoyed by existing reinforcement learning approaches based on optimism in the face of uncertainty by complementing them with principles from action elimination. Importantly, our framework circumvents the major challenges posed by naively applying action elimination in the RL setting, as formalized by a lower bound we demonstrate. Our framework yields efficient algorithms that (a) attain near-optimal regret in the absence of corruptions and (b) adapt to unknown levels of corruption, enjoying regret guarantees that degrade gracefully in the total corruption encountered. To showcase the generality of our approach, we derive results for both tabular settings (where states and actions are finite) and linear Markov decision process settings (where the dynamics and rewards admit a linear underlying representation). Notably, our work provides the first sublinear regret guarantee that accommodates any deviation from purely independent and identically distributed transitions in the bandit-feedback model for episodic reinforcement learning. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0202 .

Client selection for federated learning using combinatorial multi-armed bandit under long-term energy constraint

In a federated learning system, it is often the case that the more clients it involves, the less increment of the outcome it achieves. It is thus essential to design a client selection strategy to choose an appropriate subset of the clients to participate in federated learning. However, client selection is not easy due to the heterogeneity of clients and the long-term energy budget of each client. Moreover, long-term energy budgets intertwined with the short-term client selection often make the problem NP-hard. In this paper, we propose an online strategy Energy-Aware Client Selection for Federated Learning (EACS-FL) to address this problem. The problem is formulated with a joint energy and delay optimization objective, and the Combinatorial Multi-Armed Bandit (CMAB) is introduced to solve the problem in an online manner. We take advantage of Lyapunov optimization to manage energy consumption of clients, which enables us to deal with independent energy budgets through minimizing virtual energy deficit queues. Theoretical analysis shows that EACS-FL achieves sublinear regret and keeps all queues stable. Experiment results exhibit that the proposed approach outperforms the existing works and achieves close-to-optimal delay and energy consumption performance.

Anytime-valid off-policy Inference for Contextual Bandits

Contextual bandit algorithms are ubiquitous tools for active sequential experimentation in healthcare and the tech industry. They involve online learning algorithms that adaptively learn policies over time to map observed contexts X t to actions A t in an attempt to maximize stochastic rewards R t . This adaptivity raises interesting but hard statistical inference questions, especially counterfactual ones: for example, it is often of interest to estimate the properties of a hypothetical policy that is different from the logging policy that was used to collect the data—a problem known as “off-policy evaluation” (OPE). Using modern martingale techniques, we present a comprehensive framework for OPE inference that relaxes unnecessary conditions made in some past works (such as performing inference at prespecified sample sizes, uniformly bounded importance weights, constant logging policies, and constant policy values, among others), significantly improving on them both theoretically and empirically. Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post hoc), when the logging policy may be itself changing (due to learning), and even if the context distributions are a highly dependent time series (such as if they are drifting over time). More concretely, we derive confidence sequences for various functionals of interest in OPE. These include doubly robust ones for time-varying off-policy mean reward values, but also confidence bands for the entire cumulative distribution function of the off-policy reward distribution. All of our methods (a) are valid at arbitrary stopping times; (b) only make nonparametric assumptions; (c) do not require importance weights to be uniformly bounded, and if they are, we do not need to know these bounds; and (d) adapt to the empirical variance of our estimators. In summary, our methods enable anytime-valid off-policy inference using adaptively collected contextual bandit data.

Personalized Image Generation Through Swiping

Generating preferred images from GANs is a challenging task due to the high-dimensional nature of latent space. In this study, we propose a novel approach that uses simple user-swipe interactions to generate preferred images from users. To effectively explore the latent space with only swipe interactions, we apply principal component analysis to the latent space of StyleGAN, creating meaningful subspaces. Additionally, we use a multi-armed bandit algorithm to decide which dimensions to explore, focusing on the user's preferences. Our experiments show that our method is more efficient in generating preferred images than the baseline.

A new Hyper-heuristic based on Adaptive Simulated Annealing and Reinforcement Learning for the Capacitated Electric Vehicle Routing Problem

Electric vehicles (EVs) have been adopted in urban areas to reduce environmental pollution and global warming due to the increasing number of freight vehicles. However, there are still deficiencies in routing the trajectories of last-mile logistics that continue to impact social and economic sustainability. For that reason, in this paper, a hyper-heuristic (HH) approach called Hyper-heuristic Adaptive Simulated Annealing with Reinforcement Learning (HHASARL) is proposed. It is composed of a multi-armed bandit method and the self-adaptive Simulated Annealing (SA) metaheuristic algorithm for solving the problem called Capacitated Electric Vehicle Routing Problem (CEVRP). Due to the limited number of charging stations and the travel range of EVs, the EVs must require battery recharging moments in advance and reduce travel times and costs. The implementation of the HH improves multiple minimum best-known solutions and obtains the best mean values for some high-dimensional instances for the proposed benchmark for the IEEE WCCI2020 competition.

Exploring Multi-Armed Bandit (MAB) as an AI Tool for Optimising GMA-WAAM Path Planning

Conventional path-planning strategies for GMA-WAAM may encounter challenges related to geometrical features when printing complex-shaped builds. One alternative to mitigate geometry-related flaws is to use algorithms that optimise trajectory choices—for instance, using heuristics to find the most efficient trajectory. The algorithm can assess several trajectory strategies, such as contour, zigzag, raster, and even space-filling, to search for the best strategy according to the case. However, handling complex geometries by this means poses computational efficiency concerns. This research aimed to explore the potential of machine learning techniques as a solution to increase the computational efficiency of such algorithms. First, reinforcement learning (RL) concepts are introduced and compared with supervised machining learning concepts. The Multi-Armed Bandit (MAB) problem is explained and justified as a choice within the RL techniques. As a case study, a space-filling strategy was chosen to have this machining learning optimisation artifice in its algorithm for GMA-AM printing. Computational and experimental validations were conducted, demonstrating that adding MAB in the algorithm helped to achieve shorter trajectories, using fewer iterations than the original algorithm, potentially reducing printing time. These findings position the RL techniques, particularly MAB, as a promising machining learning solution to address setbacks in the space-filling strategy applied.

Multi‐armed bandit based online model selection for concept‐drift adaptation

AbstractEnsemble methods are among the most effective concept‐drift adaptation techniques due to their high learning performance and flexibility. However, they are computationally expensive and pose a challenge in applications involving high‐speed data streams. In this paper, we present a computationally efficient heterogeneous classifier ensemble entitled OMS‐MAB which uses online model selection for concept‐drift adaptation by posing it as a non‐stationary multi‐armed bandit (MAB) problem. We use a MAB to select a single adaptive learner within the ensemble for learning and prediction while systematically exploring promising alternatives. Each ensemble member is made drift resistant using explicit drift detection and is represented as an arm of the MAB. An exploration factor controls the trade‐off between predictive performance and computational resource requirements, eliminating the need to continuously train and evaluate all the ensemble members. A rigorous evaluation on 20 benchmark datasets and 9 algorithms indicates that the accuracy of OMS‐MAB is statistically at par with state‐of‐the‐art (SOTA) ensembles. Moreover, it offers a significant reduction in execution time and model size in comparison to several SOTA ensemble methods, making it a promising ensemble for resource constrained stream‐mining problems.

Multi-class boosting for the analysis of multiple incomplete views on microbiome data

BackgroundMicrobiome dysbiosis has recently been associated with different diseases and disorders. In this context, machine learning (ML) approaches can be useful either to identify new patterns or learn predictive models. However, data to be fed to ML methods can be subject to different sampling, sequencing and preprocessing techniques. Each different choice in the pipeline can lead to a different view (i.e., feature set) of the same individuals, that classical (single-view) ML approaches may fail to simultaneously consider. Moreover, some views may be incomplete, i.e., some individuals may be missing in some views, possibly due to the absence of some measurements or to the fact that some features are not available/applicable for all the individuals. Multi-view learning methods can represent a possible solution to consider multiple feature sets for the same individuals, but most existing multi-view learning methods are limited to binary classification tasks or cannot work with incomplete views.ResultsWe propose irBoost.SH, an extension of the multi-view boosting algorithm rBoost.SH, based on multi-armed bandits. irBoost.SH solves multi-class classification tasks and can analyze incomplete views. At each iteration, it identifies one winning view using adversarial multi-armed bandits and uses its predictions to update a shared instance weight distribution in a learning process based on boosting. In our experiments, performed on 5 multi-view microbiome datasets, the model learned by irBoost.SH always outperforms the best model learned from a single view, its closest competitor rBoost.SH, and the model learned by a multi-view approach based on feature concatenation, reaching an improvement of 11.8% of the F1-score in the prediction of the Autism Spectrum disorder and of 114% in the prediction of the Colorectal Cancer disease.ConclusionsThe proposed method irBoost.SH exhibited outstanding performances in our experiments, also compared to competitor approaches. The obtained results confirm that irBoost.SH can fruitfully be adopted for the analysis of microbiome data, due to its capability to simultaneously exploit multiple feature sets obtained through different sequencing and preprocessing pipelines.

Bandit approach for unmanned aerial vehicle-centric low earth orbit satellite selection

In this paper, the problem of optimal low earth orbit satellite (LEO-Sat) selection by unmanned aerial vehicle (UAV) in space air ground integrated networks (SAGINs) will be investigated. The primary objective is to maximize the UAV's achievable data rate while ensuring a prolonged remaining visible time for the selected LEO-Sat. This problem poses significant challenges due to the highly dynamic nature caused by the substantial relative velocity between UAV and LEO-Sats, making LEO-Sats' features, e.g., elevation angle, remaining visible time, and the available bandwidth strongly time-dependent. Moreover, some LEO-Sats will appear/disappear from UAV visibility within short time intervals. An online learning approach will be proposed to efficiently address this highly dynamic problem, where it will be modeled as contextual multi-armed bandit (MAB) with dynamic arms. Through this modeling, the time varying LEO-Sats' features as well as their dynamic appearance/disappearance will be taken into account while selecting the best LEO-Sat. Through extensive numerical analysis, the performance of the proposed approach nearly matches that of the highly complex optimal performance based on exhaustive searching with negligible time complexity, and it outperforms conventional schemes that individually maximize one of the LEO-Sats' features.

Approximate information for efficient exploration-exploitation strategies.

This paper addresses the exploration-exploitation dilemma inherent in decision-making, focusing on multiarmed bandit problems. These involve an agent deciding whether to exploit current knowledge for immediate gains or explore new avenues for potential long-term rewards. We here introduce a class of algorithms, approximate information maximization (AIM), which employs a carefully chosen analytical approximation to the gradient of the entropy to choose which arm to pull at each point in time. AIM matches the performance of Thompson sampling, which is known to be asymptotically optimal, as well as that of Infomax from which it derives. AIM thus retains the advantages of Infomax while also offering enhanced computational speed, tractability, and ease of implementation. In particular, we demonstrate how to apply it to a 50-armed bandit game. Its expression is tunable, which allows for specific optimization in various settings, making it possible to surpass the performance of Thompson sampling at short and intermediary times.

An algorithm for multi-armed bandit based on variance change sensitivity

The Multi-Arm Bandit problem is becoming increasingly popular as it enables real-world sequential decision making across application domains, including clinical trials, recommender systems, and online decision making. The Multi-Arm Bandit problem is a classical problem of exploration and exploitation dilemma in reinforcement learning and it needs to decide on the optimal strategy based on the reward situation of each rocker arm. However, the existing Multi-Arm Bandit algorithms have many shortcomings, such as blind exploration, weak generalization ability and endless exploration. Aiming at the shortcomings of the existing Multi-Armed Bandit algorithms, this paper proposes a Multi-Armed Bandit algorithm based on variance change sensitivity. The algorithm takes the variance change of reward as a clue, and adjusts the exploration probability by the average variance change of all actions, and selects the action with the largest variance change when exploring. At the same time, in order to reduce the waste of action selection times and maximize the cumulative reward, A parameter Ncon was introduced to record the number of consecutive selections of the same action. The exploration stops when Ncon reaches a certain value. Experiments show that the exploration algorithm can obtain higher reward value and lower regret value in the end.

In-depth Exploration and Implementation of Multi-Armed Bandit Models Across Diverse Fields

This paper presents an in-depth analysis of the Multi-Armed Bandit (MAB) problem, tracing its evolution from its origins in the gambling domain of the 1940s to its current prominence in machine learning and artificial intelligence. The analysis begins with a historical overview, noting key developments like Herbert Robbins' probabilistic framework and the expansion of the problem into strategic decision-making in the 1970s. The emergence of algorithms like the Upper Confidence Bound (UCB) and Thompson Sampling in the late 20th century is highlighted, demonstrating the MAB problem's transition to practical applications. The integration of MAB algorithms with machine learning, particularly in the era of reinforcement learning, is explored, emphasizing their application in various domains such as online advertising, financial market trading, and clinical trials. The paper discusses the critical role of decision theory and probabilistic models in MAB problems, focusing on the balance between exploration and exploitation strategies. Recent advancements in Contextual Bandits, non-stationary reward distributions, and Multi-agent Bandits are examined, showcasing the ongoing evolution and adaptability of MAB problems.

Enhancing UCB-tuned and Asymptotically Optimal UCB Algorithms through Weighted Average Techniques in Multi-Armed Bandit Scenarios

This paper delves into the complexities of the Multi-Armed Bandit (MAB) problem, a fundamental concept in reinforcement learning and probability theory, with a focus on its application in recommendation systems and dynamic fields such as dynamic pricing and investment. It begins by shedding light on the essential paradox at the heart of the MAB problem – the balance between exploration and exploitation within limited parameters. The study primarily centers on Upper Confidence Bound (UCB) policies, especially UCB-tuned and Asymptotically Optimal UCB, noted for their adept balance between exploration and utilization. The novel contribution of this research is the enhancement of these UCB policies via an innovative weighted average method, leading to the development of WA-UCB-tuned and WA Asymptotically Optimal UCB algorithms. The research rigorously compares these optimized iterations with traditional UCB1, UCB-tuned, and Asymptotically Optimal UCB across varied MAB models featuring different numbers of arms. This study provides an exhaustive introduction to the MAB problem and pertinent UCB policies, the methodology behind the weighted average optimization, extensive experimental analysis, and comprehensive evaluations of the findings. The results showcase marked improvements in algorithmic performance, suggesting significant advancements in the domain of recommendation systems and other applications of the MAB problem.