Multi-armed Bandit Research Articles

Interpreting pretext tasks for active learning: a reinforcement learning approach

As the amount of labeled data increases, the performance of deep neural networks tends to improve. However, annotating a large volume of data can be expensive. Active learning addresses this challenge by selectively annotating unlabeled data. There have been recent attempts to incorporate self-supervised learning into active learning, but there are issues in utilizing the results of self-supervised learning, i.e., it is uncertain how these should be interpreted in the context of active learning. To address this issue, we propose a multi-armed bandit approach to handle the information provided by self-supervised learning in active learning. Furthermore, we devise a data sampling process so that reinforcement learning can be effectively performed. We evaluate the proposed method on various image classification benchmarks, including CIFAR-10, CIFAR-100, Caltech-101, SVHN, and ImageNet, where the proposed method significantly improves previous approaches.

An online frequency allocation strategy for multi‐carrier radar against spot jammer

AbstractSpot jamming poses a significant threat to radar detection due to its ability to rapidly intercept radar signals and emit high‐power interference. A novel Cognitive Multi‐Carrier Radar (CMCR) is proposed, which utilises a detection waveform comprising multiple narrow bands to reduce the disruption from the spot jammer. Additionally, it can analyse the target’s Radar Cross‐Section (RCS) characteristics and proactively select frequencies with high RCS for detection, thereby enhancing target detection capability. The online frequency allocation problem of CMCR is a Multi‐Armed Bandit (MAB) problem with a combinatorial action space. To tackle the challenge from the huge action space, an efficient aDaptive Accept Reject (DART) method is employed, which significantly reduces the regret of CMCR in the detection task. Moreover, a slide‐window change point indicator is proposed to monitor the change of target’s RCS, which enables CMCR to be applicable in scenarios where target’s RCS varies. Finally, the effectiveness and robustness of CMCR are through extensive numerical experiments.

Strategy Selection Using Multi-Armed Bandit Algorithms in Financial Markets

This paper aims to evaluate the effectiveness of Multi-Armed Bandit (MAB) algorithms in choosing the optimal trading strategy among the sub optimal ones within financial markets. The research aims to addresses the challenge of adapting to dynamic market conditions. By introducing a Composite Trading Strategy that integrates trend-following, mean reversion, and momentum strategies, the study investigates whether increased trading frequency enhances the performance of profitability of various MAB algorithms, including UCB, Thompson Sampling, and epsilon-greedy. The experimental results indicate that while the introduction of complex strategies greatly improves returns in favorable market conditions, MAB algorithms still face limitations in adverse market environments. The findings highlight the potential of MAB algorithms in financial strategy selection and suggest directions for future research to enhance their adaptability in adverse markets.

Comparison of Multi-Armed Bandit Algorithms in Advertising Recommendation Systems

In today's rapidly evolving online environment, advertising recommendation systems utilize multi-armed bandit algorithms like dynamic collaborative filtering Thompson sampling (DCTS), upper confidence bound based on recommender system (UCB-RS), and dynamic -greedy algorithm (DEG) to optimize ad displays and enhance click-through rates (CTR). These algorithms must adapt to limited information and update strategies based on immediate feedback.This study employs an experimental comparison to assess the performance of the DCTS, UCB-RS, and DEG algorithms using the click-through rate prediction database from Kaggle. Five experimental sets under varied parameter settings were analyzed, employing the Receiver Operating Characteristic (ROC) curve, accuracy, and area under the curve (AUC) metrics.Results show that the DEG algorithm consistently outperforms the others, achieving higher AUC values and demonstrating robust sample identification capabilities. DEG also exhibits superior precision at high recall levels, showcasing its potential in dynamic advertising environments. Its dynamic adjustment strategy effectively balances exploration and exploitation, optimizing ad displays.The findings suggest that DEG's adaptability and stability make it particularly suitable for dynamic ad recommendation scenarios. Future research should focus on optimizing DEG's parameter settings and possibly integrating UCB-RS's exploration mechanisms to enhance performance and develop more effective strategies for advertising recommendation systems.

Do More Experienced Gamblers Choose Slot Machines with Better Odds? A Large-Scale Multi-Armed Bandit Problem at a Casino

Do More Experienced Gamblers Choose Slot Machines with Better Odds? A Large-Scale Multi-Armed Bandit Problem at a Casino

Multi-armed bandit games

Multi-armed bandit games

Harnessing Multi-Armed Bandits for Smarter Digital Marketing Decisions

In the current digital age, software programs and applications provide convenience and added value in a number of ways. Multi-armed Bandit algorithms (MAB) are a prime example of this; In capital markets, they assist in the adaptive design of trading strategies that adjust to market shifts and investor behavior. In e-commerce, MAB helps optimize product recommendations by learning customer preferences in real-time. In SaaS, MAB can optimize user experience by personalizing services or pricing models based on user behavior, continuously adjusting to maximize engagement or revenue. In cloud engineering MAB are indispensable for optimizing resource allocation based on user demand. The objective of this paper is to demonstrate the adaptability of MAB algorithms through an examination of their applications, specifically focusing on the Digital Marketing domain and the insight they offer for optimal decision-making. The article highlights the usefulness of major MAB algorithms in digital advertising, e-commerce content suggestion, SaaS and strategic pricing by carefully examining Thompson Sampling, UCB, Restless Bandit, and Structured Bandit as key types of MAB algorithms. The study emphasizes how these algorithms adjust to fluctuating conditions, balance a trade-off between exploration and exploitation, and eventually improve marketing strategies. This article aims to improve knowledge of MAB algorithms and encourage more research in this promising area by providing a detailed analysis of their applications.

Harvesting heterogeneity: Selective expertise versus machine learning.

The heterogeneity of outcomes in behavioral research has long been perceived as a challenge for the validity of various theoretical models. More recently, however, researchers have started perceiving heterogeneity as something that needs to be not only acknowledged but also actively addressed, particularly in applied research. A serious challenge, however, is that classical psychological methods are not well suited for making practical recommendations when heterogeneous outcomes are expected. In this article, we argue that heterogeneity requires a separation between basic and applied behavioral methods, and between different types of behavioral expertise. We propose a novel framework for evaluating behavioral expertise and suggest that selective expertise can easily be automated via various machine learning methods. We illustrate the value of our framework via an empirical study of the preferences towards battery electric vehicles. Our results suggest that a basic multiarm bandit algorithm vastly outperforms human expertise in selecting the best interventions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The Effect of Reward Magnitude on Different Types of Exploration in Human Reinforcement Learning

AbstractHow humans resolve the explore–exploit dilemma in complex environments is an important open question. Previous studies suggested that environmental richness may affect the degree of exploration in a type-specific manner and reduce random exploration while increasing uncertainty-based exploration. Our study examined this possibility by extending a recently developed two-armed bandit task that can dissociate the uncertainty and novelty of stimuli. To extract the pure effect of environmental richness, we manipulated the reward by its magnitude, not its probability, across blocks because reward probability affects outcome controllability. Participants (N = 198) demonstrated increased optimal choices when the relative reward magnitude was higher. A behavioral analysis with computational modeling revealed that a higher reward magnitude reduced the degree of random exploration but had little effect on the degree of uncertainty- and novelty-based exploration. These results suggest that humans modulate their degree of random exploration depending on the relative level of environmental richness. Combined with findings from previous studies, our findings indicate the possibility that outcome controllability also influences the exploration–exploitation balance in human reinforcement learning.

Adversarial Bandits With Multi-User Delayed Feedback: Theory and Application

Adversarial Bandits With Multi-User Delayed Feedback: Theory and Application

Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration

Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration

Psilocybin increases optimistic engagement over time: computational modelling of behaviour in rats.

Psilocybin has shown promise as a novel pharmacological intervention for treatment of depression, where post-acute effects of psilocybin treatment have been associated with increased positive mood and decreased pessimism. Although psilocybin is proving to be effective in clinical trials for treatment of psychiatric disorders, the information processing mechanisms affected by psilocybin are not well understood. Here, we fit active inference and reinforcement learning computational models to a novel two-armed bandit reversal learning task capable of capturing engagement behaviour in rats. The model revealed that after receiving psilocybin, rats achieve more rewards through increased task engagement, mediated by modification of forgetting rates and reduced loss aversion. These findings suggest that psilocybin may afford an optimism bias that arises through altered belief updating, with translational potential for clinical populations characterised by lack of optimism.

Comparative Study of Multi-Armed Bandit Algorithms in Clinical Trials

In recent years, with the rapid development of the information age, the influence of Multi Armed Bandit Algorithms (MAB) models in clinical trials for disease prevention has been increasing. In this study, based on Python programming language, Multi-Armed Bandit Algorithms (MAB) algorithm, Upper Confidence Bound (UCB) algorithm, Adaptive Epsilon-Greedy Algorithm, and Thompson Sampling (TS) algorithms to validate the idea of preventing, controlling and predicting the occurrence of diseases. The results show that the MAB model can effectively solve various decision-making problems in clinical trials, improve the efficiency of access to medical care, save doctors diagnosis time, and at the same time achieve the prevention and treatment of diseases while minimising patients pain. This study is dedicated to proposing a more effective decision-making method and verifies that the method has a wide range of applications and great potential for development today.

Understanding explore-exploit dynamics in child development: current insights and future directions

Examining children's decisions to explore or exploit the environment provides a window into their developing metacognition and reflection capacities. Reinforcement learning, characterized by the balance between exploring new options (exploration) and utilizing known ones (exploitation), is central to this discussion. Children initially exhibit broad and intensive exploration, which gradually shifts toward exploitation as they grow. We review major theories and empirical findings, highlighting two main exploration strategies: random and directed. The former involves stochastic choices without considering information or rewards, while the latter is driven by reducing uncertainty for information gain. Behavioral tasks such as n-armed bandit, horizon, and patch foraging tasks are used to study these strategies. Findings on the n-armed bandit and horizon tasks showed mixed results on whether random exploration decreases over time. Directed exploration consistently decreases with age, but its emergence depends on task difficulty. In patch-foraging tasks, adults tend to overexploit (staying too long in one patch) and children overexplore (leaving too early), whereas adolescents display the most optimal balance. The paper also addresses open questions regarding the mechanisms supporting early exploration and the application of these strategies in real-life contexts like persistence. Future research should further investigate the relation between cognitive control, such as executive function and metacognition, and explore-exploit strategies, and examine their practical implications for adaptive learning and decision-making in children.

Multi-armed Bandit Experimental Design: Online Decision-Making and Adaptive Inference

Multi-armed bandit has been well known for its efficiency in online decision-making in terms of minimizing the loss of the participants’ welfare during experiments (i.e., the regret). In clinical trials and many other scenarios, the statistical power of inferring the treatment effects (i.e., the gaps between the mean outcomes of different arms) is also crucial. Nevertheless, minimizing the regret entails harming the statistical power of estimating the treatment effect because the observations from some arms can be limited. In this paper, we investigate the trade-off between efficiency and statistical power by casting the multi-armed bandit experimental design into a minimax multi-objective optimization problem. We introduce the concept of Pareto optimality to mathematically characterize the situation in which neither the statistical power nor the efficiency can be improved without degrading the other. We derive a useful sufficient and necessary condition for the Pareto optimal solutions to the minimax multi-objective optimization problem. Additionally, we design an effective Pareto optimal multi-armed bandit experiment that can be tailored to different levels of the trade-off between the two objectives. Moreover, we extend the design and analysis to the setting where the outcome of each arm consists of an adversarial baseline reward and a stochastic treatment effect, demonstrating the robustness of our design. Finally, motivated by clinical trials, we examine the setting where the employed experiment must split the experimental units into a small number of batches, and we propose a flexible Pareto optimal design. This paper was accepted by George Shanthikumar, data science. Funding: The authors thank the Massachusetts Institute of Technology (MIT)-IBM partnership in Artificial Intelligence and the MIT Data Science Laboratory for support. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.00492 .

Simple fixes that accommodate switching costs in multi-armed bandits

When switching costs are added to the multi-armed bandit (MAB) problem where the arms’ random reward distributions are previously unknown, usually quite different techniques than those for pure MAB are required. We find that two simple fixes on the existing upper-confidence-bound (UCB) policy can work well for MAB with switching costs (MAB-SC). Two cases should be distinguished. One is with positive-gap ambiguity where the performance gap between the leading and lagging arms is known to be at least some δ>0. For this, our fix is to erect barriers that discourage frivolous arm switchings. The other is with zero-gap ambiguity where absolutely nothing is known. We remedy this by forcing the same arms to be pulled in increasingly prolonged intervals. As usual, the effectivenesses of our fixes are measured by the worst average regrets over long time horizons T. When the barriers are fixed at δ/2, we can accomplish a ln(T)-sized regret bound for the positive-gap case. When intervals are such that n of them occupy n2 periods, we can achieve the best possible T1/2-sized regret bound for the zero-gap case. Other than UCB, these fixes can be applied to a learning while doing (LWD) heuristic to reach satisfactory results as well. While not yet with the best theoretical guarantees, the LWD-based policies have empirically outperformed those based on UCB and other known alternatives. Numerically competitive policies still include ones resulting from interval-based fixes on Thompson sampling (TS).

Secure Dynamic Scheduling for Federated Learning in Underwater Wireless IoT Networks

Federated learning (FL) is a distributed machine learning approach that can enable Internet of Things (IoT) edge devices to collaboratively learn a machine learning model without explicitly sharing local data in order to achieve data clustering, prediction, and classification in networks. In previous works, some online multi-armed bandit (MAB)-based FL frameworks were proposed to enable dynamic client scheduling for improving the efficiency of FL in underwater wireless IoT networks. However, the security of online dynamic scheduling, which is especially essential for underwater wireless IoT, is increasingly being questioned. In this work, we study secure dynamic scheduling for FL frameworks that can protect against malicious clients in underwater FL-assisted wireless IoT networks. Specifically, in order to jointly optimize the communication efficiency and security of FL, we employ MAB-based methods and propose upper-confidence-bound-based smart contracts (UCB-SCs) and upper-confidence-bound-based smart contracts with a security prediction model (UCB-SCPs) to address the optimal scheduling scheme over time-varying underwater channels. Then, we give the upper bounds of the expected performance regret of the UCB-SC policy and the UCB-SCP policy; these upper bounds imply that the regret of the two proposed policies grows logarithmically over communication rounds under certain conditions. Our experiment shows that the proposed UCB-SC and UCB-SCP approaches significantly improve the efficiency and security of FL frameworks in underwater wireless IoT networks.

Integrating testing volume into bandit algorithms for infectious disease surveillance

Abstract Mobile testing services provide opportunities for active surveillance of infectious diseases for hard-to-reach and/or high-risk individuals who do not know their disease status. Identifying as many infected individuals as possible is important for mitigating disease transmission. Recently, multi-armed bandit sampling approaches have been adapted and applied in this setting to maximize the cumulative number of positive tests collected over time. However, these algorithms have not considered the possibility of variability in the number of tests administered across testing sites. What impact this variability has on the ability of these approaches to maximize yield is currently unknown. Therefore, we investigate this question by extending existing sampling frameworks to directly account for variability in testing volume while also maintaining the computational tractability of the previous methods. Through a simulation study based on human immunodeficiency virus infection characteristics in the Republic of the Congo (Congo-Brazzaville) as well as an application to COVID-19 testing data in Connecticut, we find improved long- and short-term performances of the new methods compared to several existing approaches. Based on these findings and the ease of computation, we recommend use of the newly developed methods for active surveillance of infectious diseases when variability in testing volume may be present.

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning.

Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison among a wide set of cognitive models based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation contributes to instrumental task performance in mice and is reflected in mesolimbic dopamine signaling.

B - 62 Cognitive Training Targeting Reward Responsiveness Promotes more Efficient Affective Processing in an Interference Go/NoGo Task

Abstract Objective Abnormalities in emotional processing are commonly seen in depression and anxiety, disorders also associated with decreased reward sensitivity i.e., anhedonia. We tested a noninvasive, cognitive training program aimed at boosting reward responsiveness and gaged if it may also improve affective processing efficiency. Methods Individuals (N = 50; 54% Female; 18–65 yo) with low hedonic tone (Positive and Negative Affect Scale/PANAS score < 25) and diagnostic levels of depression and/or anxiety (Patient Health Questionnaire-8/PHQ-8 or General Anxiety Disorder-7/GAD-7 score ≥ 10) who responded to community-posted flyers completed an affective go/no-go (AGN) task before and after completing an exploratory reward learning task, i.e., a multi-arm bandit (MAB). Participants were randomly assigned to a high-reward MAB training with either high variance (HV) or low variance (LV) in reward rates, HV being expected to boost reward maximization. In the AGN, participants saw negative and positive pictures and had to respond as fast as possible to pictures of a given target category (positive or negative). Results Following the HV vs LV training MAB, participants were more accurate (higher likelihood of hits), which was more evident in negative target blocks relative to positive target blocks (odds ratio/OR:0.61, 95% CI = [0.38,0.98]; z = −2.045, p = 0.041). Participants exhibited stronger decreases in hits response times following the HV vs LV training MAB, but across all target blocks (B:-37 ms, 95% CI = [−56,-18]; t = −3.806, p < 0.001). Conclusions A novel cognitive training program aimed at boosting reward sensitivity may help improve patients’ processing of emotionally taxing stimuli. Targeting reward sensitivity is a critical component of improving clinical outcomes in depressed and anxious patients..