Multi-armed Bandit Research Articles

Comparative Analysis of Multi-armed Bandits Models for Recommendation Systems

With the booming development of personalized recommendation systems, numerous algorithms have been designed to bring more bliss and engagement to users. Multi-armed bandits (MAB) models have been increasingly used in this context, as they appropriately balance exploration and exploitation. This work provides an in-depth comparative study of diverse MAB algorithms like -greedy, Upper Confidence Bound (UCB1), and Thompson Sampling in the recommendation systems literature. This paper benchmark the computational efficiency, cumulative reward, and adaptability of these models by running them at interactive speeds with simulations of real-world user interactions. The results suggest that UCB1 works well in stable environments, whereas Thompson Sampling excels in volatile settings. The paper examines the attributes of these MAB algorithms through a systematic review of recent research. The paper also discusses the utility of MAB models in areas such as online advertising, streaming services, and e-commerce. Future research will likely focus on integrating deep learning approaches with MAB to further improve recommendation systems.

EXPRESS: Bridging Adversarial and Nonstationary Multi-armed Bandit

In the multi-armed bandit framework, there are two formulations that are commonly employed to handle time-varying reward distributions: adversarial bandit and nonstationary bandit. Although their oracles, algorithms, and regret analysis differ significantly, we provide a unified formulation in this paper that smoothly bridges the two as special cases. The formulation uses an oracle that takes the best action sequences within a switch budget. Depending on the switch budget, it turns into the oracle in hindsight in the adversarial bandit and the dynamic oracle in the nonstationary bandit. We provide algorithms that attain the optimal regret with the matching lower bound. The optimal regret displays distinct behavior in two regimes.

AutoFuzz: automatic fuzzer-sanitizer scheduling with multi-armed bandit

AutoFuzz: automatic fuzzer-sanitizer scheduling with multi-armed bandit

Competing Bandits: The Perils of Exploration Under Competition

Most online platforms learn from interactions with users, and engage in exploration : making potentially suboptimal choices in order to acquire new information. We study the interplay between exploration and competition : how such platforms balance the exploration for learning and competition for users. We consider a stylized duopoly in which two firms face the same multi-armed bandit problem. Users arrive one by one and choose between the two firms, so that each firm makes progress on its bandit problem only if it is chosen. We study whether competition incentivizes the adoption of better algorithms. We find that stark competition disincentivizes exploration, leading to low welfare. However, weaker competition incentivizes better exploration algorithms and increases welfare. We investigate two channels for weakening the competition: stochastic user choice models and a first-mover advantage. Our findings speak to the competition-innovation relationship and the first-mover advantage in the digital economy.

Thompson Sampling for Non-Stationary Bandit Problems

Thompson Sampling for Non-Stationary Bandit Problems

A Novel Hyper-Heuristic Algorithm with Soft and Hard Constraints for Causal Discovery Using a Linear Structural Equation Model

Artificial intelligence plays an indispensable role in improving productivity and promoting social development, and causal discovery is one of the extremely important research directions in this field. Acyclic directed graphs (DAGs) are the most commonly used tool in causal modeling because of their excellent interpretability and structural properties. However, in the face of insufficient data, the accuracy and efficiency of DAGs learning are greatly reduced, resulting in a false perception of causality. As intuitive expert knowledge, structural constraints control DAG learning by limiting the causal relationship between variables, which is expected to solve the above-mentioned problem. However, it is often impossible to build a DAG by relying on expert knowledge alone. To solve this problem, we propose the use of expert knowledge as a hard constraint and the structural prior gained via data learning as a soft constraint. In this paper, we propose a fitness-rate-rank-based multiarmed bandit (FRRMAB) hyper-heuristic that integrates soft and hard constraints into the DAG learning process. For a linear structural equation model (SEM), soft constraints are obtained via partial correlation analysis. The experimental results on different networks show that the proposed method has higher scalability and accuracy.

Multi-Armed Bandits With Costly Probes

Multi-Armed Bandits With Costly Probes

Adaptive Multi-Armed Bandit Reinforcement Learning-Based Energy Management and Real-Time Control System for Autonomous Vehicles

Adaptive Multi-Armed Bandit Reinforcement Learning-Based Energy Management and Real-Time Control System for Autonomous Vehicles

Power-of-2-Arms for Adversarial Bandit Learning With Switching Costs

Power-of-2-Arms for Adversarial Bandit Learning With Switching Costs

An exact bandit model for the risk-volatility tradeoff

An exact bandit model for the risk-volatility tradeoff

The Fragility of Optimized Bandit Algorithms

What Does the Distribution of Regret Look Like for Bandit Algorithms? In multiarmed bandit theory, the expected regret has long been the predominant performance measure studied in the literature. For the first time, in “The fragility of optimized bandit algorithms,” Fan and Glynn study the tail of the regret distribution. For a given bandit model, it is natural to optimize to achieve the minimum possible expected regret growth rate. However, the authors show that optimizing for expected regret alone can result in severe fragility. Optimized expected regret rates come with the side effect of extremely heavy regret tails (like those of a Cauchy distribution) and thus, significant chance of catastrophically large regret. Moreover, optimized algorithms essentially operate on a knife’s edge such that the slightest degree of model misspecification in algorithm design can result in a much larger expected regret rate. Among other insights, the authors characterize sharp trade-offs between the expected regret rate and the heaviness of the regret tail. They also show how to robustify an algorithm and lighten the regret tail through increased exploration.

Exploiting full-duplex opportunities in WLANs via a reinforcement learning-based medium access control protocol

In-band full-duplex communication has the potential to double the wireless channel capacity. However, how to efficiently transform the full-duplex gain at the physical layer into network throughput improvement is still a challenge, especially in dynamic communication environments. This paper presents a reinforcement learning-based full-duplex (RLFD) medium access control (MAC) protocol for wireless local-area networks (WLANs) with full-duplex access points. To solve the channel contention problem and fully utilize the full-duplex transmission opportunities, we first design a two-way handshake transmission mechanism and make an investigation on the effects of transmission scheduling in full-duplex transmission. Then the transmission scheduling problem is theoretically formulated as a non-stationary multi-armed bandit problem in which our objective is to maximize the network throughput. Thus, we develop a Window-Constraint Bayesian (WCB) algorithm to generate optimized scheduling policies online. And full-duplex opportunities are fully utilized by transmitting packets according to the optimized scheduling policies. Besides, an analytical model is developed to characterize the performance of RLFD. The performance of RLFD is evaluated by simulation. And the results show that RLFD can improve the network throughput by 80% compared with the IEEE 802.11 distributed coordination function with Request-To-Send/Clear-To-Send. Moreover, with the proposed WCB algorithm, the network throughput can remain steady as the communication environment dynamically changes.

Multi-Armed Bandit Approaches for Location Planning with Dynamic Relief Supplies Allocation Under Disaster Uncertainty

Natural disasters (e.g., floods, earthquakes) significantly impact citizens, economies, and the environment worldwide. Due to their sudden onset, devastating effects, and high uncertainty, it is crucial for emergency departments to take swift action to minimize losses. Among these actions, planning the locations of relief supply distribution centers and dynamically allocating supplies is paramount, as governments must prioritize citizens’ safety and basic living needs following disasters. To address this challenge, this paper develops a three-layer emergency logistics network to manage the flow of emergency materials, from warehouses to transfer stations to disaster sites. A bi-objective, multi-period stochastic integer programming model is proposed to solve the emergency location, distribution, and allocation problem under uncertainty, focusing on three key decisions: transfer station selection, upstream emergency material distribution, and downstream emergency material allocation. We introduce a multi-armed bandit algorithm, named the Geometric Greedy algorithm, to optimize transfer station planning while accounting for subsequent dynamic relief supply distribution and allocation in a stochastic environment. The new algorithm is compared with two widely used multi-armed bandit algorithms: the ϵ-Greedy algorithm and the Upper Confidence Bound (UCB) algorithm. A case study in the Futian District of Shenzhen, China, demonstrates the practicality of our model and algorithms. The results show that the Geometric Greedy algorithm excels in both computational efficiency and convergence stability. This research offers valuable guidelines for emergency departments in optimizing the layout and flow of emergency logistics networks.

A new bandit setting balancing information from state evolution and corrupted context

We propose a new sequential decision-making setting, combining key aspects of two established online learning problems with bandit feedback. The optimal action to play at any given moment is contingent on an underlying changing state that is not directly observable by the agent. Each state is associated with a context distribution, possibly corrupted, allowing the agent to identify the state. Furthermore, states evolve in a Markovian fashion, providing useful information to estimate the current state via state history. In the proposed problem setting, we tackle the challenge of deciding on which of the two sources of information the agent should base its action selection. We present an algorithm that uses a referee to dynamically combine the policies of a contextual bandit and a multi-armed bandit. We capture the time-correlation of states through iteratively learning the action-reward transition model, allowing for efficient exploration of actions. Our setting is motivated by adaptive mobile health (mHealth) interventions. Users transition through different, time-correlated, but only partially observable internal states, determining their current needs. The side information associated with each internal state might not always be reliable, and standard approaches solely rely on the context risk of incurring high regret. Similarly, some users might exhibit weaker correlations between subsequent states, leading to approaches that solely rely on state transitions risking the same. We analyze our setting and algorithm in terms of regret lower bound and upper bounds and evaluate our method on simulated medication adherence intervention data and several real-world data sets, showing improved empirical performance compared to several popular algorithms.

Thompson sampling-based recursive block elimination for dynamic assignment under limited budget in pure-exploration

In this paper, we investigate Thompson sampling-based sequential block elimination approaches for dynamic assignment problems in a pure-exploration Multi-Armed Bandit (MAB) setting with limited budget constraints. The problem can be considered as a bandit game-play between the environment and a decision-maker in a metric space. Many instances of problems in fields such as e-commerce, logistics, mobility management, data management and operations research can be framed as dynamic assignment problems with budget constraints. Given an l-dimensional action space representing l variants of an entity and a budget for exploring the action space, the optimal dynamic assignment problem refers to the task of identifying the values to be assigned to different variants of the entity that maximizes the total reward by utilizing at most the given budget of rounds of play. We contribute a class of block elimination-based MAB algorithms specifically designed for the dynamic assignment problem with limited budget. Our algorithms begin by discretizing the continuous action space into a finite set of discrete actions, then proceed with a recursive block elimination procedure to remove sub-optimal actions. The elimination is carried out by calculating confidence bounds over blocks of actions. We explore two different confidence bound estimation techniques. We perform comprehensive experiments on two problem instances from distributed data management and logistics. Our results showcase that our approach yields a lower misidentification probability (i.e., the probability of recommending a non-optimal action) compared to state-of-the-art elimination-based pure-exploration MAB algorithms.

Learning to Be Fair: A Consequentialist Approach to Equitable Decision Making

In an attempt to make algorithms fair, the machine learning literature has largely focused on equalizing decisions, outcomes, or error rates across race or gender groups. To illustrate, consider a hypothetical government rideshare program that provides transportation assistance to low-income people with upcoming court dates. Following this literature, one might allocate rides to those with the highest estimated treatment effect per dollar while constraining spending to be equal across race groups. That approach, however, ignores the downstream consequences of such constraints and, as a result, can induce unexpected harm. For instance, if one demographic group lives farther from court, enforcing equal spending would necessarily mean fewer total rides provided and potentially more people penalized for missing court. Here we present an alternative framework for designing equitable algorithms that foregrounds the consequences of decisions. In our approach, one first elicits stakeholder preferences over the space of possible decisions and the resulting outcomes—such as preferences for balancing spending parity against court appearance rates. We then optimize over the space of decision policies, making trade-offs in a way that maximizes the elicited utility. To do so, we develop an algorithm for efficiently learning these optimal policies from data for a large family of expressive utility functions. In particular, we use a contextual bandit algorithm to explore the space of policies while solving a convex optimization problem at each step to estimate the best policy based on the available information. This consequentialist paradigm facilitates a more holistic approach to equitable decision making. This paper has been This paper was accepted by Catherine Tucker, Special Issue on The Human-Algorithm Connection. Funding: This work was supported by Stanford Impact Labs [Start-Up Impact Labs/GEJBU], the Stanford Institute for Human-Centered Artificial Intelligence, and Harvard Data Science Initiative. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.00345 .

Reinforcement Learning-Based Multi-Phase Seed Scheduling for Network Protocol Fuzzing

In network protocol fuzzing, effective seed scheduling plays a critical role in improving testing efficiency. Traditional state-driven seed scheduling methods in network protocol fuzzing are often limited by imbalanced seed selection, monolithic scheduling strategies, and ineffective power allocation. To overcome these limitations, we propose SCFuzz, specifically by employing a multi-armed bandit model to dynamically balance exploration and exploitation across multiple fuzzing phases. The fuzzing process is divided into initial, middle, and final phases with seed selection strategies adapted at each phase to optimize the discovery of new states, paths, and code coverage. Additionally, SCFuzz employs a power allocation method based on state weights, focusing power on high-potential messages to improve the overall fuzzing efficiency. Experimental evaluations on open-source protocol implementations show that SCFuzz significantly improves state and code coverage, achieving up to 17.10% more states, 22.92% higher state transitions, and 7.92% greater code branch coverage compared to AFLNet. Moreover, SCFuzz improves seed selection effectiveness by 389.37% and increases power utilization by 45.61%, effectively boosting the overall efficiency of fuzzing.

AutoFR: Automated Filter Rule Generation for Adblocking

Adblocking relies on filter lists, which are manually curated and maintained by a community of filter list authors. Filter list curation is a laborious process that does not scale well to a large number of sites or over time. In this article, we introduce AutoFR, a reinforcement learning framework to fully automate the process of filter rule creation and evaluation for sites of interest. We design an algorithm based on multi-arm bandits to generate filter rules that block ads while controlling the trade-off between blocking ads and avoiding visual breakage. We test AutoFR on thousands of sites and show that it is efficient: It takes only a few minutes to generate filter rules for a site of interest. AutoFR is effective: It optimizes filter rules for a particular site that can block 86% of the ads, as compared to 87% by EasyList, while achieving comparable visual breakage. Using AutoFR as a building block, we devise three methodologies that generate filter rules across sites based on: (1) a modified version of AutoFR, (2) rule popularity, and (3) site similarity. We conduct an in-depth comparative analysis of these approaches by considering their effectiveness, efficiency, and maintainability. We demonstrate that some of them can generalize well to new sites in both controlled and live settings. We envision that AutoFR can assist the adblocking community in automatically generating and updating filter rules at scale.

Smart topology detection using multi-armed bandit reinforcement learning method

ABSTRACT Inventory management in network infrastructures currently relies on static methods, leading to data consistency issues due to the difficulty of keeping static reports and lists up to date with the dynamic nature of real-world networks. An alternative approach is the use of automatically generated network topology diagrams, which provide a visual representation of the network’s structure and organization, depicting the configuration and interconnections of nodes and links. These diagrams serve purposes such as network planning, management, and security management. However, the creation of automatic network topology diagrams remains a challenge. This paper addresses this challenge by proposing a Reinforcement Learning-based approach that utilizes network traffic data to automatically generate network topology diagrams. The preliminary results demonstrate promising outcomes, enabling the automatic generation of efficient network topology diagrams for various purposes, even in the case of large-scale networks. These automatically created diagrams can further facilitate security analysis, planning, and optimization activities. By focusing on enhancing security outcomes and enabling robust network analysis, this research contributes to advancing the field of network management, particularly in the domain of automatic network visualization and comprehensive security analysis.

Block pruning residual networks using Multi-Armed Bandits

ABSTRACT Recently, deep neural networks have been adopted in many application domains showing impressive performances that sometimes even exceed those of humans. However, the considerable size of deep learning models and their computational cost limits their use in mobile and embedded systems. Pruning is an important technique that has been used in the literature to compress a trained model by removing some of its elements (weights, neurons, filters, etc.) that do not notably contribute to its performance. In this paper, we propose a new block pruning method for residual networks that removes holes layers of the network based on the Multi-Armed Bandits (MAB) framework. The proposed method takes advantage of the existence of multiple trainable paths in residual models to identify and prune the less important residual blocks for inference. To evaluate our approach, we conducted extensive experiments using three different MAB algorithms, namely ε -greedy, Win Stay Lose Shift (WSLS), and the Upper Confidence Bound (UCB). The obtained results on different datasets show that our block pruning method can considerably reduce the model size and FLOPs with little to no impact on overall accuracy. We conducted a comparative analysis of our method with three existing techniques to demonstrate that our approach is highly competitive in terms of accuracy, while significantly reducing both model size and FLOPs. Moreover, our experimental results demonstrate that when our block pruning method is combined with a filter-level pruning technique, it leads to even greater reductions in model size and FLOPs while maintaining a relatively high accuracy.