Bandit Strategy Research Articles

Distribution Fairness in Multiplayer AI Using Shapley Constraints

Experience management (EM) agents in multiplayer serious games face unique challenges and responsibilities regarding the fair treatment of players. One such challenge is the Greedy Bandit Problem that arises when using traditional Multi-Armed Bandits (MABs) as EM agents, which results in some players routinely prioritized while others may be ignored. We will show that this problem can be a cause of player non-adherence in a multiplayer serious game played by human users. To mitigate this effect, we propose a new bandit strategy, the Shapley Bandit, which enforces fairness constraints in its treatment of players based on the Shapley Value. We evaluate our approach via simulation with virtual players, finding that the Shapley Bandit can be effective in providing more uniform treatment of players while incurring only a slight cost in overall performance to a typical greedy approach. Our findings highlight the importance of fair treatment among players as a goal of multiplayer EM agents and discuss how addressing this issue may lead to more effective agent operation overall. The study contributes to the understanding of player modeling and EM in serious games and provides a promising approach for balancing fairness and engagement in multiplayer environments.

Unbiased Identification of Broadly Appealing Content Using a Pure Exploration Infinitely-Armed Bandit Strategy

Podcasting is an increasingly popular medium for entertainment and discourse around the world, with tens of thousands of new podcasts released on a monthly basis. We consider the problem of identifying from these newly-released podcasts those with the largest potential audiences so they can be considered for personalized recommendation to users. We first study and then discard a supervised approach due to the inadequacy of either content or consumption features for this task, and instead propose a novel non-contextual bandit algorithm in the fixed-budget infinitely-armed pure-exploration setting. We demonstrate that our algorithm is well-suited to the best-arm identification task for a broad class of arm reservoir distributions, out-competing a large number of state-of-the-art algorithms. We then apply the algorithm to identifying podcasts with broad appeal in a simulated study, and show that it efficiently sorts podcasts into groups by increasing appeal while avoiding the popularity bias inherent in supervised approaches. Finally, we study a setting in which users are more likely to stream more-streamed podcasts independent of their general appeal and find that our proposed algorithm is robust to this type of popularity bias.

Portfolio allocation strategy for active learning Kriging-based structural reliability analysis

Recently, numerous studies have focused on structural reliability analysis, with the Kriging-based active learning method being particularly popular. A variety of Kriging-based learning functions have been proposed, and shown to perform well in various tasks. However, no single learning function has been demonstrated to consistently outperformed the others in all tasks, and selecting the most appropriate learning function for a given task remains a challenge in engineering applications. In this paper, inspired by the multi-armed bandit strategy, a portfolio allocation of different learning functions is proposed to resolve the issue of selecting a single one, where the better learning functions are selected online according to their past performance. Finally, three classical numerical examples and two engineering applications are adopted to validate the effectiveness of the proposed method.

Intelligent short term traffic forecasting using deep learning models with Bayesian contextual hyperband tuning

AbstractAn intelligent transport system (ITS) is fully valuable only if it can dynamically and aptly integrate all the latest cutting‐edge technologies. An ITS focuses on providing services like promptly offering real‐time road traffic information to interested parties, finding ways to reduce the average waiting time and offer secure and reliable services for commuters using past statistics. Short‐term traffic prediction is one such area in which the research community has focused in the past decade. Existing models developed for prediction has scope to improve in terms of accuracy and training time. There is a necessity to develop a best‐performing model that is computationally affluent to train with the optimal hyperparameter configuration as input which leads to improved performance. This article proposes a model that captures the traffic flow trend present in the past data to predict the flow for a future time interval. This model is an amalgamation of seasonal global trend (SGT) model and long short‐term memory (LSTM) model with attention mechanism. A novel hyperparameter tuning algorithm is also proposed which is based on multi‐armed bandit strategy with context, incorporating the right trade‐off between exploitation and exploration of the hyperparameter space using successive halving. Experimental results conducted proves that our forecast model in combination with the proposed hyperparameter tuning algorithm outperforms the existing models like SGT, LSTM models in terms of accuracy and time.

Improved differential evolution based on multi-armed bandit for multimodal optimization problems

The main aim of multimodal optimization problems (MMOPs) is to find and deal with multiple optimal solutions using an objective function. MMOPs perform the exploration and exploitation simultaneously in the search space. The novelty of this paper includes the following improvements in differential evolution to be able to solve MMOPs. Clusters are formed from the whole population by applying a niching technique which uses the softmax strategy to assign a cutting probability to the species. Then iterative mutation strategy is followed to generate the unbiased mutant vector. Further, Multi-Armed Bandit (MAB) strategy is used to ensure that new individuals are generated in promising areas. The experimentation of the proposed algorithm has been performed on 20 benchmark functions from IEEE Congress on Evolutionary Computation 2013 (CEC2013). The results depict that the proposed algorithm can be compared with 15 state-of-the-art multimodal optimization algorithms in terms of locating accurate optimal solutions.

Multi-Armed Bandit Regularized Expected Improvement for Efficient Global Optimization of Expensive Computer Experiments With Low Noise

Computer experiments are widely used to mimic expensive physical processes as black-box functions. A typical challenge of expensive computer experiments is to find the set of inputs that produce the desired response. This study proposes a multi-armed bandit regularized expected improvement (BREI) method to adaptively adjust the balance between exploration and exploitation for efficient global optimization of long-running computer experiments with low noise. The BREI adds a stochastic regularization term to the objective function of the expected improvement to integrate the information of additional exploration and exploitation into the optimization process. The proposed study also develops a multi-armed bandit strategy based on Thompson sampling for adaptive optimization of the tuning parameter of the BREI based on the preexisting and newly tested points. The performance of the proposed method is validated against some of the existing methods in the literature under different levels of noise using a case study on optimization of the collision avoidance algorithm in mobile robot motion planning as well as extensive simulation studies.

Nonparametric Bayesian multiarmed bandits for single-cell experiment design

The problem of maximizing cell type discovery under budget constraints is a fundamental challenge for the collection and analysis of single-cell RNA-sequencing (scRNA-seq) data. In this paper we introduce a simple, computationally efficient and scalable Bayesian nonparametric sequential approach to optimize the budget allocation when designing a large-scale experiment for the collection of scRNA-seq data for the purpose of, but not limited to, creating cell atlases. Our approach relies on the following tools: (i) a hierarchical Pitman–Yor prior that recapitulates biological assumptions regarding cellular differentiation, and (ii) a Thompson sampling multiarmed bandit strategy that balances exploitation and exploration to prioritize experiments across a sequence of trials. Posterior inference is performed by using a sequential Monte Carlo approach which allows us to fully exploit the sequential nature of our species sampling problem. We empirically show that our approach outperforms state-of-the-art methods and achieves near-Oracle performance on simulated and scRNA-seq data alike. HPY-TS code is available at https://github.com/fedfer/HPYsinglecell.

Upper confidence tree for planning restart strategies in multi-modal optimization

In the context of gradient-free multi-modal optimization, numerous algorithms are based on restarting evolution strategies. Such an algorithm classically performs many local searches, for finding all the global optima of the objective function. The strategy used to select the restarting points (i.e., the initial points of the local searches) is a crucial step of the method. In previous works, a strategy based on reinforcement learning has been proposed: the search space is partitioned and a multi-armed bandit algorithm is used to select an interesting region to sample. This strategy significantly improves the main optimization algorithm but is limited to small dimensional problems. In this paper, we propose an algorithm tackling this problem. The main contributions are (1) a tree-based scheme for hierarchically partition the search space and (2) a multi-armed bandit strategy for traversing this tree. Thus, a node in the tree corresponds to a region of the search space, and its children partition this region according to one dimension. The multi-armed bandit strategy is used to traverse the tree by selecting interesting children recursively. We have experimented our algorithm on difficult multi-modal functions, with small and large dimensions. For small dimensions, we observe performances comparable to previous state-of-the-art algorithms. For large dimensions, we observe better results as well as lower memory consumption.

Non-Stationary Bandit Strategy for Rate Adaptation With Delayed Feedback

Rate adaptation is an efficient mechanism to utilize the channel capacity by adjusting the modulation and coding scheme in a dynamic wireless environment. The channel feedback, such as acknowledgment/negative acknowledgment (ACK/NACK) messages or the channel measurement such as received signal strength indicator (RSSI) can be applied to the rate adaptation. Existing rate adaptation algorithms are mainly driven by heuristics. They can not achieve satisfactory transmission rates in the time-varying environment. In this paper, we focus on the rate adaptation problem in a time-division duplex (TDD) system. A multi-armed bandit (MAB) strategy is applied to learn the changes of the channel condition from both RSSI and ACK/NACK signals. A discounted upper confidence bound based rate adaptation (DUCB-RA) algorithm is proposed. We show that the performance of the proposed algorithm is converged to the optimal with mathematical proofs. Simulation results demonstrate that the proposed algorithm can adapt to the time-varying channel and achieve better transmission throughput compared to existing rate adaptation algorithms.

Selfish Bandit-Based Cognitive Anti-Jamming Strategy for Aeronautic Swarm Network in Presence of Multiple Jammer

In order to enhance the anti-jamming capability of aeronautic swarm tactical network in the complicated electromagnetic environment, we address the problem of bandit-based cognitive anti-jamming strategy for enabling reliable information transmission. We first present an adversarial multiuser multi-armed bandit model for the aeronautic swarm network employing airborne cognitive radios with the same-frequency simultaneous transmit and receive feature. Then, we utilize the improved energy detection method to perform jamming sensing and derive the closed expression of false alarm probability, false detection probability, and the optimal decision threshold in the case of single and multi-jammer. Finally, using the jamming sensing output to calculate reward and with the objective of maximizing the throughput of each airborne radio, a decentralized selfish doubling trick kl-UCB <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">++</sup> anti-jamming strategy is developed to allocate an optimal configuration of transmitting power and spectrum channel to each radio. This anytime bandit strategy is simultaneously minimaxed optimal and asymptotically optimal. The simulation results validate that the aggregate average throughput, cumulative regret obtained with the proposed anti-jamming strategy outperform the well-known UCB, kl-UCB <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">++</sup> bandit algorithm.

Fast Bayesian hyperparameter optimization on large datasets

Bayesian optimization has become a successful tool for optimizing the hyperparameters of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed FABOLAS, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that FABOLAS often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.

Lobster and Groundfish Management in the Gulf of Maine: A Rational Choice Perspective

Many of the world’s most important fisheries are in a state of crisis. The Maine lobster industry is one of the very few exceptions, with catches at an all time high. Much of the success of this fishery stems from effective conservation legislation, passed largely through the long-term lobbying activities of powerful factions within the industry itself. In stark contrast is the Maine groundfishery, where stocks and catches have never been so low. This industry has had to deal with a top-down management system that has given it little incentive to conserve and has made it virtually impossible to get rules the industry considers effective. They have responded by using a roving bandit strategy. From the perspective of rational choice theory, the basic difference is that the lobster industry has been able to organize politically to get legislation to solve its communal action dilemma, while the groundfishing industry, like most fishing industries, has not. This paper explores the biological, technical, econom...