Multi-armed Bandit Research Articles

The Impact of Movie Text Information on Movie Recommendation Systems based on Multi-Armed Bandit

Abstract. The multi-armed bandit problem has been extensively studied and applied in various fields such as recommendation systems and resource allocation. However, there is still a lack of research in the movie recommendation domain, such as the comparison of algorithm performance and the impact of movie text information. This paper uses the Movielens 100K movie review data set to compare and analyze the performance of classic algorithms-Greedy, UCB, and TS-in the multi-armed bandit problem. Additionally, to explore the influence of movie text information (such as movie summaries and user reviews) on system algorithms, this experiment utilizes web scraping technology through HTTP requests and HTML parsing, to extract data from the Douban website, obtaining the information and forming a new data set. In this movie review data set, the Greedy algorithm outperforms both UCB and TS algorithms. This study finds that incorporating movie text information improves the accuracy of all three algorithms, thus contributing to the enhancement of movie recommendation systems. This experiment did not consider larger datasets or movie image information, which could be addressed in future improvements.

Online Learning and Decision Making Under Generalized Linear Model with High-Dimensional Data

We propose a minimax concave penalized multiarmed bandit algorithm under the generalized linear model (G-MCP-Bandit) for decision-makers facing high-dimensional data in an online learning and decision-making environment. We demonstrate that in the data-rich regime, the G-MCP-Bandit algorithm attains the optimal cumulative regret in the sample size dimension and a tight bound in the covariate dimension and the significant covariate dimension. In the data-poor regime, the G-MCP-Bandit algorithm maintains a tight regret upper bound. In addition, we develop a local linear approximation method, the two-step weighted Lasso procedure, to identify the minimax concave penalty (MCP) estimator for the G-MCP-Bandit algorithm when samples are not independent and identically distributed. Under this procedure, the MCP estimator can match the oracle estimator with high probability and converge to the true parameters at the optimal convergence rate. Finally, through experiments based on both synthetic and real data sets, we show that the G-MCP-Bandit algorithm outperforms other benchmarking algorithms in terms of cumulative regret and that the benefits of the G-MCP-Bandit algorithm increase in the data’s sparsity level and the size of the decision set. This paper was accepted by J. George Shanthikumar, big data analytics. Funding: This work was supported by the National Natural Science Foundation of China (NSFC) [Grants W2441021, 72371172, 72342023, and 71929101] and the National Science Foundation (NSF) [Grant DMS 1820702]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01557 .

Approximate planning in spatial search.

How people plan is an active area of research in cognitive science, neuroscience, and artificial intelligence. However, tasks traditionally used to study planning in the laboratory tend to be constrained to artificial environments, such as Chess and bandit problems. To date there is still no agreed-on model of how people plan in realistic contexts, such as navigation and search, where values intuitively derive from interactions between perception and cognition. To address this gap and move towards a more naturalistic study of planning, we present a novel spatial Maze Search Task (MST) where the costs and rewards are physically situated as distances and locations. We used this task in two behavioral experiments to evaluate and contrast multiple distinct computational models of planning, including optimal expected utility planning, several one-step heuristics inspired by studies of information search, and a family of planners that deviate from optimal planning, in which action values are estimated by the interactions between perception and cognition. We found that people's deviations from optimal expected utility are best explained by planners with a limited horizon, however our results do not exclude the possibility that in human planning action values may be also affected by cognitive mechanisms of numerosity and probability perception. This result makes a novel theoretical contribution in showing that limited planning horizon generalizes to spatial planning, and demonstrates the value of our multi-model approach for understanding cognition.

A Meta-Learning Approach to Mitigating the Estimation Bias of Q-Learning

It is a longstanding problem that Q-learning suffers from the overestimation bias. This issue originates from the fact that Q-learning uses the expectation of maximum Q-value to approximate the maximum expected Q-value. A number of algorithms, such as Double Q-learning, were proposed to address this problem by reducing the estimation of maximum Q-value, but this may lead to an underestimation bias. Note that this underestimation bias may have a larger performance penalty than the overestimation bias. Different from previous algorithms, this article studies this issue from a fresh perspective, i.e., meta-learning view, which leads to our Meta-Debias Q-learning. The main idea is to extract the maximum expected Q-value with meta-learning over multiple tasks to remove the estimation bias of maximum Q-value and help the agent choose the optimal action more accurately. However, there are two challenges: (1) How to automatically select suitable training tasks? (2) How to positively transfer the meta-knowledge from selected tasks to remove the estimation bias of maximum Q-value? To address the two challenges mentioned above, we quantify the similarity between the training tasks and the test task. This similarity enables us to select appropriate “partial” training tasks and helps the agent extract the maximum expected Q-value to remove the estimation bias. Extensive experiment results show that our Meta-Debias Q-learning outperforms SOTA baselines drastically in three evaluation indicators, i.e., maximum Q-value, policy, and reward. More specifically, our Meta-Debias Q-learning only underestimates \(1.2*10^{-3}\) than the maximum expected Q-value in the multi-armed bandit environment and only differs \(5.04\%-5\%=0.04\%\) than the optimal policy in the two states MDP environment. In addition, we compare the uniform weight and our similarity weight. Experiment results reveal fundamental insights into why our proposed algorithm outperforms in the maximum Q-value, policy, and reward.

Exploring the efficacy of Multi-Armed Bandit Algorithms in dynamic decision-making

Abstract. Originating from the scenario of gambling machines in casinos, the Multi-Armed Bandit problem aims to optimize decision-making processes under limited resources to achieve maximum returns. This article delves into the principles, classifications, and practical applications of this problem. Researchers have proposed various algorithms to address this issue, including -greedy, Upper Confidence Bound, and Thompson Sampling, which have demonstrated good performance across different scenarios. The article further elaborates on the fundamental principles of Multi-Armed Bandit algorithms, encompassing the trade-off between exploration and exploitation, and provides a detailed classification of algorithms based on probability (e.g., -greedy) and value (e.g., UCB). These algorithms not only provide a framework for addressing real-world problems such as advertisement placement and resource allocation, but also possess significant theoretical value in the fields of machine learning and reinforcement learning. By balancing exploration and exploitation, Multi-Armed Bandit algorithms offer effective tools for making optimal decisions in uncertain environments, thus driving the development of related fields.

Learning-based cooperative content caching and sharing for multi-layer vehicular networks

Caching and sharing the content files is critical and fundamental for various future vehicular applications. However, how to satisfy the content demands in a timely manner with limited storage is an open issue owing to the high mobility of vehicles and the unpredictable distribution of dynamic requests. To better serve the requests from the vehicles, a cache-enabled multi-layer architecture, consisting of a Micro Base Station (MBS) and several Small Base Stations (SBSs), is proposed in this paper. Considering that vehicles usually travel through the coverage of multiple SBSs in a short time period, the cooperative caching and sharing strategy is introduced, which can provide comprehensive and stable cache services to vehicles. In addition, since the content popularity profile is unknown, we model the content caching problems in a Multi-Armed Bandit (MAB) perspective to minimize the total delay while gradually estimating the popularity of content files. The reinforcement learning-based algorithms with a novel Q-value updating module are employed to update the caching files in different timescales for MBS and SBSs, respectively. Simulation results show the proposed algorithm outperforms benchmark algorithms with static or varying content popularity. In the high-speed environment, the cooperation between SBSs effectively improves the cache hit rate and further improves service performance.

A Truthful Pricing-Based Defending Strategy Against Adversarial Attacks in Budgeted Combinatorial Multi-Armed Bandits

A Truthful Pricing-Based Defending Strategy Against Adversarial Attacks in Budgeted Combinatorial Multi-Armed Bandits

Dual Moral Hazard and the Tyranny of Success

I explain why current success can undermine an organization’s ability to innovate. I consider a standard bandit problem between a safe and a risky arm with two modifications. First, a principal allocates resources. Second, an agent must install the risky arm, which is not contractible. If the principal cannot commit to a resource policy, a dual moral hazard problem emerges: The agent’s pay must be tied to the risky arm’s success to encourage installation, inducing the principal to stop experimenting with the arm prematurely. This problem intensifies as the safe arm becomes more profitable, potentially leaving the organization worse off. (JEL D21, D82, D86, O31)

Performance Comparison of UCB, TS, and -Greedy TS Algorithms through Simulation of Multi-Armed Bandit Machine

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

Applications and Advances of UCB Algorithms in Dynamic and Contaminated Environments

The Upper Confidence Bound (UCB) algorithm is a widely used approach in the Multi-Armed Bandit (MAB) problem, where the goal is to maximize cumulative rewards over time by selecting the best possible action among several options. The UCB algorithm uses confidence bounds to balance exploration and exploitation, guiding its decision-making. In recent years, researchers have identified significant challenges in applying the UCB algorithm to dynamic and contaminated environments. In such scenarios, the underlying conditions may change over time, making it difficult for standard UCB to adapt, or the data may be polluted by noise and outliers, leading to incorrect estimations of reward distributions. To address these challenges, several variants of the UCB algorithm have been developed. These new approaches are designed to better handle the complexities of changing environments and data contamination, ensuring more robust and reliable performance in these difficult settings. This paper aims to provide a comprehensive review of Robust-UCB (cr-UCB), Sliding Window UCB (SW-UCB) and bandit-over-bandit UCB (BOB-UCB). Focusing on their theoretical foundations, practical applications, and empirical performance. By examining how these algorithms have been adapted to handle the complexities of dynamic and contaminated environments, we found that the adaptability of these algorithms in dynamic environments is significantly improved, and they can effectively reduce decision-making errors caused by data pollution, thus providing a more reliable solution to the multi-armed bandit problem.

On the Performance of the Minimax Optimal Strategy in the Stochastic Case of Logistic Bandits

The multi-armed bandit problem is a well-established model for examining the exploration/exploitation trade-offs in sequential decision-making tasks. This study focuses on the logistic bandit, where rewards are derived from two distinct datasets of movie ratings, ranging from 1 to 5, each characterized by different variances. Previous research has shown that regret bounds for multi-armed bandit algorithms can be unstable across varying environments. However, this paper provides new insights by demonstrating the robustness of the Minimax Optimal Strategy in the Stochastic (MOSS) algorithm across environments with differing reward variances. Unlike prior studies, this research shows that MOSS maintains superior performance in both dense and sparse reward settings, consistently outperforming widely used algorithms like UCB and TS, particularly in high variance conditions and over sufficient trials. The findings indicate that MOSS achieves logarithmic expected regret for both types of environments, effectively balancing exploration and exploitation. Specifically, with K arms and T time steps, the regret R(T) of MOSS is bounded by O((KT logT )). This work highlights MOSS as a robust solution for handling diverse stochastic conditions, filling a crucial gap in the understanding of its practical adaptability across different reward distributions.

Advanced MiniMax Optimality Strategy in Bandit Problems

Nowadays, with the development of the Internet, recommending personalized information for every user becomes economically valuable. How to properly recommend to users the content they prefer is important. Mostly, bandit strategies are useful for this problem. As one of the most time-saving algorithms, Minimax Optimal Strategy (MOSS) performs well in most data sets. However, the traditional MOSS algorithm applies a conservative strategy in order to minimize the cumulative regrets in the worst situations. This strategy will over-explore the bandit models and spend too much time exploring every arm at the initial stage. This means the MOSS strategy cannot fully take advantage of the known information and then the accuracy and efficiency will be lower. In real life, people have a lot of Internet information to look through and they will not stay on one website or one advertisement for a long time. So, designing an efficient algorithm that performs well in all situations and can make good use of the known knowledge to do quick exploitation becomes so important. Therefore, this research aims to develop an advanced MOSS algorithm that can properly explore each of the bandit arms and improve the performance of the exploitation phase. By achieving this goal, this research adds an enhancement factor on the confidence bound part to speed up the exploration process. The research bases on the MovieLens data set and the data set is divided into 18 categories (arms) according to its genre attribute. Then apply this new advanced MOSS strategy to the designed data set and compare its performance with normal MOSS and UCB algorithms. Results display that the advanced MOSS strategys selection provides higher total rewards and lower average cumulative regrets than others with the advantage of time-saving. This shows the improvement of the advanced MOSS strategy in balancing the exploitation and exploration phases.

A Strategy for Advertisement Placement based on the Multi-Armed Tiger Problem

The purpose of this study is to investigate the effectiveness of using Multi armed bandit model, which contains -greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms, to optimize online advertisement placement. Through simulating different types of ad placements using different algorithms and comparing them, this paper intends to demonstrate the feasibility of the Multi-Armed Robber model for the ad placement problem. The results show that the multi-armed bandit model can improve the ad click rate compared with the traditional ad placement strategy. Thompson sampling algorithm outperforms the -greedy algorithm and UCB algorithm in this paper's experiments, which can better balance exploration and application and reduce regret. The algorithm provides a more efficient method of allocating ad resources. These findings provide new insights into the field of digital marketing and may have an impact on the development of actual ad placement strategies.

Analysis of the Effectiveness of Multi-Armed Bandit Algorithms in Crop Pricing

This study explores the application of multi-armed bandit (MAB) algorithms in dynamic pricing of crops, with a focus on evaluating the adaptive upper confidence bound (asUCB) and Thompson Sampling (TS) algorithms. Through simulation experiments on historical data, the study analyzed the performance of these algorithms in fitting actual market price trends and responding to future price fluctuations. The results indicate that the asUCB algorithm performed particularly well in both the training dataset and simulation tests, demonstrating low mean squared error (MSE) and minimal cumulative regret, reflecting its rapid convergence and stable pricing capabilities. In contrast, although the TS algorithm initially performed slightly less effectively, it demonstrated unique advantages in dealing with market volatility due to its strong adaptability. This study demonstrates the potential application of MAB algorithms in dynamic pricing, providing valuable insights for pricing strategies in the agricultural product market.

Thompson sampling for multi-armed bandits in big data environments

Thompson sampling for multi-armed bandits in big data environments

The Investigation of Progress and Application in the Multi-Armed Bandit Algorithm

This article delves deeply into the development process and practical applications of the multi-armed bandit algorithm in the current digital era. With the continuous popularity of online advertising and online learning, information has grown explosively, making decision optimization crucial. The multi-armed bandit algorithm, as a sequential decision-making model, encompasses common algorithms such as the greedy algorithm, -greedy algorithm, UCB algorithm, and Thompson sampling. Its main role is to seek the best balance between exploration and exploitation to solve the fundamental problems in reinforcement learning. The article introduces an internationally released datasets, namely MovieLens, and elaborates in detail a series of indicators for evaluation, including the average number of friends per user, the average number of listened-to artists per user, the average number of movie rating times, the average number of tags added by users, content diversity indicators, and statistics on the differences in click-through rates of recommendations for different types of movies. In addition, the article also presents the specific methods of literature collection, screening, analysis, and review. Its purpose is to understand the multi-armed bandit algorithm more deeply and provide strong guidance for the future development and wide application of this algorithm in various fields.

UCB and Thompson Sampling Algorithms in Long-Term Investing

In the rapidly evolving financial markets, traditional investment strategies are increasingly being replaced by advanced algorithms. Among these, the Upper Confidence Bound (UCB) and TS algorithms, which originate from the multi-armed bandit framework, have demonstrated significant potential across various sectors. This study utilizes historical stock data from companies during the period from 2019 to 2024 to evaluate the performance of these algorithms in long-term investment strategies. The research aims to identify how these algorithms manage investment risks and maximize returns under varying market conditions. The results reveal distinct behaviors of UCB and TS in the financial markets: UCB performs better in markets with lower volatility, while TS exhibits stronger adaptability in high-volatility environments, thereby enhancing potential returns. Through a detailed comparison of UCB and TS in long-term investment scenarios, this study provides valuable insights into the strategic application of these algorithms. It offers investment strategy guidance for investors and financial strategists and supplies effective information for economic decision-making in relevant fields.

PERFECT: Personalized Exercise Recommendation Framework and architECTure

Background : The health benefits of regular physical activity (PA) are well-established and widely acknowledged. Through the integration of wearable trackers, the Internet of Things (IoT)—a network of interconnected devices capable of collecting and exchanging data—coupled with mobile health (mHealth), which refers to the use of mobile devices to support medical and public health practices, it is now feasible to systematically gather and present individual exercise behaviors. This advanced approach enables the precise correlation of users’ physiological data and daily activities with their specific fitness needs, offering a personalized pathway to improving health outcomes. Objective : This study aims to enhance PA levels among individuals by developing a personalized exercise recommendation system. Utilizing reinforcement learning, the system proposes tailored exercise plans based on biomarkers and the user’s specific context. Methods : In this study, we developed applications for smartphones and smartwatches designed to gather, monitor, and recommend exercise routines through the application of a contextual multi-arm bandit algorithm. To evaluate the efficacy of this mHealth exercise regimen, we enlisted the participation of twenty female college students. Results : The outcomes of our investigation revealed a significant enhancement in the average daily duration of exercise (P \({\lt}\) . 001). Participants expressed high levels of satisfaction with both the walking program and the recommendation system, achieving average ratings of 4.31 (SD \(=\) 0.60) and 3.69 (SD \(=\) 0.95), respectively, on a 5-point scale. Furthermore, the average scores for participants’ confidence in safely performing the recommended walking exercises, as well as their perception of the study’s effectiveness in meeting their PA needs, were both above 4, indicating a positive reception and confidence in the program’s design and implementation. Conclusions : The evolution of the IoT and wearable technology has marked the beginning of a new era for mHealth systems, particularly in the personalization of health interventions. Such advancements enable the precise personalization of PA recommendations, potentially enhancing user engagement and performance outcomes. This paper introduces a novel exercise recommendation system that utilizes reinforcement learning to personalize walking exercises based on the user’s biomarkers and context, aiming to improve the user’s aerobic capacity significantly.

A Simple and Optimal Policy Design with Safety Against Heavy-Tailed Risk for Stochastic Bandits

We study the stochastic multi-armed bandit problem and design new policies that enjoy both optimal regret expectation and light-tailed risk for regret distribution. We first find that any policy that obtains the optimal instance-dependent expected regret could incur a heavy-tailed regret tail risk that decays slowly with T. We then focus on policies that achieve optimal worst-case expected regret. We design a novel policy that (i) enjoys the worst-case optimality for regret expectation and (ii) has the worst-case tail probability of incurring a regret larger than any regret threshold that decays exponentially with respect to T. The decaying rate is proved to be optimal for all worst-case optimal policies. Our proposed policy achieves a delicate balance between doing more exploration at the beginning of the time horizon and doing more exploitation when approaching the end, compared with standard confidence-bound-based policies. We also enhance the policy design to accommodate the “any-time” setting where T is unknown a priori, highlighting “lifelong exploration”, and prove equivalently desired policy performances as compared with the “fixed-time” setting with known T. From a managerial perspective, we show through numerical experiments that our new policy design yields similar efficiency and better safety compared to celebrated policies. Our policy design is preferable especially when (i) there is a risk of underestimating the volatility profile, or (ii) there is a challenge of tuning policy hyper-parameters. We conclude by extending our proposed policy design to the stochastic linear bandit setting that leads to both worst-case optimality in terms of regret expectation and light-tailed risk on regret distribution. This paper was accepted by J. George Shanthikumar, data science. Funding: The work of D. Simchi-Levi and F. Zhu is partially supported by the MIT Data Science Laboratory. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.03512 .

Interpreting pretext tasks for active learning: a reinforcement learning approach

As the amount of labeled data increases, the performance of deep neural networks tends to improve. However, annotating a large volume of data can be expensive. Active learning addresses this challenge by selectively annotating unlabeled data. There have been recent attempts to incorporate self-supervised learning into active learning, but there are issues in utilizing the results of self-supervised learning, i.e., it is uncertain how these should be interpreted in the context of active learning. To address this issue, we propose a multi-armed bandit approach to handle the information provided by self-supervised learning in active learning. Furthermore, we devise a data sampling process so that reinforcement learning can be effectively performed. We evaluate the proposed method on various image classification benchmarks, including CIFAR-10, CIFAR-100, Caltech-101, SVHN, and ImageNet, where the proposed method significantly improves previous approaches.