Thompson Sampling Research Articles

Exploring the efficacy of Multi-Armed Bandit Algorithms in dynamic decision-making

Abstract. Originating from the scenario of gambling machines in casinos, the Multi-Armed Bandit problem aims to optimize decision-making processes under limited resources to achieve maximum returns. This article delves into the principles, classifications, and practical applications of this problem. Researchers have proposed various algorithms to address this issue, including -greedy, Upper Confidence Bound, and Thompson Sampling, which have demonstrated good performance across different scenarios. The article further elaborates on the fundamental principles of Multi-Armed Bandit algorithms, encompassing the trade-off between exploration and exploitation, and provides a detailed classification of algorithms based on probability (e.g., -greedy) and value (e.g., UCB). These algorithms not only provide a framework for addressing real-world problems such as advertisement placement and resource allocation, but also possess significant theoretical value in the fields of machine learning and reinforcement learning. By balancing exploration and exploitation, Multi-Armed Bandit algorithms offer effective tools for making optimal decisions in uncertain environments, thus driving the development of related fields.

Hierarchical Federated Deep Reinforcement Learning based Joint Communication and Computation for UAV Situation Awareness

The computation-intensive situational awareness (SA) task of unmanned aerial vehicle (UAV) is greatly affected by its limited power and computing capability. To solve this challenge, we consider the joint communication and computation (JCC) design for UAV network in this paper. Firstly, a multi-objective optimization (MOO) model, which can optimize UAV computation offloading, transmit power, and local computation resources simultaneously, is built to minimize energy consumption and task execution delay. Then, we develop Thompson sampling based double-DQN (TDDQN) learning algorithm which allows the agent to explore more deeply and effectively, and propose a joint optimization algorithm that combines TDDQN and sequential least squares quadratic programming (SLSQP) to handle the MOO problem. Finally, to enhance the training speed and quality, we incorporate federated learning (FL) into the presented joint optimization algorithm and propose hierarchical federated TDDQN with SLSQP (HF TDDQN-S) to implement the JCC design. Simulation results show that the introduced HF TDDQN-S can efficiently learn the best JCC strategy and minimize the average cost contrasted with the DDQN with SLSQP (DDQN-S) and TDDQN with SLSPQ (TDDQN-S) approach, and achieve the low average delay SA with power efficient.

Performance Comparison of UCB, TS, and -Greedy TS Algorithms through Simulation of Multi-Armed Bandit Machine

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

The Investigation of Progress and Application in the Multi-Armed Bandit Algorithm

This article delves deeply into the development process and practical applications of the multi-armed bandit algorithm in the current digital era. With the continuous popularity of online advertising and online learning, information has grown explosively, making decision optimization crucial. The multi-armed bandit algorithm, as a sequential decision-making model, encompasses common algorithms such as the greedy algorithm, -greedy algorithm, UCB algorithm, and Thompson sampling. Its main role is to seek the best balance between exploration and exploitation to solve the fundamental problems in reinforcement learning. The article introduces an internationally released datasets, namely MovieLens, and elaborates in detail a series of indicators for evaluation, including the average number of friends per user, the average number of listened-to artists per user, the average number of movie rating times, the average number of tags added by users, content diversity indicators, and statistics on the differences in click-through rates of recommendations for different types of movies. In addition, the article also presents the specific methods of literature collection, screening, analysis, and review. Its purpose is to understand the multi-armed bandit algorithm more deeply and provide strong guidance for the future development and wide application of this algorithm in various fields.

Analysis of the Effectiveness of Multi-Armed Bandit Algorithms in Crop Pricing

This study explores the application of multi-armed bandit (MAB) algorithms in dynamic pricing of crops, with a focus on evaluating the adaptive upper confidence bound (asUCB) and Thompson Sampling (TS) algorithms. Through simulation experiments on historical data, the study analyzed the performance of these algorithms in fitting actual market price trends and responding to future price fluctuations. The results indicate that the asUCB algorithm performed particularly well in both the training dataset and simulation tests, demonstrating low mean squared error (MSE) and minimal cumulative regret, reflecting its rapid convergence and stable pricing capabilities. In contrast, although the TS algorithm initially performed slightly less effectively, it demonstrated unique advantages in dealing with market volatility due to its strong adaptability. This study demonstrates the potential application of MAB algorithms in dynamic pricing, providing valuable insights for pricing strategies in the agricultural product market.

Thompson sampling for multi-armed bandits in big data environments

Thompson sampling for multi-armed bandits in big data environments

UCB and Thompson Sampling Algorithms in Long-Term Investing

In the rapidly evolving financial markets, traditional investment strategies are increasingly being replaced by advanced algorithms. Among these, the Upper Confidence Bound (UCB) and TS algorithms, which originate from the multi-armed bandit framework, have demonstrated significant potential across various sectors. This study utilizes historical stock data from companies during the period from 2019 to 2024 to evaluate the performance of these algorithms in long-term investment strategies. The research aims to identify how these algorithms manage investment risks and maximize returns under varying market conditions. The results reveal distinct behaviors of UCB and TS in the financial markets: UCB performs better in markets with lower volatility, while TS exhibits stronger adaptability in high-volatility environments, thereby enhancing potential returns. Through a detailed comparison of UCB and TS in long-term investment scenarios, this study provides valuable insights into the strategic application of these algorithms. It offers investment strategy guidance for investors and financial strategists and supplies effective information for economic decision-making in relevant fields.

A Strategy for Advertisement Placement based on the Multi-Armed Tiger Problem

The purpose of this study is to investigate the effectiveness of using Multi armed bandit model, which contains -greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms, to optimize online advertisement placement. Through simulating different types of ad placements using different algorithms and comparing them, this paper intends to demonstrate the feasibility of the Multi-Armed Robber model for the ad placement problem. The results show that the multi-armed bandit model can improve the ad click rate compared with the traditional ad placement strategy. Thompson sampling algorithm outperforms the -greedy algorithm and UCB algorithm in this paper's experiments, which can better balance exploration and application and reduce regret. The algorithm provides a more efficient method of allocating ad resources. These findings provide new insights into the field of digital marketing and may have an impact on the development of actual ad placement strategies.

Analysis of the Application of Artificial Intelligence in Mahjong

Over the past few years, AI-based models for Mahjong have received a substantial amount of contribution due to the advancements in machine learning and game theory. In this review, these state-of-the-art AI models are considered, including Tjong, Kanachan, Suphx, and Zhejiang University’s developed model. The deployed approaches are disparately represented by reinforcement learning, Monte Carlo tree search (MCTS), and deep neural networks, all of which aim at solving the issues that come with the game’s structure, such as incomplete and overburdening information. The Tjong model complements the power of the transformer architecture with hierarchical decision-making and in turn, brings strategic depth to the gameplay. Kanachan employs Q-learning and MCTS to optimize decision-making, while Suphx combines these methods with the novel ideas of Double Q-learning and Thompson Sampling to achieve higher performance than seen before. The integration of reinforcement learning and a new evaluation model imbues the model with the unique properties of both learning machines and human expertise, namely depth and intelligence. As a result of all these exploits, obstacles still exist, the most notable among them are the management of completely unknown information, handling long-distance games, and the strategic balance between offense and defense. In the future, determinants of probabilistic reasoning, the improvement of deep learning systems, and the prowess of reward shaping will result in AI improvement in Mahjong.

Learning from different perspectives for regret reduction in reinforcement learning: A free energy approach

Reinforcement learning (RL) is the core method for interactive learning in living and artificial creatures. Nevertheless, in contrast to humans and animals, artificial RL agents are very slow in learning and suffer from the curse of dimensionality. This is partially due to using RL in isolation; i.e. lack of social learning and social diversity. We introduce a free energy-based social RL for learning novel tasks. Society is formed by the learning agent and some diverse virtual ones. That diversity is in their perception while all agents use the same interaction samples for learning and share the same action set. Individual difference in perception is mostly the cause of perceptual aliasing however, it can result in virtual agents’ faster learning in early trials. Our free energy method provides a knowledge integration method for the main agent to benefit from that diversity to reduce its regret. It rests upon Thompson sampling policy and behavioral policy of main and virtual agents. Therefore, it is applicable to a variety of tasks, discrete or continuous state space, model-free, and model-based tasks as well as to different reinforcement learning methods. Through a set of experiments, we show that this general framework highly improves learning speed and is clearly superior to previous existing methods. We also provide convergence proof.

Strategy Selection Using Multi-Armed Bandit Algorithms in Financial Markets

This paper aims to evaluate the effectiveness of Multi-Armed Bandit (MAB) algorithms in choosing the optimal trading strategy among the sub optimal ones within financial markets. The research aims to addresses the challenge of adapting to dynamic market conditions. By introducing a Composite Trading Strategy that integrates trend-following, mean reversion, and momentum strategies, the study investigates whether increased trading frequency enhances the performance of profitability of various MAB algorithms, including UCB, Thompson Sampling, and epsilon-greedy. The experimental results indicate that while the introduction of complex strategies greatly improves returns in favorable market conditions, MAB algorithms still face limitations in adverse market environments. The findings highlight the potential of MAB algorithms in financial strategy selection and suggest directions for future research to enhance their adaptability in adverse markets.

Comparison of Multi-Armed Bandit Algorithms in Advertising Recommendation Systems

In today's rapidly evolving online environment, advertising recommendation systems utilize multi-armed bandit algorithms like dynamic collaborative filtering Thompson sampling (DCTS), upper confidence bound based on recommender system (UCB-RS), and dynamic -greedy algorithm (DEG) to optimize ad displays and enhance click-through rates (CTR). These algorithms must adapt to limited information and update strategies based on immediate feedback.This study employs an experimental comparison to assess the performance of the DCTS, UCB-RS, and DEG algorithms using the click-through rate prediction database from Kaggle. Five experimental sets under varied parameter settings were analyzed, employing the Receiver Operating Characteristic (ROC) curve, accuracy, and area under the curve (AUC) metrics.Results show that the DEG algorithm consistently outperforms the others, achieving higher AUC values and demonstrating robust sample identification capabilities. DEG also exhibits superior precision at high recall levels, showcasing its potential in dynamic advertising environments. Its dynamic adjustment strategy effectively balances exploration and exploitation, optimizing ad displays.The findings suggest that DEG's adaptability and stability make it particularly suitable for dynamic ad recommendation scenarios. Future research should focus on optimizing DEG's parameter settings and possibly integrating UCB-RS's exploration mechanisms to enhance performance and develop more effective strategies for advertising recommendation systems.

Epsilon-Greedy Thompson Sampling to Bayesian Optimization

Abstract Bayesian optimization (BO) has become a powerful tool for solving simulation-based engineering optimization problems thanks to its ability to integrate physical and mathematical understandings, consider uncertainty, and address the exploitation–exploration dilemma. Thompson sampling (TS) is a preferred solution for BO to handle the exploitation–exploration trade-off. While it prioritizes exploration by generating and minimizing random sample paths from probabilistic models—a fundamental ingredient of BO—TS weakly manages exploitation by gathering information about the true objective function after it obtains new observations. In this work, we improve the exploitation of TS by incorporating the e-greedy policy, a well-established selection strategy in reinforcement learning. We first delineate two extremes of TS, namely the generic TS and the sample-average TS. The former promotes exploration, while the latter favors exploitation. We then adopt the e-greedy policy to randomly switch between these two extremes. Small and large values of e govern exploitation and exploration, respectively. By minimizing two benchmark functions and solving an inverse problem of a steel cantilever beam, we empirically show that e-greedy TS equipped with an appropriate e is more robust than its two extremes, matching or outperforming the better of the generic TS and the sample-average TS.

Harnessing Multi-Armed Bandits for Smarter Digital Marketing Decisions

In the current digital age, software programs and applications provide convenience and added value in a number of ways. Multi-armed Bandit algorithms (MAB) are a prime example of this; In capital markets, they assist in the adaptive design of trading strategies that adjust to market shifts and investor behavior. In e-commerce, MAB helps optimize product recommendations by learning customer preferences in real-time. In SaaS, MAB can optimize user experience by personalizing services or pricing models based on user behavior, continuously adjusting to maximize engagement or revenue. In cloud engineering MAB are indispensable for optimizing resource allocation based on user demand. The objective of this paper is to demonstrate the adaptability of MAB algorithms through an examination of their applications, specifically focusing on the Digital Marketing domain and the insight they offer for optimal decision-making. The article highlights the usefulness of major MAB algorithms in digital advertising, e-commerce content suggestion, SaaS and strategic pricing by carefully examining Thompson Sampling, UCB, Restless Bandit, and Structured Bandit as key types of MAB algorithms. The study emphasizes how these algorithms adjust to fluctuating conditions, balance a trade-off between exploration and exploitation, and eventually improve marketing strategies. This article aims to improve knowledge of MAB algorithms and encourage more research in this promising area by providing a detailed analysis of their applications.

TS-based Mobility-aware Multi-hierarchical Caching Model with Vehicle Clustering and Content Popularity Prediction

This research is concerned with the fusion of artificial intelligence (AI) and machine learning within multi-hierarchical caching systems, specifically targeting vehicular and edge caching domains. This study introduces an innovative architecture harmonizing Thompson sampling learning-based caching policies with advanced vehicle clustering and content-popularity prediction methods (TS-MMCM). Simulations show substantial performance improvements and a big impact of the proposed approach on system efficiency in dynamic network environments. The proposal demonstrates a notable gain in cache hit rates and decreased latency levels, highlighting the potential of AI to improve caching techniques in dynamic network environments.

A Survey on Variants of Thompson Sampling

Thompson Sampling has become a prominent algorithmic approach in recent years. This review focuses on the evolution of TS and its variants, showing the innovative aspects of Neural Thompson Sampling (NeuralTS) and Meta-Thompson Sampling (Meta-TS), explaining the aggressive strategy used by Feel-Good Thompson Sampling (FGTS) and the introduction to Safe-LTS for Linear Thompson Sampling (LTS) problem. The survey first systematically review the literature, then examine the theoretical underpinnings, algorithmic frameworks and innovations of those TS variants, in the end provide our insights in future directions. In short, NeuralTS handles high-dimensional reward functions through deep learning integration, Meta-TS takes advantage of meta-learning for adapting to unknown prior distributions, FGTS applies an aggressive exploration strategy to handle pessimistic scenarios. In the end, this paper suggests that future research should emphasis on enhancing generalizability, bridging the gap between theory and practice, and improving adaptability to complex and dynamic environments.

Simple fixes that accommodate switching costs in multi-armed bandits

When switching costs are added to the multi-armed bandit (MAB) problem where the arms’ random reward distributions are previously unknown, usually quite different techniques than those for pure MAB are required. We find that two simple fixes on the existing upper-confidence-bound (UCB) policy can work well for MAB with switching costs (MAB-SC). Two cases should be distinguished. One is with positive-gap ambiguity where the performance gap between the leading and lagging arms is known to be at least some δ>0. For this, our fix is to erect barriers that discourage frivolous arm switchings. The other is with zero-gap ambiguity where absolutely nothing is known. We remedy this by forcing the same arms to be pulled in increasingly prolonged intervals. As usual, the effectivenesses of our fixes are measured by the worst average regrets over long time horizons T. When the barriers are fixed at δ/2, we can accomplish a ln(T)-sized regret bound for the positive-gap case. When intervals are such that n of them occupy n2 periods, we can achieve the best possible T1/2-sized regret bound for the zero-gap case. Other than UCB, these fixes can be applied to a learning while doing (LWD) heuristic to reach satisfactory results as well. While not yet with the best theoretical guarantees, the LWD-based policies have empirically outperformed those based on UCB and other known alternatives. Numerically competitive policies still include ones resulting from interval-based fixes on Thompson sampling (TS).

Human-in-the-Loop Trajectory Optimization Based on sEMG Biofeedback for Lower-Limb Exoskeleton.

Lower-limb exoskeletons (LLEs) can provide rehabilitation training and walking assistance for individuals with lower-limb dysfunction or those in need of functionality enhancement. Adapting and personalizing the LLEs is crucial for them to form an intelligent human-machine system (HMS). However, numerous LLEs lack thorough consideration of individual differences in motion planning, leading to subpar human performance. Prioritizing human physiological response is a critical objective of trajectory optimization for the HMS. This paper proposes a human-in-the-loop (HITL) motion planning method that utilizes surface electromyography signals as biofeedback for the HITL optimization. The proposed method combines offline trajectory optimization with HITL trajectory selection. Based on the derived hybrid dynamical model of the HMS, the offline trajectory is optimized using a direct collocation method, while HITL trajectory selection is based on Thompson sampling. The direct collocation method optimizes various gait trajectories and constructs a gait library according to the energy optimality law, taking into consideration dynamics and walking constraints. Subsequently, an optimal gait trajectory is selected for the wearer using Thompson sampling. The selected gait trajectory is then implemented on the LLE under a hybrid zero dynamics control strategy. Through the HITL optimization and control experiments, the effectiveness and superiority of the proposed method are verified.

Learning-based dynamic pricing strategy with pay-per-chapter mode for online publisher with case study of COL

We consider how to make dynamic pricing decision for Chinese Online (COL) at T time-points, an online publisher that allow authors to sell their ongoing book projects. Instead of paying for a book, readers pay for each chapter (pay-per-chapter mode) of the ongoing book project. This mode allows readers to pay for as many chapters as they want without taking the risk that the releasing of new chapters might be delayed or stopped. Despite of the dynamics of chapter-by-chapter released of COL products, the fixed pricing strategy (FPS) does not make fully use of the reading data generated by releasing chapters of the ongoing book. We propose a learning-based dynamic pricing strategy (LDPS) that exploits the newly information to maximize cumulative revenue for the publisher. The LDPS captures the ever changing features of readers. It employs the Thompson sampling method to balance the exploration of investigating different prices sufficiently with the exploitation of settling on the optimal price. Taking COL as a case study and implementing our strategy in the context of the aforementioned real-life data set, we show that LDPS outperform several classical strategies such as Greedy, Prior-Free TS and Prior-Given TS, and average revenue of LDPS is increased by 0.5 % average per time-point compared to the publisher's historical decisions. We also provide some management implications for the COL publisher by analyzing the pricing range of different genres of books and the choice of the exploration threshold parameter.

Mating with Multi-Armed Bandits: Reinforcement Learning Models of Human Mate Search.

Mate choice requires navigating an exploration-exploitation trade-off. Successful mate choice requires choosing partners who have preferred qualities; but time spent determining one partner's qualities could have been spent exploring for potentially superior alternatives. Here I argue that this dilemma can be modeled in a reinforcement learning framework as a multi-armed bandit problem. Moreover, using agent-based models and a sample of k = 522 real-world romantic dyads, I show that a reciprocity-weighted Thompson sampling algorithm performs well both in guiding mate search in noisy search environments and in reproducing the mate choices of real-world participants. These results provide a formal model of the understudied psychology of human mate search. They additionally offer implications for our understanding of person perception and mate choice.