Thompson Sampling Algorithm Research Articles

Performance Comparison of UCB, TS, and -Greedy TS Algorithms through Simulation of Multi-Armed Bandit Machine

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

A Strategy for Advertisement Placement based on the Multi-Armed Tiger Problem

The purpose of this study is to investigate the effectiveness of using Multi armed bandit model, which contains -greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms, to optimize online advertisement placement. Through simulating different types of ad placements using different algorithms and comparing them, this paper intends to demonstrate the feasibility of the Multi-Armed Robber model for the ad placement problem. The results show that the multi-armed bandit model can improve the ad click rate compared with the traditional ad placement strategy. Thompson sampling algorithm outperforms the -greedy algorithm and UCB algorithm in this paper's experiments, which can better balance exploration and application and reduce regret. The algorithm provides a more efficient method of allocating ad resources. These findings provide new insights into the field of digital marketing and may have an impact on the development of actual ad placement strategies.

Analysis of the Effectiveness of Multi-Armed Bandit Algorithms in Crop Pricing

This study explores the application of multi-armed bandit (MAB) algorithms in dynamic pricing of crops, with a focus on evaluating the adaptive upper confidence bound (asUCB) and Thompson Sampling (TS) algorithms. Through simulation experiments on historical data, the study analyzed the performance of these algorithms in fitting actual market price trends and responding to future price fluctuations. The results indicate that the asUCB algorithm performed particularly well in both the training dataset and simulation tests, demonstrating low mean squared error (MSE) and minimal cumulative regret, reflecting its rapid convergence and stable pricing capabilities. In contrast, although the TS algorithm initially performed slightly less effectively, it demonstrated unique advantages in dealing with market volatility due to its strong adaptability. This study demonstrates the potential application of MAB algorithms in dynamic pricing, providing valuable insights for pricing strategies in the agricultural product market.

Mating with Multi-Armed Bandits: Reinforcement Learning Models of Human Mate Search.

Mate choice requires navigating an exploration-exploitation trade-off. Successful mate choice requires choosing partners who have preferred qualities; but time spent determining one partner's qualities could have been spent exploring for potentially superior alternatives. Here I argue that this dilemma can be modeled in a reinforcement learning framework as a multi-armed bandit problem. Moreover, using agent-based models and a sample of k = 522 real-world romantic dyads, I show that a reciprocity-weighted Thompson sampling algorithm performs well both in guiding mate search in noisy search environments and in reproducing the mate choices of real-world participants. These results provide a formal model of the understudied psychology of human mate search. They additionally offer implications for our understanding of person perception and mate choice.

Optimizing Commercial Site Selection Using the Thompson Sampling Algorithm

Optimizing Commercial Site Selection Using the Thompson Sampling Algorithm

Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis.

We study stochastic linear contextual bandits (CB) where the agent observes a noisy version of the true context through a noise channel with unknown channel parameters. Our objective is to design an action policy that can "approximate" that of a Bayesian oracle that has access to the reward model and the noise channel parameter. We introduce a modified Thompson sampling algorithm and analyze its Bayesian cumulative regret with respect to the oracle action policy via information-theoretic tools. For Gaussian bandits with Gaussian context noise, our information-theoretic analysis shows that under certain conditions on the prior variance, the Bayesian cumulative regret scales as O˜(mT), where m is the dimension of the feature vector and T is the time horizon. We also consider the problem setting where the agent observes the true context with some delay after receiving the reward, and show that delayed true contexts lead to lower regret. Finally, we empirically demonstrate the performance of the proposed algorithms against baselines.

Optimizing video click-through rates with bandit algorithms

In recent years, videos have increasingly influenced public perception, making video platforms a focal point of digital consumption. One critical challenge for platform operators is identifying videos that resonate most with users, as user ratings directly reflect viewer preferences and experiences. This study explores the use of bandit algorithms to predict and strategize the overall ratings of various anime videos on the Bilibili platform. Bandit algorithms, a subset of the multi-armed bandit model, dynamically adjust selection strategies based on prior feedback to maximize cumulative rewards. Our empirical research assessed multiple gambling algorithms, including the -greedy, Upper Confidence Bound (UCB), Explore-then-Commit (ETC), and Thompson Sampling (TS) algorithms. The findings indicate that the Thompson Sampling algorithm, in particular, achieved the lowest cumulative regret in selecting optimal videos on the Bilibili platform, showcasing its superior performance. This study highlights the potential of bandit algorithms to enhance video selection processes, ensuring that platforms can effectively cater to user preferences and enhance viewer satisfaction.

Analyzing the strengths and weaknesses of diverse algorithms for solving Multi-Armed Bandit problems using Python

With the rapid advancement of science and technology, the internet has become an integral part of daily life, revolutionizing how people access information and make decisions. In this context, algorithms play a pivotal role in helping individuals make informed choices tailored to their preferences across various domains. Utilizing the MovieLens dataset (https://grouplens.org/datasets/movielens/1m/), which contains a rich compilation of movie ratings and metadata, this study conducts a thorough analysis using Python to assess the performance of four distinct algorithms: Explore-then-Commit (ETC), Upper Confidence Bound (UCB), Thompson Sampling (TS), and Epsilon-Greedy. The comparison reveals that the ETC algorithm excels in applications such as online advertising recommendation and autonomous driving. The UCB algorithm proves more advantageous in financial analysis, where risk management is critical. The TS algorithm is particularly effective in short video recommendation systems, while the Epsilon-Greedy algorithm is well-suited for balancing exploration with reward. Overall, the results indicate that the TS algorithm outperforms the others in general efficacy.

Ad Optimization Via Machine Learning: A Focus on Upper Confidence Bound and Thompson Sampling Algorithms

The objective of this project is to improve the effectiveness and efficiency of advertising on various platforms by utilizing advanced algorithms, namely the Upper Confidence Bound and Thompson Sampling Algorithm. The project aims to find a balance between exploring new advertising strategies and exploiting proven high-performing approaches. By implementing these bandit algorithms, the project aims to dynamically optimize ad placements, formats, and targeting to maximize user engagement and ad revenue. The methodology involves an iterative process of data collection, analysis, and adaptation. The initial phases include defining project objectives, understanding the target audience, and reviewing the current ad strategy. The Upper Confidence Bound algorithm enables intelligent decision-making by assigning confidence bounds to different ad strategies, allowing for efficient exploration and exploitation. On the other hand, the Thompson Sampling algorithm, rooted in Bayesian principles, dynamically adapts based on observed outcomes, striking a balance between exploration and exploitation through probabilistic reasoning. In summary, this Ads Optimization Project utilizes the power of the Upper Confidence Bound and Thompson Sampling algorithms to create a data-driven, adaptive, and user-centric approach to advertising. The ultimate goal is to achieve optimal user engagement and ad revenue.

Utilizing Multi-Armed Bandit Algorithms for Advertising: An In-Depth Case Study on an Online Retail Platform's Advertising Campaign

In recent years, the advertising sector has increasingly embraced Multi-armed Bandit Algorithms (MAB) for their versatile applications. This article delves into four stochastic MAB algorithms: the Explore-Then-Commit (ETC), the Upper Confidence Bound (UCB), the Thompson-Sampling (TS), and their respective variants, applied in an online shopping platform's advertising campaign to optimize click-through rates. Our findings indicate that each of these algorithms successfully identifies the most effective advertisement. Both the UCB and TS algorithms demonstrate logarithmic regret relative to the time horizon. The ETC algorithm exhibits a delayed onset of logarithmic regret but surprisingly holds up well against the UCB algorithms. Notably, the TS algorithm class outshines the UCB class due to its inherent randomness, offering a more robust performance. Furthermore, two variants, the UCB algorithm with the Mini-max Optimal Strategy in the Stochastic case (MOSS) and the paralleled TS algorithm, show promising results. They not only maintain logarithmic regret but also enhance efficiency, indicating further potential in the UCB and TS algorithm classes for advanced applications.

Personalized Dynamic Pricing Based on Improved Thompson Sampling

This study investigates personalized pricing with demand learning. We first encode consumer-personalized feature information into high-dimensional vectors, then establish the relationship between this feature vector and product demand using a logit model, and finally learn demand parameters through historical transaction data. To address the balance between learning and revenue, we introduce the Thompson Sampling algorithm. Considering the difficulty of Bayesian inference in Thompson Sampling owing to high-dimensional feature vectors, we improve the basic Thompson Sampling by approximating the likelihood function of the logit model with the Pólya-Gamma (PG) distribution and by proposing a Thompson Sampling algorithm based on the PG distribution. To validate the proposed algorithm’s effectiveness, we conduct experiments using both simulated data and real loan data provided by the Columbia University Revenue Management Center. The study results demonstrate that the Thompson Sampling algorithm based on the PG distribution proposed outperforms traditional Laplace approximation methods regarding convergence speed and regret value in both real and simulated data experiments. The real-time personalized pricing algorithm developed here not only enriches the theoretical research of personalized dynamic pricing, but also provides a theoretical basis and guidance for enterprises to implement personalized pricing.

Finite-Time Frequentist Regret Bounds of Multi-Agent Thompson Sampling on Sparse Hypergraphs

We study the multi-agent multi-armed bandit (MAMAB) problem, where agents are factored into overlapping groups. Each group represents a hyperedge, forming a hypergraph over the agents. At each round of interaction, the learner pulls a joint arm (composed of individual arms for each agent) and receives a reward according to the hypergraph structure. Specifically, we assume there is a local reward for each hyperedge, and the reward of the joint arm is the sum of these local rewards. Previous work introduced the multi-agent Thompson sampling (MATS) algorithm and derived a Bayesian regret bound. However, it remains an open problem how to derive a frequentist regret bound for Thompson sampling in this multi-agent setting. To address these issues, we propose an efficient variant of MATS, the epsilon-exploring Multi-Agent Thompson Sampling (eps-MATS) algorithm, which performs MATS exploration with probability epsilon while adopts a greedy policy otherwise. We prove that eps-MATS achieves a worst-case frequentist regret bound that is sublinear in both the time horizon and the local arm size. We also derive a lower bound for this setting, which implies our frequentist regret upper bound is optimal up to constant and logarithm terms, when the hypergraph is sufficiently sparse. Thorough experiments on standard MAMAB problems demonstrate the superior performance and the improved computational efficiency of eps-MATS compared with existing algorithms in the same setting.

Investigation of progress and application related to Multi-Armed Bandit algorithms

This paper discusses four Multi-armed Bandit algorithms: Explore-then-Commit (ETC), Epsilon-Greedy, Upper Confidence Bound (UCB), and Thompson Sampling algorithm. ETC algorithm aims to spend the majority of rounds on the best arm, but it can lead to a suboptimal outcome if the environment changes rapidly. The Epsilon-Greedy algorithm is designed to explore and exploit simultaneously, while it often tries sub-optimal arm even after the algorithm finds the best arm. Thus, the Epsilon-Greedy algorithm performs well when the environment continuously changes. UCB algorithm is one of the most used Multi-armed Bandit algorithms because it can rapidly narrow the potential optimal decisions in a wide range of scenarios; however, the algorithm can be influenced by some specific pattern of reward distribution or noise presenting in the environment. Thompson Sampling algorithm is also one of the most common algorithms in the Multi-armed Bandit algorithm due to its simplicity, effectiveness, and adaptability to various reward distributions. The Thompson Sampling algorithm performs well in multiple scenarios because it explores and exploits simultaneously, but its variance is greater than the three algorithms mentioned above. Today, Multi-armed bandit algorithms are widely used in advertisement, health care, and website and app optimization. Finally, the Multi-armed Bandit algorithms are rapidly replacing the traditional algorithms; in the future, the advanced Multi-armed Bandit algorithm, contextual Multi-armed Bandit algorithm, will gradually replace the old one.

An investigation of progress related to stochastic stationary bandit algorithms

The Multi-armed Bandit algorithm stands as a consequential tool for informed decision-making, distinct from reliance on intuitive selections, given its systematic proclivity to meticulously assess accessible alternatives with the intent of discerning the most auspicious outcome. Amid the repertoire of algorithmic variations, the Stochastic Stationary Bandit algorithm assumes a foundational and enduring role, finding versatile application across diverse domains, including but not limited to digital advertising, price optimization, and recommendation systems. With these considerations in view, the present study embarks upon a comprehensive scrutiny of this subject matter. This paper reviews on the Explore-Then-Commit algorithm, Upper Confidence Bound algorithm, and Thompson Sampling algorithm by explaining, comparing their formulation, features, and expected results. Explore-Then-Commit algorithm has distinct phase to explore all the choices uniformly. Upper Confidence Bound algorithm make decisions by calculate an upper confidence index which is an overestimate for each choice. Thompson Sampling algorithm depends on randomness to make choices. Explore-Then-Commit algorithm faces the problem of when to explore and when to stop. Upper Confidence Bound algorithm and Thompson Sampling algorithm solve this problem by avoid certain phases. Multi-armed Bandit algorithm could deal with the process of displaying items of potential interest to users in a recommendation system, the delivery of resources in resource allocation, or the way to maximize revenue in a business.

The investigation and application of Muti-Armed algorithms in recommendation systems

In the contemporary Internet recommendation systems in various fields, muti-armed algorithms will show high accuracy and practicability. Internet users can always get abundant positive feedback through the recommendation system via utilizing these algorithms. In this paper, there are three most typical muti-armed algorithms as examples are provided. Explore-Then-Commit (ETC) algorithm is the first to mentioned. The physical meaning of this algorithm is that in the exploration stage, the action will be selected in a certain order, and after a certain number of rounds, the action with the largest average reward will be directly selected. Moreover, Upper Confidence Bond (UCB) algorithm is also a type of pivotal tool. The main function of UCB algorithm is to make the selection by calculating the upper bound of each arm confidence interval instead of the expected reward of the slot machine. Thus, it is an optimistic algorithm. First, select each arm in random order, then calculate the value of the upper bound of each arm confidence interval, and finally, select the arm with the largest value. The last mentioned Thompson Sampling (TS) Algorithm take Bayesian optimization as the theoretical basis. This algorithm takes out the candidate parameters, generates a random number, and selects the maximum value for input. This paper also introduced two fields related to the topic of the application of muti-armed algorithms in the recommendation systems in modern fields. News recommendation algorithms and personalized recommendation systems can be more comprehensive and representative to illustrate the practicality of these algorithms. Therefore, muti-armed algorithms reflect the importance in the recommendation fields via the balance of exploration and exploitation.

Optimization strategy for the pull arm function in the Multi-Armed Bandit algorithm

Within the domain of Multi-Armed Bandit (MAB) quandaries, the efficacy exhibited by the arm-pulling function stands as a determinant factor governing algorithmic performance. This study undertakes a systematic exploration of enhancing this function's optimization vis--vis the MovieLens dataset, with a specific focus on three prominent MAB algorithms: Upper Confidence Bound (UCB), Explore-Then-Commit (ETC), and Thompson Sampling (TS). The present exposition introduces an innovative methodology tailored to the Python 3.11 computational environment. Central to this approach is the pre-calculation of arm-specific reward frequencies, obviating the necessity for dataset look-up and thereby streamlining the reward selection process. Subsequently, a binomial distribution is fitted to these frequencies, ensuring rapid and accurate reward sampling. When benchmarked, the optimized version of the pull arm function showcased a remarkable reduction in execution time compared to their initial counterparts, all while maintaining algorithmic accuracy. The Thompson Sampling algorithm notably outperformed the others, exhibiting the lowest cumulative regret across diverse trial sizes. This research emphasizes the significance of refining core components in MAB algorithms and paves the way for practical enhancements in real-world applications.

Multiarmed Bandit Algorithms on Zynq System-on-Chip: Go Frequentist or Bayesian?

Multiarmed Bandit (MAB) algorithms identify the best arm among multiple arms via exploration-exploitation trade-off without prior knowledge of arm statistics. Their usefulness in wireless radio, Internet of Things (IoT), and robotics demand deployment on edge devices, and hence, a mapping on system-on-chip (SoC) is desired. Theoretically, the Bayesian-approach-based Thompson sampling (TS) algorithm offers better performance than the frequentist-approach-based upper confidence bound (UCB) algorithm. However, TS is not synthesizable due to Beta function. We address this problem by approximating it via a pseudorandom number generator (PRNG)-based architecture and efficiently realize the TS algorithm on Zynq SoC. In practice, the type of arms distribution (e.g., Bernoulli, Gaussian) is unknown, and hence, a single algorithm may not be optimal. We propose a reconfigurable and intelligent MAB (RI-MAB) framework. Here, intelligence enables the identification of appropriate MAB algorithms in an unknown environment, and reconfigurability allows on-the-fly switching between algorithms on the SoC. This eliminates the need for parallel implementation of algorithms resulting in huge savings in resources and power consumption. We analyze the functional correctness, area, power, and execution time of the proposed and existing architectures for various arm distributions, word length, and hardware-software codesign approaches. We demonstrate the superiority of the RI-MAB algorithm and its architecture over the TS and UCB algorithms.

Multi-armed bandits for performance marketing

AbstractThis paper deals with the problem of optimising bids and budgets of a set of digital advertising campaigns. We improve on the current state of the art by introducing support for multi-ad group marketing campaigns and developing a highly data efficient parametric contextual bandit. The bandit, which exploits domain knowledge to reduce the exploration space, is shown to be effective under the following settings; few clicks and/or small conversion rate, short horizon scenarios, rapidly changing markets and low budget. Furthermore, a bootstrapped Thompson sampling algorithm is adapted to fit the parametric bandit. Extensive numerical experiments, performed on synthetic and real-world data, show that, on average, the parametric bandit gains more conversions than state-of-the-art bandits. Gains in performance are particularly high when an optimisation algorithm is needed the most, i.e. with tight budget or many ad groups, though gains are present also in the case of a single-ad group.

Application of multi-armed bandits to dose-finding clinical designs

Multi-armed bandits are very simple and powerful methods to determine actions to maximize a reward in a limited number of trials. An early phase in dose-finding clinical trials needs to identify the maximum tolerated dose among multiple doses by repeating the dose-assignment. We consider applying the superior selection performance of multi-armed bandits to dose-finding clinical designs. Among the multi-armed bandits, we first consider the use of Thompson sampling which determines actions based on random samples from a posterior distribution. In the small sample size, as shown in dose-finding trials, because the tails of posterior distribution are heavier and random samples are too much variability, we also consider an application of regularized Thompson sampling and greedy algorithm. The greedy algorithm determines a dose based on a posterior mean. In addition, we also propose a method to determine a dose based on a posterior mode. We evaluate the performance of our proposed designs for nine scenarios via simulation studies.

Model-Based Thompson Sampling for Frequency and Rate Selection in Underwater Acoustic Communications

Due to the harsh propagation environment, limited bandwidth, and constrained battery life, transmission efficiency is a crucial issue for underwater acoustic (UWA) communications. This paper studies the link adaptation problem of a single UWA link by jointly selecting the transmission frequency and data rate. Since the current UWA channel lacks a universal model, we formulate this joint selection problem as a model-based stochastic multi-armed bandit (SMAB) problem. Thereafter, we propose three algorithms to solve this model-based SMAB problem under the settings of the stationary channel, non-stationary channel, and large arm (i.e., frequency and rate pair) space. For the stationary channel, we propose a unimodal objective-based Thompson sampling (UO-TS) algorithm by exploiting the unimodal feature of the objective function. For the non-stationary channel, we put forth a hybrid change detection UO-TS (HCD-UO-TS) algorithm based on the features of the unimodal objective function and non-stationary channel. For the large arm space, we propose an iterative boundary-shrinking TS (IBS-TS) algorithm by using the logistic regression-based arm classification model. These algorithms are all data-driven and have low complexity and a fast convergence rate. In addition, we derive an upper regret bound for the UO-TS algorithm. Numerical results show that the proposed algorithms outperform the state-of-the-art bandit algorithms and are not sensitive to the arm space.