Thompson Sampling Research Articles

Multi-armed bandits, Thomson sampling and unsupervised machine learning in phylogenetic graph search.

A phylogenetic graph search relies on a large number of highly parameterized search procedures (e.g. branch-swapping, perturbation, simulated annealing, genetic algorithm). These procedures vary in effectiveness over datasets and at alternative points in analytical pipelines. The multi-armed bandit problem is applied to phylogenetic graph searching to more effectively utilize these procedures. Thompson sampling is applied to a collection of search and optimization "bandits" to favour productive search strategies over those that are less successful. This adaptive random sampling strategy is shown to be more effective in producing heuristically optimal phylogenetic graphs and more time efficient than existing uniform probability randomized search strategies. The strategy acts as a form of unsupervised machine learning that can be applied to a diversity of phylogenetic datasets without prior knowledge of their properties.

High-throughput screening of catalytically active inclusion bodies using laboratory automation and Bayesian optimization

BackgroundIn recent years, the production of inclusion bodies that retain substantial catalytic activity was demonstrated. These catalytically active inclusion bodies (CatIBs) are formed by genetic fusion of an aggregation-inducing tag to a gene of interest via short linker polypeptides. The resulting CatIBs are known for their easy and cost-efficient production, recyclability as well as their improved stability. Recent studies have outlined the cooperative effects of linker and aggregation-inducing tag on CatIB activities. However, no a priori prediction is possible so far to indicate the best combination thereof. Consequently, extensive screening is required to find the best performing CatIB variant.ResultsIn this work, a semi-automated cloning workflow was implemented and used for fast generation of 63 CatIB variants with glucose dehydrogenase of Bacillus subtilis (BsGDH). Furthermore, the variant BsGDH-PT-CBDCell was used to develop, optimize and validate an automated CatIB screening workflow, enhancing the analysis of many CatIB candidates in parallel. Compared to previous studies with CatIBs, important optimization steps include the exclusion of plate position effects in the BioLector by changing the cultivation temperature. For the overall workflow including strain construction, the manual workload could be reduced from 59 to 7 h for 48 variants (88%). After demonstration of high reproducibility with 1.9% relative standard deviation across 42 biological replicates, the workflow was performed in combination with a Bayesian process model and Thompson sampling. While the process model is crucial to derive key performance indicators of CatIBs, Thompson sampling serves as a strategy to balance exploitation and exploration in screening procedures. Our methodology allowed analysis of 63 BsGDH-CatIB variants within only three batch experiments. Because of the high likelihood of TDoT-PT-BsGDH being the best CatIB performer, it was selected in 50 biological replicates during the three screening rounds, much more than other, low-performing variants.ConclusionsAt the current state of knowledge, every new enzyme requires screening for different linker/aggregation-inducing tag combinations. For this purpose, the presented CatIB toolbox facilitates fast and simplified construction and screening procedures. The methodology thus assists in finding the best CatIB producer from large libraries in short time, rendering possible automated Design-Build-Test-Learn cycles to generate structure/function learnings.

Survey of dynamic pricing based on Multi-Armed Bandit algorithms

Dynamic pricing seeks to determine the most optimal selling price for a product or service, taking into account factors like limited supply and uncertain demand. This study aims to provide a comprehensive exploration of dynamic pricing using the multi-armed bandit problem framework in various contexts. The investigation highlights the prevalence of Thompson sampling in dynamic pricing scenarios with a Bayesian backdrop, where the seller possesses prior knowledge of demand functions. On the other hand, in non-Bayesian situations, the Upper Confidence Bound (UCB) algorithm family gains traction due to their favorable regret bounds. As markets often exhibit temporal fluctuations, the domain of non-stationary multi-armed bandits within dynamic pricing emerges as crucial. Future research directions include enhancing traditional multi-armed bandit algorithms to suit online learning settings, especially those involving dynamic reward distributions. Additionally, merging prior insights into demand functions with contextual multi-armed bandit approaches holds promise for advancing dynamic pricing strategies. In conclusion, this study sheds light on dynamic pricing through the lens of multi-armed bandit problems, offering insights and pathways for further exploration.

Investigation of progress and application related to Multi-Armed Bandit algorithms

This paper discusses four Multi-armed Bandit algorithms: Explore-then-Commit (ETC), Epsilon-Greedy, Upper Confidence Bound (UCB), and Thompson Sampling algorithm. ETC algorithm aims to spend the majority of rounds on the best arm, but it can lead to a suboptimal outcome if the environment changes rapidly. The Epsilon-Greedy algorithm is designed to explore and exploit simultaneously, while it often tries sub-optimal arm even after the algorithm finds the best arm. Thus, the Epsilon-Greedy algorithm performs well when the environment continuously changes. UCB algorithm is one of the most used Multi-armed Bandit algorithms because it can rapidly narrow the potential optimal decisions in a wide range of scenarios; however, the algorithm can be influenced by some specific pattern of reward distribution or noise presenting in the environment. Thompson Sampling algorithm is also one of the most common algorithms in the Multi-armed Bandit algorithm due to its simplicity, effectiveness, and adaptability to various reward distributions. The Thompson Sampling algorithm performs well in multiple scenarios because it explores and exploits simultaneously, but its variance is greater than the three algorithms mentioned above. Today, Multi-armed bandit algorithms are widely used in advertisement, health care, and website and app optimization. Finally, the Multi-armed Bandit algorithms are rapidly replacing the traditional algorithms; in the future, the advanced Multi-armed Bandit algorithm, contextual Multi-armed Bandit algorithm, will gradually replace the old one.

Thompson Sampling─An Efficient Method for Searching Ultralarge Synthesis on Demand Databases.

Over the last five years, virtual screening of ultralarge synthesis on-demand libraries has emerged as a powerful tool for hit identification in drug discovery programs. As these libraries have grown to tens of billions of molecules, we have reached a point where it is no longer cost-effective to screen every molecule virtually. To address these challenges, several groups have developed heuristic search methods to rapidly identify the best molecules on a virtual screen. This article describes the application of Thompson sampling (TS), an active learning approach that streamlines the virtual screening of large combinatorial libraries by performing a probabilistic search in the reagent space, thereby never requiring the full enumeration of the library. TS is a general technique that can be applied to various virtual screening modalities, including 2D and 3D similarity search, docking, and application of machine-learning models. In an illustrative example, we show that TS can identify more than half of the top 100 molecules from a docking-based virtual screen of 335 million molecules by evaluating 1% of the data set.

Exploring Multi-Armed Bandit algorithms: Performance analysis in dynamic environments

The Multi-armed Bandit algorithm, a proficient solver of the exploration-and-exploitation trade-off predicament, furnishes businesses with a robust tool for resource allocation that predominantly aligns with customer preferences. However, varying Multi-armed Bandit algorithm types exhibit dissimilar performance characteristics based on contextual variations. Hence, a series of experiments is imperative, involving alterations to input values across distinct algorithms. Within this study, three specific algorithms were applied, Explore-then-commit (ETC), Upper Confident Bound (UCB) and its asymptotically optimal variant, and Thompson Sampling (TS), to the extensively utilized MovieLens dataset. This application aimed to gauge their effectiveness comprehensively. The algorithms were translated into executable code, and their performance was visually depicted through multiple figures. Through cumulative regret tracking within defined conditions, algorithmic performance was scrutinized, laying the groundwork for subsequent parameter-based comparisons. A dedicated experimentation framework was devised to evaluate the robustness of each algorithm, involving deliberate parameter adjustments and tailored experiments to elucidate distinct performance nuances. The ensuing graphical depictions distinctly illustrated Thompson Sampling's persistent minimal regrets across most scenarios. UCB algorithms displayed steadfast stability. ETC manifested excellent performance with a low number of runs but escalate significantly along the number of runs growing. It also warranting constraints on exploratory phases to mitigate regrets. This investigation underscores the efficacy of Multi-armed Bandit algorithms while elucidating their nuanced behaviors within diverse contextual contingencies.

Optimization strategy for the pull arm function in the Multi-Armed Bandit algorithm

Within the domain of Multi-Armed Bandit (MAB) quandaries, the efficacy exhibited by the arm-pulling function stands as a determinant factor governing algorithmic performance. This study undertakes a systematic exploration of enhancing this function's optimization vis--vis the MovieLens dataset, with a specific focus on three prominent MAB algorithms: Upper Confidence Bound (UCB), Explore-Then-Commit (ETC), and Thompson Sampling (TS). The present exposition introduces an innovative methodology tailored to the Python 3.11 computational environment. Central to this approach is the pre-calculation of arm-specific reward frequencies, obviating the necessity for dataset look-up and thereby streamlining the reward selection process. Subsequently, a binomial distribution is fitted to these frequencies, ensuring rapid and accurate reward sampling. When benchmarked, the optimized version of the pull arm function showcased a remarkable reduction in execution time compared to their initial counterparts, all while maintaining algorithmic accuracy. The Thompson Sampling algorithm notably outperformed the others, exhibiting the lowest cumulative regret across diverse trial sizes. This research emphasizes the significance of refining core components in MAB algorithms and paves the way for practical enhancements in real-world applications.

The investigation and application of Muti-Armed algorithms in recommendation systems

In the contemporary Internet recommendation systems in various fields, muti-armed algorithms will show high accuracy and practicability. Internet users can always get abundant positive feedback through the recommendation system via utilizing these algorithms. In this paper, there are three most typical muti-armed algorithms as examples are provided. Explore-Then-Commit (ETC) algorithm is the first to mentioned. The physical meaning of this algorithm is that in the exploration stage, the action will be selected in a certain order, and after a certain number of rounds, the action with the largest average reward will be directly selected. Moreover, Upper Confidence Bond (UCB) algorithm is also a type of pivotal tool. The main function of UCB algorithm is to make the selection by calculating the upper bound of each arm confidence interval instead of the expected reward of the slot machine. Thus, it is an optimistic algorithm. First, select each arm in random order, then calculate the value of the upper bound of each arm confidence interval, and finally, select the arm with the largest value. The last mentioned Thompson Sampling (TS) Algorithm take Bayesian optimization as the theoretical basis. This algorithm takes out the candidate parameters, generates a random number, and selects the maximum value for input. This paper also introduced two fields related to the topic of the application of muti-armed algorithms in the recommendation systems in modern fields. News recommendation algorithms and personalized recommendation systems can be more comprehensive and representative to illustrate the practicality of these algorithms. Therefore, muti-armed algorithms reflect the importance in the recommendation fields via the balance of exploration and exploitation.

An investigation of progress related to stochastic stationary bandit algorithms

The Multi-armed Bandit algorithm stands as a consequential tool for informed decision-making, distinct from reliance on intuitive selections, given its systematic proclivity to meticulously assess accessible alternatives with the intent of discerning the most auspicious outcome. Amid the repertoire of algorithmic variations, the Stochastic Stationary Bandit algorithm assumes a foundational and enduring role, finding versatile application across diverse domains, including but not limited to digital advertising, price optimization, and recommendation systems. With these considerations in view, the present study embarks upon a comprehensive scrutiny of this subject matter. This paper reviews on the Explore-Then-Commit algorithm, Upper Confidence Bound algorithm, and Thompson Sampling algorithm by explaining, comparing their formulation, features, and expected results. Explore-Then-Commit algorithm has distinct phase to explore all the choices uniformly. Upper Confidence Bound algorithm make decisions by calculate an upper confidence index which is an overestimate for each choice. Thompson Sampling algorithm depends on randomness to make choices. Explore-Then-Commit algorithm faces the problem of when to explore and when to stop. Upper Confidence Bound algorithm and Thompson Sampling algorithm solve this problem by avoid certain phases. Multi-armed Bandit algorithm could deal with the process of displaying items of potential interest to users in a recommendation system, the delivery of resources in resource allocation, or the way to maximize revenue in a business.

Multiarmed Bandit Algorithms on Zynq System-on-Chip: Go Frequentist or Bayesian?

Multiarmed Bandit (MAB) algorithms identify the best arm among multiple arms via exploration-exploitation trade-off without prior knowledge of arm statistics. Their usefulness in wireless radio, Internet of Things (IoT), and robotics demand deployment on edge devices, and hence, a mapping on system-on-chip (SoC) is desired. Theoretically, the Bayesian-approach-based Thompson sampling (TS) algorithm offers better performance than the frequentist-approach-based upper confidence bound (UCB) algorithm. However, TS is not synthesizable due to Beta function. We address this problem by approximating it via a pseudorandom number generator (PRNG)-based architecture and efficiently realize the TS algorithm on Zynq SoC. In practice, the type of arms distribution (e.g., Bernoulli, Gaussian) is unknown, and hence, a single algorithm may not be optimal. We propose a reconfigurable and intelligent MAB (RI-MAB) framework. Here, intelligence enables the identification of appropriate MAB algorithms in an unknown environment, and reconfigurability allows on-the-fly switching between algorithms on the SoC. This eliminates the need for parallel implementation of algorithms resulting in huge savings in resources and power consumption. We analyze the functional correctness, area, power, and execution time of the proposed and existing architectures for various arm distributions, word length, and hardware-software codesign approaches. We demonstrate the superiority of the RI-MAB algorithm and its architecture over the TS and UCB algorithms.

Multi-armed bandits for performance marketing

AbstractThis paper deals with the problem of optimising bids and budgets of a set of digital advertising campaigns. We improve on the current state of the art by introducing support for multi-ad group marketing campaigns and developing a highly data efficient parametric contextual bandit. The bandit, which exploits domain knowledge to reduce the exploration space, is shown to be effective under the following settings; few clicks and/or small conversion rate, short horizon scenarios, rapidly changing markets and low budget. Furthermore, a bootstrapped Thompson sampling algorithm is adapted to fit the parametric bandit. Extensive numerical experiments, performed on synthetic and real-world data, show that, on average, the parametric bandit gains more conversions than state-of-the-art bandits. Gains in performance are particularly high when an optimisation algorithm is needed the most, i.e. with tight budget or many ad groups, though gains are present also in the case of a single-ad group.

Bandit algorithms: A comprehensive review and their dynamic selection from a portfolio for multicriteria top-k recommendation

This paper discusses the use of portfolio approaches based on bandit algorithms to optimize multicriteria decision-making in recommender systems (accuracy and diversity). While previous research has primarily focused on single-item recommendations, this study extends the research to consider the recommendation of several items per iteration. Two methods, Multiple-play Gorthaur and Budgeted-Gorthaur, are proposed to solve the algorithm selection problem and their performances on real-world datasets are compared. Both methods provide a generalization of the Gorthaur method, which enables it to operate with any Multi-Armed Bandit (MAB) and Contextual Multi-Armed Bandit (CMAB) algorithm as meta-algorithm in a multi-item recommendation scenario. For Multiple-play Gorthaur, an empirical evaluation shows that the use of Thompson Sampling for algorithm selection (Gorthaur-TS) yields better results than the original EXP3 method (Gorthaur-EXP3) and the exclusive use of the optimal algorithm in the portfolio in contextual recommendation problems. Additionally, the paper includes a theoretical regret analysis based on the TS sketch proof applied for this variant of the method. Concerning Budgeted-Gorthaur, experiments show that it allows more flexibility to achieve a suitable trade-off between criteria and a broader coverage of the Pareto set of solutions, overcoming a natural limit of “a-priori” methods. Finally, this paper provides a detailed review, including pseudocodes and theoretical bounds, for all the fundamental MAB and CMAB algorithms used in this study.

Joint user association and pairing in multi-uav-assisted noma networks: a decaying-epsilon thompson sampling framework

Joint user association and pairing in multi-uav-assisted noma networks: a decaying-epsilon thompson sampling framework

Consensus-Based Thompson Sampling for Stochastic Multiarmed Bandits

Consensus-Based Thompson Sampling for Stochastic Multiarmed Bandits

MFTTS: A Mean-Field Transfer Thompson Sampling Approach for Distributed Power Allocation in Unsourced Multiple Access

MFTTS: A Mean-Field Transfer Thompson Sampling Approach for Distributed Power Allocation in Unsourced Multiple Access

A novel two-stage dynamic pricing model for logistics planning using an exploration–exploitation framework: A multi-armed bandit problem

Dynamic pricing is utilized as a tool for determining pricing strategies in different sectors, such as the food industries, airline companies, and transportation services. Dynamic pricing aims to maximize revenue based on learning the price-demand relationship. It is imperative to perform price change testing within a short time frame since rapid changes can negatively impact the market. Therefore, there are two conflicting objectives: maximizing revenue and learning the demand function, which is called the exploration–exploitation concept. In this study, we propose an exploration–exploitation framework, focused on the domestic delivery service of Tipax Express Pars Company (TEPC), to determine the optimal selling price. In the exploration process, the initial demand function is proposed based on previous sales data using weighted linear regression. Then, the selling price and delivery lead-time parameters are incorporated into the demand function hypotheses. After that, the value of the optimal demand function parameters is determined based on the Z-Price Changes Algorithm (ZPCA). In the exploitation process, price optimization is designed as a Multi-armed Bandit (MAB) problem and solved using (i) Upper Confidence Bound Method (UCBM) (i.e., deterministic approach), and (ii) Thompson Sampling Method (TSM) (i.e., stochastic approach). The proposed exploration–exploitation framework is evaluated in 361 days, of which the exploration and exploitation durations are 31 and 330 days, respectively. In this study, 10,000 demand functions are investigated, which leads to the complexity of the developed model. The cross-validation approach, based on Mean Squared Error (MSE), is developed to decrease the complexity of the proposed model. A simulation study of 500 simulations demonstrates that TSM is more useful than UCBM because its solution is more robust. Furthermore, the results demonstrate that TEPC can gain maximum total expected revenue, maximum total number of selections during the exploitation stage, and maximum actual demand using a gaining market share strategy.

Gaussian process classification bandits

Classification bandits are multi-armed bandit problems whose task is to classify a given set of arms into either positive or negative class depending on whether the rate of the arms with the expected reward of at least h is not less than w for given thresholds h and w. We study a special classification bandit problem in which arms correspond to points x in d-dimensional real space with expected rewards f(x) which are generated according to a Gaussian process prior. We develop a framework algorithm for the problem using various arm selection policies and propose policies called FCB (Farthest Confidence Bound) and FTSV (Farthest Thompson Sampling Value). We show a smaller sample complexity upper bound for FCB than that for the existing algorithm of the level set estimation, in which whether f(x) is at least h or not must be decided for every arm’s x. Arm selection policies depending on an estimated rate of arms with mean rewards of at least h are also proposed and shown to improve empirical sample complexity. According to our experimental results, the rate-estimation versions of FCB and FTSV, together with that of the popular active learning policy which selects the point with the maximum variance, outperform other policies for synthetic functions, and the rate-estimation version of FTSV is also the best performer for our real-world dataset.

Multinomial Thompson sampling for rating scales and prior considerations for calibrating uncertainty

Bandit algorithms such as Thompson sampling (TS) have been put forth for decades as useful tools for conducting adaptively-randomised experiments. By skewing the allocation toward superior arms, they can substantially improve particular outcomes of interest for both participants and investigators. For example, they may use participants’ ratings for continuously optimising their experience with a program. However, most of the bandit and TS variants are based on either binary or continuous outcome models, leading to suboptimal performances in rating scale data. Guided by behavioural experiments we conducted online, we address this problem by introducing Multinomial-TS for rating scales. After assessing its improved empirical performance in unique optimal arm scenarios, we explore potential considerations (including prior’s role) for calibrating uncertainty and balancing arm allocation in scenarios with no unique optimal arms.

Weak Signal Asymptotics for Sequentially Randomized Experiments

We use the lens of weak signal asymptotics to study a class of sequentially randomized experiments, including those that arise in solving multiarmed bandit problems. In an experiment with n time steps, we let the mean reward gaps between actions scale to the order [Formula: see text] to preserve the difficulty of the learning task as n grows. In this regime, we show that the sample paths of a class of sequentially randomized experiments—adapted to this scaling regime and with arm selection probabilities that vary continuously with state—converge weakly to a diffusion limit, given as the solution to a stochastic differential equation. The diffusion limit enables us to derive refined, instance-specific characterization of stochastic dynamics and to obtain several insights on the regret and belief evolution of a number of sequential experiments including Thompson sampling (but not upper-confidence bound, which does not satisfy our continuity assumption). We show that all sequential experiments whose randomization probabilities have a Lipschitz-continuous dependence on the observed data suffer from suboptimal regret performance when the reward gaps are relatively large. Conversely, we find that a version of Thompson sampling with an asymptotically uninformative prior variance achieves near-optimal instance-specific regret scaling, including with large reward gaps, but these good regret properties come at the cost of highly unstable posterior beliefs. This paper was accepted by Baris Ata, stochastic models and simulation. Supplemental Material: The data and online appendix are available at https://doi.org/10.1287/mnsc.2023.4964 .

Data-driven product-process optimization of N-isopropylacrylamide microgel flow-synthesis

Microgels are cross-linked, colloidal polymer networks with great potential for stimuli-response release in drug-delivery applications, as their small size allows them to pass human cell boundaries. For applications with specified requirements regarding size, producing tailored microgels in a continuous flow reactor is advantageous because the microgel properties can be controlled tightly. However, no fully-specified mechanistic models are available for continuous microgel synthesis, as the physical properties of the included components are only studied partly. To address this gap and accelerate tailor-made microgel development, we propose a data-driven optimization in a hardware-in-the-loop approach to efficiently synthesize microgels with defined sizes. We optimize the synthesis regarding conflicting objectives (maximum production efficiency, minimum energy consumption, and the desired microgel radius) by applying Bayesian optimization via the solver “Thompson sampling efficient multi-objective optimization” (TS-EMO). We validate the optimization using the deterministic global solver “McCormick-based Algorithm for mixed-integer Nonlinear Global Optimization” (MAiNGO) and verify three computed Pareto optimal solutions via experiments. The proposed framework can be applied to other desired microgel properties and reactor setups and has the potential of efficient development by minimizing number of experiments and modeling effort needed.