Thompson Sampling Research Articles

Generative Design by Using Exploration Approaches of Reinforcement Learning in Density-Based Structural Topology Optimization

A central challenge in generative design is the exploration of vast number of solutions. In this work, we extend two major density-based structural topology optimization (STO) methods based on four classes of exploration algorithms of reinforcement learning (RL) to STO problems, which approaches generative design in a new way. The four methods are: first, using ε -greedy policy to disturb the search direction; second, using upper confidence bound (UCB) to add a bonus on sensitivity; last, using Thompson sampling (TS) as well as information-directed sampling (IDS) to direct the search, where the posterior function of reward is fitted by Beta distribution or Gaussian distribution. Those combined methods are evaluated on some structure compliance minimization tasks from 2D to 3D, including the variable thickness design problem of an atmospheric diving suit (ADS). We show that all methods can generate various acceptable design options by varying one or two parameters simply, except that IDS fails to reach the convergence for complex structures due to the limitation of computation ability. We also show that both Beta distribution and Gaussian distribution work well to describe the posterior probability.

Beamforming for Maximal Coverage in mmWave Drones: A Reinforcement Learning Approach

Drone as a base station can provide wireless services in a variety of situations. In this letter, we employ a uniform linear array (ULA) to produce a directional beam to increase the quality of service (QoS) of users in the downlink of cellular networks. Due to the strict power limitations of a drone base station (DBS), we envision a single radio frequency (RF) chain architecture. A beamforming design methodology in an unknown environment is presented over a mmWave channel with the aim of maximizing the number of covered users while taking into account the human body blockage effects. Regarding the ambiguity of the environment, we model the problem of finding the optimal beam direction as a multi-armed bandit (MAB). Due to its fast convergence property, Thompson sampling (TS) is used for solving the MAB problem. Simulation results show that the DBS is able to find the optimal beam angle in only tens of iterations.

A synergy Thompson sampling hyper‐heuristic for the feature selection problem

SummaryTo classify high‐dimensional data, feature selection plays a key role to eliminate irrelevant attributes and enhance the classification accuracy and efficiency. Since feature selection is an NP‐Hard problem, many heuristics and metaheuristics have been used to tackle in practice this problem. In this article, we propose a novel approach that consists in a probabilistic selection hyper‐heuristic called the synergy Thompson sampling hyper‐heuristic. The Thompson sampling selection strategy is a probabilistic reinforcement learning mechanism to assess the behavior of the low‐level heuristics, and to predict which one will be more efficient at each point during the search process. The proposed hyper‐heuristic is combined with a 1 nearest neighbor classifier from the Weka framework. It aims to find the best subset of features that maximizes the classification accuracy rate. Experimental results show a good performance in favor of the proposed method when comparing with other existing approaches.

Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible. In this work, we focus on learning to coordinate. Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We provide a regret bound that is sublinear in time and low-order polynomial in the highest number of actions of a single agent for sparse coordination graphs. Additionally, we empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks, and a novel benchmark with Poisson distributions. An example of a loosely-coupled multi-agent system is a wind farm. Coordination within the wind farm is necessary to maximize power production. As upstream wind turbines only affect nearby downstream turbines, we can use MATS to efficiently learn the optimal control mechanism for the farm. To demonstrate the benefits of our method toward applications we apply MATS to a realistic wind farm control task. In this task, wind turbines must coordinate their alignments with respect to the incoming wind vector in order to optimize power production. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in practical applications with sparse neighbourhood structures.

Incentivized Exploration for Multi-Armed Bandits under Reward Drift

We study incentivized exploration for the multi-armed bandit (MAB) problem where the players receive compensation for exploring arms other than the greedy choice and may provide biased feedback on reward. We seek to understand the impact of this drifted reward feedback by analyzing the performance of three instantiations of the incentivized MAB algorithm: UCB, ε-Greedy, and Thompson Sampling. Our results show that they all achieve O(log T) regret and compensation under the drifted reward, and are therefore effective in incentivizing exploration. Numerical examples are provided to complement the theoretical analysis.

Bayesian Optimization for Categorical and Category-Specific Continuous Inputs

Many real-world functions are defined over both categorical and category-specific continuous variables and thus cannot be optimized by traditional Bayesian optimization (BO) methods. To optimize such functions, we propose a new method that formulates the problem as a multi-armed bandit problem, wherein each category corresponds to an arm with its reward distribution centered around the optimum of the objective function in continuous variables. Our goal is to identify the best arm and the maximizer of the corresponding continuous function simultaneously. Our algorithm uses a Thompson sampling scheme that helps connecting both multi-arm bandit and BO in a unified framework. We extend our method to batch BO to allow parallel optimization when multiple resources are available. We theoretically analyze our method for convergence and prove sub-linear regret bounds. We perform a variety of experiments: optimization of several benchmark functions, hyper-parameter tuning of a neural network, and automatic selection of the best machine learning model along with its optimal hyper-parameters (a.k.a automated machine learning). Comparisons with other methods demonstrate the effectiveness of our proposed method.

A dynamic thompson sampling hyper-heuristic framework for learning activity planning in personalized learning

Personalized learning is emerging in schools as an alternative to one-size-fits-all education. This study introduces and explores a weekly demand-driven flexible learning activity planning problem of own-pace own-method personalized learning. The introduced problem is a computationally intractable optimization problem involving many decision dimensions and also many soft constraints. We propose batch and decomposition methods to generate good-quality initial solutions and a dynamic Thompson sampling based hyper-heuristic framework, as a local search mechanism, which explores the large solution space of this problem in an integrative way. The characteristics of our test instances comply with average secondary schools in the Netherlands and are based on expert opinions and surveys. The experiments, which benchmark the proposed heuristics against Gurobi MIP solver on small instances, illustrate the computational challenge of this problem numerically. According to our experiments, the batch method seems quicker and also can provide better quality solutions for the instances in which resource levels are not scarce, while the decomposition method seems more suitable in resource scarcity situations. The dynamic Thompson sampling based online learning heuristic selection mechanism is shown to provide significant value to the performance of our hyper-heuristic local search. We also provide some practical insights; our experiments numerically demonstrate the alleviating effects of large school sizes on the challenge of satisfying high-spread learning demands.

Analysis of Distributed Thompson Sampling based on Consensus Control

Analysis of Distributed Thompson Sampling based on Consensus Control

Bayesian optimisation in unknown bounded search domains

Bayesian optimisation (BO) is one of the most sample efficient methods for determining the optima of expensive, noisy black-box functions. Despite its tremendous success in scientific discovery and hyperparameter tuning, it still requires a bounded search space. The search spaces boundaries are, however, often chosen heuristically with an educated guess. If the boundaries are misspecified, then the search space may either be unnecessarily large and hence more expensive to optimise, or it may simply not contain the global optimum. In this paper, we introduce a method for dynamically determining the bound directly from the data. This is done using a distribution of the bound derived in a Bayesian setting. The prior is chosen by the user and the likelihood is derived with Thompson sampling. This results in a bound that is both cheap to optimise and has a high probability of containing the global optimum. We compare the performance of our method with the alternative methods on a range of synthetic and real-world problems and demonstrate that our method achieves consistently superior results.

Collaborative Thompson Sampling

Thompson sampling is one of the most effective strategies to balance exploration-exploitation trade-off. It has been applied in a variety of domains and achieved remarkable success. Thompson sampling makes decisions in a noisy but stationary environment by accumulating uncertain information over time to improve prediction accuracy. In highly dynamic domains, however, the environment undergoes frequent and unpredictable changes. Making decisions in such an environment should rely on current information. Therefore, standard Thompson sampling may perform poorly in these domains. Here we present collaborative Thompson sampling to apply the exploration-exploitation strategy to highly dynamic settings. The algorithm takes collaborative effects into account by dynamically clustering users into groups, and the feedback of all users in the same group will help to estimate the expected reward in the current context to find the optimal choice. Incorporating collaborative effects into Thompson sampling allows to capture real-time changes of the environment and adjust decision making strategy accordingly. We compare our algorithm with standard Thompson sampling algorithms on two real-world datasets. Our algorithm shows accelerated convergence and improved prediction performance in collaborative environments. We also provide regret analyses of our algorithm in both contextual and non-contextual settings.

Comparing Robot and Human guided Personalization: Adaptive Exercise Robots are Perceived as more Competent and Trustworthy

Learning and matching a user’s preference is an essential aspect of achieving a productive collaboration in long-term Human–Robot Interaction (HRI). However, there are different techniques on how to match the behavior of a robot to a user’s preference. The robot can be adaptable so that a user can change the robot’s behavior to one’s need, or the robot can be adaptive and autonomously tries to match its behavior to the user’s preference. Both types might decrease the gap between a user’s preference and the actual system behavior. However, the Level of Automation (LoA) of the robot is different between both methods. Either the user controls the interaction, or the robot is in control. We present a study on the effects of different LoAs of a Socially Assistive Robot (SAR) on a user’s evaluation of the system in an exercising scenario. We implemented an online preference learning system and a user-adaptable system. We conducted a between-subject design study (adaptable robot vs. adaptive robot) with 40 subjects and report our quantitative and qualitative results. The results show that users evaluate the adaptive robots as more competent, warm, and report a higher alliance. Moreover, this increased alliance is significantly mediated by the perceived competence of the system. This result provides empirical evidence for the relation between the LoA of a system, the user’s perceived competence of the system, and the perceived alliance with it. Additionally, we provide evidence for a proof-of-concept that the chosen preference learning method (i.e., Double Thompson Sampling (DTS)) is suitable for online HRI.

Generalized Probabilistic Bisection for Stochastic Root Finding

We consider numerical schemes for root finding of noisy responses through generalizing the Probabilistic Bisection Algorithm (PBA) to the more practical context where the sampling distribution is unknown and location dependent. As in standard PBA, we rely on a knowledge state for the approximate posterior of the root location. To implement the corresponding Bayesian updating, we also carry out inference of oracle accuracy, namely learning the probability of the correct response. To this end we utilize batched querying in combination with a variety of frequentist and Bayesian estimators based on majority vote, as well as the underlying functional responses, if available. For guiding sampling selection we investigate both entropy-directed sampling and quantile sampling. Our numerical experiments show that these strategies perform quite differently; in particular, we demonstrate the efficiency of randomized quantile sampling, which is reminiscent of Thompson sampling. Our work is motivated by the root-finding subroutine in pricing of Bermudan financial derivatives, illustrated in the last section of the article.

Self-accelerated Thompson sampling with near-optimal regret upper bound

Thompson sampling utilizes Bayesian heuristic strategy to balance the exploration-exploitation trade-off. It has been applied in a variety of practical domains and achieved great success. Despite being empirically efficient and powerful, Thompson sampling has eluded theoretical analysis. Existing analyses of Thompson sampling only provide regret upper bound of O˜(d3/2T) for linear contextual bandits, which is worse than the information-theoretic lower bound by a factor of d. In this paper, we design and analyze self-accelerated Thompson Sampling algorithm for the stochastic contextual multi-armed bandit problem. Our analysis establishes that the regret upper bound of self-accelerated Thompson sampling is O˜(dT), which is the best upper bound achieved by any efficient contextual bandit algorithm in infinite action space. Our experiment on simulated data and real-world dataset shows that self-accelerated Thompson sampling outperforms standard Thompson sampling in both convergence rate and prediction accuracy.

Thompson Sampling-Based Channel Selection Through Density Estimation Aided by Stochastic Geometry

We propose a sophisticated channel selection scheme based on multi-armed bandits and stochastic geometry analysis. In the proposed scheme, a typical user attempts to estimate the density of active interferers for every channel via the repeated observations of signal-to-interference power ratio (SIR), which demonstrates the randomness induced by randomized interference sources and fading effects. The purpose of this study involves enabling a typical user to identify the channel with the lowest density of active interferers while considering the communication quality during exploration. To resolve the trade-off between obtaining more observations on uncertain channels and using a channel that appears better, we employ a bandit algorithm called Thompson sampling (TS), which is known for its empirical effectiveness. We consider two ideas to enhance TS. First, noticing that the SIR distribution derived through stochastic geometry is useful for updating the posterior distribution of the density, we propose incorporating the SIR distribution into TS to estimate the density of active interferers. Second, TS requires sampling from the posterior distribution of the density for each channel, while it is significantly more complicated for the posterior distribution of the density to generate samples than well-known distribution. The results indicate that this type of sampling process is achieved via the Markov chain Monte Carlo method (MCMC). The simulation results indicate that the proposed method enables a typical user to determine the channel with the lowest density more efficiently than the TS without density estimation aided by stochastic geometry, and $\epsilon $ -greedy strategies.

Dynamic Decision-Making Process in the Opportunistic Spectrum Access

We investigate in this paper many problems related to the decision-making process in the Cognitive Radio (CR), where a Secondary User (SU) tries to maximize its opportunities by finding the most vacant channel. Recently, Multi-Armed Bandit (MAB) problems attracted the attention to help a single SU, in the context of CR, makes an optimal decision using the well-known MAB algorithms, such as: Thompson Sampling, Upper Confidence Bound, e-greedy, etc. However, the big challenge for multiple SUs remains to learn collectively or separately the vacancy of channels and decrease the number of collisions among users. To solve the latter issue for multiple users, the All-Powerful Learning (APL) policy is proposed; this new policy considers the priority access and the dynamic multi-user access, where the number of SUs may change over time. Based on our APL policy, we consider as well as the Quality of Service (QoS), where SUs should estimate and then access best channels in terms of both quality and availability. The experimental results show the superiority of APL compared to existing algorithms, and it has also been shown that the SUs are able to learn channels qualities and availabilities and further enhance the QoS. © 2020 ASTES Publishers. All rights reserved.

From Average Customer to Individual Traveler: A Field Experiment in Airline Ancillary Pricing

Ancillaries in the travel industry are now a major stream for revenue and profitability. Ancillaries are optional products or services whose sales depend on an individual's personal preference and their trip context. Conventional pricing strategies for ancillaries based on poorly optimized or static business rules do not respond to changing market conditions or trip context. We present a dynamic pricing model developed in conjunction with Deepair solutions, an AI technology provider for travel suppliers. Our models provide dynamic, customer-interaction-specific pricing recommendations, to increase revenue. The unique nature of personalized pricing provides the opportunity to search over the market space to find the optimal price-point for each customer, without violating customer privacy. We present an A/B testing deployment framework on an airline's website. Embedded in it are three models for dynamic pricing of ancillaries, with increasing levels of sophistication: (1) a two-stage forecasting and pricing model using a logistic mapping function; (2) a two-stage model with a deep neural network for forecasting, followed by pricing using discrete exhaustive search; (3) a single-stage end-to-end deep neural network that recommends the optimal price. In an outer loop, we introduce an online adaptive model-selection framework that adaptively routes customer requests to the above models. This is modeled as multi-armed bandit problem, which we solve using Thompson sampling. We evaluate the performance of these models based on offfine and online evaluations, and their real-world business impact. Offline experiments show that deep learning algorithms outperform traditional machine learning techniques for this problem. In online testing, our AI-driven pricing outperforms human rule-based approaches, improving conversion by 17% and revenue per offer by 25%. Additionally, our adaptive model-selection approach outperforms a uniformly random selection policy by improving the expected revenue per offer by 43% and conversion score by 58% in a simulation environment.

Learning-Based Beamforming for Multi-User Vehicular Communications: A Combinatorial Multi-Armed Bandit Approach

There is an urgent need to support high data rate among connected vehicles for both safety and infotainment purposes. However, high rate for vehicular communications is impacted by several challenges such as optimal beam tracking for multiple simultaneous-transmitting vehicles in highly-dynamic environments. In this paper, we aim to maximize the overall network throughput for multi-vehicular communications. We propose a reinforcement learning (RL) approach called combinatorial multi-armed bandit (CMAB) framework, where multiple arms (i.e., actions) form a super arm and can be played together, to handle the beam selection problem in a vehicular network. More specifically, we propose an adaptive combinatorial Thompson sampling algorithm, namely adaptive CTS, that CMAB embodies for appropriate selection of simultaneous beams in a high-mobility vehicular environment. The proposed approach applies the smart exploration-exploitation trade-off for the fast selection of beams. As the proposed adaptive CTS scheme produces a higher complexity over the search space which increases exponentially with the number of user, we also propose a sequential Thompson sampling (TS) algorithm where beam selection is performed in a one-by-one manner. We analyze the regret bounds for the proposed beam tracking algorithms. The performance of the proposed strategies are evaluated for multi-vehicle millimeter-wave (mmWave) simulation environments. Our results suggest that the proposed sequential approach performs almost similar to the simultaneous adaptive CTS scheme for tracking optimal beams in a multi-vehicular network with much reduced complexity. Simulation results also show that both of our proposed strategies approach the optimal achievable rate achieved by the genie-aided solution.

Combining Bayesian optimization and Lipschitz optimization

Bayesian optimization and Lipschitz optimization have developed alternative techniques for optimizing black-box functions. They each exploit a different form of prior about the function. In this work, we explore strategies to combine these techniques for better global optimization. In particular, we propose ways to use the Lipschitz continuity assumption within traditional BO algorithms, which we call Lipschitz Bayesian optimization (LBO). This approach does not increase the asymptotic runtime and in some cases drastically improves the performance (while in the worst case the performance is similar). Indeed, in a particular setting, we prove that using the Lipschitz information yields the same or a better bound on the regret compared to using Bayesian optimization on its own. Moreover, we propose a simple heuristics to estimate the Lipschitz constant, and prove that a growing estimate of the Lipschitz constant is in some sense “harmless”. Our experiments on 15 datasets with 4 acquisition functions show that in the worst case LBO performs similar to the underlying BO method while in some cases it performs substantially better. Thompson sampling in particular typically saw drastic improvements (as the Lipschitz information corrected for its well-known “over-exploration” pheonemon) and its LBO variant often outperformed other acquisition functions.

Online Learning and Pricing for Service Systems with Reusable Resources

We consider a price-based revenue management problem with finite reusable resources over a finite time horizon. Stochastically arrived customers request an exponentially distributed service time and may balk and renege given insufficient resource. The resource unit, upon completion of serving one customer, can be released to serve the next customer immediately. The arrival, service, balking, and reneging rates all depend on the price being offered. In this paper, we assume that the firm does not know the mappings between these rates and prices, and thus it makes adaptive pricing decisions in each period based only on past sales to maximize the cumulative revenue. We propose two new multi-armed bandit (MAB) based learning algorithms, termed Batch Upper Confidence Bound (BUCB) algorithm and Batch Thompson Sampling (BTS) algorithm, for finding near-optimal pricing policies. Compared with prior pricing and MAB literature, the salient difficulties of this problem lie in (i) the unknown rate-and-price mapping information, (ii) the dynamic nature of reusable resources being committed over time, (iii) the transient behavior of the service system when the price changes, and (iv) unbounded and heavy-tailed distributions of observed random variables. Our proposed algorithms contain a Warm-up Phase to eliminate the heavy-tail effects and a Learning Phase to identify the optimal price. Our algorithms separate the Learning Phase into successive operational batches and select a price from a prescribed set in each batch using past sales collected in previous batches. The performance measure is cumulative regret, which is the difference between the revenue attained by our approach and by a clairvoyant optimal pricing policy under full distributional information. We prove that the cumulative regret is $O(\sqrt{PT\log (T)})$, where $T$ is the total number of time periods and $P$ is the cardinality of the feasible price set, and the result matches the lower bound up to a logarithmic factor. As an intermediate step, we also develop a coupling analysis for analyzing the time for a queue to reach the steady state from an empty state or from a steady state under another set of system parameters. Our numerical experiments demonstrate and confirm the efficacy of the proposed BUCB and BTS algorithms.

Estimating prediction error for complex samples

With a growing interest in using non‐representative samples to train prediction models for numerous outcomes it is necessary to account for the sampling design that gives rise to the data in order to assess the generalized predictive utility of a proposed prediction rule. After learning a prediction rule based on a non‐uniform sample, it is of interest to estimate the rule's error rate when applied to unobserved members of the population. Efron (1986) proposed a general class of covariance penalty inflated prediction error estimators that assume the available training data are representative of the target population for which the prediction rule is to be applied. We extend Efron's estimator to the complex sample context by incorporating Horvitz–Thompson sampling weights and show that it is consistent for the true generalization error rate when applied to the underlying superpopulation. The resulting Horvitz–Thompson–Efron estimator is equivalent to dAIC, a recent extension of Akaike's information criteria to survey sampling data, but is more widely applicable. The proposed methodology is assessed with simulations and is applied to models predicting renal function obtained from the large‐scale National Health and Nutrition Examination Study survey. The Canadian Journal of Statistics 48: 204–221; 2020 © 2019 Statistical Society of Canada