Two-armed Bandit Problem Research Articles

Optimal strategy for Bayesian two-armed bandit problem with an arched reward function

In the classical two-armed bandit (TAB) problems, the goal of maximizing the total return specifies the optimal decision rule and fosters the development of abundant practicable methods, such as $ \epsilon $-greedy and upper confidence bound for TAB procedures. However, the existence of an arched reward function for the total return breaks the previously updated algorithms, like traditional myopic strategy, and prompts the generation of novel computation and theory for maximizing its expectation. Here we develop a special class of Bayesian two-armed bandit problems by imposing a prior probability on which arm has a greater expectation of the returns and propose a myopic strategy to specify the decision rule. We prove that the proposed myopic strategy is optimal under the structure of the arched reward function by exploring the dynamic programming principle. This article also demonstrates that arched functions are extensions of the monotone functions including the classical linear functions. Meanwhile, we establish a corresponding law of large numbers for our myopic strategy. Simulation studies pose supportive evidence that the newly proposed strategy performs well.

A confirmation of a conjecture on Feldman’s two-armed bandit problem

AbstractThe myopic strategy is one of the most important strategies when studying bandit problems. In 2018, Nouiehed and Ross put forward a conjecture about Feldman’s bandit problem (J. Appl. Prob. (2018) 55, 318–324). They proposed that for Bernoulli two-armed bandit problems, the myopic strategy stochastically maximizes the number of wins. In this paper we consider the two-armed bandit problem with more general distributions and utility functions. We confirm this conjecture by proving a stronger result: if the agent playing the bandit has a general utility function, the myopic strategy is still optimal if and only if this utility function satisfies reasonable conditions.

Optimal distributions of rewards for a two-armed slot machine

In this paper we consider the continuous time two-armed bandit (TAB) problem where the slot machine has two different arms in the sense that the two arms have different expected rewards and variances. We explore the optimal distribution of rewards for two-armed bandit problems, and obtain the explicit distribution function as well as the searching rules of optimal strategy. As a by-product, we find two new counter-intuitive phenomena in nonlinear probability framework (optimal strategic framework). The first is that the combination of losing arm and winning arm can make the winning arm achieve a greater coverage probability to win expected reward, which is also referred to “good + bad = better”. The discovery implies that the traditional advice of always pursuing the arm with larger expected reward (i.e., stay on a winner rule) is not optimal in the probability framework. The second is that the combination sequence out of two independent and normal distribution-based arms is not normally distributed if the two arms are different, which is straightforward understood as “mutually independent normal + normal = unnormal”. Furthermore, we provide the optimal sequential strategy to construct the “combination” arm and numerically examine the underlying mechanism.

Theory of Acceleration of Decision-Making by Correlated Time Sequences

Photonic accelerators have been intensively studied to provide enhanced information processing capability to benefit from the unique attributes of physical processes. Recently, it has been reported that chaotically oscillating ultrafast time series from a laser, called laser chaos, provides the ability to solve multi-armed bandit (MAB) problems or decision-making problems at GHz order. Furthermore, it has been confirmed that the negatively correlated time-domain structure of laser chaos contributes to the acceleration of decision-making. However, the underlying mechanism of why decision-making is accelerated by correlated time series is unknown. In this study, we demonstrate a theoretical model to account for accelerating decision-making by correlated time sequence. We first confirm the effectiveness of the negative autocorrelation inherent in time series for solving two-armed bandit problems using Fourier transform surrogate methods. We propose a theoretical model that concerns the correlated time series subjected to the decision-making system and the internal status of the system therein in a unified manner, inspired by correlated random walks. We demonstrate that the performance derived analytically by the theory agrees well with the numerical simulations, which confirms the validity of the proposed model and leads to optimal system design. This study paves the way for improving the effectiveness of correlated time series for decision-making, impacting artificial intelligence and other applications.

Two-Armed Bandit Problem and Batch Version of the Mirror Descent Algorithm

Two-Armed Bandit Problem and Batch Version of the Mirror Descent Algorithm

Universal strategies for the two-alternative big data processing

We consider the two-alternative processing of big data in the framework of the two-armed bandit problem. We assume that there are two processing methods with different, fixed but a priori unknown efficiencies which are due to different reasons including those caused by legislation. Results of data processing are interpreted as random incomes. During control process, one has to determine the most efficient method and to provide its primary usage. The difficulty of the problem is caused by the fact that its solution essentially depends on distributions of one-step incomes corresponding to results of data processing. However, in case of big data we show that there are universal processing strategies for a wide class of distributions of one-step incomes. To this end, we consider Gaussian two-armed bandit which naturally arises when batch data processing is analyzed. Minimax risk and minimax strategy are searched for as Bayesian ones corresponding to the worst-case prior distribution. We present recursive integro-difference equation for computing Bayesian risk and Bayesian strategy with respect to the worst-case prior distribution and a second order partial differential equation into which integro-difference equation turns in the limiting case as the control horizon goes to infinity. We also show that, in case of big data, processing of data one-by-one is not more efficient than optimal batch data processing for some types of distributions of one-step incomes, e.g. for Bernoulli and Poissonian distributions. Numerical experiments are presented and show that proposed universal strategies provide high performance of two-alternative big data processing.

Two-armed bandit problem and batch version of the mirror descent algorithm

We consider the minimax setup for the two-armed bandit problem as applied to data processing if there are two alternative processing methods with different a priori unknown efficiencies. One should determine the most efficient method and provide its predominant application. To this end, we use the mirror descent algorithm (MDA). It is well-known that corresponding minimax risk has the order of $N^{1/2$ with $N$ being the number of processed data and this bound is unimprovable in order. We propose a batch version of the MDA which allows processing data by packets that is especially important if parallel data processing can be provided. In this case, the processing time is determined by the number of batches rather than by the total number of data. Unexpectedly, it turned out that the batch version behaves unlike the ordinary one even if the number of packets is large. Moreover, the batch version provides significantly smaller value of the minimax risk, i.e., it considerably improves a control performance. We explain this result by considering another batch modification of the MDA which behavior is close to behavior of the ordinary version and minimax risk is close as well. Our estimates use invariant descriptions of the algorithms based on Gaussian approximations of incomes in batches of data in the domain of ``close'' distributions and are obtained by Monte-Carlo simulations.

Three heads are better than two: Comparing learning properties and performances across individuals, dyads, and triads through a computational approach.

Although it is considered that two heads are better than one, related studies argued that groups rarely outperform their best members. This study examined not only whether two heads are better than one but also whether three heads are better than two or one in the context of two-armed bandit problems where learning plays an instrumental role in achieving high performance. This research revealed that a U-shaped correlation exists between performance and group size. The performance was highest for either individuals or triads, but the lowest for dyads. Moreover, this study estimated learning properties and determined that high inverse temperature (exploitation) accounted for high performance. In particular, it was shown that group effects regarding the inverse temperatures in dyads did not generate higher values to surpass the averages of their two group members. In contrast, triads gave rise to higher values of the inverse temperatures than their averages of their individual group members. These results were consistent with our proposed hypothesis that learning coherence is likely to emerge in individuals and triads, but not in dyads, which in turn leads to higher performance. This hypothesis is based on the classical argument by Simmel stating that while dyads are likely to involve more emotion and generate greater variability, triads are the smallest structure which tends to constrain emotions, reduce individuality, and generate behavioral convergences or uniformity because of the ''two against one" social pressures. As a result, three heads or one head were better than two in our study.

Strategic Experimentation with Congestion

This paper considers a two-player game of strategic experimentation with competition. Each agent faces a two-armed bandit problem where she continually chooses between her private, risky arm and a common, safe arm. Each agent has exclusive access to her private arm. However, the common arm can only be activated by one agent at a time. This congestion creates negative payoff externalities. Our main finding is that congestion gives rise to new strategic considerations: players perceive a strategic option value from occupying the common arm, making it more attractive than in the absence of competition or when switching is irreversible. (JEL C72, C73, D62, D83)

Two-alternative optimization of moderate batch data processing

We consider optimization of moderate batch data processing in the framework of Bernoulli two-armed bandit problem with indefinite control horizon. We assume that there are two alternative processing methods with different a priori unknown efficiencies which are caused by different reasons including those related to legislation. One has to find the most efficient method and to provide its predominant usage. The arising batches of data with close properties have moderate and possibly uncertain sizes. The problem is considered in minimax setting. According to the main theorem of the game theory minimax risk and minimax strategy are searched for as Bayesian ones corresponding to approximately the worst-case prior distribution concentrated on the finite set of parameters. Numerical experiments show that this approach provides good approximations of minimax strategy and minimax risk.

Learning from failures: Optimal contracts for experimentation and production

Before embarking on a project, a principal must often rely on an agent to learn about its profitability. We model this learning as a two-armed bandit problem and highlight the interaction between learning (experimentation) and production. We derive the optimal contract for both experimentation and production when the agent has private information about the efficiency of experimentation. This private information in the experimentation stage generates asymmetric information in the production stage even though there was no disagreement about the profitability of the project at the outset. The degree of asymmetric information is endogenously determined by the length of the experimentation stage. An optimal contract uses the length of experimentation, the production scale, and the timing of payments to screen the agent. We find that over-experimentation and over-production can be used to reduce the agent's rent. An efficient type is rewarded early since he is more likely to succeed in experimenting, while an inefficient type is rewarded at the very end of the experimentation stage. This result is robust to the introduction of ex post moral hazard.

A Bandit Approach for Mode Selection in Ambient Backscatter-Assisted Wireless-Powered Relaying

Backscattering assisted wireless-powered communication combines ultralow-power backscatter transmitters with energy harvesting devices. This paper investigates the transmission mode selection problem of a hybrid relay that forwards data by switching between the active wireless-powered transmission and the passive ambient backscattering. It first presents a hybrid relay system model and derives its end-to-end success probability under theoretically optimal, but practically unrealistic, conditions. The transmission mode selection is then formulated as a stochastic two-armed bandit problem in a varying environment where the distributions of rewards are nonstationary. The proposed model selection scheme does not assume to have access to any channel states or network conditions, but merely relies on learning from past transmission records. Numerical analyses are performed to validate the proposed bandit-based mode selection approach.

Decision irrationalities involving deadly risks

This article provides an experimental analysis of two-armed bandit problems that have a different structure in which the first unsuccessful outcome leads to termination of the game. It differs from a conventional two-armed bandit problem in that there is no opportunity to alter behavior after an unsuccessful outcome. Introducing the risk of death into a sequential decision problem alters the structure of the problem. Even though play ends after an unsuccessful outcome, Bayesian learning after successful outcomes has a potential function in this class of two-armed bandit problems. Increasing uncertainty boosts the chance of long-term survival since ambiguous probabilities of survival are increased more after each successful outcome. In the independent choice experiments, a slim majority of participants displayed a preference for greater risk ambiguity. Particularly in the interdependent choice experiments, participants were overly deterred by ambiguity. For both independent and interdependent choices, there were several dimensions on which participants displayed within session rationality. However, participants failed to learn and improve their strategy over a series of rounds, which is consistent with evidence of bounded rationality in other challenging games.

Optimization of two-alternative batch data processing

We consider optimization of batch data processing if there are two alternative processing methods available with different unknown efficiencies. One should determine more efficient method and provide its predominant usage. Formally, the problem is presented as Gaussian two-armed bandit problem with a priori unknown mathematical expectations and variances of incomes. We consider the problem in robust (minimax) setting. According to the main theorem of game theory, minimax strategy and minimax risk are sought for as Bayesian ones corresponding to the worst-case prior distribution of parameter. We describe the properties of the worst-case prior distribution and present corresponding recursive equations for determining Bayesian risk and expected losses. Some numerical examples are presented. We show that the control performance almost does not depend on the number of processed batches if this number is large enough.

Category Theoretic Analysis of Photon-Based Decision Making

Decision making is a vital function in the age of machine learning and artificial intelligence; however, its physical realization and theoretical fundamentals are not yet well understood. In our former study, we demonstrated that single photons can be used to make decisions in uncertain, dynamically changing environments. The two-armed bandit problem was successfully solved using the dual probabilistic and particle attributes of single photons. In this study, we present a category theoretic modeling and analysis of single-photon-based decision making, including a quantitative analysis that agrees well with the experimental results. The category theoretic model unveils complex interdependencies of the entities of the subject matter in the most simplified manner, including a dynamically changing environment. In particular, the octahedral structure and the braid structure in triangulated categories provide better understandings and quantitative metrics of the underlying mechanisms for the single-photon decision maker. This study provides insight and a foundation for analyzing more complex and uncertain problems for machine learning and artificial intelligence.

Scalable photonic reinforcement learning by time-division multiplexing of laser chaos

Reinforcement learning involves decision-making in dynamic and uncertain environments and constitutes a crucial element of artificial intelligence. In our previous work, we experimentally demonstrated that the ultrafast chaotic oscillatory dynamics of lasers can be used to efficiently solve the two-armed bandit problem, which requires decision-making concerning a class of difficult trade-offs called the exploration–exploitation dilemma. However, only two selections were employed in that research; hence, the scalability of the laser-chaos-based reinforcement learning should be clarified. In this study, we demonstrated a scalable, pipelined principle of resolving the multi-armed bandit problem by introducing time-division multiplexing of chaotically oscillated ultrafast time series. The experimental demonstrations in which bandit problems with up to 64 arms were successfully solved are presented where laser chaos time series significantly outperforms quasiperiodic signals, computer-generated pseudorandom numbers, and coloured noise. Detailed analyses are also provided that include performance comparisons among laser chaos signals generated in different physical conditions, which coincide with the diffusivity inherent in the time series. This study paves the way for ultrafast reinforcement learning by taking advantage of the ultrahigh bandwidths of light wave and practical enabling technologies.

Strategic Experimentation With Congestion

This paper considers a two-player game of strategic experimentation with competition. Each agent faces a two-armed bandit problem where she continually chooses between her private risky arm and a common, safe arm. Each agent has exclusive access to her private arm. However, the common arm can only be activated by one agent at a time. This congestion creates negative payoff externalities. Our main finding is that congestion gives rise to new strategic considerations: players perceive a strategic option value from occupying the common arm, making it more attractive than in the absence of competition or when switching is irreversible.

A Note on Optimal Experimentation Under Risk Aversion

This paper solves the two-armed bandit problem when decision makersare risk averse. It shows, counterintuitively, that a more risk-averse decisionmaker might be more willing to take risky actions. The reason relates tothe fact that pulling the risky arm in bandit models produces informationon the environment – thereby reducing the risk that a decision maker willface in the future. This finding gives reason for caution when inferring riskpreferences from observed actions: in a bandit setup, observing a greaterappetite for risky actions can actually be indicative of more risk aversion,not less. Studies which do not take this into account may produce biasedestimates.

Learning from Failures: Optimal Contract for Experimentation and Production

Before embarking on a project, a principal must often rely on an agent to learn about its profitability. We model this learning as a two-armed bandit problem and highlight the interaction between learning (experimentation) and production. We derive the optimal contract for both experimentation and production when the agent has private information about his efficiency in experimentation. This private information in the experimentation stage generates asymmetric information in the production stage even though there was no disagreement about the profitability of the project at the outset. The degree of asymmetric information is endogenously determined by the length of the experimentation stage. An optimal contract uses the length of experimentation, the production scale, and the timing of payments to screen the agents. Due to the presence of an optimal production decision after experimentation, we find over-experimentation to be optimal. The asymmetric information generated during experimentation makes over-production optimal. An efficient type is rewarded early since he is more likely to succeed in experimenting, while an inefficient type is rewarded at the very end of the experimentation stage. This result is robust to the introduction of ex post moral hazard.

Learning from Failures: Optimal Contract for Experimentation and Production

Before embarking on a project, a principal must often rely on an agent to learn about its profitability. We model this learning as a two-armed bandit problem and highlight the interaction between learning (experimentation) and production. We derive the optimal contract for both experimentation and production when the agent has private information about his efficiency in experimentation. This private information in the experimentation stage generates asymmetric information in the production stage even though there was no disagreement about the profitability of the project at the outset. The degree of asymmetric information is endogenously determined by the length of the experimentation stage. An optimal contract uses the length of experimentation, the production scale, and the timing of payments to screen the agents. Due to the presence of an optimal production decision after experimentation, we find over-experimentation to be optimal. The asymmetric information generated during experimentation makes over-production optimal. An efficient type is rewarded early since he is more likely to succeed in experimenting, while an inefficient type is rewarded at the very end of the experimentation stage. This result is robust to the introduction of ex post moral hazard.