Two-armed Bandit Problem Research Articles

TWO SOLUTION TECHNIQUES FOR ADAPTIVE REINVESTMENT: A SMALL SAMPLE COMPARISON

Abstract Two adaptive control techniques are compared in the context of an adaptive reinvestment two-armed bandit problem. The first solution technique is the Bayesian-dynamic programming approach. The distinctive feature of the recently proposed second solution technique is the direct estimation and updating of the criterion function without recourse to explicit probability distribution estimation via Bayes's rule. The control selections generated by the two solution techniques are shown to closely approximate each other. An explanation for this close correspondence is provided by means of an equivalence theorem for control objective specification.

On finite memory solutions to the two-armed bandit problem (Corresp.)

The least upper bound on the asymptotic proportion of the choice of the correct coin, achievable by {\em expedient} finite-memory algorithms in certain two-armed bandit problems, is derived and schemes which achieve these bounds in a limiting sense are displayed. A deterministic automaton whose performance is close to optimal is also presented.

On the evaluation of independent binary features (Corresp.)

On the evaluation of independent binary features (Corresp.)

A Uniform Two-armed Bandit Problem--The Parameter of one Distribution is Known

A Uniform Two-armed Bandit Problem--The Parameter of one Distribution is Known

A new approach to filtering and adaptive control: stability results

A new approach to adaptive control is proposed. The principal distinguishing feature is the direct estimation and updating of the criterion function by means of a filtering operation on a vector of transitional pseudo-return functions. The data storage and computational problems often associated with explicit probability distribution updating via Bayes' rule are thus avoided. Convergence properties are established for a simple linear criterion function filter designed for a class of adaptive control problems typified by a well-known two-armed bandit problem. Optimality properties are established for the filter in a companion paper.

Bayesian Rules for the Two-Armed Bandit Problem

Bayesian Rules for the Two-Armed Bandit Problem

Interaction between stochastic automata and random environment

Stochastic automata models capable of exhibiting unconditional learning behaviour are proposed and their behaviour analysed when operating in a stationary random environment about which they have no a priori knowledge. The automaton expected expected penalty is used as a performance index to assess its capability to acquire information pertaining to the unknown features of the environment. The optimization of the performance index provides a measure of the learning capacity of the automaton. The interaction between an automaton and a random media is also considered in the context of the two-armed bandit problem. Computer simulation results show that automata structures discussed in this paper compare well with two-armed bandit models.

Bayesian rules for the two-armed bandit problem

Bayesian rules for the two-armed bandit problem

The apparent conflict between estimation and control—a survey of the two-armed bandit problem

The two-armed bandit is one of the simplest possible non-deterministic control environments which are not trivial. And yet it is astonishingly difficult to control. For the finite-time problem, dynamic programming methods provide optimal controllers. Optimal control strategies also exist for the infinite-time problem, but their implementation requires infinite storage. Storage can be restricted by allowing only the results of the last r tosses to be recorded: the finite-memory problem—or by considering finite state controllers: the finite-state problem.This paper surveys approaches to the two-armed bandit problem. After introducing the problem and discussing examples of systems where it appears, progress on the three above-mentioned variants is reviewed in turn.

The Two-Armed Bandit

The optimal dynamic programming approach to the solution of the two-armed bandit problem is discussed and the suboptimal play-the-winner and one-step-ahead sampling rules are compared with it.

A Note on the Bernoulli Two-Armed Bandit Problem

Suppose the arms of a two-armed bandit generate i.i.d. Bernoulli random variables with success probabilities $\rho$ and $\lambda$ respectively. It is desired to maximize the expected sum of $N$ trials where $N$ is fixed. If the prior distribution of $(\rho, \lambda)$ is concentrated at two points $(a, b)$ and $(c, d)$ in the unit square, a characterization of the optimal policy is given. In terms of $a, b, c$, and $d$, necessary and sufficient conditions are given for the optimality of the myopic policy.

Some Remarks on the Two-Armed Bandit

Some Remarks on the Two-Armed Bandit

A note on the two-armed bandit problem with finite memory

Robbins has proposed a finite memory constraint on the two-armed bandit problem in which the coin to be tossed at each stage may depend on the history of the previous tosses only through the outcomes of the last r tosses. Letting the choice of coin depend on the time at which the toss is made, we exhibit a deterministic rule with memory r = 2, the description of which is independent of the coin biases p1 and p2 , which achieves, with probability one, a limiting proportion of heads equal to max \{{ p1 , p2 \}}. Thus this rule is asymptotically uniformly best among the class of time-varying finite memory rules.

Ultimate choice between two attractive goals: Predictions from a model

A mathematical model for two-choice behavior in situations where both choices are desirable is discussed. According to the model, one or the other choice is ultimately preferred, and a functional equation is given for the fraction of the population ultimately preferring a given choice. The solution depends upon the learning rates and upon the initial probabilities of the choices. Several techniques for approximating the solution of this functional equation are described. One of these leads to an explicit formula that gives good accuracy. This solution can be generalized to the two-armed bandit problem with partial reinforcement in each arm, or the equivalent T-maze problem. Another suggests good ways to program the calculations for a high-speed computer.

An Elementary Survey of Statistical Decision Theory

STATISTICAL decision theory was originated some 15 years ago by Abraham Wald. In the past decade, great strides have been made in this field by Wald and others. However, so far these developments have had but little impact on experimental research in the social and physical sciences. There appear to be two basic reasons for this: one is the natural lag between theory and practice which so often occurs in science; the other, which in the present case may be more fundamental, is that decision theory to date has been too much concerned with the mathematical foundations of the subject and less with its immediate application. Curiously enough, here is a situation in which the foundational development, difficult as it is, is easier than the application to actual problems. What is decision theory? In its broadest sense, decision theory deals with the problem of decision making in the face of uncertainty. But since life is beset with all kinds of uncertainties, this definition is all-embracing and consequently nonilluminating. The truth of the matter is that the methods of decision theory are all-embracing, and could be said to encompass the whole science of inductive reasoning. But being statisticians and not philosophers, we attempt to narrow down the field. This narrowing down consists mainly in a specification of the kind of uncertainty with which we are dealing and an insistence that the decisions to be made must be based on observations obtained from an experiment. The notions of decision theory may be introduced with an illustration which, tho mathematically simple, exhibits all essential features. A few years ago Professor Merrill Flood, then at the RAND Corporation and presently at Columbia University, requested assistance in solving a problem which he and his group encountered in their experimental work with learning models. For our purpose it is not essential to know the exact nature of the experiment except that it dealt with the question of how a person utilizes available information to make decisions. The problem, which is an abstraction of the real situation, can be presented in the form of the now famous two-armed bandit problem. In this formulation, we are given a slot machine with two arms, a right and left arm. The probability of paying off is different for each arm, one having a probability co of paying off, and the other a probability 0, with co greater than 0. When either arm pays off, the amount is a monetary units. The subject is told the values of o and 0, but he is not told which arm is associated with which probability. The subject is allowed to pull either arm at any time for a total of N pulls. The problem is to study how, at