Risky Arm Research Articles

Dual Moral Hazard and the Tyranny of Success

I explain why current success can undermine an organization’s ability to innovate. I consider a standard bandit problem between a safe and a risky arm with two modifications. First, a principal allocates resources. Second, an agent must install the risky arm, which is not contractible. If the principal cannot commit to a resource policy, a dual moral hazard problem emerges: The agent’s pay must be tied to the risky arm’s success to encourage installation, inducing the principal to stop experimenting with the arm prematurely. This problem intensifies as the safe arm becomes more profitable, potentially leaving the organization worse off. (JEL D21, D82, D86, O31)

(Bad) reputation in relational contracting

Motivated by markets for “expertise,” we study a bandit model where a principal chooses between a safe and risky arm. A strategic agent controls the risky arm and privately knows whether its type is high or low. Irrespective of type, the agent wants to maximize duration of experimentation with the risky arm. However, only the high type arm can generate value for the principal. Our main insight is that reputational incentives can be exceedingly strong unless both players coordinate on maximally inefficient strategies on path. We discuss implications for online content markets, term limits for politicians, and experts in organizations.

Strategic observation with exponential bandits

We introduce strategic observation into Keller et al. (2005)'s game of experimentation with conclusive breakthroughs. There are two players who must decide when to start and when to stop observation, given that observation is costly and stopping observation is irreversible. We construct a class of symmetric Markov Perfect Equilibria in which, on path, players fully experiment before starting observation, and allocate only a fraction of the resource to the risky arm afterwards. Each equilibrium in this class outperforms the symmetric equilibrium of Keller et al. (2005) in terms of payoffs.

Strategic Experimentation with Congestion

This paper considers a two-player game of strategic experimentation with competition. Each agent faces a two-armed bandit problem where she continually chooses between her private, risky arm and a common, safe arm. Each agent has exclusive access to her private arm. However, the common arm can only be activated by one agent at a time. This congestion creates negative payoff externalities. Our main finding is that congestion gives rise to new strategic considerations: players perceive a strategic option value from occupying the common arm, making it more attractive than in the absence of competition or when switching is irreversible. (JEL C72, C73, D62, D83)

Stopping with congestion and private payoffs

This paper analyses a two-player stopping game with multiarmed bandits in which each player chooses between learning about the quality of her private risky arm and competing for the use of a single shared safe arm. The qualities of the players’ risky arms are independent. A player whose risky arm produces a success no longer competes for the safe arm. We assume that a player observes her opponent’s actions but not his realised payoffs. She is therefore never certain whether her opponent is still competing for the safe arm. When the players’ prior probabilities of success are sufficiently close, there exists no pure strategy equilibrium, and we characterise the unique mixed strategy equilibrium. Otherwise, the unique equilibrium is in pure strategies. The amount of experimentation performed in equilibrium is inefficiently low but, for many priors, higher than if successes are publicly observed.

Undiscounted bandit games

We analyze undiscounted continuous-time games of strategic experimentation with two-armed bandits. The risky arm generates payoffs according to a Lévy process with an unknown average payoff per unit of time which nature draws from an arbitrary finite set. Observing all actions and realized payoffs, plus a free background signal, players use Markov strategies with the common posterior belief about the unknown parameter as the state variable. We show that the unique symmetric Markov perfect equilibrium can be computed in a simple closed form involving only the payoff of the safe arm, the expected current payoff of the risky arm, and the expected full-information payoff, given the current belief. In particular, the equilibrium does not depend on the precise specification of the payoff-generating processes.

Learning While Experimenting

Abstract An agent performing risky experimentation can benefit from suspending it to learn directly about the state. ‘Positive’ information acquisition seeks news that would confirm the state that favours experimentation. It is used as a last-ditch effort when the agent is pessimistic about the risky arm before abandoning it. ‘Negative’ information acquisition seeks news that would demonstrate that experimentation is futile. It is used as an insurance strategy to avoid wasteful experimentation when the agent is still optimistic. A higher reward from risky experimentation expands the region of beliefs that the agent optimally chooses information acquisition rather than experimentation.

A note on optimal experimentation under risk aversion

In a standard two-armed bandit setup, this paper shows – counterintuitively – that a more risk-averse decision maker might be more willing to take risky actions. The reason relates to the fact that pulling the risky arm in bandit models produces information on the environment – thereby reducing the risk that a decision maker will face in the future. This finding gives reason for caution when inferring risk preferences from observed actions: in a bandit setup, observing a greater appetite for risky actions can actually be indicative of more risk aversion, not less.

Strategic Experimentation With Congestion

This paper considers a two-player game of strategic experimentation with competition. Each agent faces a two-armed bandit problem where she continually chooses between her private risky arm and a common, safe arm. Each agent has exclusive access to her private arm. However, the common arm can only be activated by one agent at a time. This congestion creates negative payoff externalities. Our main finding is that congestion gives rise to new strategic considerations: players perceive a strategic option value from occupying the common arm, making it more attractive than in the absence of competition or when switching is irreversible.

A Note on Optimal Experimentation Under Risk Aversion

This paper solves the two-armed bandit problem when decision makersare risk averse. It shows, counterintuitively, that a more risk-averse decisionmaker might be more willing to take risky actions. The reason relates tothe fact that pulling the risky arm in bandit models produces informationon the environment – thereby reducing the risk that a decision maker willface in the future. This finding gives reason for caution when inferring riskpreferences from observed actions: in a bandit setup, observing a greaterappetite for risky actions can actually be indicative of more risk aversion,not less. Studies which do not take this into account may produce biasedestimates.

Stopping with Congestion and Private Payoffs

This paper analyses a two-player stopping game with multiarmed bandits in which each player chooses between learning about the quality of her private risky arm and competing for the use of a single shared safe arm. The qualities of the players' risky arms are independent. A player whose risky arm produces a success stops competing for the safe arm. We assume that a player observes her opponent's actions but not his realised payoffs. She is therefore never certain whether her opponent is still competing for the safe arm. When the players' prior probabilities of success are sufficiently close, there exists no pure strategy equilibrium, and we characterise the unique mixed strategy equilibrium. The amount of experimentation performed in equilibrium is inefficiently low but, for many priors, higher than if successes are publicly observed.

Two-Armed Restless Bandits with Imperfect Information: Stochastic Control and Indexability

We present a two-armed bandit model of decision making under uncertainty where the expected return to investing in the “risky arm” increases when choosing that arm and decreases when choosing the “safe” arm. These dynamics are natural in applications such as human capital development, job search, and occupational choice. Using new insights from stochastic control, along with a monotonicity condition on the payoff dynamics, we show that optimal strategies in our model are stopping rules that can be characterized by an index which formally coincides with Gittins’ index. Our result implies the indexability of a new class of restless bandit models.

Strategic Experimentation with Erlang Bandits

Risks related to events that arrive randomly play important role in many real life decisions, and models of learning and experimentation based on two-armed Poisson bandits addressed several important aspects related to strategic and motivational learning in cases when events arrive at jump times of the standard Poisson process. At the same time, these models remain mostly abstract theoretical models with few direct economic applications. We suggest a new class of models of strategic experimentation which are almost as tractable as exponential models, but incorporate such realistic features as dependence of the expected rate of news arrival on the time elapsed since the start of an experiment and judgement about the quality of a risky arm based on evidence of a series of trials as opposed to a single evidence of success or failure as in exponential models with conclusive experiments. We demonstrate that, unlike in the exponential models, players may stop experimentation before the first failure happens. Moreover, ceteris paribus, experimentation in a model with breakthroughs may last longer than experimentation in the corresponding model with failures.

A Risky Arms Race in Space.

A Risky Arms Race in Space.

Strategic experimentation with private payoffs

We consider a game of strategic experimentation in which players face identical discrete-time bandit problems with a safe and a risky arm. In any period, the risky arm yields either a success or a failure, and the first success reveals the risky arm to dominate the safe one. When payoffs are public information, the ensuing free-rider problem is so severe that equilibrium experimentation ceases at the same threshold belief at which a single agent would stop, even if players can coordinate their actions through mediated communication. When payoffs are private information and the success probability on the risky arm is not too high, however, the socially optimal symmetric experimentation profile can be supported as a sequential equilibrium for sufficiently optimistic prior beliefs, even if players can only communicate via binary cheap-talk messages.

Strategic learning in teams

This paper analyzes a two-player game of strategic experimentation with three-armed exponential bandits in continuous time. Players play bandits of identical types, with one arm that is safe in that it generates a known payoff, whereas the likelihood of the risky armsʼ yielding a positive payoff is initially unknown. When the types of the two risky arms are perfectly negatively correlated, the efficient policy is an equilibrium if and only if the stakes are high enough. If the negative correlation is imperfect and stakes are high, there exists an equilibrium that leads to efficiency for optimistic enough prior beliefs.

On games of strategic experimentation

We study a class of symmetric strategic experimentation games. Each of two players faces an (exponential) two-armed bandit problem, and must decide when to stop experimenting with the risky arm. The equilibrium amount of experimentation depends on the degree to which experimentation outcomes are observed, and on the correlation between the two individual bandit problems. When experimentation outcomes are public, the game is basically one of strategic complementarities. When experimentation decisions are public, but outcomes are private, the strategic interaction is more complex. We fully characterize the equilibrium behavior in both informational setups, leading to a clear comparison between the two. In particular, equilibrium payoffs are higher when equilibrium outcomes are public.

Bandit Problems with Lévy Processes

Bandit problems model the trade-off between exploration and exploitation in various decision problems. We study two-armed bandit problems in continuous time, where the risky arm can have two types: High or Low; both types yield stochastic payoffs generated by a Lévy process. We show that the optimal strategy is a cut-off strategy and we provide an explicit expression for the cut-off and for the optimal payoff.

Strategic Experimentation with Private Payoffs

We consider a game of strategic experimentation in which players face identical discrete-time bandit problems with a safe and a risky arm. In any period, the risky arm yields either a success or a failure, and the first success reveals the risky arm to dominate the safe one. When payoffs are public information, the ensuing free-rider problem is so severe that equilibrium experimentation ceases at the same threshold belief at which a single agent would stop, even if players can coordinate their actions through mediated communication. When payoffs are private information and the success probability on the risky arm is not too high, however, the socially optimal symmetric experimentation profile can be supported as a perfect Bayesian equilibrium for sufficiently optimistic prior beliefs, even if players can only communicate via binary cheap-talk messages.

Negatively Correlated Bandits

We analyse a two-player game of strategic experimentation with two-armed bandits. Either player has to decide in continuous time whether to use a safe arm with a known pay-off or a risky arm whose expected pay-off per unit of time is initially unknown. This pay-off can be high or low and is negatively correlated across players. We characterize the set of all Markov perfect equilibria in the benchmark case where the risky arms are known to be of opposite type and construct equilibria in cut-off strategies for arbitrary negative correlation. All strategies and pay-offs are in closed form. In marked contrast to the case where both risky arms are of the same type, there always exists an equilibrium in cut-off strategies, and there always exists an equilibrium exhibiting efficient long-run patterns of learning. These results extend to a three-player game with common knowledge that exactly one risky arm is of the high pay-off type.