Optimal strategy for Bayesian two-armed bandit problem with an arched reward function

Abstract

In the classical two-armed bandit (TAB) problems, the goal of maximizing the total return specifies the optimal decision rule and fosters the development of abundant practicable methods, such as $ \epsilon $-greedy and upper confidence bound for TAB procedures. However, the existence of an arched reward function for the total return breaks the previously updated algorithms, like traditional myopic strategy, and prompts the generation of novel computation and theory for maximizing its expectation. Here we develop a special class of Bayesian two-armed bandit problems by imposing a prior probability on which arm has a greater expectation of the returns and propose a myopic strategy to specify the decision rule. We prove that the proposed myopic strategy is optimal under the structure of the arched reward function by exploring the dynamic programming principle. This article also demonstrates that arched functions are extensions of the monotone functions including the classical linear functions. Meanwhile, we establish a corresponding law of large numbers for our myopic strategy. Simulation studies pose supportive evidence that the newly proposed strategy performs well.