Best Arm Identification for Both Stochastic and Adversarial Multi-armed Bandits

Abstract

We study the problem of best arm identification in multi-armed bandits, where the underlying setting can be either stochastic or adversarial and is not revealed to the forecaster a priori. We propose an algorithm, called S3-BA, whose key idea is to simultaneously explore the arms and learn the problem model, and switch when adversarial is declared. We analyze the performance of the proposed algorithm for both stochastic and adversarial bandits. For stochastic bandits we prove an upper bound on the error rate of S3-BA that is on the same order as UCB-E and SH. For adversarial bandits we show that the best arm can be guaranteed as long as the budget is above a threshold. More importantly, we prove that the “learning loss” due to the formulation uncertainty is negligible compared to the arm selection policies, for both stochastic and adversarial bandits. The performance of S3-BA is evaluated using the real-world US stock market dataset, and it is observed that it outperforms existing algorithms designed for either stochastic or adversarial best arm identification problems.