Involvement of the variance in the UCB algorithm regarding risk aversion and the regret bound

Abstract

The classical Upper Confidence Bound (UCB) algorithm implemented overestimates of the true mean of reward distributions based on the sample mean and the number of times such arms were chosen to decide the best arm. In this case, the variances, the essential components of any distributions, of reward distributions are dismissed. Moreover, in real-world applications, arms with relatively high means and small variances sometimes are preferable to arms with the highest mean and large variance. Such concerns are considered risk-aversion in the Multi-Armed Bandits algorithm. Additionally, since the combination of the variance and mean could help estimate the range of the distribution, proper utilization of the sample variance might help people to construct a tighter upper confidence bound to perform the UCB algorithm, leading to a smaller regret. In this paper, the author investigates and summarizes the current algorithm regarding risk aversion based on the estimations and implementations of variances in constructing the upper confidence bound in the UCB algorithm.