Multi-armed Bandit with Time-variant Budgets

Abstract

Based on the traditional paradigm of multi-armed bandit with a fixed total budget, we further investigate the multi-armed bandit with time-variant budgets (MAB-TV), which captures the situation where the learner's actions are constrained by a random budget in each round of arm-pulling. Besides, we assume that the learner needs to pay a random cost for pulling a particular arm and could obtain a random reward accordingly. The learner can pull multiple arms in each round, as long as the corresponding random budget is satisfied. To solve MAB-TV, we design a Greedy-TV algorithm. During each round, we first estimate the real average values of the reward and the cost of each arm, and then under the budget constraint, we pull some arms according to the estimated ratio of the average reward to the average cost regarding each arm. We derive the regret bound of Greedy-Tvregarding the achievable sum of rewards. Evaluations results validate the superiority of Greedy-TV, compared with three existing benchmark algorithms under varying distributions of rewards and costs and the different number of arms.