Comparative analysis and applications of classic multi-armed bandit algorithms and their variants

Abstract

The multi-armed bandit problem, a pivotal aspect of Reinforcement Learning (RL), presents a classic dilemma in sequential decision-making, balancing exploration with exploitation. Renowned bandit algorithms like Explore-Then-Commit, Epsilon-Greedy, SoftMax, Upper Confidence Bound (UCB), and Thompson Sampling have demonstrated efficacy in addressing this issue. Nevertheless, each algorithm exhibits unique strengths and weaknesses, necessitating a detailed comparative evaluation. This paper executes a series of implementations of various established bandit algorithms and their derivatives, aiming to assess their stability and efficacy. The study engages in empirical analysis utilizing a real dataset, generating charts and data for a thorough examination of the pros and cons associated with each algorithm. A significant aspect of the research focuses on the parameter sensitivity of these algorithms and the impact of parameter tuning on their performance. Findings reveal that the SoftMax algorithm's effectiveness is markedly influenced by the initial estimated mean reward value for each arm. Conversely, algorithms like Epsilon-Greedy and UCB exhibit enhanced performance with optimal parameter settings. Furthermore, the study explores the limitations inherent in classic bandit algorithms and introduces some innovative models and methodologies pertinent to the multi-armed bandit problem, along with their applications.