Performance variance in Multi-Armed Bandits: In-depth analysis of three core algorithms

Abstract

In real-time strategy games, players grapple with uncertainty regarding the best strategy for victory. This paper delves into multi-armed bandit (MAB) algorithms as potential solutions. The theoretical foundations of MAB are explored, with a focus on the crucial balance between exploration and exploitation. An experimental comparison of the Explore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS) algorithms is conducted, showcasing their varied performance. Beyond gaming, the paper also examines the broader applications of MAB algorithms in fields such as healthcare, finance, and dynamic pricing in online retail, highlighting their versatility. A significant portion of the study is dedicated to implementing the UCB and TS algorithms in StarCraft, a popular real-time strategy game. The performance of these algorithms is assessed by calculating cumulative regret values, a metric critical to understanding their effectiveness in decision-making contexts. The results indicate that both UCB and TS algorithms substantially improve players' win rates in StarCraft. However, the study acknowledges existing challenges and the need for further research in this area. The use of MAB algorithms in complex, dynamic environments like real-time strategy games presents a rich avenue for exploration and holds significant promise for enhancing decision-making strategies in diverse domains. This research, therefore, not only contributes to the understanding of MAB algorithms in gaming but also underscores their potential in various other sectors.