Initial state diversification for efficient AlphaZero-style training

Abstract

AlphaZero has achieved superhuman performance in Go, chess, and shogi with a general reinforcement learning (RL) algorithm. This achievement is remarkable because AlphaZero does not rely on any training dataset of strong players. However, AlphaZero-style training requires substantial computational resources. Gumbel AlphaZero, a recently introduced more efficient version of AlphaZero, reduces the computational cost of AlphaZero training. The goal of this study is to further improve the playing strength of Gumbel AlphaZero under a limited amount of computational resources. We focus on the diversity in training games, inspired by procedural generation and domain randomization in RL studies, and propose a novel method, initial state diversification. This method diversifies the initial states of a self-play game to encourage the RL agent to understand the game in a more general manner through diverse experiences. For example, in shogi, the initial state of each self-play game is diversified by rearranging the pieces under realistic domain constraints. Experiments demonstrated that training with initial state diversification improves the playing strength of Gumbel AlphaZero in shogi, within the same computational budget for training.