Multi-agent Dual Level Reinforcement Learning of Strategy and Tactics in Competitive Games

Abstract

Reinforcement learning has been used extensively to learn the low-level tactical choices during gameplay; however, less effort is invested in the strategic decisions governing the effective engagement of a diverse set of opponents. In this paper, a two-tier reinforcement learning model is created to play competitive games and effectively engage in matches with different opponents to maximize earnings. The multi-agent environment has four types of learners, which vary in their ability to learn gameplay directly (tactics) and their ability to learn to bet or withdraw from gameplay (strategy). The players are tested in three different competitive games: Connect 4, Dots and Boxes, and Tic-Tac-Toe. Analyzing the behavior of players as they progress from naivety to game mastery reveals some interesting features: (1) learners who optimize strategy and tactics outperform all learners, (2) learners who initially optimize their strategy to engage in matches outperform those who focus on optimizing tactical gameplay, and (3) the advantage of strategy optimization versus tactical gameplay optimization diminishes as more games are played. A reinforcement learning model with a dual learning scheme presents possible applications in adversarial scenarios where both strategic and tactical learning are critical. We present detailed results in a systematic manner, providing strong support for our claim.