Empirically evaluating multiagent reinforcement learning algorithms in repeated games

Abstract

This dissertation presents a platform for running experiments on multiagent reinforcement learning algorithms and an empirical evaluation that was conducted on the platform. The setting under consideration is game theoretic in which a single normal form game is repeatedly played. There has been a large body of work focusing on introducing new algorithms to achieve certain goals such as guaranteeing values in a game, converging to a Nash equilibrium or minimizing total regret. Currently, we have an understanding of how some of these algorithms work in limited settings, but lack a broader understanding of which algorithms perform well against each other and how they perform on a larger variety of games. We describe our development of a platform that allows large scale tests to be run, where multiple algorithms are played against one another on a variety of games. The platform has a set of builtin metrics that can be used to measure the performance of an algorithm, including convergence to a Nash equilibrium, regret, reward and number of wins. Visualising the results of the test can be automatically achieved through the platform, with all interaction taking place through graphical user interfaces. We also present the results of an empirical test that to our knowledge includes the largest combination of game instances and algorithms used in the multiagent learning literature. To demonstrate the usefulness of the platform, we provide evidence for a number of claims and hypotheses. This includes claims related to convergence to a Nash equilibrium, reward, regret and best response metrics and claims dealing with estimating an opponent’s strategy. Some of our claims include that (1) no algorithm does best across all metrics and over all opponents, (2) algorithms do not often converge to an exact Nash equilibrium, but (3) do often reach a small window around a Nash equilibrium, (4) there is no apparent link between converging to a Nash equilibrium and obtaining high reward and (5) there is no linear trend between reward and the size of the game for any agent. The two major contributions of this work are a software platform for running large experimental tests and empirical results that provide insight into the performance of various algorithms.