Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

Abstract

In this paper, the fully distributed neural learning algorithms by neural network approximation for networked multi-agent reinforcement learning (NMARL) are studied. To tackle the convergence analysis of methods in NMARL with tremendous state-action space, most of the existing distributed algorithms are designed by linear function approximation, which however would fall into a situation of poor expression. To conquer such limitation, the distributed neural learning algorithms are developed by using a novel neural network approximation that bridges the theory and practice of deep multi-agent reinforcement learning (DMARL). Specifically, inspired by the overparametrization method for minimizing mean-squared projected bellman error (MSPBE), the distributed neural learning algorithms with population semigradients and stochastic semigradients are respectively proposed to solve the NMARL problem. Furthermore, the convergence of the proposed algorithms are strictly given by employing the overparametrization method to establish the approximate stationary point of MSPBE to characterizes the algorithms toward the global optimum. Finally, some numerical simulations demonstrate the effectiveness of the distributed neural learning algorithms.