Decentralized Adaptive Optimal Tracking Control for Massive Multi-agent Systems: An Actor-Critic-Mass Algorithm

Abstract

In this paper, decentralized optimal tracking control problem has been studied for multi-agent systems (MAS) with a large population, i.e., massive MAS. Due to the curse of dimensionality and limited communication resource, solving optimal tracking control for massive multi-agent systems in a decentralized manner is much more preferred but also challenging. Therefore, the emerging Mean Field game theory has been adopted and integrated with online reinforcement learning approaches and further produced a novel Actor-Critic-Mass algorithm. In the developed scheme, each agent has three neural networks (NN), i.e., 1) mass neural network (NN) that learned the MAS large population behaviors, 2) critic NN that estimated optimal cost function by using local information and mass behaviors learned from mass NN, and 3) actor NN that online solved the decentralized adaptive optimal tracking control based on information obtained from mass NN and critic NN. According to mean-field game theory, the Hamiltonian-Jacobian-Bellman (HJB) and Fokker-Planck-Kolmogorov (FPK) equations are two key components that can be used for tuning actor, critic, and mass NNs effectively. Moreover, Lyapunov theory is used to prove that all the closed-loop signals and NN weights are uniformly bounded in the meanwhile the approximated control input converges close to its near optimal cost with time. Eventually, a series of comprehensive simulation demonstrated the effectiveness of the proposed framework.