Efficient multi-agent cooperation: Scalable reinforcement learning with heterogeneous graph networks and limited communication

Abstract

This paper addresses the challenge of scalable multi-agent reinforcement learning (MARL) under partial observability and communication constraints. An Efficient Multi-Agent Cooperation (EMAC) approach is proposed, which comprises a Heterogeneous Graph Network (HGN) and a Frame-wise Communication Reduction Algorithm (FCRA). HGN can effectively extract semantic information from local observations, while FCRA utilizes historical embeddings to achieve an approximation of multi-hop communication with 1-hop communication. Through comparative experiments, EMAC showed a 30% improvement in efficiency over the best-performing benchmark, scalability to arbitrary numbers of agents, and adaptation to unstructured environments. A t-SNE analysis of agent embeddings further reveals EMAC’s capability to understand and leverage semantic similarities among agents. These results highlight EMAC’s potential as a robust framework for decentralized multi-agent decision-making in complex and dynamic environments.