Hierarchical Relational Graph Learning for Autonomous Multirobot Cooperative Navigation in Dynamic Environments

Abstract

As a specific kind of Cyber-Physical Systems (CPSs), autonomous robot clusters play an important role in various intelligent manufacturing fields. However, due to the increasing design complexity of robot clusters, it is becoming more and more challenging to guarantee the safety and efficiency for multi-robot cooperative navigation in dynamic and complex environments. Although Deep Reinforcement Learning (DRL) shows great potential in learning multi-robot cooperative navigation policies, existing DRL-based approaches suffer from scalability issues and rarely consider the transferability of trained policies to new tasks. To address these problems, this paper presents a novel DRL-based multi-robot cooperative navigation approach named HRMR-Navi that equips each robot with both a two-layered hierarchical graph network model and an attention-based communication model. In our approach, the hierarchical graph network model can efficiently figure out hierarchical relations among all agents that either cooperate for efficiency or avoid obstacles for safety to derive more advanced strategies, and the communication model can accurately form a global view of the environment for a specific robot, thus the multi-robot cooperation efficiency can be further strengthened. Meanwhile, we propose an improved Proximal Policy Optimization (PPO) algorithm based on the Maximum Entropy reinforcement learning, named MEPPO, to enhance the robot exploration ability. Comprehensive experimental results demonstrate that, compared with state-of-the-art approaches, HRMR-Navi can achieve more efficient cooperative navigation with less time cost, lower collision rate, higher scalability, and better knowledge transferability.