A centralised training algorithm with D3QN for scalable regular unmanned ground vehicle formation maintenance

Abstract

The unmanned ground vehicle (UGV) has been widely used to accomplish various missions in civilian or military environments. Formation of the UGVs group is an important technique to support the broad applications of multi-functional UGVs. This study proposes a scalable regular UGV formation maintenance (SRUFM) algorithm based on deep reinforcement learning (DRL), which aims to use a unified DRL framework to improve the lateral control and longitudinal control performance of UGV in different situations of the formation. Based on the tailored design of state, action space, related inter-vehicle information, and the dueling double deep Q-network (D3QN), SRUFM outperforms nature deep Q-network and double deep Q-network in the exploration efficiency and the convergence speed in same CARLA training environments with fixed size formation. Furthermore, when the formation's scale is extended with similar initialisation conditions, the SRUFM can still get a nearly 90% success rate to execute all experiment formation maintenance missions after 4000 episodes of training. Each UGV in the formation can keep distance within the upper and lower error threshold of 0.15 m. The simulation experiments show that the proposed centralised training frame with D3QN is suitable to solve scalable regular UGV formation maintenance missions.