CESDQL: Communicative experience-sharing deep Q-learning for scalability in multi-robot collaboration with sparse reward

Abstract

Owing to the massive transformation in industrial processes and logistics, warehouses are also undergoing advanced automation. The application of Autonomous Mobile Robots (a.k.a. multi-robots) is one of the important elements of overall warehousing automation. The autonomous collaborative behaviour of the multi-robots can be considered as employment on a control task and, thus, can be optimised using multi-agent reinforcement learning (MARL). Consequently, an autonomous warehouse is to be represented by an MARL environment. An MARL environment replicating an autonomous warehouse poses the challenge of exploration due to sparse reward leading to inefficient collaboration. This challenge aggravates further with an increase in the number of robots and the grid size, i.e., scalability. This research proposes Communicative Experience-Sharing Deep Q-Learning (CESDQL) based on Q-learning, a novel hybrid multi-robot communicative framework for scalability for MARL collaboration with sparse rewards, where exploration is challenging and makes collaboration difficult. CESDQL makes use of experience-sharing through collective sampling from the Experience (Replay) buffer and communication through Communicative Deep recurrent Q-network (CommDRQN), a Q-function approximator. Through empirical evaluation of CESDQL in a variety of collaborative scenarios, it is established that CESDQL outperforms the baselines in terms of convergence and stable learning. Overall, CESDQL achieves 5%, 69%, 60%, 211%, 171%, 3.8% & 10% more final accumulative training returns than the closest performing baseline by scenario, and, 27%, 10.33% & 573% more final average training returns than the closest performing baseline by the big-scale scenario.