Smart mobile robot fleet management based on hierarchical multi-agent deep Q network towards intelligent manufacturing

Abstract

With the advent of intelligent manufacturing era, smart mobile robots have taken the major roles on transporting materials through intelligent dynamic production environment. It is paramount to efficiently manage the mobile robot fleet to complete the material transportation task so as to facilitate the smooth production in the workshop. However, many mobile robot fleet management systems adopt centralized control where the interaction between mobile robots and other resources cannot be updated promptly, and the in-time dynamic environment variance cannot be considered. To cope with the unexpected real-time change of the system, reinforcement learning (RL) method is suggested to handle the problem. To overcome the sparse reward problem of RL, we propose a hierarchical multi-agent deep Q network (HMDQN) algorithm of a two-layer structure, in which the goal layer is controlled by a main controller for selecting a current goal and the sub controller from action layer is for coordinating multi-agent controlled smart mobile robots. The main controller is aimed at learning how to determine the current goal based on the order status and production states. Simultaneously, the sub controller is learning to seek an optimal way that the smart mobile robot fleet jointly executes the transportation tasks through the information exchange between robot agent and production environment under the current goal. A smart mobile robot fleet management case in a PCBA company is studied to validate the feasibility of our approach. In addition, we utilized alternative methods to solve the same problem and compared the performance to prove our approach’s superiority. Furthermore, we demonstrated the adaptability of the proposed method by changing the problem scales.