Learning to schedule dynamic distributed reconfigurable workshops using expected deep Q-network

Abstract

Distributed manufacturing has become common in the globalized production environment. Facing the tendency of mass customization, distributed workshops are required to have the reconfiguration capability to produce multiple products. Based on the scheduling demands of distributed workshops, this paper studied the dynamic distributed reconfigurable flowshop scheduling problem (DRFSP) with new job arrivals using deep reinforcement learning (DRL) for the first time. The DRL-based intelligent scheduling and reconfiguration systems are modeled by designing efficient rewards, actions, and state features for both scheduling and reconfiguration agents. A new state feature normalization approach is proposed to address the variability in production configurations and dynamics. A novel expected deep Q-network (EDQN) algorithm is proposed to provide more precise state-action values rather than using estimations. Moreover, the DRL-based training and execution procedures for the dynamic DRFSP are provided by combining decision-making, agent learning, and reconfigurable production. Training curves validate the reasonableness of system modeling and the good learning efficiency of the EDQN algorithm. Extensive comparison experiments show that our EDQN algorithm outperforms three popular DRL algorithms and four well-known priority dispatching rules (PDRs) by around 20 % and 89 % respectively, with good generalization ability and time efficiency. The average decision time under dynamics is only 0.65 ms, which enables real-time scheduling.