An End-to-end Hierarchical Reinforcement Learning Framework for Large-scale Dynamic Flexible Job-shop Scheduling Problem

Abstract

The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the modern manufacturing industry. As the intelligent manufacturing paradigm evolves, it is urgent to design a dynamic scheduling framework for handling the uncertainty and complexity in the real-time production line control. DFJSP aims to dynamically address new job random arrival so that the desired objective could be optimized, such as the minimization of makespan. The intractability of this problem can be directly reflected by the following two points. 1) the arrival times of new jobs are unknown in advance, so the scheduling framework needs to schedule them in real-time optimization; 2) Two optimization tasks, including job operation selection and machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. This paper proposes a novel end-to-end hierarchical reinforcement learning framework to cope with the large-scale DFJSP. For generality, a higher-level layer is designed to automatically divide the DFJSP into a series of sub-problems with different scales, i.e., static FJSPs, which aims to achieve global optimization. Moreover, two lower-level layers are constructed to efficiently solve the sub-problems generated from the higher-level layer. One layer's policy based on the graph neural network is trained to schedule a job operation, and another policy based on the multi-layer perceptron is trained to assign a machine to process the job operation. Especially a Markov decision process (MDP), including state, action, and reward function, is designed for each layer of the hierarchy. Numerical experiments, including offline training and online testing, are conducted on several large-scale instances with diverse production configurations. The results verify the effectiveness of the proposed framework compared with the existing dynamic scheduling methods such as well-known dispatching rules and the existing heuristics.