Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm

Abstract

Dynamic scheduling methods are essential and critical to manufacturing systems because of uncertain events in the production process, such as new job insertions, order cancellations, worker absences, and machine breakdowns. Emerging digital twin (DT) technology can help detect disturbances by continuously comparing physical space with virtual space and triggering a rescheduling policy immediately after a disturbance. This enables dynamic scheduling and greatly reduces the deviation between preschedules and actual schedules. This study focuses on a DT-enabled integrated optimisation problem of flexible job shop scheduling and flexible preventive maintenance (PM) considering both machine and worker resources. A double-layer Q-learning algorithm (DLQL) is designed as the underlying key optimisation method to simultaneously learn the selection process of machines and operations to achieve efficient real-time scheduling. The superior solution performance of DLQL was verified by comparing it with two well-known metaheuristic algorithms and a single-layer Q-learning algorithm under several benchmarks. Furthermore, different disturbance settings were designed to illustrate the DLQL-based dynamic scheduling process in detail. The proposed reinforcement learning (RL)-driven DT enables efficient collaborative scheduling between production and maintenance departments and helps manufacturing companies improve the real-time decision-making process under uncertain perturbations.