Large-scale dynamic surgical scheduling under uncertainty by hierarchical reinforcement learning

Abstract

Dynamic surgical scheduling within a workday is a complicated decision-making process. The critical challenge is that the actual duration of surgery and the arrival process of emergency patients are uncertain and unknown in advance. In this work, we propose a two-level dynamic scheduling framework based on hierarchical reinforcement learning to solve dynamic surgical scheduling problems considering both elective and emergency patients. Specifically, with the realisation of uncertainty, the upper-level agent (UA) dynamically decides whether to trigger rescheduling to optimise the workday total cost. The lower-level agent (LA) aims at obtaining subscheduling solutions when rescheduling is triggered. The subproblem at the LA can be formulated as a mixed integer programming model, which can be generalised to unrelated parallel machine scheduling with machine eligibility restrictions and sequence- and machine-dependent setup times. Such problems can be solved in small-scale cases and suffers the combinatorial explosion in large scale cases. To address this issue, we propose a heuristic method that is built upon deep reinforcement learning to obtain high-quality solutions. We conduct extensive simulation experiments with real data to test the effective of our framework. The results for different scenarios show that our proposed framework outperforms existing methods in terms of overall performance and has strong generalisation ability.