Dynamic Job Shop Scheduling Problem Research Articles

A deep reinforcement learning model for dynamic job-shop scheduling problem with uncertain processing time

The dynamic job-shop scheduling problem (DJSP) is a type of scheduling tasks where rescheduling is performed when encountering the uncertainties such as the uncertain operation processing time. However, the current deep reinforcement learning (DRL) scheduling approaches are hard to train convergent scheduling policies as the problem scale increases, which is very important for rescheduling under uncertainty. In this paper, we propose a DRL scheduling method for DJSP based on the proximal policy optimization (PPO) with hybrid prioritized experience replay. The job shop scheduling problem is formulated as a sequential decision-making problem based on Markov Decision Process (MDP) where a novel state representation is designed based on the feasible solution matrix which depicts the scheduling order of a scheduling task, a set of paired priority dispatching rules (PDR) are used as the actions and a new intuitive reward function is established based on the machine idle time. Moreover, a new hybrid prioritized experience replay method for PPO is proposed to reduce the training time where samples with positive temporal-difference (TD) error are replayed. Static experiments on classic benchmark instances show that the make-span obtained by our scheduling agent has been reduced by 1.59% on average than the best known DRL methods. In addition, dynamic experiments demonstrate that the training time of the reused scheduling policy is reduced by 27% compared with the retrained policy when encountering uncertainties such as uncertain operation processing time.

Dynamic Job-Shop Scheduling Based on Transformer and Deep Reinforcement Learning

The dynamic job-shop scheduling problem is a complex and uncertain task that involves optimizing production planning and resource allocation in a dynamic production environment. Traditional methods are limited in effectively handling dynamic events and quickly generating scheduling solutions; in order to solve this problem, this paper proposes a solution by transforming the dynamic job-shop scheduling problem into a Markov decision process and leveraging deep reinforcement learning techniques. The proposed framework introduces several innovative components, which make full use of human domain knowledge and machine computing power, to realize the goal of man–machine collaborative decision-making. Firstly, we utilize disjunctive graphs as the state representation, capturing the complex relationships between various elements of the scheduling problem. Secondly, we select a set of dispatching rules through data envelopment analysis to form the action space, allowing for flexible and efficient scheduling decisions. Thirdly, the transformer model is employed as the feature extraction module, enabling effective capturing of state relationships and improving the representation power. Moreover, the framework incorporates the dueling double deep Q-network with prioritized experience replay, mapping each state to the most appropriate dispatching rule. Additionally, a dynamic target strategy with an elite mechanism is proposed. Through extensive experiments conducted on multiple examples, our proposed framework consistently outperformed traditional dispatching rules, genetic algorithms, and other reinforcement learning methods, achieving improvements of 15.98%, 17.98%, and 13.84%, respectively. These results validate the effectiveness and superiority of our approach in addressing dynamic job-shop scheduling problems.

Dynamic scheduling for job shop with machine failure based on data mining technologies

PurposeInfluenced by the constantly changing manufacturing environment, no single dispatching rule (SDR) can consistently obtain better scheduling results than other rules for the dynamic job-shop scheduling problem (DJSP). Although the dynamic SDR selection classifier (DSSC) mined by traditional data-mining-based scheduling method has shown some improvement in comparison to an SDR, the enhancement is not significant since the rule selected by DSSC is still an SDR.Design/methodology/approachThis paper presents a novel data-mining-based scheduling method for the DJSP with machine failure aiming at minimizing the makespan. Firstly, a scheduling priority relation model (SPRM) is constructed to determine the appropriate priority relation between two operations based on the production system state and the difference between their priority values calculated using multiple SDRs. Subsequently, a training sample acquisition mechanism based on the optimal scheduling schemes is proposed to acquire training samples for the SPRM. Furthermore, feature selection and machine learning are conducted using the genetic algorithm and extreme learning machine to mine the SPRM.FindingsResults from numerical experiments demonstrate that the SPRM, mined by the proposed method, not only achieves better scheduling results in most manufacturing environments but also maintains a higher level of stability in diverse manufacturing environments than an SDR and the DSSC.Originality/valueThis paper constructs a SPRM and mines it based on data mining technologies to obtain better results than an SDR and the DSSC in various manufacturing environments.

A spatial pyramid pooling-based deep reinforcement learning model for dynamic job-shop scheduling problem

The dynamic job-shop scheduling problem (DJSP) is a typical of scheduling tasks where rescheduling is performed when encountering unexpected events such as random job arrivals and rush order. However, the current rescheduling approaches cannot reuse the trained scheduling policies or the experiences due to the variant size of scheduling problems. In this paper, we propose a deep reinforcement learning (DRL) scheduling model for DJSP based on spatial pyramid pooling networks (SPP-Net). A new state representation is proposed based on the machine matrix and remaining time matrix which is decomposed from the scheduling instance matrix. And a new reward function is derived from the area of total scheduling time where the accumulated reward is negatively linearly dependent with the make-span of a scheduling task. Moreover, a size-agnostic scheduling policy is designed based on the SPP-Net and SoftMax function, which is trained by the proximal policy optimization (PPO). Besides, various paired priority dispatching rules (PDR) are used as available actions. Static experiments on classic benchmark instances show that our scheduling model achieves better results on average than existing DRL methods. In addition, dynamic scheduling experiments are tested and our model obtains better results than the PDR scheduling methods in reasonable time when encountering unexpected events such as random job arrivals and rush order.

A data-driven simulation-optimization framework for generating priority dispatching rules in dynamic job shop scheduling with uncertainties

Modeling and optimizing dynamic job shop scheduling problems (DJSSP) without ample assumptions is inherently challenging due to the increasing complexity and uncertainty arising from various disruptive events. Moreover, establishing job due dates in dynamic job shop environments has become increasing difficult due to unforeseen events, while it remains a pivotal factor in determining tardiness-related objectives. A research gap exists in exploring the potential synergy between the two aforementioned aspects and its resulting implications. In this study, we propose a novel framework that combines data-driven simulation, machine learning, and evolutionary algorithm to generate new Priority Dispatching Rules (PDRs) for optimizing makespan and total tardiness. The framework incorporates a data-exchange mechanism for real-time updates of the simulation model based on the shop floor. Moreover, an online due date prediction model using XGBoost is developed, replacing the previous rough estimation of the Total Work Content (TWK) rule. Simulation experiments of various scales demonstrate that XGBoost-assisted due date prediction leads to an average reduction of 1.6% in makespan and 65.0% in total tardiness compared to the TWK rule. The proposed method outperforms existing well-known dispatching rules, yielding considerable improvements with average reductions of 90.8%, 40.3%, and 44.6% in total tardiness for vary-sized experiments. Finally, the approach is validated through a real industrial case study.

Evolution strategies-based optimized graph reinforcement learning for solving dynamic job shop scheduling problem

The job shop scheduling problem (JSSP) with dynamic events and uncertainty is a strongly NP-hard combinatorial optimization problem (COP) with extensive applications in the manufacturing system. Recently, growing interest has been aroused in utilizing machine learning techniques to solve the JSSP. However, most prior arts cannot handle dynamic events and barely consider uncertainties. To close this gap, this paper proposes a framework to solve a dynamic JSSP (DJSP) with machine breakdown and stochastic processing time based on Graph Neural Network (GNN) and deep reinforcement learning (DRL). To this end, we first formulate the DJSP as a Markov Decision Process (MDP), where disjunctive graph represent the states. Secondly, we propose a GNN-based model to effectively extract the embeddings of the state by considering the features of the dynamic events and the stochasticity of the problem, e.g., the machine breakdown and stochastic processing time. Then, the model constructs solutions by dispatching optimal operations to machines based on the learned embeddings. Notably, we propose to use the evolution strategies (ES) to find optimal policies that are more stable and robust than conventional DRL algorithms. The extensive experiments show that our method substantially outperforms existing reinforcement learning-based and traditional methods on multiple classic benchmarks.

A deep multi-agent reinforcement learning approach to solve dynamic job shop scheduling problem

Manufacturing industry is experiencing a revolution in the creation and utilization of data, the abundance of industrial data creates a need for data-driven techniques to implement real-time production scheduling. However, existing dynamic scheduling techniques have been mainly developed to solve problems of invariable size, and are incapable of addressing the increasing volatility and complexity of practical production scheduling problems. To facilitate near real-time decision-making on the shop floor, we propose a deep multi-agent reinforcement learning-based approach to solve the dynamic job shop scheduling problem. Double deep Q-network algorithm, attached to decentralized scheduling agents, is used to learn the relationships between production information and scheduling objectives, and to make near real-time scheduling decisions. Proposed framework utilizes centralized training and decentralized execution scheme and parameter-sharing technique to tackle the non-stationary problem in the multi-agent reinforcement learning task. Several enhancements are also developed, including the novel state and action representation that can handle size-agnostic dynamic scheduling problems, a chronological joint-action framework to alleviate the credit-assignment difficulty, and knowledge-based reward-shaping techniques to encourage cooperation. Simulation study shows that the proposed architecture significantly improves the learning effectiveness, and delivers superior performance compared to existing scheduling strategies and state-of-the-art deep reinforcement learning-based dynamic scheduling approaches.

Combining Reinforcement Learning Algorithms with Graph Neural Networks to Solve Dynamic Job Shop Scheduling Problems

Smart factories have attracted a lot of attention from scholars for intelligent scheduling problems due to the complexity and dynamics of their production processes. The dynamic job shop scheduling problem (DJSP), as one of the intelligent scheduling problems, aims to make an optimized scheduling decision sequence based on the real-time dynamic job shop environment. The traditional reinforcement learning (RL) method converts the scheduling problem with a Markov process and combines its own reward method to obtain scheduling sequences in different real-time shop states. However, the definition of shop states often relies on the scheduling experience of the model constructor, which undoubtedly affects the optimization capability of the reinforcement learning model. In this paper, we combine graph neural network (GNN) and deep reinforcement learning (DRL) algorithm to solve DJSP. An agent model from job shop state analysis graph to scheduling rules is constructed, thus avoiding the problem that traditional reinforcement learning methods rely on scheduling experience to artificially set the state feature vectors. In addition, a new reward function is defined, and the experimental results prove that our proposed reward method is more effective. The effectiveness and feasibility of our model is demonstrated by comparing with general deep reinforcement learning algorithms on minimizing the earlier and later completion time, which also lays the foundation for solving the DJSP later.

An Efficient Metaheuristic Algorithm for Job Shop Scheduling in a Dynamic Environment

This paper proposes an Improved Multi-phase Particle Swarm Optimization (IMPPSO) to solve a Dynamic Job Shop Scheduling Problem (DJSSP) known as an non-deterministic polynomial-time hard (NP-hard) problem. A cellular neighbor network, a velocity reinitialization strategy, a randomly select sub-dimension strategy, and a constraint handling function are introduced in the IMPPSO. The IMPPSO is used to solve the Kundakcı and Kulak problem set and is compared with the original Multi-phase Particle Swarm Optimization (MPPSO) and Heuristic Kalman Algorithm (HKA). The results show that the IMPPSO has better global exploration capability and convergence. The IMPPSO has improved fitness for most of the benchmark instances of the Kundakcı and Kulak problem set, with an average improvement rate of 5.16% compared to the Genetic Algorithm-Mixed (GAM) and of 0.74% compared to HKA. The performance of the IMPPSO for solving real-world problems is verified by a case study. The high level of operational efficiency is also evaluated and demonstrated by proposing a simulation model capable of using the decision-making algorithm in a real-world environment.

Dynamic production scheduling towards self-organizing mass personalization: A multi-agent dueling deep reinforcement learning approach

Mass personalization is rapidly approaching. In response, manufacturing systems should be capable of autonomously changing production plans, configurations and schedules under dynamic manufacturing environments for producing personalized products. Self-organizing manufacturing network is a promising paradigm for mass personalization. The backbone of a self-organizing manufacturing network is an adaptive production scheduling method to dynamically allocate and sequence manufacturing jobs under dynamic settings, such as stochastic processing time or unplanned machine breakdown. However, existing production scheduling methods (i.e., heuristic rules, meta-heuristic algorithms, and existing reinforcement learning models) fail to automatically optimize production schedules while maintaining stable manufacturing performance, under dynamic settings. In this paper, we designed a reinforcement learning-based static-training-dynamic-execution approach for dynamic job shop scheduling problems. The scheduling policies are learned from static scheduling instances by a multi-agent dueling deep reinforcement learning approach. Under this approach, we proposed new representations of observation, action, reward, and cooperation mechanisms between agents. The learned scheduling policies are then deployed to a dynamic scheduling system where stochastic processing time and unplanned machine breakdown randomly occur. Extensive simulation experiments demonstrated that our approach outperforms heuristic rules on makespan under two dynamic manufacturing settings.

A self-learning discrete salp swarm algorithm based on deep reinforcement learning for dynamic job shop scheduling problem

A self-learning discrete salp swarm algorithm based on deep reinforcement learning for dynamic job shop scheduling problem

An End-to-End Deep Learning Method for Dynamic Job Shop Scheduling Problem

Job shop scheduling problem (JSSP) is essential in the production, which can significantly improve production efficiency. Dynamic events such as machine breakdown and job rework frequently occur in smart manufacturing, making the dynamic job shop scheduling problem (DJSSP) methods urgently needed. Existing rule-based and meta-heuristic methods cannot cope with dynamic events in DJSSPs of different sizes in real time. This paper proposes an end-to-end transformer-based deep learning method named spatial pyramid pooling-based transformer (SPP-Transformer), which shows strong generalizability and can be applied to different-sized DJSSPs. The feature extraction module extracts the production environment features that are further compressed into fixed-length vectors by the feature compression module. Then, the action selection module selects the simple priority rule in real time. The experimental results show that the makespan of SPP-Transformer is 11.67% smaller than the average makespan of dispatching rules, meta-heuristic methods, and RL methods, proving that SPP-Transformer realizes effective dynamic scheduling without training different models for different DJSSPs. To the best of our knowledge, SPP-Transformer is the first application of an end-to-end transformer in DJSSP, which not only improves the productivity of industrial scheduling but also provides a paradigm for future research on deep learning in DJSSP.

Feature selection approach for evolving reactive scheduling policies for dynamic job shop scheduling problem using gene expression programming

Dispatching rules are one of the most widely applied methods for solving Dynamic Job Shop Scheduling problems (DJSSP) in real-world manufacturing systems. Hence, the automated design of effective rules has been an important subject in the scheduling literature for the past several years. High computational requirements and difficulty in interpreting generated rules are limitations of literature methods. Also, feature selection approaches in the field of automated design of scheduling policies have been developed for the tree-based GP approach only. Therefore, the aim of this study is to propose a feature selection approach for the Gene Expression Programming (GEP) algorithm to evolve high-quality rules in simple structures with an affordable computational budget. This integration speeds up the search process by restricting the GP search space using the linear representation of the GEP algorithm and creates concise rules with only meaningful features using the feature selection approach. The proposed algorithm is compared with five algorithms and 30 rules from the literature under different processing conditions. Three performance measures are considered including total weighted tardiness, mean tardiness, and mean flow time. The results show that the proposed algorithm can generate smaller rules with high interpretability in a much shorter training time.

Effective and interpretable dispatching rules for dynamic job shops via guided empirical learning

The emergence of Industry 4.0 is making production systems more flexible and also more dynamic. In these settings, schedules often need to be adapted in real-time by dispatching rules. Although substantial progress was made until the ’90s, the performance of these rules is still rather limited. The machine learning literature is developing a variety of methods to improve them. However, the resulting rules are difficult to interpret and do not generalise well for a wide range of settings. This paper is the first major attempt at combining machine learning with domain problem reasoning for scheduling. The idea consists of using the insights obtained with the latter to guide the empirical search of the former. We hypothesise that this guided empirical learning process should result in effective and interpretable dispatching rules that generalise well to different scenarios. We test our approach in the classical dynamic job shop scheduling problem minimising tardiness, one of the most well-studied scheduling problems. The simulation experiments include a wide spectrum of scenarios for the first time, from highly loose to tight due dates and from low utilisation conditions to severely congested shops. Nonetheless, results show that our approach can find new state-of-the-art rules, which significantly outperform the existing literature in the vast majority of settings. Overall, the average improvement over the best combination of benchmark rules is 19%. Moreover, the rules are compact, interpretable, and generalise well to extreme, unseen scenarios. Therefore, we believe that this methodology could be a new paradigm for applying machine learning to dynamic optimisation problems.

A novel feature selection for evolving compact dispatching rules using genetic programming for dynamic job shop scheduling

Because of advances in computational power and machine learning algorithms, the automated design of scheduling rules using Genetic Programming (GP) is successfully applied to solve dynamic job shop scheduling problems. Although GP-evolved rules usually outperform dispatching rules reported in the literature, intensive computational costs and rule interpretability persist as important limitations. Furthermore, the importance of features in the terminal set varies greatly among scenarios. The inclusion of irrelevant features broadens the search space. Therefore, proper selection of features is necessary to increase the convergence speed and to improve rule understandability using fewer features. In this paper, we propose a new representation of the GP rules that abstracts the importance of each terminal. Moreover, an adaptive feature selection mechanism is developed to estimate terminals’ weights from earlier generations in restricting the search space of the current generation. The proposed approach is compared with three GP algorithms from the literature and 30 human-made rules from the literature under different job shop configurations and scheduling objectives, including total weighted tardiness, mean tardiness, and mean flow time. Experimentally obtained results demonstrate that the proposed approach outperforms methods from the literature in generating more interpretable rules in a shorter computational time without sacrificing solution quality.

Joint optimisation for dynamic flexible job-shop scheduling problem with transportation time and resource constraints

Dynamic flexible job-shop scheduling is traditionally a challenge in real-world manufacturing systems, especially considering the constraints of transportation resources and transportation time. To address the dynamic optimisation problem in flexible manufacturing systems, this paper proposes a novel proactive-reactive methodology to adapt to the dynamic changes in working environments and addresses the joint scheduling problem for machine tools and transportation resources. The joint optimisation model is first formulated as a mixed-integer programming model considering production efficiency and transportation constraints. The flowchart of the dynamic scheduling system is then designed for dynamic decision-making, and a novel particle swarm optimisation algorithm integrated with genetic operators is developed to respond to dynamic events and generate the reschedule plan in time. Finally, several numerical experiments and case studies in reality are applied to verify the efficiency of the developed methodology. Common dispatching rules and heuristic methods are also applied to test and evaluate the efficiency of the developed algorithm. Computational results demonstrate that the developed methods and decision models are efficient for dynamic job-shop scheduling problems in flexible manufacturing systems, which can acquire rather a good effect in practical production.

Genetic programming-based hyper-heuristic approach for solving dynamic job shop scheduling problem with extended technical precedence constraints

Extended technical precedence constraints (ETPC) in dynamic job shop scheduling problem (DJSP) are the precedence constraints existing between different jobs instead of the conventional technical precedence constraints existing in the operations of the same job. This paper presents the mathematical programming model of the DJSP with ETPC to minimize the mean weighted tardiness of the jobs. The mathematical model contributes to the solution and modelling of the DJSP with ETPC and it is used to solve small-sized problems to optimality. To solve industry-sized problems, a constructive heuristic called the dispatching rule (DR) is employed. This paper investigates the use of genetic programming (GP) as a hyper-heuristic in the automated generation of the problem-specific DRs for solving the problem under consideration. The genetic programming-based hyper heuristic (GPHH) approach constructs the DRs which are learned from the training instances and then verified on the test instances by the simulation experiments. To enhance the efficiency of the approach when evolving effective DRs to solve the problem, the approach is improved with strategies which consist of a problem-specific attribute selection for GP and a threshold condition mechanism for fitness evaluation. The simulation results verify the effectiveness and efficiency of the evolved DRs to the problem under consideration by comparing against the existing classical DRs. The statistical analysis of the simulation results shows that the evolved DRs outperform the selected benchmark DRs on the problem under study. The sensitivity analysis also shows that the DRs generated by the GPHH approach are robust under different scheduling performance measures. Moreover, the effects of the model parameters, including the percentage of jobs with ETPC and the machine utilization, on the performance of the DRs are investigated.

Advanced Metaheuristic Method for Decision-Making in a Dynamic Job Shop Scheduling Environment

As a well-known NP-hard problem, the dynamic job shop scheduling problem has significant practical value, so this paper proposes an Improved Heuristic Kalman Algorithm to solve this problem. In Improved Heuristic Kalman Algorithm, the cellular neighbor network is introduced, together with the boundary handling function, and the best position of each individual is recorded for constructing the cellular neighbor network. The encoding method is introduced based on the relative position index so that the Improved Heuristic Kalman Algorithm can be applied to solve the dynamic job shop scheduling problem. Solving the benchmark example of dynamic job shop scheduling problem and comparing it with the original Heuristic Kalman Algorithm and Genetic Algorithm-Mixed, the results show that Improved Heuristic Kalman Algorithm is effective for solving the dynamic job shop scheduling problem. The convergence rate of the Improved Heuristic Kalman Algorithm is reduced significantly, which is beneficial to avoid the algorithm from falling into the local optimum. For all 15 benchmark instances, Improved Heuristic Kalman Algorithm and Heuristic Kalman Algorithm have obtained the best solution obtained by Genetic Algorithm-Mixed. Moreover, for 9 out of 15 benchmark instances, they achieved significantly better solutions than Genetic Algorithm-Mixed. They have better robustness and reasonable running time (less than 30 s even for large size problems), which means that they are very suitable for solving the dynamic job shop scheduling problem. According to the dynamic job shop scheduling problem applicability, the integration-communication protocol was presented, which enables the transfer and use of the Improved Heuristic Kalman Algorithm optimization results in the conventional Simio simulation environment. The results of the integration-communication protocol proved the numerical and graphical matching of the optimization results and, thus, the correctness of the data transfer, ensuring high-level usability of the decision-making method in a real-world environment.

An improved deep reinforcement learning approach for the dynamic job shop scheduling problem with random job arrivals

Deep reinforcement learning (DRL) method is a powerful way to solve the dynamic job shop scheduling problems (DJSSP). However, these DRL approaches are dispatching rules-based, meaning they are problem-specific, dependent on experience, and code effort. We propose a double loop deep Q-network (DLDQN) method with exploration loop and exploitation loop to solve DJSSP under random job arrivals, aiming to minimize the makespans of DJSSP. Simultaneously, by integrating into a single agent scheduling system, the proposed method could avoid complicated dispatching rules, enhancing the proposed method’s versatility. The experiment results have confirmed the superiority of our method compared to other algorithms.

A dynamic job-shop scheduling model based on deep learning

Ideally, the solution to job-shop scheduling problem (JSP) should effectively reduce the cost of manpower and materials, thereby enhancing the core competitiveness of the manufacturer. Deep learning (DL) neural networks have certain advantages in handling complex dynamic JSPs with a massive amount of historical data. Therefore, this paper proposes a dynamic job-shop scheduling model based on DL. Firstly, a data prediction model was established for dynamic job-shop scheduling, with long short-term memory network (LSTM) as the basis; the Dropout technology and adaptive moment estimation (ADAM) were introduced to enhance the generalization ability and prediction effect of the model. Next, the dynamic JSP was described in details, and three objective functions, namely, maximum makespan, total device load, and key device load, were chosen for optimization. Finally, the multi-objective problem of dynamic JSP scheduling was solved by the improved multi-objective genetic algorithm (MOGA). The effectiveness of the algorithm was proved experimentally.