Dynamic Job Shop Research Articles

Probing an LSTM-PPO-Based reinforcement learning algorithm to solve dynamic job shop scheduling problem

With the growth of personalized demand and the continuous improvement in social productivity, the large-scale and few-variety centralized production model is gradually transitioning towards a personalized model of small batches and multiple varieties, which makes the manufacturing process of the job shop increasingly complex. Furthermore, disruptive events such as machinery failures and rush orders in the job shop increase the uncertainty and variability of the production environment. Traditional scheduling methods are usually based on fixed rules and heuristic algorithms, which are difficult to adapt to constantly changing production environments and demands. This may lead to inaccurate scheduling decisions and hinder the optimal allocation of job shop resources. To solve the dynamic job shop scheduling problem (JSP) more effectively, this paper proposes a Reinforcement Learning (RL) optimization algorithm integrating long short-term memory (LSTM) neural network and proximal policy optimization (PPO). It can dynamically adjust scheduling strategies according to the changing production environment, achieving comprehensive status awareness of the job shop environment to make optimal scheduling decisions. First, a state-aware network framework based on LSTM-PPO is proposed to achieve real-time perception of job shop state changes. Then, the state and action space of the job shop are described within the context of the state-aware network framework. Finally, an experimental environment is established to verify the algorithm’s effectiveness. Training the LSTM-PPO algorithm makes it feasible to achieve better performance than other scheduling methods. By comparing the initial planning time with the actual completion time of the rescheduling decision under different dynamic disturbances, the efficiency of the proposed algorithm is verified for the dynamic JSP.

Dynamic Job-Shop Scheduling via Graph Attention Networks and Deep Reinforcement Learning

Dynamic Job-Shop Scheduling via Graph Attention Networks and Deep Reinforcement Learning

Dynamic Job Shop Scheduling Problem With New Job Arrivals Using Hybrid Genetic Algorithm

Dynamic Job Shop Scheduling Problem With New Job Arrivals Using Hybrid Genetic Algorithm

Multitask Linear Genetic Programming with Shared Individuals and its Application to Dynamic Job Shop Scheduling

Multitask genetic programming methods have been applied to various domains, such as classification, regression, and combinatorial optimization problems. Most existing multitask genetic programming methods are designed based on tree-based structures, which are not good at reusing building blocks since each sub-tree passes its outputs to only one parent. It may limit the design and performance of knowledge sharing in multitask optimization. Different from tree-based genetic programming, building blocks in linear genetic programming can be easily reused by more than one parent. Besides, existing multitask genetic programming methods always allocate each individual to a specific task and have to duplicate genetic materials from task to task in knowledge transfer, which is inefficient and often produces redundancy. Contrarily, it is natural for a linear genetic programming individual to produce multiple distinct outputs, which enables each linear genetic programming individual to solve multiple tasks simultaneously. With this in mind, we propose a new multitask linear genetic programming method that transfers knowledge via multi-output individuals (i.e., shared individuals among tasks). By integrating different solutions into one multi-output individual, the proposed method efficiently reuses common knowledge among tasks and maintains distinct behaviors for each task. The empirical results show that the proposed method has a significantly better test performance than state-of-the-art multitask genetic programming methods. Further analyses verify that the new knowledge transfer mechanism can adjust the transfer rate automatically and thus improves its effectiveness.

Dynamic Job-Shop Scheduling Based on Transformer and Deep Reinforcement Learning

The dynamic job-shop scheduling problem is a complex and uncertain task that involves optimizing production planning and resource allocation in a dynamic production environment. Traditional methods are limited in effectively handling dynamic events and quickly generating scheduling solutions; in order to solve this problem, this paper proposes a solution by transforming the dynamic job-shop scheduling problem into a Markov decision process and leveraging deep reinforcement learning techniques. The proposed framework introduces several innovative components, which make full use of human domain knowledge and machine computing power, to realize the goal of man–machine collaborative decision-making. Firstly, we utilize disjunctive graphs as the state representation, capturing the complex relationships between various elements of the scheduling problem. Secondly, we select a set of dispatching rules through data envelopment analysis to form the action space, allowing for flexible and efficient scheduling decisions. Thirdly, the transformer model is employed as the feature extraction module, enabling effective capturing of state relationships and improving the representation power. Moreover, the framework incorporates the dueling double deep Q-network with prioritized experience replay, mapping each state to the most appropriate dispatching rule. Additionally, a dynamic target strategy with an elite mechanism is proposed. Through extensive experiments conducted on multiple examples, our proposed framework consistently outperformed traditional dispatching rules, genetic algorithms, and other reinforcement learning methods, achieving improvements of 15.98%, 17.98%, and 13.84%, respectively. These results validate the effectiveness and superiority of our approach in addressing dynamic job-shop scheduling problems.

Dynamic Job-Shop Scheduling Problems Using Graph Neural Network and Deep Reinforcement Learning

Dynamic Job-Shop Scheduling Problems Using Graph Neural Network and Deep Reinforcement Learning

Instance-Rotation-Based Surrogate in Genetic Programming With Brood Recombination for Dynamic Job-Shop Scheduling

Genetic programming has achieved great success for learning scheduling heuristics in dynamic job shop scheduling. In theory, generating a large number of offspring for genetic programming, known as brood recombination, can improve its heuristic generation ability. However, it is time-consuming to evaluate extra individuals. Phenotypic characterisation based surrogates with K-nearest neighbours have been successfully used for genetic programming to preselect only promising individuals for real fitness evaluations in dynamic job shop scheduling. However, sample individuals used by surrogate are from only the current generation, since the fitness of individuals across generations are not comparable due to the rotation of training instances. The surrogate cannot accurately estimate the fitness of an offspring that is far away from all the limited sample individuals at the current generation. This paper proposes an effective instance rotation based surrogate to address the above issue. Specifically, the surrogate uses the samples extracted from individuals across multiple generations with different instances. More importantly, we propose a fitness mapping strategy to make the fitness evaluated by different instances comparable. The results show that the GP with brood recombination and the proposed surrogate can significantly improve the quality of scheduling heuristics. The results also reveal that the proposed algorithm has successfully reduced the number of omitted promising offspring due to the higher accuracy of the surrogate. The samples in the new surrogate spread better in the phenotypic space, and the nearest neighbour tends to be closer to the predicted offspring. This makes the estimated fitness more accurate.

A data-driven simulation-optimization framework for generating priority dispatching rules in dynamic job shop scheduling with uncertainties

Modeling and optimizing dynamic job shop scheduling problems (DJSSP) without ample assumptions is inherently challenging due to the increasing complexity and uncertainty arising from various disruptive events. Moreover, establishing job due dates in dynamic job shop environments has become increasing difficult due to unforeseen events, while it remains a pivotal factor in determining tardiness-related objectives. A research gap exists in exploring the potential synergy between the two aforementioned aspects and its resulting implications. In this study, we propose a novel framework that combines data-driven simulation, machine learning, and evolutionary algorithm to generate new Priority Dispatching Rules (PDRs) for optimizing makespan and total tardiness. The framework incorporates a data-exchange mechanism for real-time updates of the simulation model based on the shop floor. Moreover, an online due date prediction model using XGBoost is developed, replacing the previous rough estimation of the Total Work Content (TWK) rule. Simulation experiments of various scales demonstrate that XGBoost-assisted due date prediction leads to an average reduction of 1.6% in makespan and 65.0% in total tardiness compared to the TWK rule. The proposed method outperforms existing well-known dispatching rules, yielding considerable improvements with average reductions of 90.8%, 40.3%, and 44.6% in total tardiness for vary-sized experiments. Finally, the approach is validated through a real industrial case study.

Correction to: Smart scheduling of dynamic job shop based on discrete event simulation and deep reinforcement learning

Correction to: Smart scheduling of dynamic job shop based on discrete event simulation and deep reinforcement learning

Effect of Setup Times and Workload Trigger Levels on Order Review and Release Policies Performance in a Stochastic Dynamic Job Shop with Sequence-Dependent Setup Time — A Simulation Approach

Order review and release (ORR) in a job shop considering sequence-dependent setup time (SDST) is among the most challenging job shop problems. This study investigates the effect of varying setup times and workload trigger levels on ORR policies performance in a stochastic dynamic job shop considering SDST. It is essential to examine the effect of varying setup times and workload trigger levels on ORR policies, because, for different ORR policies, workload trigger level plays a major role to control the work-in-process on the shop floor. Further, in manufacturing industries, less setup time is required in some cases, but it can also be equal to or greater than the processing time in others. Promodel[Formula: see text] simulation software is used to develop the simulation model for conducting experiments. The performance of the system is evaluated in terms of mean lead time (MLT), mean throughput time (MTT), number of tardy jobs (NOTJ), and total setups (TS) performance measures. From results, it is found that for a given ORR policy and at a given setup time level, lower workload trigger level provides minimum MTT and maximum MLT, TS, and NOTJ performance measures. As the workload trigger level increases, MTT performance measure increases, whereas MLT, TS, and NOTJ performance measures decrease. Further, for a given ORR policy and workload trigger level, as the setup time level increases, MTT and MLT performance measures increase. In contrast, TS and NOTJ performance measures decrease. A stochastic dynamic job shop considering ORR and SDST where setup time is either less than, equal to or more than operation processing time is the novelty of the work.

Evolution strategies-based optimized graph reinforcement learning for solving dynamic job shop scheduling problem

The job shop scheduling problem (JSSP) with dynamic events and uncertainty is a strongly NP-hard combinatorial optimization problem (COP) with extensive applications in the manufacturing system. Recently, growing interest has been aroused in utilizing machine learning techniques to solve the JSSP. However, most prior arts cannot handle dynamic events and barely consider uncertainties. To close this gap, this paper proposes a framework to solve a dynamic JSSP (DJSP) with machine breakdown and stochastic processing time based on Graph Neural Network (GNN) and deep reinforcement learning (DRL). To this end, we first formulate the DJSP as a Markov Decision Process (MDP), where disjunctive graph represent the states. Secondly, we propose a GNN-based model to effectively extract the embeddings of the state by considering the features of the dynamic events and the stochasticity of the problem, e.g., the machine breakdown and stochastic processing time. Then, the model constructs solutions by dispatching optimal operations to machines based on the learned embeddings. Notably, we propose to use the evolution strategies (ES) to find optimal policies that are more stable and robust than conventional DRL algorithms. The extensive experiments show that our method substantially outperforms existing reinforcement learning-based and traditional methods on multiple classic benchmarks.

Smart scheduling of dynamic job shop based on discrete event simulation and deep reinforcement learning

Smart scheduling of dynamic job shop based on discrete event simulation and deep reinforcement learning

A deep multi-agent reinforcement learning approach to solve dynamic job shop scheduling problem

Manufacturing industry is experiencing a revolution in the creation and utilization of data, the abundance of industrial data creates a need for data-driven techniques to implement real-time production scheduling. However, existing dynamic scheduling techniques have been mainly developed to solve problems of invariable size, and are incapable of addressing the increasing volatility and complexity of practical production scheduling problems. To facilitate near real-time decision-making on the shop floor, we propose a deep multi-agent reinforcement learning-based approach to solve the dynamic job shop scheduling problem. Double deep Q-network algorithm, attached to decentralized scheduling agents, is used to learn the relationships between production information and scheduling objectives, and to make near real-time scheduling decisions. Proposed framework utilizes centralized training and decentralized execution scheme and parameter-sharing technique to tackle the non-stationary problem in the multi-agent reinforcement learning task. Several enhancements are also developed, including the novel state and action representation that can handle size-agnostic dynamic scheduling problems, a chronological joint-action framework to alleviate the credit-assignment difficulty, and knowledge-based reward-shaping techniques to encourage cooperation. Simulation study shows that the proposed architecture significantly improves the learning effectiveness, and delivers superior performance compared to existing scheduling strategies and state-of-the-art deep reinforcement learning-based dynamic scheduling approaches.

Effective and Interpretable Rule Mining for Dynamic Job-Shop Scheduling via Improved Gene Expression Programming with Feature Selection

Gene expression programming (GEP) is frequently used to create intelligent dispatching rules for job-shop scheduling. The proper selection of the terminal set is a critical factor for the success of GEP. However, there are various job features and machine features that can be included in the terminal sets to capture the different characteristics of the job-shop state. Moreover, the importance of features in the terminal set varies greatly between scenarios. The irrelevant and redundant features may lead to high computational requirements and increased difficulty in interpreting generated rules. Consequently, a feature selection approach for evolving dispatching rules with improved GEP has been proposed, so as to select the proper terminal set for different dynamic job-shop scenarios. First, the adaptive variable neighborhood search algorithm was embedded into the GEP to obtain a diverse set of good rules for job-shop scenarios. Secondly, based on the fitness of the good rules and the contribution of features to the rules, a weighted voting ranking method was used to select features from the terminal set. The proposed approach was then compared with GEP-based algorithms and benchmark rules in the different job-shop conditions and scheduling objectives. The experimentally obtained results illustrated that the performance of the dispatching rules generated using the improved GEP algorithm after the feature selection process was better than that of both the baseline dispatching rules and the baseline GEP algorithm.

Learning dispatching rules via novel genetic programming with feature selection in energy-aware dynamic job-shop scheduling

The incorporation of energy conservation measures into production efficiency is widely recognized as a crucial aspect of contemporary industry. This study aims to develop interpretable and high-quality dispatching rules for energy-aware dynamic job shop scheduling (EDJSS). In comparison to the traditional modeling methods, this paper proposes a novel genetic programming with online feature selection mechanism to learn dispatching rules automatically. The idea of the novel GP method is to achieve a progressive transition from exploration to exploitation by relating the level of population diversity to the stopping criteria and elapsed duration. We hypothesize that diverse and promising individuals obtained from the novel GP method can guide the feature selection to design competitive rules. The proposed approach is compared with three GP-based algorithms and 20 benchmark rules in the different job shop conditions and scheduling objectives considered energy consumption. Experiments show that the proposed approach greatly outperforms the compared methods in generating more interpretable and effective rules. Overall, the average improvement over the best-evolved rules by the other three GP-based algorithms is 12.67%, 15.38%, and 11.59% in the meakspan with energy consumption (EMS), mean weighted tardiness with energy consumption (EMWT), and mean flow time with energy consumption (EMFT) scenarios, respectively.

Event-driven dynamic Job-shop scheduling method with strong process constraints

In the actual production environment, there are many uncertainties that need to be handled. Since the occurrence time and frequency of dynamic events cannot be predicted, frequent global scheduling calculation will cause waste of time and resources. Therefore, facing the challenge of job shop scheduling problem in dynamic scenarios, this paper studied the dynamic event-driven job shop scheduling problem with complex process constraints, and proposed an improved NSGAII algorithm. At the end of this paper, the designed dynamic scheduling method is applied to a variety of benchmarks through experiments to verify the effectiveness of the proposed method in solving the job shop scheduling problem in a dynamic environment.

Combining Reinforcement Learning Algorithms with Graph Neural Networks to Solve Dynamic Job Shop Scheduling Problems

Smart factories have attracted a lot of attention from scholars for intelligent scheduling problems due to the complexity and dynamics of their production processes. The dynamic job shop scheduling problem (DJSP), as one of the intelligent scheduling problems, aims to make an optimized scheduling decision sequence based on the real-time dynamic job shop environment. The traditional reinforcement learning (RL) method converts the scheduling problem with a Markov process and combines its own reward method to obtain scheduling sequences in different real-time shop states. However, the definition of shop states often relies on the scheduling experience of the model constructor, which undoubtedly affects the optimization capability of the reinforcement learning model. In this paper, we combine graph neural network (GNN) and deep reinforcement learning (DRL) algorithm to solve DJSP. An agent model from job shop state analysis graph to scheduling rules is constructed, thus avoiding the problem that traditional reinforcement learning methods rely on scheduling experience to artificially set the state feature vectors. In addition, a new reward function is defined, and the experimental results prove that our proposed reward method is more effective. The effectiveness and feasibility of our model is demonstrated by comparing with general deep reinforcement learning algorithms on minimizing the earlier and later completion time, which also lays the foundation for solving the DJSP later.

A Multi-Agent Reinforcement Learning Approach to the Dynamic Job Shop Scheduling Problem

In a production environment, scheduling decides job and machine allocations and the operation sequence. In a job shop production system, the wide variety of jobs, complex routes, and real-life events becomes challenging for scheduling activities. New, unexpected events disrupt the production schedule and require dynamic scheduling updates to the production schedule on an event-based basis. To solve the dynamic scheduling problem, we propose a multi-agent system with reinforcement learning aimed at the minimization of tardiness and flow time to improve the dynamic scheduling techniques. The performance of the proposed multi-agent system is compared with the first-in–first-out, shortest processing time, and earliest due date dispatching rules in terms of the minimization of tardy jobs, mean tardiness, maximum tardiness, mean earliness, maximum earliness, mean flow time, maximum flow time, work in process, and makespan. Five scenarios are generated with different arrival intervals of the jobs to the job shop production system. The results of the experiments, performed for the 3 × 3, 5 × 5, and 10 × 10 problem sizes, show that our multi-agent system overperforms compared to the dispatching rules as the workload of the job shop increases. Under a heavy workload, the proposed multi-agent system gives the best results for five performance criteria, which are the proportion of tardy jobs, mean tardiness, maximum tardiness, mean flow time, and maximum flow time.

An Efficient Metaheuristic Algorithm for Job Shop Scheduling in a Dynamic Environment

This paper proposes an Improved Multi-phase Particle Swarm Optimization (IMPPSO) to solve a Dynamic Job Shop Scheduling Problem (DJSSP) known as an non-deterministic polynomial-time hard (NP-hard) problem. A cellular neighbor network, a velocity reinitialization strategy, a randomly select sub-dimension strategy, and a constraint handling function are introduced in the IMPPSO. The IMPPSO is used to solve the Kundakcı and Kulak problem set and is compared with the original Multi-phase Particle Swarm Optimization (MPPSO) and Heuristic Kalman Algorithm (HKA). The results show that the IMPPSO has better global exploration capability and convergence. The IMPPSO has improved fitness for most of the benchmark instances of the Kundakcı and Kulak problem set, with an average improvement rate of 5.16% compared to the Genetic Algorithm-Mixed (GAM) and of 0.74% compared to HKA. The performance of the IMPPSO for solving real-world problems is verified by a case study. The high level of operational efficiency is also evaluated and demonstrated by proposing a simulation model capable of using the decision-making algorithm in a real-world environment.

Dynamic production scheduling towards self-organizing mass personalization: A multi-agent dueling deep reinforcement learning approach

Mass personalization is rapidly approaching. In response, manufacturing systems should be capable of autonomously changing production plans, configurations and schedules under dynamic manufacturing environments for producing personalized products. Self-organizing manufacturing network is a promising paradigm for mass personalization. The backbone of a self-organizing manufacturing network is an adaptive production scheduling method to dynamically allocate and sequence manufacturing jobs under dynamic settings, such as stochastic processing time or unplanned machine breakdown. However, existing production scheduling methods (i.e., heuristic rules, meta-heuristic algorithms, and existing reinforcement learning models) fail to automatically optimize production schedules while maintaining stable manufacturing performance, under dynamic settings. In this paper, we designed a reinforcement learning-based static-training-dynamic-execution approach for dynamic job shop scheduling problems. The scheduling policies are learned from static scheduling instances by a multi-agent dueling deep reinforcement learning approach. Under this approach, we proposed new representations of observation, action, reward, and cooperation mechanisms between agents. The learned scheduling policies are then deployed to a dynamic scheduling system where stochastic processing time and unplanned machine breakdown randomly occur. Extensive simulation experiments demonstrated that our approach outperforms heuristic rules on makespan under two dynamic manufacturing settings.