Dynamic Flexible Job Shop Scheduling Problem Research Articles

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

Collaborative dynamic scheduling in a self-organizing manufacturing system using multi-agent reinforcement learning

Personalized product demands have made the production mode of many varieties and small batches mainstream. Self-organizing manufacturing systems represented by multi-agent-based manufacturing systems (MAMS) are pursued by manufacturing enterprises to support collaborative dynamic scheduling among various pieces of manufacturing equipment. In contrast to traditional scheduling methods including meta-heuristic algorithms and dispatching rules, reinforcement learning (RL), whether in the form of single-agent RL (SARL) or multi-agent RL (MARL), combines high response and robust performance in dynamic environments. However, the SARL-based methods suffer from the curse of dimensionality in the action space, while current MARL-based methods fail to achieve the optimization of collaborative scheduling decisions among heterogeneous manufacturing equipment. These factors directly hinder the effective resolution of dynamic scheduling in the MAMS. Therefore, we present a novel MARL-based approach that solves the dynamic flexible job-shop scheduling problem in the MAMS (DFJSP-MAMS), with the objective of minimizing the mean tardiness. Here, various items of heterogeneous manufacturing equipment in the MAMS, including the warehouse, machines, buffers, and vehicles, are encapsulated as agents. Moreover, a partially observable Markov game is constructed to specify the collaborative scheduling process among these agents and transfer the DFJSP-MAMS into a MARL task. Furthermore, an action space consisting of several weight variables is designed for each agent to offer the weight selection of dispatching rules. Subsequently, a policy network is built for each agent to determine the weights and scheduling decisions. Finally, the multi-agent deep deterministic policy gradient algorithm is employed to train these agents for the optimization of collaborative scheduling decisions among various pieces of heterogeneous equipment. Experiment results demonstrate that the proposed approach outperforms single dispatching rules, the SARL-based method, and other MARL-based methods under 36 distinct manufacturing environment conditions.

Automatic Design of Energy-Efficient Dispatching Rules for Multi-Objective Dynamic Flexible Job Shop Scheduling Based on Dual Feature Weight Sets

Considering the requirements of the actual production scheduling process, the utilization of the genetic programming hyper-heuristic (GPHH) approach to automatically design dispatching rules (DRs) has recently emerged as a popular optimization approach. However, the decision objects and decision environments for routing and sequencing decisions are different in the dynamic flexible job shop scheduling problem (DFJSSP), leading to different required feature information. Traditional algorithms that allow these two types of scheduling decisions to share one common feature set are not conducive to the further optimization of the evolved DRs, but instead introduce redundant and unnecessary search attempts for algorithm optimization. To address this, some related studies have focused on customizing the feature sets for both routing and sequencing decisions through feature selection when solving single-objective problems. While being effective in reducing the search space, the selected feature sets also diminish the diversity of the obtained DRs, ultimately impacting the optimization performance. Consequently, this paper proposes an improved GPHH with dual feature weight sets for the multi-objective energy-efficient DFJSSP, which includes two novel feature weight measures and one novel hybrid population adjustment strategy. Instead of selecting suitable features, the proposed algorithm assigns appropriate weights to the features based on their multi-objective contribution, which could provide directional guidance to the GPHH while ensuring the search space. Experimental results demonstrate that, compared to existing studies, the proposed algorithm can significantly enhance the optimization performance and interpretability of energy-efficient DRs.

Dynamic Events in the Flexible Job-Shop Scheduling Problem: Rescheduling with a Hybrid Metaheuristic Algorithm

In the real world of manufacturing systems, production planning is crucial for organizing and optimizing various manufacturing process components. The objective of this paper is to present a methodology for both static scheduling and dynamic scheduling. In the proposed method, a hybrid algorithm is utilized to optimize the static flexible job-shop scheduling problem (FJSP) and dynamic flexible job-shop scheduling problem (DFJSP). This algorithm integrates the genetic algorithm (GA) as a global optimization technique with a simulated annealing (SA) algorithm serving as a local search optimization approach to accelerate convergence and prevent getting stuck in local minima. Additionally, variable neighborhood search (VNS) is utilized for efficient neighborhood search within this hybrid algorithm framework. For the FJSP, the proposed hybrid algorithm is simulated on a 40-benchmark dataset to evaluate its performance. Comparisons among the proposed hybrid algorithm and other algorithms are provided to show the effectiveness of the proposed algorithm, ensuring that the proposed hybrid algorithm can efficiently solve the FJSP, with 38 out of 40 instances demonstrating better results. The primary objective of this study is to perform dynamic scheduling on two datasets, including both single-purpose machine and multi-purpose machine datasets, using the proposed hybrid algorithm with a rescheduling strategy. By observing the results of the DFJSP, dynamic events such as a single machine breakdown, a single job arrival, multiple machine breakdowns, and multiple job arrivals demonstrate that the proposed hybrid algorithm with the rescheduling strategy achieves significant improvement and the proposed method obtains the best new solution, resulting in a significant decrease in makespan.

An improved genetic programming hyper-heuristic for the dynamic flexible job shop scheduling problem with reconfigurable manufacturing cells

The Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is a classical and important research direction. However, current research usually considers the case where each manufacturing cell has a fixed and constant process capability. There are often situations in non-machining shops where each manufacturing cell is capable of capability reconfiguration and performs many different types of process operations, such as assembly shops and test shops. The variability of the manufacturing cell's capabilities increases the complexity of the problem compared to traditional FJSP. In this paper, we study the dynamic flexible job shop scheduling problem considering reconfigurable manufacturing cells (DFJSP-RMCs) with completion time, delay time and reconfiguration time as optimization objectives, and propose an improved Genetic Programming Hyper-Heuristic (GPHH) method to solve it. The method weighs the solution efficiency and the effectiveness of the results. In addition, an individual simplification policy (ISP) is proposed to reduce the evaluation time of the heuristic. Finally, random instances were generated under three production conditions and 10 independent runs were performed for each. Experiments show that the proposed method significantly reduces the time consumption while ensuring the quality of the results.

Dynamic constrained evolutionary optimization based on deep Q-network

Dynamic constrained optimization problems (DCOPs) are common and important optimization problems in real-world, which have great difficulty to solve. Dynamic constrained evolutionary algorithms (DCEAs) are widely used methods for solving DCOPs. However, existing DCEAs often struggle with convergence, particularly for DCOPs with drastic dynamic changes or intricate constraints. To address this issue, this paper proposes a novel DCEA called DCEA-DQN, which leverages the powerful perception and decision-making capabilities of Deep Q-Network (DQN). DCEA-DQN integrates two DQNs to enhance its performance. The first DQN is designed to adaptively respond to dynamic changes, enabling effective handling of DCOPs with various types and degrees of changes. It provides a high-quality re-initialized population for subsequent static optimization, resulting in faster and improved convergence. The second DQN is introduced to guide the mutation direction during offspring generation. It steers the population towards better feasible regions or directs it towards the optimal individual within the current feasible region. Moreover, a penalty mechanism is employed to handle constraints during offspring generation. To evaluate the performance of DCEA-DQN, comprehensive empirical studies are conducted using a new test suite called C-GMPB and a dynamic flexible job-shop scheduling problem. The experimental results, using two commonly used metrics EB and EO in the field of DCOPs, demonstrate that DCEA-DQN outperforms six state-of-the-art DCEAs and achieved optimal performance on 80% and 75% of all 24 test problems, respectively. The source code for DCEA-DQN is available at https://github.com/CIA-SZU/YRT.

Large-Scale Dynamic Scheduling for Flexible Job-Shop With Random Arrivals of New Jobs by Hierarchical Reinforcement Learning

As the intelligent manufacturing paradigm evolves, it is urgent to design a near real-time decision-making framework for handling the uncertainty and complexity of production line control. The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the manufacturing industry. However, it is still challenging to obtain high-quality schedules for DFJSP with dynamic job arrivals in real-time, especially facing thousands of operations from a large-scale scene with complex contexts in an assembly plant. This paper aims to propose a novel end-to-end hierarchical reinforcement learning framework for solving the large-scale DFJSP in near real-time. In the DFJSP, the processing information of newly arrived jobs is unknown in advance. Besides, two optimization tasks, including job operation selection and job-to-machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. In our framework, a higher-level layer is designed to automatically divide the DFJSP into sub-problems, i.e., static FJSPs with different scales. And two lower-level layers are constructed to solve the sub-problems. In particular, one layer based on a graph neural network is in charge of sequencing job operations, and another layer based on a multi-layer perceptron is used to assign a machine to process the job operations. Numerical experiments, including offline training and online testing, are conducted on several instances with different scales. The results verify the superior performance of the proposed framework compared with existing dynamic scheduling methods, such as well-known dispatching rules and meta-heuristics.

Efficient Multi-Objective Optimization on Dynamic Flexible Job Shop Scheduling Using Deep Reinforcement Learning Approach

Previous research focuses on approaches of deep reinforcement learning (DRL) to optimize diverse types of the single-objective dynamic flexible job shop scheduling problem (DFJSP), e.g., energy consumption, earliness and tardiness penalty and machine utilization rate, which gain many improvements in terms of objective metrics in comparison with metaheuristic algorithms such as GA (genetic algorithm) and dispatching rules such as MRT (most remaining time first). However, single-objective optimization in the job shop floor cannot satisfy the requirements of modern smart manufacturing systems, and the multiple-objective DFJSP has become mainstream and the core of intelligent workshops. A complex production environment in a real-world factory causes scheduling entities to have sophisticated characteristics, e.g., a job’s non-uniform processing time, uncertainty of the operation number and restraint of the due time, avoidance of the single machine’s prolonged slack time as well as overweight load, which make a method of the combination of dispatching rules in DRL brought up to adapt to the manufacturing environment at different rescheduling points and accumulate maximum rewards for a global optimum. In our work, we apply the structure of a dual layer DDQN (DLDDQN) to solve the DFJSP in real time with new job arrivals, and two objectives are optimized simultaneously, i.e., the minimization of the delay time sum and makespan. The framework includes two layers (agents): the higher one is named as a goal selector, which utilizes DDQN as a function approximator for selecting one reward form from six proposed ones that embody the two optimization objectives, while the lower one, called an actuator, utilizes DDQN to decide on an optimal rule that has a maximum Q value. The generated benchmark instances trained in our framework converged perfectly, and the comparative experiments validated the superiority and generality of the proposed DLDDQN.

Dynamic scheduling for flexible job shop using a deep reinforcement learning approach

Due to the influence of dynamic changes in the manufacturing environment, a single dispatching rule (SDR) cannot consistently attain better results than other rules for dynamic scheduling problems. Dynamic selection of the most appropriate rule from several SDRs based on the Deep Q-Network (DQN) offers better scheduling performance than using an individual SDR. However, the discreteness of action space caused by the DQN and the simplicity of the action as an SDR limit the selection range and restrict performance improvement. Thus, in this paper, we propose a scheduling method based on deep reinforcement learning for the dynamic flexible job-shop scheduling problem (DFJSP), aiming to minimize the mean tardiness. Firstly, a Markov decision process with composite scheduling action is provided to elaborate the flexible job-shop dynamic scheduling process and transform the DFJSP into an RL task. Subsequently, a composite scheduling action aggregated by SDRs and continuous weight variables is designed to provide a continuous rule space and SDR weight selection. Moreover, a reward function related to mean tardiness performance criteria is designed such that maximizing the cumulative reward is equivalent to minimizing the mean tardiness. Finally, a policy network with states as inputs and weights as outputs is constructed to generate the scheduling decision at each decision point. Also, the deep deterministic policy gradient (DDPG) algorithm is used to train the policy network to select the most appropriate weights at each decision point, thereby aggregating the SDRs into a better rule. Results from numerical experiments reveal that the proposed scheduling method achieves significantly better scheduling results than an SDR and the DQN-based method in dynamically changeable manufacturing environments.

An effective two-stage algorithm based on convolutional neural network for the bi-objective flexible job shop scheduling problem with machine breakdown

In the actual manufacturing process, the environment of the job shop is complex. There will be many kinds of uncertainties such as random job arrivals, machine breakdowns, order cancellations and other dynamic events. In this paper, an effective two-stage algorithm based on convolutional neural network is proposed to solve the flexible job shop scheduling problem (FJSP) with machine breakdown. A bi-objective dynamic flexible job shop scheduling problem (DFJSP) model with the objective of maximum completion time and robustness is established. In the two-stage algorithm, the first stage is to train the prediction model by convolutional neural network (CNN). The second stage is to predict the robustness of scheduling through the model trained in the first stage. First, an improved imperialist competition algorithm (ICA) is proposed to generate training data. Then, a predictive model constructed by CNN was proposed, and an alternative metric called RMn was developed to evaluate robustness. RMn evaluates that the float time has an effect on the robustness through the information of machine breakdown, workload and float time of the operation. The experimental results show that the proposed two-stage algorithm is effective for solving DFJSP, and RMn can evaluate the robustness of scheduling more quickly, efficiently and accurately.

Deep Reinforcement Learning for Dynamic Flexible Job Shop Scheduling with Random Job Arrival

The production process of a smart factory is complex and dynamic. As the core of manufacturing management, the research into the flexible job shop scheduling problem (FJSP) focuses on optimizing scheduling decisions in real time, according to the changes in the production environment. In this paper, deep reinforcement learning (DRL) is proposed to solve the dynamic FJSP (DFJSP) with random job arrival, with the goal of minimizing penalties for earliness and tardiness. A double deep Q-networks (DDQN) architecture is proposed and state features, actions and rewards are designed. A soft ε-greedy behavior policy is designed according to the scale of the problem. The experimental results show that the proposed DRL is better than other reinforcement learning (RL) algorithms, heuristics and metaheuristics in terms of solution quality and generalization. In addition, the soft ε-greedy strategy reasonably balances exploration and exploitation, thereby improving the learning efficiency of the scheduling agent. The DRL method is adaptive to the dynamic changes of the production environment in a flexible job shop, which contributes to the establishment of a flexible scheduling system with self-learning, real-time optimization and intelligent decision-making.

Dynamic Self-Learning Artificial Bee Colony Optimization Algorithm for Flexible Job-Shop Scheduling Problem with Job Insertion

To solve the problem of inserting new job into flexible job-shops, this paper proposes a dynamic self-learning artificial bee colony (DSLABC) optimization algorithm to solve dynamic flexible job-shop scheduling problem (DFJSP). Through the reasonable arrangement of the processing sequence of the jobs and the corresponding relationship between the operations and the machines, the makespan can be shortened, the economic benefit of the job-shop and the utilization rate of the processing machine can be improved. Firstly, the Q-learning algorithm and the traditional artificial bee colony (ABC) algorithm are combined to form the self-learning artificial bee colony (SLABC) algorithm. Using the learning characteristics of the Q-learning algorithm, the update dimension of each iteration of the ABC algorithm can be dynamically adjusted, which improves the convergence accuracy of the ABC algorithm. Secondly, the specific method of dynamic scheduling is determined, and the DSLABC algorithm is proposed. When a new job is inserted, the new job and the operations that have not started processing will be rescheduled. Finally, through solving the Brandimarte instances, it is proved that the convergence accuracy of the SLABC algorithm is higher than that of other optimization algorithms, and the effectiveness of the DSLABC algorithm is demonstrated by solving a specific example with a new job inserted.

Joint optimisation for dynamic flexible job-shop scheduling problem with transportation time and resource constraints

Dynamic flexible job-shop scheduling is traditionally a challenge in real-world manufacturing systems, especially considering the constraints of transportation resources and transportation time. To address the dynamic optimisation problem in flexible manufacturing systems, this paper proposes a novel proactive-reactive methodology to adapt to the dynamic changes in working environments and addresses the joint scheduling problem for machine tools and transportation resources. The joint optimisation model is first formulated as a mixed-integer programming model considering production efficiency and transportation constraints. The flowchart of the dynamic scheduling system is then designed for dynamic decision-making, and a novel particle swarm optimisation algorithm integrated with genetic operators is developed to respond to dynamic events and generate the reschedule plan in time. Finally, several numerical experiments and case studies in reality are applied to verify the efficiency of the developed methodology. Common dispatching rules and heuristic methods are also applied to test and evaluate the efficiency of the developed algorithm. Computational results demonstrate that the developed methods and decision models are efficient for dynamic job-shop scheduling problems in flexible manufacturing systems, which can acquire rather a good effect in practical production.

An effective MCTS-based algorithm for minimizing makespan in dynamic flexible job shop scheduling problem

In the past several decades, most of the research methods are designed to solve the static flexible job shop scheduling problem. However, in real production environments, some inevitable dynamic events such as new jobs arrival and machine breakdown may occur frequently. In this paper, we study a dynamic flexible job shop scheduling problem (DFJSP) considering four dynamic events, which are new jobs arrival, machine breakdown, jobs cancellation and change in the processing time of operations. A rescheduling method based on Monte Carlo Tree Search algorithm (MCTS) is designed to solve the proposed DFJSP with the objective of minimizing the makespan. Several optimization techniques such as Rapid Action Value Estimates heuristic and prior knowledge are adopted to enhance the performance of the MCTS-based rescheduling method. The response time to dynamic events is critical in DFJSP but has not been solved very well. To greatly reduce the response time to dynamic events, when dynamic events occur, multiple continuous specified time windows are designed for the proposed method, according to which the corresponding subsequent partial schedule for the remaining unprocessed operations is progressively generated. Some experiments have been conducted to compare the proposed method with the commonly used completely reactive scheduling methods and the GA-based rescheduling method. The experiment results indicate that the proposed method is an efficient and promising method for dynamic scheduling both on solution quality and computation efficiency.

Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning

In modern manufacturing industry, dynamic scheduling methods are urgently needed with the sharp increase of uncertainty and complexity in production process. To this end, this paper addresses the dynamic flexible job shop scheduling problem (DFJSP) under new job insertions aiming at minimizing the total tardiness. Without lose of generality, the DFJSP can be modeled as a Markov decision process (MDP) where an intelligent agent should successively determine which operation to process next and which machine to assign it on according to the production status of current decision point, making it particularly feasible to be solved by reinforcement learning (RL) methods. In order to cope with continuous production states and learn the most suitable action (i.e. dispatching rule) at each rescheduling point, a deep Q-network (DQN) is developed to address this problem. Six composite dispatching rules are proposed to simultaneously select an operation and assign it on a feasible machine every time an operation is completed or a new job arrives. Seven generic state features are extracted to represent the production status at a rescheduling point. By taking the continuous state features as input to the DQN, the state–action value (Q-value) of each dispatching rule can be obtained. The proposed DQN is trained using deep Q-learning (DQL) enhanced by two improvements namely double DQN and soft target weight update. Moreover, a “softmax” action selection policy is utilized in real implementation of the trained DQN so as to promote the rules with higher Q-values while maintaining the policy entropy. Numerical experiments are conducted on a large number of instances with different production configurations. The results have confirmed both the superiority and generality of DQN compared to each composite rule, other well-known dispatching rules as well as the stand Q-learning-based agent.

NSGA‐III for solving dynamic flexible job shop scheduling problem considering deterioration effect

The production process of manufacturing systems is usually not static and always interrupted by stochastic events, such as the deterioration effect of cutting tools. This study focuses on the dynamic flexible job shop-scheduling problem considering the deterioration effect (DFJSP-DE). Two types of disturbances are considered, i.e. the implicit disturbance caused by the deterioration effect and the explicit disturbance caused by the reprocessing of unqualified jobs. Firstly, a step-deterioration effect model is proposed, with which the actual processing time of each operation can be predicted more accurately. A multi-objective optimisation model is formulated for the DFJSP-DE. The makespan, energy consumption, and stability of rescheduling solutions are three objectives to be optimised simultaneously. For this non-deterministic polynominal (NP)-hard problem, the non-dominated sorting genetic algorithm III is employed to search the Pareto solutions for the DFJSP-DE. Finally, the results of three numerical experiments show that the proposed approach can solve DFJSP-DE effectively and efficiently.

Towards Energy Efficient Scheduling and Rescheduling for Dynamic Flexible Job Shop Problem

Nowadays the migration to green manufacturing is the interest of many companies. Taken into account sustainability in all industrial activities is the main goal of sustainable intelligent manufacturing system. In recent years, the demand for energy and the investment in energy have continued to increase. This work addresses reducing energy consumption when resolving dynamic flexible job-shop scheduling problem under machine breakdowns. A new rescheduling method is proposed to find a reschedule with minimum makespan and with less global energy consumption. The predictive reactive approach is based Particle Swarm Optimization method.

An improved quantum genetic algorithm based on MAGTD for dynamic FJSP

For the purpose of solving the dynamic flexible job-shop scheduling problem, this paper establishes the mathematical model to minimize the makespan and stability value, an improved double chains quantum genetic algorithm was proposed. Firstly, it is proposed that the method of double chains structure coding including machine allocation chain and process chain. Secondly, it is proposed that non- dominated sorting based on the crowding distance selection strategy. Thirdly, the most satisfying solution is obtained through the multi-attribute grey target decision model. Finally, the novel method is applied to the Brandimarte example and a dynamic simulation, the result of comparing with other classical algorithms verifies its effectiveness.

A fast estimation of distribution algorithm for dynamic fuzzy flexible job-shop scheduling problem

Due to the complicated circumstances in workshop, most of the conventional scheduling algorithms fail to meet the requirements of instantaneity, complexity, and dynamicity in job-shop scheduling problems. Compared with the static algorithms, dynamic scheduling algorithms can better fulfill the requirements in real situations. Considering that both flexibility and fuzzy processing time are common in reality, this paper focuses on the dynamic flexible job-shop scheduling problem with fuzzy processing time (DfFJSP). By adopting a series of transforming procedures, the original DfFJSP is simplified as a traditional static fuzzy flexible job-shop problem, which is more suitable to take advantage of the existing algorithms. In this paper, estimation of distribution algorithm (EDA) is brought into address the post-transforming problem. An improved EDA is developed through making use of several elements omitted in original EDA, including the historical-optimal solution and the standardized solution vectors. The improved algorithm is named as fast estimation of distribution algorithm (fEDA) since it performs better in convergence speed and computation precision, compared with the original EDA. To sum up, the ingenious transformation and the effective fEDA algorithm provide an efficient and practical way to tackle the dynamic flexible fuzzy job-shop scheduling problem.

A GEP-based reactive scheduling policies constructing approach for dynamic flexible job shop scheduling problem with job release dates

Flexible job shop scheduling problem (FJSSP) is generalization of job shop scheduling problem (JSSP), in which an operation may be processed on more than one machine each of which has the same function. Most previous researches on FJSSP assumed that all jobs to be processed are available at the beginning of scheduling horizon. The assumption, however, is always violated in practical industries because jobs usually arrive over time and can not be predicted before their arrivals. In the paper, dynamic flexible job shop scheduling problem (DFJSSP) with job release dates is studied. A heuristic is proposed to implement reactive scheduling for the dynamic scheduling problem. An approach based on gene expression programming (GEP) is also proposed which automatically constructs reactive scheduling policies for the dynamic scheduling. In order to evaluate the performance of the reactive scheduling policies constructed by the proposed GEP-based approach under a variety of processing conditions three factors, such as the shop utilization, due date tightness, problem flexibility, are considered in the simulation experiments. The scheduling performance measure considered in the simulation is the minimization of makespan, mean flowtime and mean tardiness, respectively. The results show that GEP-based approach can construct more efficient reactive scheduling policies for DFJSSP with job release dates under a big range of processing conditions and performance measures in the comparison with previous approaches.