Dynamic Job Arrivals Research Articles

Real-time scheduling for two-stage assembly flowshop with dynamic job arrivals by deep reinforcement learning

Assembly flowshops differ from flowshops because their jobs have several components that are manufactured in the processing stage and then joined together in the assembly stage. Real-time scheduling of two-stage assembly flowshops is of great significance for many discrete manufacturing companies in customized order fulfilment. Therefore, we study two-stage assembly flowshop scheduling problem with the objective of total tardiness minimization considering dynamic job arrivals using deep reinforcement learning (DRL). First, an architecture of DRL-based real-time scheduling is proposed. Second, we establish a DRL-based real-time scheduling model by designing proper state space, action space and reward function. A scheduling agent is constructed and trained through the proximal policy optimization (PPO) using interactive simulation data. The experimental results show that the trained scheduling agent can learn to choose the appropriate dispatching rules efficiently so as to make a real-time scheduling quickly and effectively. The PPO algorithm outperforms six dispatching rules, genetic algorithm and two DRL algorithms (i.e., DDQN and DDPG) in terms of solution quality and computing speed. The proposed approach is highly applicable to many practical scheduling situations that requires real-time and quick decisions.

A clustering-aided multi-agent deep reinforcement learning for multi-objective parallel batch processing machines scheduling in semiconductor manufacturing

Batch processing machines are often the bottleneck in semiconductor manufacturing and their scheduling plays a key role in production management. Pioneer researches on multi-objective batch machines scheduling mainly focus on evolutionary algorithms, failing to meet the online scheduling demand. To deal with the challenges confronted by incompatible job families, dynamic job arrivals, capacitated machines and multiple objectives, we propose a clustering-aided multi-agent deep reinforcement learning approach (CA-MADRL) for the scheduling problem. Specifically, to achieve diverse nondominated solutions, an offline multi-objective scheduling algorithm named Multi-Subpopulation fast elitist Non-Dominated Sorting Genetic Algorithm (MS-NSGA-II) is firstly developed to obtain the Pareto Fronts, and a clustering algorithm based on cosine distance is employed to analyze the distribution of Pareto frontier solution, which would be used to guide reward functions design in multi-agent deep reinforcement learning. To realize multi-objective optimization, several reinforcement learning base models are trained for different optimization directions, each of which composed of batch forming agent and batch scheduling agent. To alleviate time complexity of model training, a parameter sharing strategy is introduced between different reinforcement learning base model. By validating the proposed approach with 16 instances designed based on actual production data from a semiconductor manufacturing company, it has been demonstrated that the approach not only meets the high-frequency scheduling requirements of manufacturing systems for parallel batch processing machines but also effectively reduces the total job tardiness and machine energy consumption.

Dynamic flexible scheduling with transportation constraints by multi-agent reinforcement learning

Reinforcement learning-based methods have addressed production scheduling problems with flexible processing constraints. However, delayed rewards arise due to the dynamic arrival of jobs and transportation constraints between two successive operations. The flow time of operations can only be determined after processing due to the possibility that the solution for job sequencing may change if new operations are inserted in dynamic environments. Job sequencing is often overlooked in single-agent-based scheduling methods. The lack of information sharing between multiple agents necessitates that researchers manually design reward functions to fit the relationship between optimization objectives and rewards, thereby reducing the accuracy of the learned policies. Thus, this paper proposes a multi-agent-based scheduling optimization framework that facilitates collaboration between the agents of both machines and jobs to address dynamic flexible job-shop scheduling problems (DFJSP) with transportation time constraints. Then, this paper formulates the Partial Observation Markov Decision Process and constructs a reward-sharing mechanism to tackle the delayed reward issue and facilitate policy learning. Finally, we develop an improved multi-agent dueling double deep Q network algorithm to optimize scheduling policy during long-term training. The results show that, compared with the state-of-the-art methods, the proposed method efficiently shortens the weighted flow time under the trained and unseen scenarios. Additionally, the case study results demonstrate its efficiency and responsiveness. It indicates that the proposed method efficiently addresses production scheduling problems with complex constraints, including the insertion of jobs, transportation time constraints, and flexible processing routes.

A Double Deep Q-Network framework for a flexible job shop scheduling problem with dynamic job arrivals and urgent job insertions

In the semiconductor manufacturing industry, the Dynamic Flexible Job Shop Scheduling Problem is regarded as one of the most complex and significant scheduling problems. Existing studies consider the dynamic arrival of jobs, however, the insertion of urgent jobs such as testing chips poses a challenge to the production model, and there is an urgent need for new scheduling methods to improve the dynamic response and self-adjustment of the shop floor. In this work, deep reinforcement learning is utilized to address the dynamic flexible job shop scheduling problem and facilitate near-real-time shop floor decision-making. We extracted eight state features, including machine utilization, operation completion rate, etc., to reflect real-time shop floor production data. After examining machine availability time, the machine's earliest available time is redefined and incorporated into the design of compound scheduling rules. Eight compound scheduling rules have been developed for job selection and machine allocation. By using the state features as inputs to the Double Deep Q-Network, it is possible to acquire the state action values (Q-values) of each compound scheduling rule, and the intelligent agent can learn a reasonable optimization strategy through training. Simulation studies show that the proposed Double Deep Q-Network algorithm outperforms other heuristics and well-known scheduling rules by generating excellent solutions quickly. In most scenarios, the Double Deep Q-Network algorithm outperforms the Deep Q-Network, Q-Learning, and State-Action-Reward-State-Action (SARSA) frameworks. Moreover, the intelligent agent has good generalization ability in terms of optimization for similar objectives.

Large-Scale Dynamic Scheduling for Flexible Job-Shop With Random Arrivals of New Jobs by Hierarchical Reinforcement Learning

As the intelligent manufacturing paradigm evolves, it is urgent to design a near real-time decision-making framework for handling the uncertainty and complexity of production line control. The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the manufacturing industry. However, it is still challenging to obtain high-quality schedules for DFJSP with dynamic job arrivals in real-time, especially facing thousands of operations from a large-scale scene with complex contexts in an assembly plant. This paper aims to propose a novel end-to-end hierarchical reinforcement learning framework for solving the large-scale DFJSP in near real-time. In the DFJSP, the processing information of newly arrived jobs is unknown in advance. Besides, two optimization tasks, including job operation selection and job-to-machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. In our framework, a higher-level layer is designed to automatically divide the DFJSP into sub-problems, i.e., static FJSPs with different scales. And two lower-level layers are constructed to solve the sub-problems. In particular, one layer based on a graph neural network is in charge of sequencing job operations, and another layer based on a multi-layer perceptron is used to assign a machine to process the job operations. Numerical experiments, including offline training and online testing, are conducted on several instances with different scales. The results verify the superior performance of the proposed framework compared with existing dynamic scheduling methods, such as well-known dispatching rules and meta-heuristics.

Integrated optimization of production scheduling and maintenance planning with dynamic job arrivals and mold constraints

Effective management of production often requires harmonizing maintenance and production activities. However, decsions on production scheduling and machine maintenance are usually made independently, leading to potential overlaps and inefficiencies. This paper joinly investigates the production scheduling and maintenance planning for a production machine subject to different types of jobs that arrive randomly. A unique emphasis is placed on the prerequisite of loading specific molds prior to job execution, an element often overlooked in previous studies. Maintenance considerations employ the reliability/availability framework. The overarching goal is the creation of an integrated schedule that minimizes the weighted sum of total costs and the machine's maximum unavailability. To address this intricate challenge, a novel hybrid differential evolution and genetic algorithm (DE-GA) is proposed, complemented by a solution refinement strategy. Performance evaluations indicate that DE-GA methodology consistently outperforms Gurobi solver and four other prevalent algorithms across different test instances.

Multi-criteria scheduling of realistic flexible job shop: a novel approach for integrating simulation modelling and multi-criteria decision making

Increased flexibility in job shops leads to more complexity in decision-making for shop floor engineers. Partial Flexible Job Shop Scheduling (PFJSS) is a subset of Job shop problems and has substantial application in the real world. Priority Dispatching Rules (PDRs) are simple and easy to implement for making quick decisions in real-time. The present study proposes a novel method of integrating Multi-Criteria Decision Making (MCDM) methods and the Discrete Event Simulation (DES) Model to define job priorities in large-scale problems involving multiple criteria. DES approach is employed to model the PFJSS to evaluate Makespan, Flow Time, and Tardiness-based measures considering static and dynamic job arrivals. The proposed approach is implemented in a benchmark problem and large-scale PFJSS. The integration of MCDM methods and simulation models offers the flexibility to choose the parameters that need to govern the ranking of jobs. The solution given by the proposed methods is tested with the best-performing Composite Dispatching Rules (CDR), combining several PDR, which are available in the literature. Proposed MCDM approaches perform well for Makespan, Flow Time, and Tardiness-based measures for large-scale real-world problems. The proposed methodology integrated with the DES model is easy to implement in a real-time shop floor environment.

A tabu-based adaptive large neighborhood search for scheduling unrelated parallel batch processing machines with non-identical job sizes and dynamic job arrivals

A tabu-based adaptive large neighborhood search for scheduling unrelated parallel batch processing machines with non-identical job sizes and dynamic job arrivals

Real-time scheduling for distributed permutation flowshops with dynamic job arrivals using deep reinforcement learning

Distributed manufacturing plays an important role for large-scale companies to reduce production and transportation costs for globalized orders. However, how to real-timely and properly assign dynamic orders to distributed workshops is a challenging problem. To provide real-time and intelligent decision-making of scheduling for distributed flowshops, we studied the distributed permutation flowshop scheduling problem (DPFSP) with dynamic job arrivals using deep reinforcement learning (DRL). The objective is to minimize the total tardiness cost of all jobs. We provided the training and execution procedures of intelligent scheduling based on DRL for the dynamic DPFSP. In addition, we established a DRL-based scheduling model for distributed flowshops by designing suitable reward function, scheduling actions, and state features. A novel reward function is designed to directly relate to the objective. Various problem-specific dispatching rules are introduced to provide efficient actions for different production states. Furthermore, four efficient DRL algorithms, including deep Q-network (DQN), double DQN (DbDQN), dueling DQN (DlDQN), and advantage actor-critic (A2C), are adapted to train the scheduling agent. The training curves show that the agent learned to generate better solutions effectively and validate that the system design is reasonable. After training, all DRL algorithms outperform traditional meta-heuristics and well-known priority dispatching rules (PDRs) by a large margin in terms of solution quality and computation efficiency. This work shows the effectiveness of DRL for the real-time scheduling of dynamic DPFSP.

Dynamic Scheduling of Crane by Embedding Deep Reinforcement Learning into a Digital Twin Framework

This study proposes a digital twin (DT) application framework that integrates deep reinforcement learning (DRL) algorithms for the dynamic scheduling of crane transportation in workshops. DT is used to construct the connection between the workshop service system, logical simulation environment, 3D visualization model and physical workshop, and DRL is used to support the core decision in scheduling. First, the dynamic scheduling problem of crane transportation is constructed as a Markov decision process (MDP), and the corresponding double deep Q-network (DDQN) is designed to interact with the logic simulation environment to complete the offline training of the algorithm. Second, the trained DDQN is embedded into the DT framework, and then connected with the physical workshop and the workshop service system to realize online dynamic crane scheduling based on the real-time states of the workshop. Finally, case studies of crane scheduling under dynamic job arrival and equipment failure scenarios are presented to demonstrate the effectiveness of the proposed framework. The numerical analysis shows that the proposed method is superior to the traditional dynamic scheduling method, and it is also suitable for large-scale problems.

Network-based dynamic dispatching rule generation mechanism for real-time production scheduling problems with dynamic job arrivals

Although the concept of Industrial 4.0 has been well accepted, only few studies have dealt with real-time production scheduling of smart factories. Due to the advantages of simplicity, efficiency and quick response, heuristic rules have become the most promising technology to solve such problems. However, they suffer some drawbacks, such as high development and maintenance costs, low solution quality, and excessive emphasis on local information. To design heuristics from the perspective of system optimization and ensure the performance of heuristics in real-time production scheduling environments, this study develops a network-based dynamic dispatching rule generation mechanism. The complex network theory is introduced to extract a series of low-level heuristics from the perspective of system optimization, while the automatic heuristic generation problem is formulated as a multiple attribute decision making problem. Given that the dispersity of local features indicates their value for decision-making, the entropy weighting method is employed to automatically produce an adequate combination of the provided easy-to-implement low-level heuristics. Finally, the open shop scheduling problem with dynamic job arrivals is taken as an example to evaluate the effectiveness of the proposed algorithm. Numerical results demonstrate the excellent performance of the proposed algorithm in terms of algorithm effectiveness and computational time.

SIMULTANEOUS JOB-SHOP SCHEDULING AND MAINTENANCE PLANNING WITH ENERGY CONSIDERATION

This simulation study integrates the reliability-centered preventive maintenance approach and job-shop scheduling in a stochastic environment with energy consideration. Five reliability levels are considered ranging from 0.74 to 0.90 in the step of 0.04. The shop consists of six job types and ten distinct machines with dynamic job arrival. The system performance was measured by maximum flow-time, maximum-tardiness, mean-consumed operational energy, and mean-consumed ideal energy. The results indicate that each performance measure's value increases as the reliability levels increase. However, from a maintenance planning viewpoint, lower levels of reliability (i.e., 0.74, 0.78, and 0.82) are recommended for all performance metrics.

Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing

To realise the intelligent decision-making of dynamic scheduling and reconfiguration, we studied the intelligent scheduling and reconfiguration with dynamic job arrival for a reconfigurable flow line (RFL) using deep reinforcement learning (DRL), for the first time. The system architecture of intelligent scheduling and reconfiguration in smart manufacturing is proposed, and the mathematical model is established to minimise total tardiness cost. In addition, a DRL system of scheduling and reconfiguration is proposed by designing state features, actions, and rewards for scheduling and reconfiguration agents. Moreover, the advantage actor-critic (A2C) is adapted to solve the studied problem. The training curve shows the A2C-based agents have effectively learned to generate better solutions for unseen instances. The test results show that the A2C-based approach outperforms two traditional meta-heuristics, iterated greedy (IG) and genetic algorithm (GA), in solution quality and CPU times by a large margin. Specifically, the A2C-based approach outperforms IG and GA by 57.43% and 88.30%, using only 0.46‱ and 2.20‱ CPU times of IG and GA. The trained model can generate a scheduling or reconfiguration decision within 1.47 ms, which is almost instantaneous and can satisfy real-time optimisation. Our work shows a promising prospect of using DRL for intelligent scheduling and reconfiguration.

Dynamic scheduling of manufacturing systems: a product-driven approach using hyper-heuristics

ABSTRACT Dynamic scheduling of manufacturing systems is encountered in many real-world industries such as the food and pharmaceutical industries. The scheduling of these systems must not only be efficient but also reactive to cope with dynamic job arrivals and machine breakdowns. Over the last decade, products within Product-Driven Control Systems (PDCS) have become smart entities capable of actively handling the manufacturing process. In this paper, a PDCS based on the design of Smart Products is proposed. A generic model of the decisional strategy allows Smart Products to characterize different decisional contexts and thus switch efficiently from one scheduling rule to another using a novel Hyper-Heuristics (HH) based approach. The implementation and testing of the proposed PDCS on hybrid flexible flow-shops with multiple constraints inspired by the pharmaceutical industry are presented. The comparative study with 168 combinations of scheduling rules from the literature highlighted the superiority of the HH to minimize the Mean Completion Time. Furthermore, the proposed approach enhanced both the global performance and the reactivity of the manufacturing control system.

Surrogate-Assisted Symbiotic Organisms Search Algorithm for Parallel Batch Processor Scheduling

Parallel batch processor scheduling with dynamic job arrival is complex and challenging in semiconductor manufacturing. In order to get its reliable and high-performance schedule in a reasonable time, this work decomposes this scheduling problem into two-stage solution strategy: a batch forming subproblem and a batch scheduling subproblem. The batch formation is made by a heuristic rule. Then, a surrogate-assisted symbiotic organisms search algorithm with a new encoding mechanism is utilized to search for the optimal batch schedule, which integrates a surrogate model and a parameter control scheme. The surrogate model, which can predict the sequencing result instead of time-consuming true fitness evaluation, is used to reduce the computational burden greatly. In this article, a parameter control scheme based on reinforcement learning is proposed to balance the global and local search of symbiotic organisms search algorithm, as a guide for searching an assignment scheme. Finally, the experimental results demonstrate that the proposed algorithm can significantly improve the quality of a solution and save computational time via parameter control scheme and surrogate model.

An optimization method for energy-conscious production in flexible machining job shops with dynamic job arrivals and machine breakdowns

With rising energy prices and environmental concerns, reduction of energy consumption has become a critical manufacturing focus. One appropriate way to reduce energy consumption in manufacturing systems is to develop energy-conscious optimization strategies for production planning. In a flexible machining job shop, this planning must accommodate common dynamic events, such as new job arrivals and machine breakdowns. Dynamic events could change production energy consumption, thus require plan changes in pursuit of energy consumption reduction. To this end, this paper proposes an energy-conscious optimization method in flexible machining job shops considering dynamic events. In this paper, a optimization method which updates the jobs and machine plan status when dynamic events occur is proposed. The method considers two states for machine tool energy consumption: actual machining and machine idling/stand-by. The optimization model considers the total energy consumption and makespan, and employs Non-dominated Sorting Gene Algorithm II (NSGA-II) approach to obtain a solution. The proposed method is evaluated with a test case in which a flexible machining job shop experiences new dynamic job arrivals and machine breakdowns. The results show that the proposed method is effective at adjusting the schedule in response to dynamic events.

Effective dynamic selection of smart products scheduling rules in FMS

The present paper addresses the dynamic scheduling problem of smart products in a Flexible Manufacturing System (FMS). Unrelated machines, multiple product families, operation setups and dynamic job arrivals are considered. The goal is to propose a dynamic selection of smart products scheduling rules enriched with a context perception capability that is optimized using a hyper heuristic. A proof of concept with preliminary results is given.

A Heuristic for Inserting Randomly Arriving Jobs Into an Existing Hoist Schedule

Hoist scheduling in automated electroplating lines has been extensively studied in a static environment. However, practical electroplating lines are subject to diversified unforeseen disruptions that require frequent rescheduling to maintain or optimize system performance. This paper addresses a hoist scheduling problem, where randomly arriving jobs need to be inserted into an existing schedule without changing the sequence of hoist moves already scheduled. The objective is to minimize the total completion time of all the jobs in the existing schedule and a newly inserted job. We develop a polynomial-time heuristic that adjusts the starting times of the existing hoist moves to a limited extent but does not bring about a severe disturbance of the existing hoist moves. We compare our algorithm with two existing approaches with different rescheduling policies (i.e., partial and zero adjustment of the existing schedule). We empirically analyze the productivity and the stability of the schedules generated by the three approaches. Computational results demonstrate that our algorithm can generate more productive and stable schedules than the two existing approaches. Note to Practitioners —Electroplating and chemical surface treatment lines with automated material handling hoists are commonplace in electronics, semiconductor, and many other manufacturing industries. In an uncertain environment, hoist rescheduling plays an important role in improving the productivity and reducing the impact of disruptions. This paper presents a hoist scheduling algorithm to deal with dynamic job arrivals by considering the impact of the disturbance incurred by rescheduling. Our algorithm can generate a better schedule with smaller total completion time and slighter disturbance than the existing algorithms. The proposed algorithm runs in polynomial time and can be used to control hoist operations in practical electroplating lines. A comparative analysis provides useful insights on the implementation of rescheduling approaches and policies to industry practitioners.

A multi-objective differential evolution algorithm for parallel batch processing machine scheduling considering electricity consumption cost

The manufacturing industry consumes massive amounts of energy and produces great numbers of greenhouse gases every year. Recently, an increasing attention has been paid to the energy efficiency of the manufacturing industry. This paper considers a parallel batch processing machine (BPM) scheduling problem in the presence of dynamic job arrivals and a time-of-use pricing scheme. The objective is to simultaneously minimize makespan, a measure of production efficiency and minimize total electricity cost (TEC), an indicator for environmental sustainability. A BPM is capable of processing multiple jobs at a time, which has wide applications in many manufacturing industries such as electronics manufacturing facilities and steel-making plants. We formulate this problem as a mixed integer programming model. Considering the problem is strongly NP-hard, a multi-objective differential evolution algorithm is proposed for effectively solving the problem at large scale. The performance of the proposed algorithm is evaluated by comparing it to the well-known NSGA-II algorithm and another multi-objective optimization algorithm AMGA. Experimental results show that the proposed algorithm performs better than NSGA-II and AMGA in terms of solution quality and distribution.

An Efficient Bi-objective Genetic Algorithm for the Single Batch-Processing Machine Scheduling Problem with Sequence Dependent Family Setup Time and Non-identical Job Sizes

This paper considers the problem of minimizing make-span and maximum tardiness simultaneously for scheduling jobs under non-identical job sizes, dynamic job arrivals, incompatible job families,and sequence-dependentfamily setup time on the single batch- processor, where split size of jobs is allowed between batches. At first, a new Mixed Integer Linear Programming (MILP) model is proposed for this problem; then, it is solved by -constraint method.Since this problem is NP-hard, a bi-objective genetic algorithm (BOGA) is offered for real-sized problems. The efficiency of the proposed BOGA is evaluated to be comparedwith many test problemsby -constraint method based on performance measures. The results show that the proposed BOGAis found to be more efficient and faster than the -constraint method in generating Pareto fronts in most cases.