Real-time Scheduling Method Research Articles

Deep reinforcement learning-based optimal scheduling of integrated energy systems for electricity, heat, and hydrogen storage

The increasing load demands and the extensive usage of renewable energy in integrated energy systems pose a challenge to the most efficient scheduling of integrated energy systems (IES) because of the unpredictability and volatility of both the load side and renewable energy.Integrating heat storage and hydrogen storage technologies into integrated energy systems can effectively alleviate the phenomena of wind and solar power desertion caused by the instability of renewable energy sources. This study introduces a real-time optimal scheduling method based on the Soft Actor-Critic (SAC) algorithm, aimed at reducing system operating costs and enhancing the consumption rate of renewable energy. First, the established mathematical model is converted into a Markov Decision Process (MDP), the objective function is converted by designing the action state space and the reward function, and the deep reinforcement learning framework is established. Finally, the decision-making outcomes of intelligence in various energy storage scenarios of renewable energy consumption and extreme cases are analyzed and compared, and the results show that the heat storage and hydrogen storage system significantly improve the rate of renewable energy consumption and the economy of the system. This method is also shown to significantly improve the rate of renewable energy consumption.

Proactive real-time scheduling method for apron service vehicles based on mixed strategies

Due to the uncertainty of flight arrival and departure times, the scheduling of apron vehicles in airports has always faced significant challenges. This paper considers the real-time scheduling problem of apron vehicles under dynamic requests, establishing a new parking rule to allow apron vehicles to dynamically adjust parking areas. In order to maximize service level and minimize operational costs, this paper proposes a proactive real-time scheduling method for apron vehicles based on mixed strategies. Through strategies such as flexible waiting and dynamic relocation, this paper responds proactively to future requests. The proposed strategies are validated through implementation in large and medium-sized scale cases at Beijing Capital International Airport and Nanjing Lukou International Airport in China, respectively. Numerical results indicate that the proposed scheduling method is effective and applicable. The flexible waiting strategy significantly reduces vehicle travel distance, while the dynamic relocation strategy is significant in shortening request response times.

Deep reinforcement learning for dynamic distributed job shop scheduling problem with transfers

Dynamic events and transportation constraints would significantly affect the full utilization of resources and the reduction of production costs in distributed job shops. Therefore, in this paper, a deep reinforcement learning algorithm (DRL)-based real-time scheduling method is developed to minimize the mean tardiness of the dynamic distributed job shop scheduling problem with transfers (DDJSPT) considering random job arrivals. Firstly, the proposed DDJSPT is modeled as a Markov decision process (MDP). Then, ten problem-oriented state features covering four aspects of factories, machines, jobs, and operations are elaborately extracted from the dynamic distributed job shop. After that, eleven composite rules considering the uniqueness of DDJSPT are constructed as a pool of actions to intelligently prioritize unfinished jobs and allocate the selected job to an appropriate factory. Moreover, a justified reward function adapted from the objective is designed for better convergence of DRLs. Subsequently, five DRLs are employed to address the DDJSPT, encompassing deep Q-network (DQN), double DQN (DDQN), dueling DQN (DlDQN), trust region policy optimization (TRPO), and proximal policy optimization (PPO). Finally, grounded in numerical comparison experiments under 243 production configurations of the DDJSPT, the effectiveness and generalization of DRL-based scheduling methods are credibly verified and confirmed.

Optimal scheduling of CCHP-based resilient energy distribution system considering active microgrids' multi-carrier energy transactions

This paper introduces a two-stage two-level optimization method for optimal day-ahead and real-time scheduling of multicarrier energy distribution systems and microgrids. The model considers the incentive-based and price-based demand response programs to encourage microgrids to transact electrical, heating, and cooling energy carriers with the energy distribution system, which is named hereafter as the energy system. Further, the model formulates the resilient operation of the energy system considering the energy transactions with the electrical, heating, and cooling markets. The main contribution of this paper is the integration of demand response procedures of microgrids in energy transactions with the energy system considering the switching of electrical switches and heating and cooling control valves. The optimization process is another contribution of this paper that is decomposed into two stages that consist of day-ahead and real-time horizons. The first stage is also decomposed into two levels that determine the optimal scheduling of the energy system and microgrids in day-ahead markets. The second stage is comprised of two levels that commit the energy system and microgrids resources. A resiliency index is proposed to assess the resiliency of the energy system in shock conditions. The proposed method was simulated for the 123-bus test system. Different types of microgrids, incentive-based and price-based demand response processes were considered. Simulation results confirmed that the proposed method can reduce the costs of residential, industrial, and commercial microgrids by about 4.47%, 3.88%, and 5.47% concerning only the real-time pricing process. Further, the model can increase the aggregated benefits of the energy system in the day-ahead and real-time markets by about 0.608 Million Monetary Units (MMUs) and 1.10 MMUs, respectively.

Adaptive real-time reconfiguration gate scheduling scheme using time perceptive stream

An adaptive real-time gate scheduling scheme for time perceptive stream or packet flow is proposed to improve the standards of Ultra Low Latency during data transmission. For highly dynamic network conditions, the conventional configuration scheme is not suitable and therefore an adaptive real-time gate scheduling method is proposed for time perceptive streams. This dynamicity reconfiguration is difficult and the scheduling problem is formulated using Field Programmable Gate Array – Boolean Satisfiability Problem (FPGA-BSP) solver. This proposed scheme highly helps in network dynamicity conditions with good bandwidth utilization and high flexibility. End-to-end latency is required to be on sub-milliseconds order deal with the applications such as Industrial Internet of Things, 5G and 6G mobile, tactile internet and so on. Simulation analysis is carried out to prove the efficiency of the proposed model.

Edge computing-based real-time scheduling for digital twin flexible job shop with variable time window

• A new R-DTFJSS idea was developed to realise high-efficiency production. • Edge computing-based R-DTFJSS framework can enhance the data processing ability. • A variable time window-based RS strategy was used to achieve operation allocation. • An IHA-based solve method was designed to reduce the computational complexity. Production scheduling is the central link between enterprise production and operation management and is also the key to realising efficient, high-quality and sustainable production. However, in real-world manufacturing, the frequent occurrence of abnormal disturbance leads to the deviation of scheduling, which affects the accuracy and reliability of scheduling execution. The traditional dynamic scheduling methods (TDSMs) cannot solve this problem effectively. This paper presents a real-time digital twin flexible job shop scheduling (R-DTFJSS) method with edge computing to address the issue. Firstly, an overall framework of R-DTFJSS is proposed to realise real-time scheduling (RS) through real-time interaction between physical workshop (PW) and virtual workshop (VW). Secondly, the implementation process of R-DTFJSS is designed to realise real-time operation allocation. Then, to obtain the optimal RS result, an improved Hungarian algorithm (IHA) is adopted. Finally, a case simulation from an industrial case of a cooperative enterprise is described and analysed to verify the effectiveness of the proposed R-DTFJSS method. The results show that compared with the TDSMs, the R-DTFJSS method can effectively deal with unexpected and frequent abnormal disturbances in the production process.

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

Multi-uncertainties from power sources and loads have brought significant challenges to the stable demand supply of various resources at islands. To address these challenges, a comprehensive scheduling framework is proposed by introducing a model-free deep reinforcement learning (DRL) approach based on modeling an island integrated energy system (IES). In response to the shortage of freshwater on islands, in addition to the introduction of seawater desalination systems, a transmission structure of “hydrothermal simultaneous transmission” (HST) is proposed. The essence of the IES scheduling problem is the optimal combination of each unit’s output, which is a typical timing control problem and conforms to the Markov decision-making solution framework of deep reinforcement learning. Deep reinforcement learning adapts to various changes and timely adjusts strategies through the interaction of agents and the environment, avoiding complicated modeling and prediction of multi-uncertainties. The simulation results show that the proposed scheduling framework properly handles multi-uncertainties from power sources and loads, achieves a stable demand supply for various resources, and has better performance than other real-time scheduling methods, especially in terms of computational efficiency. In addition, the HST model constitutes an active exploration to improve the utilization efficiency of island freshwater.

Research on an Adaptive Real-Time Scheduling Method of Dynamic Job-Shop Based on Reinforcement Learning

With the rapid development of modern industrialization in our country and the continuous improvement of people’s living standards, the changing market has put forward new requirements for traditional manufacturing enterprises. The order product demand of many manufacturing enterprises is changing from a single variety, large batch to multiple varieties, small batch. In view of this change, the traditional job-shop scheduling method is far from enough, which greatly affects the efficiency of the production job-shop. In order to solve the above problems, this paper proposes a real-time scheduling method based on reinforcement learning applied in the dynamic job-shop and a new type of neural network is designed at the same time. The neural network is designed with the high-dimensional data in the above problem as input, and a policy-based reinforcement learning algorithm is proposed based on this network. In the process of research, it was found that the reinforcement learning method not only enables the agent to use historical data for learning, but also enables it to explore and learn other possible high-reward actions within a certain range, so as to realize the optimization of production goals under real-time scheduling. The effectiveness of the proposed real-time scheduling method is verified by comparing with other common rule-based scheduling methods in the manufacturing environment.

Real-Time Scheduling for Dynamic Partial-No-Wait Multiobjective Flexible Job Shop by Deep Reinforcement Learning

In modern discrete flexible manufacturing systems, dynamic disturbances frequently occur in real time and each job may contain several special operations in partial-no-wait constraint due to technological requirements. In this regard, a hierarchical multiagent deep reinforcement learning (DRL)-based real-time scheduling method named hierarchical multi-agent proximal policy optimization (HMAPPO) is developed to address the dynamic partial-no-wait multiobjective flexible job shop scheduling problem (DMOFJSP-PNW) with new job insertions and machine breakdowns. The proposed HMAPPO contains three proximal policy optimization (PPO)-based agents operating in different spatiotemporal scales, namely, objective agent, job agent, and machine agent. The objective agent acts as a higher controller periodically determining the temporary objectives to be optimized. The job agent and machine agent are lower actuators, respectively, choosing a job selection rule and machine assignment rule to achieve the temporary objective at each rescheduling point. Five job selection rules and six machine assignment rules are designed to select an uncompleted job and assign the next operation of which together with its successors in no-wait constraint on the corresponding processing machines. A hierarchical PPO-based training algorithm is developed. Extensive numerical experiments have confirmed the effectiveness and superiority of the proposed HMAPPO compared with other well-known dynamic scheduling methods. Note to Practitioners—The motivation of this article stems from the need to develop real-time scheduling methods for modern discrete flexible manufacturing factories, such as aerospace product manufacturing and steel manufacturing, where dynamic events frequently occur, and each job may contain several operations subjected to the no-wait constraint. Traditional dynamic scheduling methods, such as metaheuristics or dispatching rules, either suffer from poor time efficiency or fail to ensure good solution quality for multiple objectives in the long-term run. Meanwhile, few of the previous studies have considered the partial-no-wait constraint among several operations from the same job, which widely exists in many industries. In this article, we propose a hierarchical multiagent deep reinforcement learning (DRL)-based real-time scheduling method named HMAPPO to address the dynamic partial-no-wait multiobjective flexible job shop scheduling problem (DMOFJSP-PNW) with new job insertions and machine breakdowns. The proposed HMAPPO uses three DRL-based agents to adaptively select the temporary objectives and choose the most feasible dispatching rules to achieve them at different rescheduling points, through which the rescheduling can be made in real time and a good compromise among different objectives can be obtained in the long-term schedule. Extensive experimental results have demonstrated the effectiveness and superiority of the proposed HMAPPO. For industrial applications, this method can be extended to many other production scheduling problems, such as hybrid flow shops and open shop with different uncertainties and objectives.

Digital twin and deep reinforcement learning enabled real-time scheduling for complex product flexible shop-floor

Real-time scheduling methods are essential and critical to complex product flexible shop-floor due to the dynamic events in the production process, such as new job insertions, machine breakdowns and frequent rework. Recently, digital twin (DT) technology can help identify disturbances by continuously comparing physical space with virtual space, which enables real-time scheduling and greatly reduces the deviation between pre-schedule and actual schedule. However, the conventional scheduling models and algorithms cannot satisfy the adaptiveness and timeliness requirements of optimization in DT enabled shop-floor (DTS). To address above challenges, an overall framework of DT enabled real-time scheduling (DTE-RS) for complex product shop-floor is proposed to effectively reduce adverse impacts of the dynamic disturbances and minimize the makespan. Firstly, complex product flexible job shop scheduling problem (CPFJSP) is formulated as Markov Decision Process (MDP), taking into account machine breakdown and new job insertions. Then, deep Q-network (DQN) based solution is developed to achieve optimal task dispatching according to real-time production state. Finally, the case study for aircraft overhaul shop-floor is conducted to demonstrate effectiveness and feasibility of the proposed real-time scheduling method. Through experimental comparison, it is indicated that the proposed method could effectively respond to dynamic disturbances and outperform the dynamic scheduling method in terms of makespan.

Day-ahead and real-time congestion scheduling method for distribution network with multiple access to electric vehicle charging piles

Abstract The day-ahead and real-time congestion scheduling method for distribution network with multiple access to electric vehicle charging piles is studied to effectively solve the day-ahead and real-time congestion scheduling problem of distribution network. The charge adjustment strategy of charge and discharge service fee is established to realize the double response regulation between the distribution system’s scheduling organization and the charging pile operator; considering the adjustment strategy of charging service fee, a day-ahead congestion scheduling model is established with the goal of minimizing the charging cost of electric vehicles; based on the day-ahead congestion scheduling, a real-time congestion scheduling model is established to minimize the regional power fluctuation; the day-ahead and real-time congestion scheduling model is solved by differential evolution algorithm, and the optimal scheduling scheme is obtained. Experiments show that this method can reduce the line load rate of distribution network, avoid re-congestion, reduce the congestion scheduling cost and improve the security and economy of power grid operation.

Dynamic long-term expansion planning of electric vehicle parking lots considering lost opportunity cost and energy saving

In this paper, a practical approach is presented to long-term expansion planning of electric vehicle (EV) parking lots which aims to maximize the parking lot operator’s profit. The optimal number, location, capacity, and time for construction or expansion of EV parking lots have also been determined. Moreover, a method based on drivers’ travel patterns is proposed to estimate the number of EVs referring to each area, in which the unscented transformation method (UTM) is used to calculate the annual growth rate of EVs. An innovative and real-time method for optimal charge and discharge scheduling of EVs is also provided, which can calculate the parking lots’ demand at different times. The most important advantages of this method are its independence from iterative-based optimization algorithms and from modeling some input parameters such as the initial state of charge (SOC) and the arrival time of EVs into the parking lot as uncertain parameters. Also, a Lagrange interpolation-based method is proposed to minimize the energy consumption required for EVs to reach a parking lot. In addition, in order to prevent the reduction of parking lot operator’s profit in case of non-fulfillment of the obligations of the EV owners, an approach based on paying the lost opportunity cost (LOC) is presented. Real-world data has been used for numerical studies. The results show that based on travel patterns and estimating the number of EVs available in each municipal district and using the proposed real-time scheduling method, parking lot’s demand can be calculated as the main prerequisite for parking lots expansion planning. Also, the proposed method for parking selection compared to two other methods will save energy by 26.6% and 8.33%, respectively. Moreover, the proposed approach increases the total parking lot operator’s profit by 4.23%.

A Two-Layer Model for Microgrid Real-Time Scheduling Using Approximate Future Cost Function

Microgrids incorporate an increasing number of distributed energy resources (DERs), which induce a higher variability and faster dispatch capabilities in power systems. This paper proposes a two-layer real-time scheduling model for microgrids, based on approximate future cost function (AFCF), where the future cost represents the opportunity cost for the microgrid operation in subsequent periods. At the upper layer, the look-ahead rolling scheduling is adopted to optimize microgrid operations, in which the future cost function (FCF) in deterministic and stochastic scenarios is approximated by a piecewise linear function. At the lower layer, a real-time parameter updating strategy based on real-time data is proposed. In this case, the real-time scheduling readjusts the look-ahead schedule using the immediate cost in the current period and the future cost calculated by the updated AFCF. The proposed two-layer real-time scheduling model uses an offline optimization, in which most of the computation tasks are completed at the upper layer, and applies a real-time optimization, in which the time-consuming problem is avoided at the lower layer. The effectiveness of the proposed two-layer real-time scheduling model of microgrids is validated by using a grid-connected microgrid system. For comparison, other existing real-time scheduling methods are also implemented in the same microgrid system.

HiPGen: An approach for fast generation of multi-satellite observation plans via a hierarchical multi-channel transformer network

Effective and fast Earth Observation Satellite (EOS) scheduling plays an essential role in new generation space-based information services, especially in emergency scenarios. Most existing research regards the EOS scheduling as a combinatorial optimization problem and adopts metaheuristic local search-based methods to solve it in a batch-wise fashion. To the best of our knowledge, no practical scheduling approach has been able to generate an optimized multi-EOS observation plan in an immediate response style. This paper proposes a novel real-time multi-satellite scheduling method consisting of a machine learning-based hierarchical prediction model and a heuristic local search algorithm. Firstly, the hierarchical prediction model based on a stacked multi-channel transformer network can learn from existing historical multi-EOS observation plans and generate a high-quality initial solution for the current scheduling scenario. Then, the local search algorithm based on Random Hill Climbing does further improvement and heuristic constraint handling on the initial solution and generates the final solution. To verify the effectiveness of the proposed method, simulation experiments are carried out. The results show that the method proposed in this paper consumes a short calculation time and has high planning profits compared with the state-of-the-art approaches.

Real-time scheduling simulation optimisation of job shop in a production-logistics collaborative environment

ABSTRACT In a complex and dynamic job shop containing logistics factor, schedule needs to be generated rapidly, so the real-time scheduling method is more suitable for such scenario. Such method takes advantage of local information within a short time due to the rapid changes of information under uncertain environment. Therefore, how to make use of the future information by prediction while ensuring the robustness of schedule is a valuable problem. To solve it, firstly, a new real-time scheduling model and algorithm is proposed. There is a new kind of release moment of task information which can give AGVs the longest time to prepare for the task than existing research. Secondly, a real-time information update mechanism is designed to increase schedule’s robustness. Finally, a large-scale and dynamic job shop simulation experimental platform is developed. Dynamic factors include the random insertion of orders and failures of equipment. Results show that the method proposed outperforms existing research in terms of customer satisfaction, equipment utilisation and energy consumption. The robustness of schedule can also be acceptable. This paper also finds a rule that in job shop with the large proportion of logistics transportation time, the above method can achieve more competitive results.

Reinforcement learning and digital twin-based real-time scheduling method in intelligent manufacturing systems

Optimization efficiency and decision-making responsiveness are two conflicting objectives to be considered in intelligent manufacturing. Therefore, we proposed a reinforcement learning and digital twin-based real-time scheduling method, called twins learning, to satisfy multiple objectives simultaneously. First, the interaction of multiple resources is constructed in a virtual twin, including physics, behaviors, and rules to support the decision-making. Then, the real-time scheduling problems are modeled as Markov Decision Process and reinforcement learning algorithms are developed to learn better scheduling policies. The case study indicates the proposed method has excellent adaptability and learning capacity in intelligent manufacturing.

Impact analysis of cyber system in microgrids: Perspective from economy and reliability

With the increasing requirements of power supply quality and operation economy of the microgrid in smart grid field, real-time energy scheduling methods that highly rely on the cyber system are inevitable supports for the optimal operation of microgrids. As a typical cyber-physical system, the disturbance and failure on the cyber side may make the performance of microgrids deviate from the expected ones. To quantify the degree to which these cyber contingencies influence the microgrid system, we propose a novel quantification and analytical framework. In this framework, the cyber system of a centrally controlled microgrid is abstracted consisting of information conversion, transmission and routing forward, as well as fusion and calculation. On this basis, the general information flow is modeled through information mapping between measured state and control command. Furthermore, an equivalent model for a cyber-physical microgrid system is formulated by integrating information mapping model into the state transition of physical microgrid. With this equivalent model, a unified impact analytical procedure is presented to accurately quantify the physical impact of multi-cyber-contingencies in both economy and reliability aspects. Numerical simulations on a microgrid validate the reasonableness and effectiveness of the proposed method.

An adaptive real‐time energy management system for a renewable energy‐based microgrid

This paper proposes an adaptive real-time energy scheduling method (RT-EMS) for a microgrid, using a Lyapunov optimization-based real-time approach. Inaccuracy in day-ahead predictions can result in non-optimal solutions to the energy scheduling problem. Although the real-time optimization method eliminates the need to deal with the prediction uncertainties, it ignores the valuable statistical information used in day-ahead stochastic approaches and provides suboptimal solutions to the problem. The proposed adaptive approach combines the advantages of both the stochastic day-ahead and the RT-EMS and reduces the real-time operational cost of the microgrid. The proposed method moves the RT-EMS solution towards the optimal solution, by adding a penalty term to the objective function. Numerical results are provided to demonstrate the improved performance of the proposed adaptive method.

Efficient Real-Time EV Charging Scheduling via Ordinal Optimization

The surge of plug-in electric vehicles (PEV) on the roads poses the issue to handle the substantial charging demand. Particularly, the operation of charging stations requires an efficient and scalable real-time scheduling method to accommodate the dramatic charging requests in an economical (i.e., to best utilize the local renewable generation) and friendly (i.e., to reduce the impact on the electric grid) manner. This paper fulfills the objectives and makes the following main contributions. First, we develop a parameterized aggregated PEV charging model using the energy boundaries to express the charging flexibility. We propose to parameterize the aggregated charging policy by the incomplete Beta function based on the problem structures. The proposed model can scale down the decision variables from O(NH) to O(2) where N is the number of PEVs and H is the number of prediction horizon. Second, we develop an ordinal optimization (OO) based method (denoted as OO-P) to search for good enough charging policies within seconds while still providing probabilistic performance guarantee. Third, we demonstrate the performance of the proposed OO-P via simulations. The numerical results show that the solution of OO-P is only 4% worse than the optima. However, OO-P shows high computation efficiency and scalability. Compared with the existing real-time PEV charging scheduling method, OO-P can reduce 6% of the operation cost for the charging station. OO-P is also shown to outperform the existing heuristic rule.

Real-time scheduling for carrier-borne aircraft support operations: a reinforcement learning approach

The carrier-borne aircraft dispatch rate is an important indicator to measure the combat performance of an aircraft carrier. The key factor affecting the carrier-borne aircraft dispatch rate is the efficiency of the carrier aircraft support operation scheduling. Shipboard aircraft support operation scheduling refers to a rational arrangement of the order of support operations required by the carrier aircraft as well as to an efficient completion of the support operations for the carrier-borne aircraft under constraints of limited time, space, and resources. The existing solution strategies based on optimization methods (dynamic programming, linear programming, etc.) and heuristic methods (genetic algorithm, particle swarm optimization, etc.) are only suitable for operation scheduling in the case of predictable operations, and it is challenging to meet the real-time support operation scheduling requirements in highly dynamic combat scenarios. For this reason, a new real-time scheduling method for carrier-borne aircraft support operations based on deep Q-networks (DQNs) is proposed. This method consists of modeling the scheduling problem of aircraft support operations as a partially observable Markov decision process (POMDP) problem. The global and long-term benefits are used to optimize the scheduling process, and the decision-making framework of offline learning and online deployment is used to solve the scheduling problem of aircraft support operations. The simulation results demonstrate that this new method can significantly improve the efficiency of shipboard aircraft support operation scheduling, thus enabling to meet the needs of real-time decision-making environments.