Heuristic Dynamic Programming Research Articles

Solving dynamic facility layout problem using a hybridized heuristic dynamic programming approach

Today’s market volatility shortens product lifecycles and drives constant changes in product mix and demand. Changes in product mix and demand necessitate, in turn, changes to the shopfloor layout. The dynamic facility layout problem (DFLP) addresses layout development over multiple periods, with rearrangements from one period to another. To optimize the DFLP using dynamic programming (DP), the DP state space should be restricted as the problem is NP-hard. Therefore, a two-phased hybridized solution algorithm is being proposed and developed in this article. In the first phase, a heuristic approach is used to determine the set of layouts to be considered in each period. In the second phase, a metaheuristic approach is used to solve the recursive formulation of DP. A genetic algorithm (GA) searches for the best subsets of layouts, each represented by one chromosome. Notably, the GA incorporates a heuristic selection operator guided by a deep neural network algorithm. The best subset of layouts that results in the best multi-period layout plan is found throughout the different GA generations. The proposed method’s efficiency is statistically validated through rigorous statistical tests, affirming its superior performance, particularly for large-sized instances of the problem, and showcasing more efficient solutions.

Action-dependent heuristic dynamic programming for hybrid-order systems based on dynamic event-triggered mechanism

We investigate the distributed optimal consensus control problem based on action-dependent heuristic dynamic programming (ADHDP). In order to strike a stable balance between the learning cost of reinforcement learning and the resource utilization efficiency of the hybrid-order multi-agent systems (MASs), we propose an improved dynamic event-triggered ADHDP (dET-ADHDP) method. This approach can non-periodically explore the control policy distribution using the online action-dependent actor–critic (ADAC) learning framework. Meanwhile, it can dynamically adjust the trigger lower bound by exploiting the designed trigger threshold function, and adaptively decide the signal trigger moment during the ADAC learning process. In addition, we demonstrate the boundedness of the ADAC network weights and show that under the designed dynamic event-triggering rules, the MASs can asymptotically achieve optimal tracking control without Zeno phenomenon. Finally, compared with the traditional static counterparts, simulation experiments demonstrate that the proposed dynamic event-triggered ADAC (dET-ADAC) algorithm has more efficient resource utilization while maintaining satisfactory learning performance.

Event-Triggered Constrained Optimal Control for Organic Rankine Cycle Systems via Safe Reinforcement Learning.

The organic Rankine cycle (ORC) is an effective application for converting low-grade heat sources into power and is crucial for environmentally friendly production and energy recovery. However, the inherent complexity of the mechanism, its strong and unidentified nonlinearity, and the presence of control constraints severely impair the design of its optimal controller. To solve these issues, this study provides a novel event-triggered (ET) constrained optimal control approach for the ORC systems based on a safe reinforcement learning technique to find the optimal control law. Instead of employing the usual non-quadratic integral form to solve the control-limited optimal control problems, a constraint handling strategy based on a relaxed weighted barrier function (BF) technique is proposed. By adding the BF terms to the original value function, a modified value iteration algorithm is developed to make the control input solutions that tend to violate the constraints be pushed back and maintained in their safe sets. In addition, the ET mechanism proposed in this article is critically required for the ORC systems, and it can significantly reduce the computational load. The combination of these two techniques allows the ORC systems to achieve set-point tracking control and satisfy the control restrictions. The proposed approach is conducted based on a heuristic dynamic programming framework with three neural networks (NNs) involved. The safety and convergence of the proposed approach and the stability of the closed-loop system are analyzed. Simulation results and comparisons are presented to demonstrate its effectiveness.

Adaptive neural event‐triggered near‐optimal control for affined uncertain nonlinear discrete‐time system

AbstractA novel event‐triggered heuristic dynamic programming (HDP) algorithm is proposed for the near‐optimal control of uncertain discrete‐time nonlinear input‐affine systems. Based on input‐to‐state stability (ISS) analysis, a new event‐triggered mechanism (ETM) is designed. Under constant coefficients, a Lipschitz‐like assumption that forms the basis of the event‐triggering condition is considered to be conservative. To further reduce the conservativeness of the triggering condition and enlarge the average interevent time, an adaptive threshold parameter is utilized in the proposed ETM. In the HDP algorithm framework, model, critic, and action network are adopted to achieve state estimation, approximation to the optimal cost function, and solution to Hamilton–Jacobian–Bellman (HJB) equation. Under the proposed event‐triggered HDP algorithm, the closed system is proved to possess semiglobal uniform ultimate boundedness (SGUUB). Finally, by conducting simulation, it shows that on the premise of satisfying control performance, the event‐triggered strategy can realize reduction on the updating frequency of the controller.

An efficient energy management strategy based on heuristic dynamic programming specialized for hybrid electric unmanned delivery aerial vehicles

Large-sized unmanned delivery aerial vehicles (UDAVs) stand out as prominent vertical take-off and landing aircraft, benefiting from integrating a hybrid electric propulsion system (HEPS) to enhance the power-to-weight ratio. Despite this advancement, UDAVs encounter challenges regarding range limitations and the need for swift computational solutions. Addressing these issues necessitates a computationally efficient energy management strategy (EMS) that optimizes power distribution to properly plan engine working points, ultimately aimed at enhancing fuel economy to increase flight range. This study proposes a heuristic dynamic programming (HDP)-based EMS tailored for the HEPS of a large-sized UDAV. In this strategy, the HDP framework is adopted to improve computational efficiency by iteratively updating control variables. To account for the specific flight characteristics, a flight state identifier incorporating selected flight command flags applicable to UDAVs is designed. This identifier aids in predicting velocity changes for different flight states by amalgamating corresponding prediction networks into an integrated velocity prediction network (IVPN). Substituting the model network with IVPN in the HDP framework significantly enhances prediction accuracy and guides power distribution towards optimal results, signifying an efficient EMS specialized for UDAVs. The proposed strategy is evaluated against baseline strategies in designed flight scenarios, highlighting a remarkable 51.37% improvement in prediction accuracy, a noteworthy 6.04% reduction in equivalent fuel consumption, and an impressive 95.81% reduction in global computation time. Finally, the actual applicability of the proposed strategy is validated in a hardware-in-loop test. Consequently, the proposed strategy can provide theoretical support for the efficient energy management of diverse large-sized UDAVs.

Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes

Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes

Supplementary Control for Quantized Discrete-Time Nonlinear Systems Under Goal Representation Heuristic Dynamic Programming.

This article is concerned with supplementary control of discrete-time nonlinear systems with multiple controllers in the framework of goal representation heuristic dynamic programming (GrHDP), where a logarithmic quantizer is used to govern the network communication. For the addressed problem, a neural network (NN)-based observer is first proposed to estimate the unknown system state in the simultaneous presence of quantized influence. In light of the estimated states and the ideal control inputs via a zero-sum game, a GrHDP algorithm with a reinforced term is developed to implement the supplementary control task, where some novel weight updating rules are constructed by virtue of an additional tunable parameter to improve the system performance. Furthermore, a set of conditions about the stability of estimated error dynamics of both observer states and updated NNs' weights are derived by resorting to the Lyapunov stability theory. Finally, the effectiveness of the developed method is verified by a power system and a numerical experiment.

Optimizing location-routing and demand allocation in the household waste collection system using a branch-and-price algorithm

In catering to the rapid development of urbanization and the growing urban population, it is essential to design an efficient and effective household waste collection system for guaranteeing a favorable ecological environment in cities. This paper focuses on a location-routing and demand allocation problem (LRDAP) in the household waste collection system where the household waste is initially allocated and transported to a set of intermediate transfer stations using a category of small garbage vehicles, and subsequently a fleet of large garbage vehicles visits each transfer station and transports the collected household waste to the waste disposal center for harmless treatment. The purpose is to simultaneously determine the location of transfer stations, the demand allocation of household waste collection and the routing plan of garbage vehicles while maximizing the efficiency of household waste collection within the whole system. Then, we formulate the proposed LRDAP into a mixed integer program. To solve our LRDAP, an exact branch-and-price (B&P) algorithm is developed along with several acceleration strategies involving column generation (CG) stabilization and heuristic dynamic programming (DP). Finally, we illustrate the proposed model and solution methodology by dealing with real-world problem instances with respect to household waste collection in China. Computational results are presented that explore the performance of our proposed B&P algorithm, analyze the optimized location-routing and demand allocation plan, discuss the impact of selected parameters, and gain some managerial insights in designing an efficient and effective household waste collection system under different decision-making conditions.

Multi‐event‐triggered adaptive dynamic programming for non‐zero‐sum game of unknown nonlinear system

AbstractIn this article, a multi‐event‐triggered (MET) near‐optimal tracking control algorithm is developed for a discrete‐time (DT) nonlinear non‐zero‐sum game (NZSG). Firstly, the iterative dual heuristic dynamic programming (DHP) approach is utilized to obtain the optimal tracking control set. Subsequently, distinct independent triggering conditions are formulated for control inputs to improve the utilization efficiency of resources and ensure that the independence between them and the closed‐loop system has an excellent control performance. Meanwhile, input‐to‐state stability (ISS) is proven for the MET tracking plant. Additionally, neural networks (NNs) are designed in the iterative DHP algorithm to construct the model module, the critic module and the action modules, with aims to identify the unknown system, approximate costate function and estimate optimal tracking control strategy set, respectively. Finally, two numerical experimental simulations are conducted to verify the effectiveness of the proposed scheme.

Adaptive Critic Control Design With Knowledge Transfer for Wastewater Treatment Applications

The wastewater treatment process (WWTP) is of great significance to environmental protection. To improve the efficiency of the WWTP, it is crucial to ensure that the dissolved oxygen (DO) concentration tracks the set value efficiently. Due to the nonlinear and time-varying dynamics of the WWTP, traditional control methods cannot accurately control the DO concentration. To overcome these challenges, this paper proposes an online transferred heuristic dynamic programming (TrHDP) control design by combining transfer learning with adaptive critic design. First, we use the historical sample data to construct a mathematical model of the WWTP and learn the prior knowledge from the model. Then, the online control process of the DO concentration is guided by utilizing the prior knowledge. In order to avoid negative transfer and save computing resources, we design a novel decay function with the truncation mechanism. In addition, we prove the stability of the TrHDP control scheme by constructing a Lyapunov function. Finally, the performance of the TrHDP scheme is verified by the Benchmark Simulation Model No. 1. Compared with other methods, the TrHDP method possesses higher control accuracy for the DO concentration and overcomes the disadvantage of low learning efficiency of general online methods.

Supplementary heuristic dynamic programming for wastewater treatment process control

With the rapid development of industry, the amount of wastewater discharge is increasing. In order to improve the efficiency of the wastewater treatment process (WWTP), we often desire that the dissolved oxygen (DO) concentration and the nitrate nitrogen (NO) concentration can be controlled to track set values. However, the wastewater treatment system is a type of unknown nonlinear plant with time-varying dynamics and strong disturbances. Some traditional control methods are difficult to achieve this goal. To overcome these challenges, a supplementary heuristic dynamic programming (SUP-HDP) control scheme is established by combining the traditional control method and heuristic dynamic programming (HDP). A parallel control structure is constructed in the SUP-HDP control scheme, which not only complements the shortcomings of traditional control schemes in learning and adaptive abilities but also improves the convergence speed and the stability of the learning process of HDP. Besides, the convergence proof of the designed control scheme is provided. The SUP-HDP control scheme is implemented utilizing neural networks. Finally, we validate the effectiveness of the SUP-HDP control method through a benchmark simulation platform for the WWTP. Compared with other control methods, SUP-HDP has better control performance.

Research on Production Scheduling of Industrial Big Data for Internet of Things Based on Dynamic Planning Algorithm

Abstract In this paper, the optimized Dual Heuristic Dynamic Programming (DHP) algorithm is studied in depth, compared and analyzed with the other three algorithms through experimental simulation, and the DHP algorithm is applied to different industrial accurate scheduling tests to explore the loads of the industrial equipment during scheduling operation. The simulation results show that the dual-heuristic dynamic programming DHP algorithm has a significant decrease in both the mean waiting time (MWT) and the mean response time (MRT), where the highest peak value of the mean waiting time is only 0.06901 μs, and the highest peak value of the mean response time is 0.24493 μs. In industrial accurate production scheduling, the improved DHP algorithm performs the best for textile industry’’s task scheduling with the best performance. Specifically, the task scheduling completion time for the textile industry is 4522 seconds when the number of automated guided vehicles (AGVs) is the minimum value of 3. In comparison, the completion time is 8541 seconds when the number of AGVs is the maximum value of 15. The textile industry shows the fastest task completion time compared to the other industry types for different settings of the number of AGVs. In addition, the analysis of the operation of the equipment engines shows that although there are high and low torques during operation, the Torque of most of the engines stays within the range of 29 Nm to 30.7 Nm, which is generally stable and has little impact on the performance of the equipment. The improved dynamic programming algorithm proposed in this paper has effective application potential in industrial extensive data production scheduling, which can improve the efficiency and responsiveness of production scheduling. It has important practical significance for industrial production management.

Evolutionary Dynamics Optimal Research-Oriented Tumor Immunity Architecture.

The article is devoted to evolutionary dynamics optimal control-oriented tumor immune differential game system. First, the mathematical model covering immune cells and tumor cells considering the effects of chemotherapy drugs and immune agents. Second, the bounded optimal control problem covering is transformed into solving Hamilton-Jacobi-Bellman (HJB) equation considering the actual constraints and infinite-horizon performance index based on minimizing the amount of medication administered. Finally, approximate optimal control strategy is acquired through iterative-dual heuristic dynamic programming (I-DHP) algorithm avoiding dimensional disaster effectively and providing optimal treatment scheme for clinical applications.

A novel energy management method for multiple residential energy systems with energy exchange

Energy saving and emission reduction are the worldwide outstanding problems in the development of economy and society, which lead to wide attention of residential energy management. This paper presents a novel energy management method for multiple residential energy systems with energy exchange based on action-dependent heuristic dynamic programming (ADHDP). The major contributions are displayed as follows: (1) a novel energy management model is established for multiple residential energy systems with real-time electricity price and time-varying residential loads; (2) a novel ADHDP based energy management method is presented to obtain the energy scheduling scheme of multiple residential energy systems; (3) the continuous control action of the overall system is transformed into discrete control space which avoids training the action network. First, the structure and system models of multiple residential energy systems are introduced, and the energy management problem is formulated. Then, a brief introduction of ADHDP method is provided. Based on this, an energy management method is presented and an effective discretization method of control action is developed for solving the presented energy management problem with less computational burden. Simulation results show that the presented method can reduce the energy cost effectively. Finally, the comparison results verify the superiorities over the traditional energy management methods.

Vibration Suppression of a Flexible Beam Structure Coupled with Liquid Sloshing via ADP Control Based on FBG Strain Measurement

In this study, an adaptive dynamic programming (ADP) control strategy based on the strain measurement of a fiber Bragg grating (FGB) sensor array is proposed for the vibration suppression of a complicated flexible-sloshing coupled system, which usually exists in aerospace engineering, such as launch vehicles with a large amount of liquid propellant as well as a flexible beam structure. To simplify the flexible-sloshing coupled dynamics model, the equivalent spring-mass-damper (SMD) model of liquid sloshing is employed, and a finite-element method (FEM) dynamic model for the beam structure coupled with the liquid sloshing is mathematically established. Then, a strain-based vibration dynamic model is derived by employing a transformation matrix based on the relationship between displacement and strain of the beam structure. To facilitate the design of a strain-based control, a tracking differentiator is designed to provide the strains’ derivative signals as partial states’ estimations. Feeding the system with the strain measurements and their derivatives’ estimations, an ADP controller with an action-dependent heuristic dynamic programming structure is proposed to suppress the vibration of the flexible-sloshing coupled system, and the corresponding Lyapunov stability of the closed-loop system is theoretically guaranteed. Numerical results show the proposed method can effectively suppress coupled vibration depending on limited strain measurements irrespective of external disturbances.

Online self-learning attitude tracking control of morphing unmanned aerial vehicle based on dual heuristic dynamic programming

This article presents a partially model-free dual heuristic dynamic programming (DHP) control approach to address the online discrete-time attitude tracking problem. The DHP method learns the control policy by solving the translated optimal regulation problem of the augmented system, which is constructed with the tracking error dynamics and the desired trajectory. In the context of the augmented system, a corresponding recursive least squares (RLS) identification strategy has been devised, and the DHP update rules have been derived. The identified augmented system model not only provides dynamic information and predicted states for the actor and critic updates but also incorporates an uncertainty compensation part to reduce the learning complexity and enhance robustness of the controller. It is noteworthy that concurrent learning (CL) is employed to improve the critic convergence by utilizing an identified model-based data generation method. This approach eliminates the need for an offline learning stage and enables flexible controller design based on the available system knowledge. Based on the proposed approach, different attitude control frameworks have been designed for a morphing unmanned aerial vehicle (UAV). In addition to numerical simulations, flight tests have been conducted to further demonstrate the capabilities of the proposed approach, highlighting its effectiveness and advantages.

Adaptive fault-tolerant control for non-minimum phase hypersonic vehicles based on adaptive dynamic programming

In this paper, a novel adaptive Fault-Tolerant Control (FTC) strategy is proposed for non-minimum phase Hypersonic Vehicles (HSVs) that are affected by actuator faults and parameter uncertainties. The strategy is based on the output redefinition method and Adaptive Dynamic Programming (ADP). The intelligent FTC scheme consists of two main parts: a basic fault-tolerant and stable controller and an ADP-based supplementary controller. In the basic FTC part, an output redefinition approach is designed to make zero-dynamics stable with respect to the new output. Then, Ideal Internal Dynamic (IID) is obtained using an optimal bounded inversion approach, and a tracking controller is designed for the new output to realize output tracking of the non-minimum phase HSV system. For the ADP-based compensation control part, an Action-Dependent Heuristic Dynamic Programming (ADHDP) adopting an actor-critic learning structure is utilized to further optimize the tracking performance of the HSV control system. Finally, simulation results are provided to verify the effectiveness and efficiency of the proposed FTC algorithm.

Model-free adaptive dynamic event-triggered robust control for unknown nonlinear systems using iterative neural dynamic programming

This paper proposes an adaptive dynamic event-triggered (ADET) robust control method for unknown nonlinear systems using iterative neural dynamic programming (INDP). Firstly, the ADET robust control problem is transformed into an ADET optimal control problem by introducing an infinite domain integral function. Then, dynamic variables and adaptive thresholds are designed for discrete-time systems with auxiliary variables. The proposed ADET method reduces computation and transmission costs compared to existing static and dynamic event triggering methods. INDP is utilized to learn the optimal solution of the ADET Hamilton-Jacobi-Bellman equation within the heuristic dynamic programming (HDP) framework, providing system stability and algorithm convergence. The INDP algorithm employs model, action, and critic neural networks and includes a method to directly minimize the iterative cost function in the back-propagation process. The neural network implementation of the INDP algorithm is detailed. Simulation results demonstrate that the proposed ADET method based on INDP achieves faster convergence with fewer transmitted data and control updates.

A Nonlinear Critical Surface With Input Constraints on Costate-Adaptive Dynamic Programming

In this brief, a costate-adaptive dynamic programming algorithm is presented to solve a class of nonlinear critical surface problems with input constraints. Different from the traditional-dual heuristic dynamic programming, costate-ADP does not rely on the exact system model, and can incubate the model-neural network through data catalysis, avoiding the super-dimensional calculation in the iterative process. Furthermore, the critical nonlinear surface with input constraints is transformed into the optimal action sequence for solving the non-quadratic HJB equation with discount factor. The costate function replaces the integral term in the repeated iterations. The algorithm is proved to be effective in determining the security properties of a class of structures.

Adaptive Critic for Event-Triggered Unknown Nonlinear Optimal Tracking Design With Wastewater Treatment Applications.

In this article, an event-based near-optimal tracking control algorithm is developed for a class of nonaffine systems. First, in order to gain the tracking control strategy, the costate function is established through the iterative dual heuristic dynamic programming (DHP) algorithm. Then, the event-based control method is employed to improve the utilization efficiency of resources and ensure that the closed-loop system has an excellent control performance. Meanwhile, the input-to-state stability (ISS) is proven for the event-based tracking plant. In addition, three kinds of neural networks are used in the event-based DHP algorithm, which aims to identify the nonaffine nonlinear system, estimate the costate function, and approximate the tracking control law. Finally, a numerical experimental simulation is conducted to verify the effectiveness of the proposed scheme. Moreover, in order to further validate the feasibility, the algorithm is applied to the wastewater treatment plant to effectively control the concentrations of dissolved oxygen and nitrate nitrogen.