Heuristic Dynamic Programming Research Articles

Virtual inertia control parameter regulator of doubly fed induction generator based on direct heuristic dynamic programming

The most widely applied doubly-fed induction generator (DFIG) leads to insufficient frequency regulation ability of power system. The virtual inertia control can adjust the active power of DFIG according to frequency fluctuation. However, the frequent changes in power flow and operating point bring great challenges to traditional virtual inertia controller with fixed parameters. Direct Heuristic Dynamic Programming (Direct HDP) based control parameter regulator is proposed in this paper to obtain the optimal control parameters when the operation condition changes. Compared with the initial parameters, the response has been significantly improved after adaptive optimization by the proposed method when the system operating point changes.

Dual ML-ADHDP method for heterogeneous discrete-time nonlinear multi-agent systems with unknown dynamics and time delay

This paper develops a new dual ML-ADHDP method to solve the optimal consensus problem (OCP) of a class of heterogeneous discrete-time nonlinear multi-agent systems (MASs) with unknown dynamics and time delay. A hierarchical and distributed control strategy is used to transform the original problem into nonlinear model reference adaptive control (MRAC) problems and an OCP of virtual linear MASs. For the nonlinear MRAC problems, a new multi-layer action-dependent heuristic dynamic programming (ML-ADHDP) method is developed to overcome the unknown dynamics and neural network estimation errors, which has higher control accuracy. In order to solve the OCP of virtual linear MASs and improve the convergence speed, a new multi-layer performance index is proposed. Then the ML-ADHDP method is used to solve the coupled Hamiltonian–Jacobi–Bellman equation and obtain the optimal virtual control. Theoretical analysis proves that the original MASs can achieve Nash equilibrium, and simulation results show that the developed dual ML-ADHDP method ensures better convergence speed and higher control accuracy of original MASs.

Dual Heuristic Dynamic Programming Based Energy Management Control for Hybrid Electric Vehicles

This paper investigates an adaptive dynamic programming (ADP)-based energy management control strategy for a series-parallel hybrid electric vehicle (HEV). This strategy can further minimize the equivalent fuel consumption while satisfying the battery level constraints and vehicle power demand. Dual heuristic dynamic programming (DHP) is one of the basic structures of ADP, combining reinforcement learning, dynamic programming (DP) optimization principle, and neural network approximation function, which has higher accuracy with a slightly more complex structure. In this regard, the DHP energy management strategy (EMS) is designed by the backpropagation neural network (BPNN) as an Action network and two Critic networks approximating the control policy and the gradient of value function concerning the state variable. By comparing with the existing results such as HDP-based and rule-based control strategies, the equivalent consumption minimum strategy (ECMS), and reinforcement learning (RL)-based strategy, simulation results verify the robustness of fuel economy and the adaptability of the power-split optimization of the proposed EMS to different driving conditions.

Model-free event-triggered optimal control with performance guarantees via goal representation heuristic dynamic programming

Model-free event-triggered optimal control with performance guarantees via goal representation heuristic dynamic programming

Global optimization energy management for multi-energy source vehicles based on “Information layer - Physical layer - Energy layer - Dynamic programming” (IPE-DP)

To reveal the energy-saving mechanisms of global energy management, we propose a global optimization framework of “information layer-physical layer-energy layer-dynamic programming” (IPE-DP), which can realize the unity of different information scenarios, different vehicle configurations and energy conversions. The deterministic dynamic programing (DP) and adaptive dynamic programming (ADP) are taken as the core algorithms. As a benchmark for assessing the optimality, DP strategy has four main challenges: standardization, real-time application, accuracy, and satisfactory drivability. To solve the above problems, the IPE-DP optimization framework is established, which consists of three main layers, two interface layers and an application layer. To be specific, the full-factor trip information is acquired from three scenarios in the information layer, and then the feasible work modes of the vehicle are determined in the physical layer based on the proposed conservation framework of “kinetic/potential energy & onboard energy“. The above lays a foundation for the optimal energy distribution in the energy layer. Then, a global domain-searching algorithm and action-dependent heuristic dynamic programming (ADHDP) model are developed for different information acquisition scenarios to obtain the optimal solution. To improve the computational efficiency under the deterministic information, a fast DP is developed based on the statistical rules of DP behavior, the core of which is to restrict the exploring region based on a reference SOC trajectory. Regarding the stochastic trip information, the ADHDP model is established, including determining the utility function, network design and training process. Finally, two case studies are given to compare the economic performance of the vehicle under different information acquisition scenarios, which lays a foundation for analyzing the relationship between the amount of information input and energy-saving potential of the vehicle. Simulation results demonstrate that the proposed method gains a better performance in both real-time performance and global optimality.

Learning‐based robust control methodologies under information constraints

Learning‐based robust control methodologies under information constraints

Incremental Dual Heuristic Dynamic Programming Based Hybrid Approach for Multi-Channel Control of Unstable Tailless Aircraft

Actor-critic based online reinforcement learning control has been proved to be promising method for control of aerial vehicles. However, it is difficult to guarantee high-level success rate of initial training and to tune the large amount of parameters for actors and critics considering unstable multi-input and multi-output (MIMO) aircraft. In order to facilitate and simplify the training of the actor and critic for unstable aircraft, classic stability augmentation system (SAS) is designed for the open-loop aircraft first. Then the online incremental model based dual heuristic dynamic programming (IDHP) method, which has been proposed recently, is extended in application to design a multi-channel robust adaptive controller, and MIMO form network structures are designed and determined for the actors and critics considering the three-channel coupling issues. Consequently, the classic SAS and the IDHP controller make up a novel hybrid control framework. In this control framework, the SAS takes charge of counteracting the unstable eigenvalues of the open-loop aircraft system, and the IDHP takes charge on guaranteeing robust and adaptive performance for high-performance tailless aircraft equipped with the SAS. Specifically, the introduction of the classic control method decreases the difficulty of the initial training for multi-channel IDHP controller. The tuning process for initial parameters of actor and critic neural networks in multiple channels is greatly facilitated. Without the help of SAS, the initial training for multi-channel IDHP controllers of unstable plants is almost impossible to succeed. Finally, the novel hybrid control architecture and method are validated using the Innovative Control Effectors (ICE) model, which has unstable modes in the longitudinal dynamics. Typical aerodynamic model uncertainties are numerically simulated to demonstrate the effectiveness of the proposed control method.

Robotic Knee Tracking Control to Mimic the Intact Human Knee Profile Based on Actor-Critic Reinforcement Learning

We address a state-of-the-art reinforcement learning (RL) control approach to automatically configure robotic prosthesis impedance parameters to enable end-to-end, continuous locomotion intended for transfemoral amputee subjects. Specifically, our actor-critic based RL provides tracking control of a robotic knee prosthesis to mimic the intact knee profile, This is a significant advance from our previous RL based automatic tuning of prosthesis control parameters which have centered on regulation control with a designer prescribed robotic knee profile as the target. In addition to presenting the tracking control algorithm based on direct heuristic dynamic programming (dHDP), we provide a control performance guarantee including the case of constrained inputs. We show that our proposed tracking control possesses several important properties, such as weight convergence of the learning networks, Bellman (sub) optimality of the cost-to-go value function and control input, and practical stability of the human-robot system. We further provide a systematic simulation of the proposed tracking control using a realistic human-robot system simulator, the OpenSim, to emulate how the dHDP enables level ground walking, walking on different terrains and at different paces. These results show that our proposed dHDP based tracking control is not only theoretically suitable, but also practically useful.

New solution procedures for the order picker routing problem in U-shaped pick areas with a movable depot

This paper develops new solution procedures for the order picker routing problem in U-shaped order picking zones with a movable depot, which has so far only been solved using simple heuristics. The paper presents the first exact solution approach, based on combinatorial Benders decomposition, as well as a heuristic approach based on dynamic programming that extends the idea of the venerable sweep algorithm. In a computational study, we demonstrate that the exact approach can solve small instances well, while the heuristic dynamic programming approach is fast and exhibits an average optimality gap close to zero in all test instances. Moreover, we investigate the influence of various storage assignment policies from the literature and compare them to a newly derived policy that is shown to be advantageous under certain circumstances. Secondly, we investigate the effects of having a movable depot compared to a fixed one and the influence of the effort to move the depot.

Resilient Adaptive Wide-Area Damping Control to Mitigate False Data Injection Attacks

The wide-area damping controller (WADC) using the remote signals obtained from wide-area measurement system (WAMS) as input is an effective measure to suppress interarea oscillations. Once the input signal of WADC is threatened by false data injection attacks (FDIAs), its performance will be decreased dramatically. To address this problem, this article proposes a resilient adaptive wide-area damping control (RAWADC) framework combined with linear state estimator (LSE) and goal representation heuristic dynamic programming (GrHDP). The LSE is proposed and adopted as a preprocessor to detect and correct the attacked data. By detecting and correcting FDIA, the integrity of the WAMS measurement signal is guaranteed. The GrHDP algorithm is employed to design the WADC, which not only does not require a detailed mathematical model of the system but can also adapt to variations of the operating conditions. Simulation results of the 16-machine 68-bus system with voltage source converter based HVdc indicate that the RAWADC can suppress interarea oscillation effectively under a wide range of operating conditions, different FDIA types and measurement noise.

Learning‐basedT‐sHDP() for optimal control of a class of nonlinear discrete‐time systems

AbstractThis article investigates the optimal control problem via reinforcement learning for a class of nonlinear discrete‐time systems. The nonlinear system under consideration is assumed to be partially unknown. A new learning‐based algorithm,T‐step heuristic dynamic programming with eligibility traces (T‐sHDP()), is proposed to tackle the optimal control problem for such partially unknown system. First, the concerned optimal control problem is turned into its equivalence problem, that is, solving a Bellman equation. Then, theT‐sHDP() is utilized to get an approximate solution of Bellman equation, and a rigorous convergence analysis is also conducted as well. Instead of the commonly used single step update approach, theT‐sHDP() stores finite step past returns by introducing a parameter, and then utilizes these knowledge to update the value function (VF) of multiple moments synchronously, so as to achieve higher convergence speed. For implementation ofT‐sHDP(), a neural network‐based actor‐critic architecture is applied to approximate VF and optimal control scheme. Finally, the feasibility of the algorithm is demonstrated by two illustrative simulation examples.

Deep Stackelberg heuristic dynamic programming for frequency regulation of interconnected power systems considering flexible energy sources

The flexible energy sources (FESs) could refuse to execute the generation commands (GCs) if the benefits fell short of their expectations when applying traditional automatic generation control methods, this problem leads to frequency instability and the waste of scheduling resources. Accordingly, a deep Stackelberg heuristic dynamic programming (DSHDP) is proposed for the frequency regulation and power dispatch of smart generation control (SGC). Firstly, the heuristic deep dynamic programming, whose networks are replaced by three deep neural networks, can be trained offline by historical data to fit the optimal performance index. Then, the GCs can be optimized through an online updating strategy for stabilizing the frequency of interconnected power systems. To maximize the benefits of FESs further, an improved SGC framework based on the Stackelberg game is built to let FESs play games with conventional power plants in a fair environment. Finally, the optimal GCs can be distributed by the Nash equilibrium to obtain higher profits than traditional methods, and the control accuracy is increased indirectly. Two cases, i.e., IEEE two-area power system and six-area power system based on China Southern Power Grid (CSPG), are provided to verify the superiority of the DSHDP. The results indicate that the excellent flexibility and stability of the proposed algorithm in complex power systems containing renewable energy and FESs. Totally, the frequency deviation obtained by DSHDP is reduced by up to 59.38%, while the total profit of FESs rises by up to 67.47% when compared with the other seven representative algorithms.

A novel adaptive heuristic dynamic programming-based algorithm for aircraft confrontation games

Intelligent confrontation has become a vital technology for future air combats. Confrontation games between a penetrating aircraft and an intercepting aircraft are essential for modern air combats. In addition, the performance indexes of both the interceptor and penetrator must be considered. Traditional methods only solve one side's guidance problem without considering the intelligence of the opponent. In this paper, an adaptive heuristic dynamic programming-based algorithm is proposed for aircraft confrontation games. This algorithm constructs a heuristic dynamic programming model for both confrontation aircraft and then updates the critical and action network parameters using the dynamic confrontation state information. Numerical simulations indicate that the proposed algorithm can optimize the guidance law for both the interceptor and penetrator and is therefore superior to traditional proportional navigation methods.

Bio-inspired heuristic dynamic programming for high-precision real-time flow control in a multi-tributary river system

Rivers are a type of time-varying, highly nonlinear systems, especially with multiple tributaries. To address low precision and poor real-time flow control of multi-tributary river systems, a heuristic dynamic programming algorithm inspired by regulation mechanisms in the human body (Bio-HDP) is proposed. Bio-HDP consists of several control units including the model network (MNCU), the action network (ANCU), the critic network (CNCU), and the central coordination (CCCU), all of which are developed based on the interactions and mechanisms in human’s NEI system (nervous, endocrine, and immune systems). The MNCU approximates the actual system through learning actual data on the control object. The ANCU calculates the control variable according to the error between the current state of the system acquired from the MNCU and the set value. The CNCU is used to evaluate whether the current control variable obtained by the ANCU can meet the control requirements. Finally, the CCCU conducts coordinated regulation of the above three control units in real-time according to the current state of the system, and makes the control mechanism faster and more accurate. By incorporating a biological feedback mechanism into the MNCU, the ANCU, and the CNCU, we adopt a neural network with multi-level feedback for the control mechanism. We select the Chongyang River system in China as the control object to verify the effects of the Bio-HDP control algorithm. The results show that the Bio-HDP algorithm improves speed, adaptability, and accuracy of the algorithm in comparison to other algorithms, and exhibits timely and accurate control effects.

Torsional oscillation-considered mode transition coordinated control for a power-split PHEV based on action dependent heuristic dynamic programming

The mode transition process (MTP) from electric mode to hybrid electric mode (EM-to-HM) will cause the deterioration in occupant comfort of PHEV, to tickle this issue, a torsional oscillation-considered mode transition coordinated control strategy and a novel general evaluation index for MTP are developed in this research, the quality of mode transition and transient torsional oscillation of gears (TTOGs) during MTP are taken into consideration comprehensively. An action dependent heuristic dynamic programming algorithm which takes the vehicle jerk, friction loss and TTOGs as evaluation index is used to optimize the pressure curve of clutch oil and the compensation torque of motor in the entire EM-to-HM process. Finally, the simulation results and hardware-in-the-loop tests show that vehicle jerk and TTOGs are suppressed, and the driving comfort can be improved accordingly.

Intelligent optimal tracking with asymmetric constraints of a nonlinear wastewater treatment system

AbstractThe wastewater treatment is an effective method for alleviating the shortage of water resources. In this article, a data‐driven iterative adaptive tracking control approach is developed to improve the control performance of the dissolved oxygen concentration and the nitrate nitrogen concentration in the nonlinear wastewater treatment plant. First, the model network is established to obtain the steady control and evaluate the new system state. Then, a nonquadratic performance functional is provided to handle asymmetric control constraints. Moreover, the new costate function and the tracking control policy are derived by using the dual heuristic dynamic programming algorithm. In the present control scheme, two neural networks are constructed to approximate the costate function and the tracking control law. Finally, the feasibility of the proposed algorithm is confirmed by applying the designed strategy to the wastewater treatment plant.

Neural optimal tracking control of constrained nonaffine systems with a wastewater treatment application

In this paper, we aim to solve the optimal tracking control problem for a class of nonaffine discrete-time systems with actuator saturation. First, a data-based neural identifier is constructed to learn the unknown system dynamics. Then, according to the expression of the trained neural identifier, we can obtain the steady control corresponding to the reference trajectory. Next, by involving the iterative dual heuristic dynamic programming algorithm, the new costate function and the tracking control law are developed. Two other neural networks are used to estimate the costate function and approximate the tracking control law. Considering approximation errors of neural networks, the stability analysis of the proposed algorithm for the specific systems is provided by introducing the Lyapunov approach. Finally, via conducting simulation and comparison, the superiority of the developed optimal tracking method is confirmed. Moreover, the trajectory tracking performance of the wastewater treatment application is also involved for further verifying the proposed approach.

Self-Learning Controllers in the Oil and Gas Industry

Recently, solving the optimization-control problems by using artificial intelligence has widelyappeared in the petroleum fields in exploration and production. This paper presents the stateof-the-art reinforcement-learning algorithm applying in the petroleum optimization-controlproblems, which is called a direct heuristic dynamic programming (DHDP). DHDP has twointeractive artificial neural networks, which are the critic network (provider acritique/evaluated signal) and the actor network (provider a control signal). This paper focuseson a generic on-line learning control system in Markov decision process principles.Furthermore, DHDP is a model-free learning design that does not require prior knowledgeabout a dynamic model; therefore, DHDP can be appllied with any petroleum equipment ordevise directly without needed to drive a mathematical model. Moreover, DHDP learns byitself (self-learning) without human intervention via repeating the interaction between anequipment and environment/process. The equipment receives the states of theenvironment/process via sensors, and the algorithm maximizes the reward by selecting thecorrect optimal action (control signal). A quadruple tank system (QTS) is taken as a benchmarktest problem, that the nonlinear model responses close to the real model, for three reasons:First, QTS is widely used in the most petroleum exploration/production fields (entire system orparts), which consists of four tanks and two electrical-pumps with two pressure control valves.Second, QTS is a difficult model to control, which has a limited zone of operating parametersto be stable; therefore, if DHDP controls on QTS by itself, DHDP can control on otherequipment in a fast and optimal manner. Third, QTS is designed with a multi-input-multioutput (MIMO) model for analysis in the real-time nonlinear dynamic system; therefore, theQTS model has a similar model with most MIMO devises in oil and gas field. The overalllearning control system performance is tested and compared with a proportional integralderivative (PID) via MATLAB programming. DHDP provides enhanced performancecomparing with the PID approach with 99.2466% improvement.

Neural Network-Based Intelligent Computing Algorithms for Discrete-Time Optimal Control with the Application to a Cyberphysical Power System

Adaptive dynamic programming (ADP), which belongs to the field of computational intelligence, is a powerful tool to address optimal control problems. To overcome the bottleneck of solving Hamilton–Jacobi–Bellman equations, several state-of-the-art ADP approaches are reviewed in this paper. First, two model-based offline iterative ADP methods including policy iteration (PI) and value iteration (VI) are given, and their respective advantages and shortcomings are discussed in detail. Second, the multistep heuristic dynamic programming (HDP) method is introduced, which avoids the requirement of initial admissible control and achieves fast convergence. This method successfully utilizes the advantages of PI and VI and overcomes their drawbacks at the same time. Finally, the discrete-time optimal control strategy is tested on a power system.

Research on ADHDP energy management strategy for fuel cell hybrid power system

Fuel cell hybrid power system is a prospective power source for electrical vehicles. To reduce hydrogen consumption and enhance dynamic performance of the system, Action Dependent Heuristic Dynamic Programming (ADHDP) energy management strategy for the fuel cell hybrid power system was proposed. Firstly, topology of the system was analyzed and mathematical model was established through mechanism analysis. Secondly, framework of the ADHDP algorithm was presented, and it was followed by training algorithm for evaluating network and executing network of ADHDP based on Back Propagation (BP) algorithm. Finally, hardware-in-the-loop (HIL) simulation of the fuel cell hybrid power system was carried out to demonstrate the proposed ADHDP algorithm under real operating conditions. The results show that evaluating network and executing network of ADHDP have good convergence performance under different operating conditions. Compared with the other algorithms, the proposed ADHDP energy management strategy has better fuel economy and dynamic performance.