Dual Heuristic Dynamic Programming Research Articles

Multi‐event‐triggered adaptive dynamic programming for non‐zero‐sum game of unknown nonlinear system

AbstractIn this article, a multi‐event‐triggered (MET) near‐optimal tracking control algorithm is developed for a discrete‐time (DT) nonlinear non‐zero‐sum game (NZSG). Firstly, the iterative dual heuristic dynamic programming (DHP) approach is utilized to obtain the optimal tracking control set. Subsequently, distinct independent triggering conditions are formulated for control inputs to improve the utilization efficiency of resources and ensure that the independence between them and the closed‐loop system has an excellent control performance. Meanwhile, input‐to‐state stability (ISS) is proven for the MET tracking plant. Additionally, neural networks (NNs) are designed in the iterative DHP algorithm to construct the model module, the critic module and the action modules, with aims to identify the unknown system, approximate costate function and estimate optimal tracking control strategy set, respectively. Finally, two numerical experimental simulations are conducted to verify the effectiveness of the proposed scheme.

Research on Production Scheduling of Industrial Big Data for Internet of Things Based on Dynamic Planning Algorithm

Abstract In this paper, the optimized Dual Heuristic Dynamic Programming (DHP) algorithm is studied in depth, compared and analyzed with the other three algorithms through experimental simulation, and the DHP algorithm is applied to different industrial accurate scheduling tests to explore the loads of the industrial equipment during scheduling operation. The simulation results show that the dual-heuristic dynamic programming DHP algorithm has a significant decrease in both the mean waiting time (MWT) and the mean response time (MRT), where the highest peak value of the mean waiting time is only 0.06901 μs, and the highest peak value of the mean response time is 0.24493 μs. In industrial accurate production scheduling, the improved DHP algorithm performs the best for textile industry’’s task scheduling with the best performance. Specifically, the task scheduling completion time for the textile industry is 4522 seconds when the number of automated guided vehicles (AGVs) is the minimum value of 3. In comparison, the completion time is 8541 seconds when the number of AGVs is the maximum value of 15. The textile industry shows the fastest task completion time compared to the other industry types for different settings of the number of AGVs. In addition, the analysis of the operation of the equipment engines shows that although there are high and low torques during operation, the Torque of most of the engines stays within the range of 29 Nm to 30.7 Nm, which is generally stable and has little impact on the performance of the equipment. The improved dynamic programming algorithm proposed in this paper has effective application potential in industrial extensive data production scheduling, which can improve the efficiency and responsiveness of production scheduling. It has important practical significance for industrial production management.

Online self-learning attitude tracking control of morphing unmanned aerial vehicle based on dual heuristic dynamic programming

This article presents a partially model-free dual heuristic dynamic programming (DHP) control approach to address the online discrete-time attitude tracking problem. The DHP method learns the control policy by solving the translated optimal regulation problem of the augmented system, which is constructed with the tracking error dynamics and the desired trajectory. In the context of the augmented system, a corresponding recursive least squares (RLS) identification strategy has been devised, and the DHP update rules have been derived. The identified augmented system model not only provides dynamic information and predicted states for the actor and critic updates but also incorporates an uncertainty compensation part to reduce the learning complexity and enhance robustness of the controller. It is noteworthy that concurrent learning (CL) is employed to improve the critic convergence by utilizing an identified model-based data generation method. This approach eliminates the need for an offline learning stage and enables flexible controller design based on the available system knowledge. Based on the proposed approach, different attitude control frameworks have been designed for a morphing unmanned aerial vehicle (UAV). In addition to numerical simulations, flight tests have been conducted to further demonstrate the capabilities of the proposed approach, highlighting its effectiveness and advantages.

Adaptive Critic for Event-Triggered Unknown Nonlinear Optimal Tracking Design With Wastewater Treatment Applications.

In this article, an event-based near-optimal tracking control algorithm is developed for a class of nonaffine systems. First, in order to gain the tracking control strategy, the costate function is established through the iterative dual heuristic dynamic programming (DHP) algorithm. Then, the event-based control method is employed to improve the utilization efficiency of resources and ensure that the closed-loop system has an excellent control performance. Meanwhile, the input-to-state stability (ISS) is proven for the event-based tracking plant. In addition, three kinds of neural networks are used in the event-based DHP algorithm, which aims to identify the nonaffine nonlinear system, estimate the costate function, and approximate the tracking control law. Finally, a numerical experimental simulation is conducted to verify the effectiveness of the proposed scheme. Moreover, in order to further validate the feasibility, the algorithm is applied to the wastewater treatment plant to effectively control the concentrations of dissolved oxygen and nitrate nitrogen.

Dual Heuristic Dynamic Programming Enables Trajectory Tracking Control

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 200271, “Dual Heuristic Dynamic Programming in the Oil and Gas Industry for Trajectory Tracking Control,” by Seaar Al-Dabooni, Basra Oil Company; Alaa Azeez Tawiq, Technical Institute of Basra; and Hussen Alshehab, Basra Oil Company. The paper has not been peer reviewed. _ The complete paper presents an artificial intelligence (AI) algorithm the authors call dual heuristic dynamic programming (DHDP) that is used to solve optimization-control problems. Fast, self-learning control based on DHDP is illustrated for trajectory-tracking levels on a quadruple tank system (QTS) consisting of four tanks and two electrical pumps with two pressure-control valves. Two artificial neural networks are constructed for the DHDP approach: the critic network (the provider of a critique or evaluated signals) and the actor network or controller (the provider of control signals). The DHDP controller is learned without human intervention. Approximate Dynamic Programming (ADP) Recently, many different types of artificial algorithms have been applied in petroleum fields to solve optimization problems. This complete paper introduces a new field of AI applicable to oil and gas, ADP. ADP is a useful tool to overcome the behavior of nonlinear systems and is a special algorithm of reinforcement learning (RL). The authors write that ADP can be viewed as consisting of three categories: heuristic dynamic programming (HDP), DHDP, and globalized HDP. ADP features two neural networks—an actor and a critic—to provide an optimal control signal and long-cost value, respectively. ADP has numerous applications. The complete paper references work that discusses control on turbo-generator and swarm-robot problems by use of DHDP and that illustrates that action-dependent HDP can obtain an optimal path by multirobot navigation. QTS is frequently used in the oil and gas industry. DHDP is used to control the voltage of the two pumps to follow the desired level (set-point-level value) of the tanks, an approach that can learn by itself (a self-learning controller). The complete paper devotes several pages to equations and parameters that describe HDP. In ADP, optimal control problems are solved, thereby allowing agents to select an optimal action to minimize a long-term cost value through solution of Bellman’s equation. RL and ADP are used to train the actor neural network to provide optimal actions based on minimizing the cost-to-go value produced from the critic network. The actor function approximator is denoted for the actor neural network. After full training of these networks, the optimal action values are obtained from the actor network. System Functionality The equipment receives the system states of the process through sensors, and the algorithm maximizes the reward by selecting the correct optimal action (control signal) to feed the equipment. The simulation results for applying DHDP with QTS as a benchmark test problem were obtained using MATLAB. QTS is illustrated as an example in the paper because QTS is widely used in most petroleum exploration or production fields as an entire system or in parts. Another reason for the authors’ choice of QTS as a test problem is that QTS features a difficult model to control that has a limited zone of operating parameters to be stable. The multi-input/multioutput (MIMO) model of QTS was a similar model as most MIMO devices in the oil and gas field. The overall learning-control-system performance was tested and compared with HDP and a well-known industrial controller, a proportional integral derivative (PID) using MATLAB programming. The simulation results of DHDP provide enhanced performance compared with the PID approach, with a 98.9002% improvement.

Technology Focus: Intelligent Operations (May 2023)

The first paper selected for this Intelligent Fields Technical Focus presents the use of a unique artificial-intelligence (AI) technique from the fourth industrial revolution—dual heuristic dynamic programming—to address business opportunities associated with optimization. The background, methodology, and novelty presented by the authors are clear; however, they need to benchmark their efforts and claims of improvement with peers. The second paper demonstrates well-structured content, though the topic is detailed, with the introduction of a digital laboratory called Ibtikar toward the enablement of connecting to open-source tools with the use of AI. The collaboration in research and development, as well as addressing challenges of upstream oil and gas requirements, is commendable. The authors, however, do not address associated security concerns concerning the open-source aspect of the technology. The third paper presents a combination of technologies used to automate daily data acquisition and model updates from production databases and run a supervised machine-learning model to continuously calibrate the network model. Recommended additional reading at OnePetro: www.onepetro.org. SPE 207188 Design and Development of a Solar-Powered Pump System With Liquid-Level Sensor and Controller Using Internet of Things Technology by Chinonyelum Ejimuda, Afe Babalola University, et al. SPE 211314 Improved Field Production Planning and Cost Optimization Using Predictive Analytical Choke Performance Model and Intelligent Alarms in an Integrated Platform by Ayesha Alsaeedi, Adnoc, et al. SPE 211367 Multidisciplinary Data Integration for Artificial-Intelligence-Assisted Well Placement and Trajectory-Design Optimization Under Uncertainty by Shi Su, SLB, et al.

Dual Event-Triggered Constrained Control Through Adaptive Critic for Discrete-Time Zero-Sum Games

In this article, through adaptive critic, a dual event-triggered (DET) constrained control scheme is established for discrete-time nonlinear zero-sum games. The neural networks are trained from the dual heuristic dynamic programming technique to obtain the approximate optimal policy pair. Two corresponding independent triggering conditions are constructed for the control input and the disturbance to improve the utilization efficiency and ensure the independence between them. In addition, in order to overcome the challenge caused by the actuator saturation, we constrain the control input to a bounded range. Meanwhile, the asymptotically stability is proved for the DET control system. Finally, experimental simulations are conducted to verify the effectiveness of the proposed algorithm.

Error Dynamics Based Dual Heuristic Dynamic Programming for Self-Learning Flight Control

A data-driven nonlinear control approach, called error dynamics-based dual heuristic dynamic programming (ED-DHP), is proposed for air vehicle attitude control. To solve the optimal tracking control problem, the augmented system is defined by the derived error dynamics and reference trajectory so that the actor neural network can learn the feedforward and feedback control terms at the same time. During the online self-learning process, the actor neural network learns the control policy by minimizing the augmented system’s value function. The input dynamics identified by the recursive least square (RLS) and output of the critic neural network are used to update the actor neural network. In addition, the total uncertainty term of the error dynamics is also identified by RLS, which can compensate for the uncertainty caused by inaccurate modeling, parameter perturbation, and so on. The outputs of ED-DHP include the rough trim surface, feedforward and feedback terms from the actor neural network, and the compensation. Based on this control scheme, the complete knowledge of system dynamics and the reference trajectory dynamics are not needed, and offline learning is unnecessary. To verify the self-learning ability of ED-DHP, two numerical experiments are carried out based on the established morphing air vehicle model. One is sinusoidal signal tracking at a fixed operating point, and the other is guidance command tracking with a morphing process at variable operating points. The simulation results demonstrate the good performance of ED-DHP for online self-learning attitude control and validate the robustness of the proposed scheme

Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios

In order to improve the driving safety and fuel economy of hybrid electric vehicles (HEVs) in car-following scenarios, this paper proposes a hierarchical energy management strategy (EMS) based on adaptive dynamic programming (ADP). In the upper-layer speed planning, the speed prediction model of the preceding vehicle is established utilizing a back-propagation neural network (BPNN), and the host vehicle speed is planned based on heuristic dynamic programming (HDP) real-time through the predicted preceding vehicle speed and the current state of the host vehicle. In the lower-layer energy management control, the EMS based on dual heuristic dynamic programming (DHP) is used to realize the power allocation of each power source according to the planned speed to reduce fuel consumption. In both HDP and DHP algorithms, BPNN is used to implement the critic network (CN) and action network (AN) to obtain a more accurate control variable and faster convergence speed. By comparing the results with various existing EMSs in the different driving cycles, the simulation results show that the proposed EMS ensures real-time planning speed and good following performance while reducing energy consumption.

Novel optimal trajectory tracking for nonlinear affine systems with an advanced critic learning structure

In this paper, a critic learning structure based on the novel utility function is developed to solve the optimal tracking control problem with the discount factor of affine nonlinear systems. The utility function is defined as the quadratic form of the error at the next moment, which can not only avoid solving the stable control input, but also effectively eliminate the tracking error. Next, the theoretical derivation of the method under value iteration is given in detail with convergence and stability analysis. Then, the dual heuristic dynamic programming (DHP) algorithm via a single neural network is introduced to reduce the amount of computation. The polynomial is used to approximate the costate function during the DHP implementation. The weighted residual method is used to update the weight matrix. During simulation, the convergence speed of the given strategy is compared with the heuristic dynamic programming (HDP) algorithm. The experiment results display that the convergence speed of the proposed method is faster than the HDP algorithm. Besides, the proposed method is compared with the traditional tracking control approach to verify its tracking performance. The experiment results show that the proposed method can avoid solving the stable control input, and the tracking error is closer to zero than the traditional strategy.

Dual Heuristic Dynamic Programming Based Energy Management Control for Hybrid Electric Vehicles

This paper investigates an adaptive dynamic programming (ADP)-based energy management control strategy for a series-parallel hybrid electric vehicle (HEV). This strategy can further minimize the equivalent fuel consumption while satisfying the battery level constraints and vehicle power demand. Dual heuristic dynamic programming (DHP) is one of the basic structures of ADP, combining reinforcement learning, dynamic programming (DP) optimization principle, and neural network approximation function, which has higher accuracy with a slightly more complex structure. In this regard, the DHP energy management strategy (EMS) is designed by the backpropagation neural network (BPNN) as an Action network and two Critic networks approximating the control policy and the gradient of value function concerning the state variable. By comparing with the existing results such as HDP-based and rule-based control strategies, the equivalent consumption minimum strategy (ECMS), and reinforcement learning (RL)-based strategy, simulation results verify the robustness of fuel economy and the adaptability of the power-split optimization of the proposed EMS to different driving conditions.

Incremental Dual Heuristic Dynamic Programming Based Hybrid Approach for Multi-Channel Control of Unstable Tailless Aircraft

Actor-critic based online reinforcement learning control has been proved to be promising method for control of aerial vehicles. However, it is difficult to guarantee high-level success rate of initial training and to tune the large amount of parameters for actors and critics considering unstable multi-input and multi-output (MIMO) aircraft. In order to facilitate and simplify the training of the actor and critic for unstable aircraft, classic stability augmentation system (SAS) is designed for the open-loop aircraft first. Then the online incremental model based dual heuristic dynamic programming (IDHP) method, which has been proposed recently, is extended in application to design a multi-channel robust adaptive controller, and MIMO form network structures are designed and determined for the actors and critics considering the three-channel coupling issues. Consequently, the classic SAS and the IDHP controller make up a novel hybrid control framework. In this control framework, the SAS takes charge of counteracting the unstable eigenvalues of the open-loop aircraft system, and the IDHP takes charge on guaranteeing robust and adaptive performance for high-performance tailless aircraft equipped with the SAS. Specifically, the introduction of the classic control method decreases the difficulty of the initial training for multi-channel IDHP controller. The tuning process for initial parameters of actor and critic neural networks in multiple channels is greatly facilitated. Without the help of SAS, the initial training for multi-channel IDHP controllers of unstable plants is almost impossible to succeed. Finally, the novel hybrid control architecture and method are validated using the Innovative Control Effectors (ICE) model, which has unstable modes in the longitudinal dynamics. Typical aerodynamic model uncertainties are numerically simulated to demonstrate the effectiveness of the proposed control method.

Intelligent optimal tracking with asymmetric constraints of a nonlinear wastewater treatment system

AbstractThe wastewater treatment is an effective method for alleviating the shortage of water resources. In this article, a data‐driven iterative adaptive tracking control approach is developed to improve the control performance of the dissolved oxygen concentration and the nitrate nitrogen concentration in the nonlinear wastewater treatment plant. First, the model network is established to obtain the steady control and evaluate the new system state. Then, a nonquadratic performance functional is provided to handle asymmetric control constraints. Moreover, the new costate function and the tracking control policy are derived by using the dual heuristic dynamic programming algorithm. In the present control scheme, two neural networks are constructed to approximate the costate function and the tracking control law. Finally, the feasibility of the proposed algorithm is confirmed by applying the designed strategy to the wastewater treatment plant.

Neural optimal tracking control of constrained nonaffine systems with a wastewater treatment application

In this paper, we aim to solve the optimal tracking control problem for a class of nonaffine discrete-time systems with actuator saturation. First, a data-based neural identifier is constructed to learn the unknown system dynamics. Then, according to the expression of the trained neural identifier, we can obtain the steady control corresponding to the reference trajectory. Next, by involving the iterative dual heuristic dynamic programming algorithm, the new costate function and the tracking control law are developed. Two other neural networks are used to estimate the costate function and approximate the tracking control law. Considering approximation errors of neural networks, the stability analysis of the proposed algorithm for the specific systems is provided by introducing the Lyapunov approach. Finally, via conducting simulation and comparison, the superiority of the developed optimal tracking method is confirmed. Moreover, the trajectory tracking performance of the wastewater treatment application is also involved for further verifying the proposed approach.

Event-triggered design for discrete-time nonlinear systems with control constraints

In order to solve the constrained-input problem and reduce the computing resources, a novel event-triggered optimal control method is proposed for a class of discrete-time nonlinear systems. In the proposed method, the event-triggered control policy is applied to the globalized dual heuristic dynamic programming (GDHP) algorithm. Compared with the traditional adaptive dynamic programming (ADP) control, the event-triggered GDHP control can reduce the computation while ensuring the system performance. In this paper, a non-quadratic function is given to code the control constraints and the trigger condition with the stability analysis is provided. Neural networks (NN) are constructed in the GDHP structure, where the model network is designed to identify the unknown nonlinear system, the critic network is used to learn the cost function and its partial derivative, and the action network is designed to obtain the approximate optimal control law. Three simulation examples are presented to demonstrate the performance of the proposed event-triggered design for constrained discrete-time nonlinear systems.

Model-Free Adaptive Optimal Control for Unknown Nonlinear Multiplayer Nonzero-Sum Game.

In this article, an online adaptive optimal control algorithm based on adaptive dynamic programming is developed to solve the multiplayer nonzero-sum game (MP-NZSG) for discrete-time unknown nonlinear systems. First, a model-free coupled globalized dual-heuristic dynamic programming (GDHP) structure is designed to solve the MP-NZSG problem, in which there is no model network or identifier. Second, in order to relax the requirement of systems dynamics, an online adaptive learning algorithm is developed to solve the Hamilton-Jacobi equation using the system states of two adjacent time steps. Third, a series of critic networks and action networks are used to approximate value functions and optimal policies for all players. All the neural network (NN) weights are updated online based on real-time system states. Fourth, the uniformly ultimate boundedness analysis of the NN approximation errors is proved based on the Lyapunov approach. Finally, simulation results are given to demonstrate the effectiveness of the developed scheme.

Modeling and Trajectory Tracking Control for Magnetic Wheeled Mobile Robots Based on Improved Dual-Heuristic Dynamic Programming

In this article, a mathematical model of a magnetic-wheeled mobile robot (MWMR) used for wall climbing in some special industrial sites is established, and to realize the precise motion control of the robot, an intelligent discrete algorithm for trajectory tracking control of the MWMR is presented. The robot is subjected to nonholonomic constraints when moving on the wall. The discrete mathematical model of the MWMR is established with the description of the kinematics and dynamics, where the dynamics are described by the second-order Lagrange's equations. Improved dual-heuristic dynamic programming (DHP) where the actor-critic structure adopts random vector functional link neural networks (RVFL NNs) is the main configuration of the trajectory tracking control algorithm. Moreover, the tracking control algorithm is supplied by a PD controller and a supervisory element to generate overall control signals. The improved strategy is to optimize the input layer weights of RVFL NNs by a genetic algorithm to improve the approximation performance of DHP. Simulations are performed to test the trajectory tracking control algorithm for this wall-climbing robot, and the results are compared with those of neural network tracking control algorithm. Comparative analysis verifies the effectiveness and advancement of the proposed method.

Data-based composite control design with critic intelligence for a wastewater treatment platform

In this paper, by integrating neural network approximators, a data-based composite control technique is developed with critic learning implementation and wastewater treatment verification. The iterative adaptive critic framework is established involving dual heuristic dynamic programming (DHP), so as to obtain an intelligent optimal controller. Besides, a steady control input is computed with the help of the neural identifier. Then, by combining the DHP controller and the steady control input, an effective composite control strategy is derived and applied to the proposed wastewater treatment platform. Through conducting experiments, it is observed that the dissolved oxygen concentration and the nitrate level can be maintained at setting points successfully, which results in an intelligent wastewater treatment system.

Data-driven based optimal distributed frequency control for islanded AC microgrids

With the great stride of clean energy technology, an increasing number of distributed generations are connected to microgrids. However, the heterogeneity and uncertainty of distributed generation may cause frequency instability in microgrids. This paper proposes a data-driven based distributed secondary frequency controller for islanded microgrids without requiring the accurate model parameters. Motivated by iterative learning algorithm, a distributed dual heuristic dynamic programming control framework is established to derive an optimal control action. The frequency controller quickly restores frequency while guaranteeing accurate active power sharing. Furthermore, convergence of the proposed algorithm and stability of the data-based error system are analyzed. Simulation examples are carried out to validate the effectiveness of our designs. Compared with conventional distributed frequency control, simulation result shows the proposed controller has a superior dynamic performance.

Event-triggered constrained control with DHP implementation for nonaffine discrete-time systems

This paper proposes an event-based near-optimal control algorithm for nonaffine discrete-time systems with constrained inputs. The method is derived from the dual heuristic dynamic programming (DHP) technique. The challenge caused by saturating actuators is overcome by using a nonquadratic performance index. Then, the event-based control technique is used to decrease the amount of computation. Meanwhile, the stability analysis is provided. It illustrates that the proposed event-based method can asymptotically stabilize the nonaffine systems by using the Lyapunov method. Furthermore, the stability conditions and the design process of the event-based controller are established. The event-based DHP algorithm is implemented by constructing three neural networks, namely, the model network, the critic network, and the action network. Finally, simulation studies are conducted to demonstrate the applicability and the performance of the proposed method.