Action-dependent Heuristic Dynamic Programming Research Articles

Action-dependent heuristic dynamic programming for hybrid-order systems based on dynamic event-triggered mechanism

We investigate the distributed optimal consensus control problem based on action-dependent heuristic dynamic programming (ADHDP). In order to strike a stable balance between the learning cost of reinforcement learning and the resource utilization efficiency of the hybrid-order multi-agent systems (MASs), we propose an improved dynamic event-triggered ADHDP (dET-ADHDP) method. This approach can non-periodically explore the control policy distribution using the online action-dependent actor–critic (ADAC) learning framework. Meanwhile, it can dynamically adjust the trigger lower bound by exploiting the designed trigger threshold function, and adaptively decide the signal trigger moment during the ADAC learning process. In addition, we demonstrate the boundedness of the ADAC network weights and show that under the designed dynamic event-triggering rules, the MASs can asymptotically achieve optimal tracking control without Zeno phenomenon. Finally, compared with the traditional static counterparts, simulation experiments demonstrate that the proposed dynamic event-triggered ADAC (dET-ADAC) algorithm has more efficient resource utilization while maintaining satisfactory learning performance.

Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes

Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes

A novel energy management method for multiple residential energy systems with energy exchange

Energy saving and emission reduction are the worldwide outstanding problems in the development of economy and society, which lead to wide attention of residential energy management. This paper presents a novel energy management method for multiple residential energy systems with energy exchange based on action-dependent heuristic dynamic programming (ADHDP). The major contributions are displayed as follows: (1) a novel energy management model is established for multiple residential energy systems with real-time electricity price and time-varying residential loads; (2) a novel ADHDP based energy management method is presented to obtain the energy scheduling scheme of multiple residential energy systems; (3) the continuous control action of the overall system is transformed into discrete control space which avoids training the action network. First, the structure and system models of multiple residential energy systems are introduced, and the energy management problem is formulated. Then, a brief introduction of ADHDP method is provided. Based on this, an energy management method is presented and an effective discretization method of control action is developed for solving the presented energy management problem with less computational burden. Simulation results show that the presented method can reduce the energy cost effectively. Finally, the comparison results verify the superiorities over the traditional energy management methods.

Vibration Suppression of a Flexible Beam Structure Coupled with Liquid Sloshing via ADP Control Based on FBG Strain Measurement

In this study, an adaptive dynamic programming (ADP) control strategy based on the strain measurement of a fiber Bragg grating (FGB) sensor array is proposed for the vibration suppression of a complicated flexible-sloshing coupled system, which usually exists in aerospace engineering, such as launch vehicles with a large amount of liquid propellant as well as a flexible beam structure. To simplify the flexible-sloshing coupled dynamics model, the equivalent spring-mass-damper (SMD) model of liquid sloshing is employed, and a finite-element method (FEM) dynamic model for the beam structure coupled with the liquid sloshing is mathematically established. Then, a strain-based vibration dynamic model is derived by employing a transformation matrix based on the relationship between displacement and strain of the beam structure. To facilitate the design of a strain-based control, a tracking differentiator is designed to provide the strains’ derivative signals as partial states’ estimations. Feeding the system with the strain measurements and their derivatives’ estimations, an ADP controller with an action-dependent heuristic dynamic programming structure is proposed to suppress the vibration of the flexible-sloshing coupled system, and the corresponding Lyapunov stability of the closed-loop system is theoretically guaranteed. Numerical results show the proposed method can effectively suppress coupled vibration depending on limited strain measurements irrespective of external disturbances.

Adaptive fault-tolerant control for non-minimum phase hypersonic vehicles based on adaptive dynamic programming

In this paper, a novel adaptive Fault-Tolerant Control (FTC) strategy is proposed for non-minimum phase Hypersonic Vehicles (HSVs) that are affected by actuator faults and parameter uncertainties. The strategy is based on the output redefinition method and Adaptive Dynamic Programming (ADP). The intelligent FTC scheme consists of two main parts: a basic fault-tolerant and stable controller and an ADP-based supplementary controller. In the basic FTC part, an output redefinition approach is designed to make zero-dynamics stable with respect to the new output. Then, Ideal Internal Dynamic (IID) is obtained using an optimal bounded inversion approach, and a tracking controller is designed for the new output to realize output tracking of the non-minimum phase HSV system. For the ADP-based compensation control part, an Action-Dependent Heuristic Dynamic Programming (ADHDP) adopting an actor-critic learning structure is utilized to further optimize the tracking performance of the HSV control system. Finally, simulation results are provided to verify the effectiveness and efficiency of the proposed FTC algorithm.

Robust Self-Learning Fault-Tolerant Control for Hypersonic Flight Vehicle Based on ADHDP

In this article, a robust self-learning fault-tolerant control (FTC) strategy is proposed to deal with the tracking control problem of the hypersonic flight vehicle (HFV) with uncertainties, actuator faults, and external disturbances. First, an adaptive baseline controller is constructed to achieve stable tracking, in which neural networks are introduced to approximate the unknown dynamics, adaptive laws are formulated to compensate the unknown lumped disturbances, and the Nussbaum technique is applied to address the time-varying actuator faults. Then, to improve the command tracking performance of the baseline controller, a data-driven auxiliary controller which can adaptively adjust the action–critic network weights over time along with the tracking deviation to obtain the optimal control signals in the sense of performance index is developed based on action-dependent heuristic dynamic programming technology. Finally, a comprehensive robust self-learning FTC law is constructed by synthesizing the baseline controller and the auxiliary controller, which leads to good robustness and tracking performance of the closed-loop HFV system. The stability and the superiority of the proposed control algorithm are verified by the Lyapunov theory and comparative numerical simulations, respectively.

Dual Heuristic Dynamic Programming Enables Trajectory Tracking Control

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 200271, “Dual Heuristic Dynamic Programming in the Oil and Gas Industry for Trajectory Tracking Control,” by Seaar Al-Dabooni, Basra Oil Company; Alaa Azeez Tawiq, Technical Institute of Basra; and Hussen Alshehab, Basra Oil Company. The paper has not been peer reviewed. _ The complete paper presents an artificial intelligence (AI) algorithm the authors call dual heuristic dynamic programming (DHDP) that is used to solve optimization-control problems. Fast, self-learning control based on DHDP is illustrated for trajectory-tracking levels on a quadruple tank system (QTS) consisting of four tanks and two electrical pumps with two pressure-control valves. Two artificial neural networks are constructed for the DHDP approach: the critic network (the provider of a critique or evaluated signals) and the actor network or controller (the provider of control signals). The DHDP controller is learned without human intervention. Approximate Dynamic Programming (ADP) Recently, many different types of artificial algorithms have been applied in petroleum fields to solve optimization problems. This complete paper introduces a new field of AI applicable to oil and gas, ADP. ADP is a useful tool to overcome the behavior of nonlinear systems and is a special algorithm of reinforcement learning (RL). The authors write that ADP can be viewed as consisting of three categories: heuristic dynamic programming (HDP), DHDP, and globalized HDP. ADP features two neural networks—an actor and a critic—to provide an optimal control signal and long-cost value, respectively. ADP has numerous applications. The complete paper references work that discusses control on turbo-generator and swarm-robot problems by use of DHDP and that illustrates that action-dependent HDP can obtain an optimal path by multirobot navigation. QTS is frequently used in the oil and gas industry. DHDP is used to control the voltage of the two pumps to follow the desired level (set-point-level value) of the tanks, an approach that can learn by itself (a self-learning controller). The complete paper devotes several pages to equations and parameters that describe HDP. In ADP, optimal control problems are solved, thereby allowing agents to select an optimal action to minimize a long-term cost value through solution of Bellman’s equation. RL and ADP are used to train the actor neural network to provide optimal actions based on minimizing the cost-to-go value produced from the critic network. The actor function approximator is denoted for the actor neural network. After full training of these networks, the optimal action values are obtained from the actor network. System Functionality The equipment receives the system states of the process through sensors, and the algorithm maximizes the reward by selecting the correct optimal action (control signal) to feed the equipment. The simulation results for applying DHDP with QTS as a benchmark test problem were obtained using MATLAB. QTS is illustrated as an example in the paper because QTS is widely used in most petroleum exploration or production fields as an entire system or in parts. Another reason for the authors’ choice of QTS as a test problem is that QTS features a difficult model to control that has a limited zone of operating parameters to be stable. The multi-input/multioutput (MIMO) model of QTS was a similar model as most MIMO devices in the oil and gas field. The overall learning-control-system performance was tested and compared with HDP and a well-known industrial controller, a proportional integral derivative (PID) using MATLAB programming. The simulation results of DHDP provide enhanced performance compared with the PID approach, with a 98.9002% improvement.

Economic Dispatch for Smart Buildings with Load Demand of High Volatility Based on Quasi-Quadratic Online Adaptive Dynamic Programming

In this paper, a quasi-quadratic online adaptive dynamic programming (QOADP) algorithm is proposed to realize optimal economic dispatch for smart buildings. Load demand of high volatility is considered, which is modeled by an uncontrollable state. To reduce residual errors of the approximation structure, a quasi-quadratic-form parametric structure was designed elaborately with a bias term to counteract effects of uncertainties. Based on action-dependent heuristic dynamic programming (ADHDP), an implementation of the QOADP algorithm is presented that involved obtaining optimal economic dispatch for smart buildings. Finally, hardware-in-loop (HIL) experiments were conducted, and the performance of the proposed QOADP algorithm is superior to that of two other typical algorithms.

A data-driven energy management method for parallel PHEVs based on action dependent heuristic dynamic programming (ADHDP) model

Due to the strong self-learning ability and adaptability of neural network, adaptive dynamic programming (ADP) method is regarded as an effective method to improve the vehicle economy in the case of uncertain driving conditions. With the look-ahead information, a data-driven energy management method is proposed based on action dependent heuristic dynamic programming (ADHDP) algorithm. Firstly, based on statistical analysis of DP behavior, a reference SOC trajectory is generated to limit the electricity consumption, which is adjusted with the dynamically updated driving information. Then, a data-driven energy management control method is developed for HEV architectures, including information acquisition module, shift scheduling module, energy distribution module. The gear shift command is designed by enumeration and the power distribution is performed by multiple ADHDP models, the core of which is to determine the utility function, action network (ANN), critic network (CNN) and training process. To ensure practical driveability, the restrictions on gear shifting and engine starting-stopping are taken as additional conditions in ADHDP optimizing process. Finally, two case studies under different driving scenarios are given. Simulation results demonstrate that the proposed method gains a good performance in both optimality approximation and adaptivity to uncertain driving conditions.

Small Leak Location for Intelligent Pipeline System via Action-Dependent Heuristic Dynamic Programming

In the construction process of intelligent pipeline system, pipeline monitoring is an important content to improve the safety of pipeline operation. Small leak location, in particular, is the primary focus of pipeline monitoring due to the unclear pressure drop point. To solve this, in this article, a data driven-based method of small leak location is proposed. First, through the collected pipeline parameters, empirical flow variables, and historical pressure values in pipeline system, a pipeline model based on pressure along pipeline is presented by a three-layer neural network to close to the industrial scenarios. Then, on the basis of the analyzed propagation process of negative pressure wave, an action-dependent heuristic dynamic programming with pressure–distance physical constraints is proposed to obtain the small leak location result. The proposed method is suitable for the collection of only pressure data scenario, which expands the application range. Finally, different cases of small leak location results indicate that the proposed method can locate the leak point, and the field tests further show that the proposed method has satisfactory performances in pipeline leak analysis.

Dual ML-ADHDP method for heterogeneous discrete-time nonlinear multi-agent systems with unknown dynamics and time delay

This paper develops a new dual ML-ADHDP method to solve the optimal consensus problem (OCP) of a class of heterogeneous discrete-time nonlinear multi-agent systems (MASs) with unknown dynamics and time delay. A hierarchical and distributed control strategy is used to transform the original problem into nonlinear model reference adaptive control (MRAC) problems and an OCP of virtual linear MASs. For the nonlinear MRAC problems, a new multi-layer action-dependent heuristic dynamic programming (ML-ADHDP) method is developed to overcome the unknown dynamics and neural network estimation errors, which has higher control accuracy. In order to solve the OCP of virtual linear MASs and improve the convergence speed, a new multi-layer performance index is proposed. Then the ML-ADHDP method is used to solve the coupled Hamiltonian–Jacobi–Bellman equation and obtain the optimal virtual control. Theoretical analysis proves that the original MASs can achieve Nash equilibrium, and simulation results show that the developed dual ML-ADHDP method ensures better convergence speed and higher control accuracy of original MASs.

Global optimization energy management for multi-energy source vehicles based on “Information layer - Physical layer - Energy layer - Dynamic programming” (IPE-DP)

To reveal the energy-saving mechanisms of global energy management, we propose a global optimization framework of “information layer-physical layer-energy layer-dynamic programming” (IPE-DP), which can realize the unity of different information scenarios, different vehicle configurations and energy conversions. The deterministic dynamic programing (DP) and adaptive dynamic programming (ADP) are taken as the core algorithms. As a benchmark for assessing the optimality, DP strategy has four main challenges: standardization, real-time application, accuracy, and satisfactory drivability. To solve the above problems, the IPE-DP optimization framework is established, which consists of three main layers, two interface layers and an application layer. To be specific, the full-factor trip information is acquired from three scenarios in the information layer, and then the feasible work modes of the vehicle are determined in the physical layer based on the proposed conservation framework of “kinetic/potential energy & onboard energy“. The above lays a foundation for the optimal energy distribution in the energy layer. Then, a global domain-searching algorithm and action-dependent heuristic dynamic programming (ADHDP) model are developed for different information acquisition scenarios to obtain the optimal solution. To improve the computational efficiency under the deterministic information, a fast DP is developed based on the statistical rules of DP behavior, the core of which is to restrict the exploring region based on a reference SOC trajectory. Regarding the stochastic trip information, the ADHDP model is established, including determining the utility function, network design and training process. Finally, two case studies are given to compare the economic performance of the vehicle under different information acquisition scenarios, which lays a foundation for analyzing the relationship between the amount of information input and energy-saving potential of the vehicle. Simulation results demonstrate that the proposed method gains a better performance in both real-time performance and global optimality.

Torsional oscillation-considered mode transition coordinated control for a power-split PHEV based on action dependent heuristic dynamic programming

The mode transition process (MTP) from electric mode to hybrid electric mode (EM-to-HM) will cause the deterioration in occupant comfort of PHEV, to tickle this issue, a torsional oscillation-considered mode transition coordinated control strategy and a novel general evaluation index for MTP are developed in this research, the quality of mode transition and transient torsional oscillation of gears (TTOGs) during MTP are taken into consideration comprehensively. An action dependent heuristic dynamic programming algorithm which takes the vehicle jerk, friction loss and TTOGs as evaluation index is used to optimize the pressure curve of clutch oil and the compensation torque of motor in the entire EM-to-HM process. Finally, the simulation results and hardware-in-the-loop tests show that vehicle jerk and TTOGs are suppressed, and the driving comfort can be improved accordingly.

Research on ADHDP energy management strategy for fuel cell hybrid power system

Fuel cell hybrid power system is a prospective power source for electrical vehicles. To reduce hydrogen consumption and enhance dynamic performance of the system, Action Dependent Heuristic Dynamic Programming (ADHDP) energy management strategy for the fuel cell hybrid power system was proposed. Firstly, topology of the system was analyzed and mathematical model was established through mechanism analysis. Secondly, framework of the ADHDP algorithm was presented, and it was followed by training algorithm for evaluating network and executing network of ADHDP based on Back Propagation (BP) algorithm. Finally, hardware-in-the-loop (HIL) simulation of the fuel cell hybrid power system was carried out to demonstrate the proposed ADHDP algorithm under real operating conditions. The results show that evaluating network and executing network of ADHDP have good convergence performance under different operating conditions. Compared with the other algorithms, the proposed ADHDP energy management strategy has better fuel economy and dynamic performance.

Artificial-intelligence-based algorithms in multi-access edge computing for the performance optimization control of a benchmark microgrid

In practical engineering, the communication networks of industrial systems are complex, and system models are generally unavailable. To overcome the requirement of mathematical models, several artificial-intelligence-based algorithms in multi-access edge computing are introduced for the performance optimization control of a benchmark microgrid in this paper. First, a neural-network-based identification scheme is proposed to combine with the online adaptive dynamic programming learning method, which avoids the requirement of system models. However, the identification errors are not taken into consideration. Next, to realize the model-free purpose without using the identification schemes, an online dual-network-based action-dependent heuristic dynamic programming method and a critic-only Q-learning approach are presented. Finally, the optimal control strategy is applied to a benchmark microgrid system to demonstrate the effectiveness of performance optimization.

Integrated adaptive dynamic programming for data-driven optimal controller design

In this paper a novel integrated adaptive dynamic programming method with an advantage function is developed to solve model-free optimal control problems and improve the control performance. The advantage function is utilized to evaluate the cost resulting from the action (control variables) which does not follow the optimal control policy. The Q function in Q-learning can thus be built from a value function and the advantage function. The control policy is then improved through minimizing the Q function. To employ the proposed algorithm, an integrated multi-layer neural network (INN) is designed for the value function and the control variables. Only one single neural network requires adaption. This avoids the iterative learning of two separate networks in the heuristic dynamic programming-based methods. Simulation for linear and non-linear optimal control problems is studied. Comparing to the optimal solutions resulting from the linear quadratic regulator and dynamic programming (DP), the proposed INN design can lead to closer control performance than the ones with action dependent heuristic dynamic programming (ADHDP). Furthermore INN is applied to optimize the energy management strategy of hybrid electric vehicles for fuel economy. The fuel consumption based on INN is lower than the one from ADHDP and much closer to the optimal results by DP. The result indicates the near fuel-optimality and an effective practical application.

An Action Dependent Heuristic Dynamic Programming Approach for Algal Bloom Prediction With Time-Varying Parameters

Algal bloom is a nonlinear and time-varying process, which brings challenges for the accurate prediction. For the existing mechanism model of algae ignores the external key factors, we propose an algae growth model (AGM) optimized by action dependent heuristic dynamic programming (ADHDP). This model has the structure of information interaction with the outside, which can predict algal bloom with well adaptive ability. In this paper, chlorophyll-a concentration is used as the representative factor of algal bloom. We use ADHDP approach to map the external key factors to the time-varying parameters, so the AGM can be adjusted to realize the self-adaptive prediction with the changes in external environments. Compared with different prediction methods, the simulation result shows that the ADHDP-AGM prediction model can accurately predict the chlorophyll-a concentration under different data distributions. Moreover, the prediction process shows that the time-varying parameters in AGM conform to the evolution trend of chlorophyll-a concentration in fact, which further improves the interpretability of prediction model. It provides a new perspective for building a data-driven prediction model with clear physical significance, and makes the mechanism research and data science further fusion.

Fuel-Efficient Gear Shift and Power Split Strategy for Parallel HEVs Based on Heuristic Dynamic Programming and Neural Networks

In this paper, an online gear shift and power split strategy for parallel hybrid electric vehicles (HEVs) is proposed to optimize the fuel consumption. The gear shift control is implemented with a neural network (NN). The training of the NN is performed with data obtained from dynamic programming (DP) for different driving cycles. Simulation results present that the gear shift sequences and fuel consumption obtained from the NN and DP are very similar, underlining the near-optimal gear shift control for HEVs based on the NN. Furthermore, the power split control is realized with action-dependent heuristic dynamic programming (ADHDP). ADHDP does not need a system model and allows learning continuously. The power split control is therefore robust with respect to uncertainties and disturbances, as well as adaptive with respect to various driving situations and driver behaviors. The gear shift control based on the NN and the power split control based on ADHDP are combined to an adaptive energy management strategy with possible real-time application. It enables online learning of the controller and prevents the curse of dimensionality from DP calculation. Simulations for different driving cycles and comparisons with optimal solutions by DP indicate that the proposed energy management strategy is robust, adaptive, and near optimal.

Ecological Adaptive Cruise Control and Energy Management Strategy for Hybrid Electric Vehicles Based on Heuristic Dynamic Programming

In this paper, an ecological adaptive cruise controller (ECO-ACC) for parallel hybrid electric vehicles (HEVs) in a car-following scenario is presented to improve the fuel economy and to maintain a desired inter-vehicle distance from the preceding vehicle. An ACC based on action dependent heuristic dynamic programming (ADHDP) is proposed to obtain an ecological velocity profile and realize an active distance control in normal driving situations. ADHDP is able to adapt internal parameters online and can thus deal with systems with disturbances. Furthermore, an adaptive energy management strategy for HEVs is introduced to control the gear shift and power split for fuel consumption optimization. The gear shift command is designed by enumeration, and the power distribution between the engine and the electric motor is performed by ADHDP. The developed ACC and energy management strategy are finally combined to an ECO-ACC to achieve a multi-objective optimization. Only the current velocity and acceleration of the preceding vehicle are used while knowledge about the future velocity is not needed. The simulations of different driving cycles indicate that the ECO-ACC can lead to near-optimal fuel economy and comfortable driving.

Morphing aircraft control based on switched nonlinear systems and adaptive dynamic programming

This paper investigates the control problem of a morphing aircraft with variable sweep wings based on switched nonlinear systems and adaptive dynamic programming (ADP). The longitudinal altitude motion of the morphing aircraft is first modeled as switched nonlinear systems in lower triangular form. Then, the designed controller is comprised of the basic part and supplementary part. For the basic part, the backstepping technique is applied and a modified dynamic surface is introduced to overcome the ‘explosion of complexity’ problem. Disturbance observers inspired from the idea of extended state observer are designed to obtain the estimations of the internal uncertainties and external disturbances. The common virtual control laws of the backstepping method are developed by the disturbance observers and radial basis function neural networks. On the other hand, for the supplementary part, an ADP approach with the name of action-dependent heuristic dynamic programming is used to further decrease the altitude tracking error, which generates an additional control input by observing the differences between the actual and desired values in the backstepping design. Finally, comparative simulations are conducted to demonstrate the improved control performance of the proposed approach.