Iterative Adaptive Dynamic Programming Algorithm Research Articles

Hybrid Iteration ADP Algorithm to Solve Cooperative, Optimal Output Regulation Problem for Continuous-Time, Linear, Multiagent Systems: Theory and Application in Islanded Modern Microgrids With IBRs

This paper proposes a novel adaptive dynamic programming (ADP) algorithm, named hybrid iteration (HI), to solve the cooperative, optimal output regulation problem (CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP) for continuous-time, linear, multi-agent systems. Unlike traditional ADP algorithms, i.e., policy iteration (PI) and value iteration (VI), HI does not need an initial stabilizing control policy required by PI. At the same time, it maintains a faster convergence rate compared with VI. First, a model-based HI algorithm is proposed to solve the CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP. Based on the proposed HI algorithm, a data-driven, adaptive, optimal controller is developed to solve the cooperative, adaptive, optimal output regulation problem without using any information about the physics of the system. Instead, the states/input information collected along the trajectories of the dynamic system is employed. The proposed data-driven HI is applied to the adaptive, optimal secondary voltage control (also known as voltage restoration control) of an islanded modern microgrid based on inverter-based resources. Compared with the VI and PI algorithms, comparative simulation results demonstrate that the proposed HI approach is significantly able to save the convergence time of the central processing unit (also known as CPU) deployed, reduce the number of learning iterations, and remove the requirement of the initial stabilizing control policy. Comparative experiments reveal the practicality and superiority of the proposed methodology.

An integrated structure of operational performance-oriented monitoring and degradation recovery based on iterative adaptive dynamic programming

Driven by the increasing industrial demands for process safety and product quality, this paper is devoted to operational performance monitoring and performance degradation recovery for closed-loop control systems under abnormal working conditions. To this end, a new operational performance indicator is designed and its benchmark with an explicit sensitivity criteria for selecting historical data, to detect the system degradation timely and accurately. Then, by embedding the operational performance indicator as a trigger condition, a data-driven performance degradation recovery method with the aid of iterative adaptive dynamic programming (ADP) algorithm is investigated. This implementation shows that iterative ADP can be applied efficiently by utilizing both value iteration and measurement data to recover the anomaly induced system performance degradation. Finally, a case study on the continuous stirred tank reactor system is provided to validate the effectiveness of these proposed methods.

Optimal Regulation Strategy for Nonzero-Sum Games of the Immune System Using Adaptive Dynamic Programming.

This article investigates the optimal control strategy problem for nonzero-sum games of the immune system based on adaptive dynamic programming (ADP). First, the main objective is approximating a Nash equilibrium between the tumor cells and the immune cell population, which is governed through chemotherapy drugs and immunoagents guided by the mathematical growth model of the tumor cells. Second, a novel intelligent nonzero-sum games-based ADP is put forward to solve the optimization control problem by reducing the growth rate of tumor cells and minimizing chemotherapy drugs and immunotherapy drugs. Meanwhile, the convergence analysis and iterative ADP algorithm are specified to prove feasibility. Finally, simulation examples are listed to account for availability and effectiveness of the research methodology.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems, in this paper, a new iterative adaptive dynamic programming algorithm, which is the discrete-time time-varying policy iteration (DTTV) algorithm, is developed. The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance. The admissibility of the iterative control law is analyzed. The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution. To implement the algorithm, neural networks are employed and a new implementation structure is established, which avoids solving the generalized Bellman equation in each iteration. Finally, the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm, where the mass and pendulum bar length are permitted to be time-varying parameters. The effectiveness of the developed method is illustrated by numerical results and comparisons.

Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors

In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, "min-Hamiltonian," is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.

Robust oxygen excess ratio control of PEMFC systems using adaptive dynamic programming

To improve the power generation efficiency and cells lifetime of proton exchange membrane fuel cells, the oxygen excess ratio (OER) control problem is investigated in this paper. With the use of the double-loop cascade control structure, our work reinforces the iteration adaptive dynamic programming (ADP) algorithm and develops a robust OER control strategy drawing upon rich tools and techniques in sliding mode control theory. Our contribution is threefold. First, the proposed OER control policy is insensitive to various dynamics and disturbances by using refined super-twisting sliding mode controller in the external control loop. Second, the ADP-based controller is employed in the internal control loop to approximate the real optimal tracking control policy. The algorithm is computationally efficient and easy to implement in hardware platforms. Third, simulation results show that the proposed OER control strategy has a quick, smooth response, which can be further refined by means of numerous advanced neural network algorithms.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

Adaptive dynamic programming (ADP) technique is adopted in this work to investigate the optimal control problem of Markovian jump systems. By utilizing Bellman’s optimality principle, a discrete Hamilton Jacobi Bellman (HJB) equation is established to design the optimal controller for the system under consideration. Then, based on value iteration, a new ADP algorithm is proposed for finding the solution of the established HJB equation. It is proven that the iterative solution sequence generated by the developed ADP iterative approach under zero initial values is monotonically convergent. Neural networks are constructed to accomplish the presented value iteration ADP algorithm. At last, simulation researches for two Markovian jump systems demonstrate the effectiveness of the proposed optimal control method.

Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

In this paper, an adaptive dynamic programming (ADP) strategy is investigated for discrete-time nonlinear systems with unknown nonlinear dynamics subject to input saturation. To save the communication resources between the controller and the actuators, stochastic communication protocols (SCPs) are adopted to schedule the control signal, and therefore the closed-loop system is essentially a protocol-induced switching system. A neural network (NN)-based identifier with a robust term is exploited for approximating the unknown nonlinear system, and a set of switch-based updating rules with an additional tunable parameter of NN weights are developed with the help of the gradient descent. By virtue of a novel Lyapunov function, a sufficient condition is proposed to achieve the stability of both system identification errors and the update dynamics of NN weights. Then, a value iterative ADP algorithm in an offline way is proposed to solve the optimal control of protocol-induced switching systems with saturation constraints, and the convergence is profoundly discussed in light of mathematical induction. Furthermore, an actor-critic NN scheme is developed to approximate the control law and the proposed performance index function in the framework of ADP, and the stability of the closed-loop system is analyzed in view of the Lyapunov theory. Finally, the numerical simulation results are presented to demonstrate the effectiveness of the proposed control scheme.

A partial policy iteration ADP algorithm for nonlinear neuro-optimal control with discounted total reward

This paper constructs a partial policy iteration adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of nonlinear systems with discounted total reward. Compared with traditional policy iteration ADP algorithm, the approach updates the iterative control law only in a local region of the global system state space. With the benefit of this feature, the overall computational burden at each iteration for processing units can be significantly reduced. Hence, this feature enables our algorithm to be successfully executed on low-performance devices such as smartphones, smartwatches and the Internet of Things (IoT) objects. We provide the convergency analysis to show that the generated sequence of value functions is monotonically nonincreasing and can finally reach a local optimum. In addition, the corresponding local policy space is developed theoretically for the first time. Besides, when the sequence of the local system state spaces is chosen properly, we prove that the developed algorithm is capable of finding the global optimal performance index function for the nonlinear systems. Finally, we present a numerical simulation to demonstrate the effectiveness of the proposed algorithm.

Approximately Optimal Control of Discrete-Time Nonlinear Switched Systems Using Globalized Dual Heuristic Programming

Based on the idea of data-driven control, a novel iterative adaptive dynamic programming (ADP) algorithm based on the globalized dual heuristic programming (GDHP) technique is used to solve the optimal control problem of discrete-time nonlinear switched systems. In order to solve the Hamilton–Jacobi–Bellman (HJB) equation of switched systems, the iterative ADP method is proposed and the strict convergence analysis is also provided. Three neural networks are constructed to implement the iterative ADP algorithm, where a novel model network is designed to identify the system dynamics, a critic network is used to approximate the cost function and its partial derivatives, and an action network is provided to obtain the approximate optimal control law. Two simulation examples are described to illustrate the effectiveness of the proposed method by comparing with the heuristic dynamic programming (HDP) and dual heuristic programming (DHP) methods.

Adaptive Dynamic Programming for Model-Free Global Stabilization of Control Constrained Continuous-Time Systems.

This article addresses the problem of global stabilization of continuous-time linear systems subject to control constraints using a model-free approach. We propose a gain-scheduled low-gain feedback scheme that prevents saturation from occurring and achieves global stabilization. The framework of parameterized algebraic Riccati equations (AREs) is employed to design the low-gain feedback control laws. An adaptive dynamic programming (ADP) method is presented to find the solution of the parameterized ARE without requiring the knowledge of the system dynamics. In particular, we present an iterative ADP algorithm that searches for an appropriate value of the low-gain parameter and iteratively solves the parameterized ADP Bellman equation. We present both state feedback and output feedback algorithms. The closed-loop stability and the convergence of the algorithm to the nominal solution of the parameterized ARE are shown. The simulation results validate the effectiveness of the proposed scheme.

Improved value iteration for neural-network-based stochastic optimal control design

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm is presented, which is called an improved value iteration ADP algorithm, to obtain the optimal policy for discrete stochastic processes. In the improved value iteration ADP algorithm, for the first time we propose a new criteria to verify whether the obtained policy is stable or not for stochastic processes. By analyzing the convergence properties of the proposed algorithm, it is shown that the iterative value functions can converge to the optimum. In addition, our algorithm allows the initial value function to be an arbitrary positive semi-definite function. Finally, two simulation examples are presented to validate the effectiveness of the developed method.

Data-Driven Nonlinear Near-Optimal Regulation Based on Multi-Dimensional Taylor Network Dynamic Programming

Using the data-driven control formulation, an iterative dynamic programming approach which is based on a multi-dimensional Taylor network is established to design the near optimal regulation of discrete-time nonlinear systems. For discrete-time general nonlinear systems, the iterative adaptive dynamic programming algorithm is developed and proved to guarantee the property of convergence and optimality. Three networks are constructed, namely, the identification network, critic network and action network. Moreover, a globalized dual heuristic programming technique with detailed implementation is developed. The cost function and its derivative can be approximated by this novel architecture. Besides, without the consideration of the system dynamics, this technique can learn the near-optimal control law simultaneously and adaptively. In addition, this technique greatly improves the existing results of the iterative adaptive dynamic programming algorithm in terms of reducing the requirement of the control matrix. Furthermore, because of the approach that is based on the multi-dimensional Taylor network, the amount of calculation needed is also greatly reduced. The simulation experiment is described to illustrate the effectiveness of the data-driven optimal regulation method proposed in this paper.

Data-Based Nonaffine Optimal Tracking Control Using Iterative DHP Approach

Abstract In this paper, a data-based optimal tracking control approach is developed by involving the iterative dual heuristic dynamic programming algorithm for nonaffine systems. In order to gain the steady control corresponding to the desired trajectory, a novel strategy is established with regard to the unknown system function. Then, according to the iterative adaptive dynamic programming algorithm, the updating formula of the costate function and the new optimal control policy for unknown nonaffine systems are provided to solve the optimal tracking control problem. Moreover, three neural networks are used to facilitate the implementation of the proposed algorithm. In order to improve the accuracy of the steady control corresponding to the desired trajectory, we employ a model network to directly approximate the unknown system function instead of the error dynamics. Finally, the effectiveness of the proposed method is demonstrated through a simulation example.

Neuro-Optimal Tracking Control for Continuous Stirred Tank Reactor With Input Constraints

This paper proposes a novel data-based optimal control algorithm for continuous stirred tank reactor (CSTR) system based on adaptive dynamic programming (ADP). To overcome the challenge of establishing an accurate mathematical model for the CSTR system, neural networks are employed to reconstruct the dynamics of the CSTR system using the production data of the system. A new nonquadratic form performance index function is provided, where the control input is constrained in order not to exceed the bound of the actuator. Then, the operational optimal control problem of CSTR is formulated. Furthermore, an iterative ADP (IADP) algorithm is developed to obtain the optimal tracking controller for the CSTR system with control constraints. In particular, the convergence analysis of the IADP algorithm is developed. The proposed IADP algorithm is implemented via the dual heuristic dynamic programming structure. Finally, the proposed approach is applied to the real CSTR system to verify the effectiveness and performance.

Neuro-Optimal Control for Discrete Stochastic Processes via a Novel Policy Iteration Algorithm

In this paper, a novel policy iteration adaptive dynamic programming (ADP) algorithm is presented which is called “local policy iteration ADP algorithm” to obtain the optimal control for discrete stochastic processes. In the proposed local policy iteration ADP algorithm, the iterative decision rules are updated in a local space of the whole state space. Hence, we can significantly reduce the computational burden for the CPU in comparison with the conventional policy iteration algorithm. By analyzing the convergence properties of the proposed algorithm, it is shown that the iterative value functions are monotonically nonincreasing. Besides, the iterative value functions can converge to the optimum in a local policy space. In addition, this local policy space will be described in detail for the first time. Under a few weak constraints, it is also shown that the iterative value function will converge to the optimal performance index function of the global policy space. Finally, a simulation example is presented to validate the effectiveness of the developed method.

Neural-network-based output-feedback control with stochastic communication protocols

This paper is concerned with the neural-network-based (NN-based) output-feedback control issue for a class of nonlinear systems. For the purpose of effectively mitigating the phenomena of data congestion/collision, the stochastic communication protocols are favorably utilized to orchestrate the data transmissions, and the resultant closed-loop plant is represented by a so-called protocol-induced Markovian jump system with uncertain transition probability matrices. Taking such an uncertainty probability into account, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to obtain the desired suboptimal solution with the help of auxiliary quasi-HJB equation, and the algorithm convergence is also investigated via the intensive use of the mathematical analysis. In this ADP framework, an NN-based observer with a novel adaptive tuning law is first adopted to reconstruct the system states. Then, based on the reconfigurable system, an actor–critic NN scheme with online learning is developed to realize the considered control strategy. Furthermore, in light of the Lyapunov theory, some sufficient conditions are derived to guarantee the stability of the zero equilibrium point of the closed-loop system as well as the boundedness of the estimation errors for critic and actor NN weights. Finally, a simulation example is employed to demonstrate the effectiveness of the developed suboptimal control scheme.

Discrete-Time Impulsive Adaptive Dynamic Programming.

In this paper, a new iterative adaptive dynamic programming (ADP) algorithm is developed to solve optimal impulsive control problems for infinite horizon discrete-time nonlinear systems. Considering the constraint of the impulsive interval, in each iteration, the iterative impulsive value function under each possible impulsive interval is obtained, and then the iterative value function and iterative control law are achieved. A new convergence analysis method is developed which proves an iterative value function to converge to the optimum as the iteration index increases to infinity. The properties of the iterative control law are analyzed, and the detailed implementation of the optimal impulsive control law is presented. Finally, two simulation examples with comparisons are given to show the effectiveness of the developed method.

Stable value iteration for two-player zero-sum game of discrete-time nonlinear systems based on adaptive dynamic programming

In this paper, a stable value iteration (SVI) algorithm is developed to solve discrete-time two-player zero-sum game (TP-ZSG) for nonlinear systems based on adaptive dynamic programming (ADP). In the SVI algorithm, both optimality and stability of nonlinear systems are considered with proofs given. First, an iterative ADP algorithm is presented to obtain the approximate optimal solutions by solving Hamilton–Jacobi–Isaacs (HJI) equation. Second, a range of the discount factor is shown, which guarantees HJI equation serving as a Lyapunov equation. Moreover, we prove that if the iteration number reaches a given number, then the iterative control inputs make the closed-loop system asymptotic stable. Third, in order to improve the practicability of the developed stability condition, a simple criteria is established based on Lyapunov stability theory. Neural networks (NNs) are used to approximate the system states, the value function, the control and disturbance inputs. Finally, simulation results are given to illustrate the performance of the developed optimal control method.

Data-driven optimal tracking control of discrete-time multi-agent systems with two-stage policy iteration algorithm

Herein, a novel adaptive dynamic programming (ADP) algorithm is developed to solve the optimal tracking control problem of discrete-time multi-agent systems. Compared to the classical policy iteration ADP algorithm with two components, policy evaluation, and policy improvement, a two-stage policy iteration algorithm is proposed to obtain the iterative control laws and the iterative performance index functions. The proposed algorithm contains a sub-iteration procedure to calculate the iterative performance index functions at the policy evaluation. The convergence proof for the iterative performance index functions and the iterative control laws are provided. Subsequently, the stability of the closed-loop error system is also provided. Further, an actor-critic neural network (NN) is used to approximate both the iterative control laws and the iterative performance index functions. The actor-critic NN can implement the developed algorithm online without knowledge of the system dynamics. Finally, simulation results are provided to illustrate the performance of our method.