Adaptive Dynamic Programming Scheme Research Articles

Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems.

In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.

Multi-H∞ Controls for Unknown Input-Interference Nonlinear System With Reinforcement Learning.

This article studies the multi-H∞ controls for the input-interference nonlinear systems via adaptive dynamic programming (ADP) method, which allows for multiple inputs to have the individual selfish component of the strategy to resist weighted interference. In this line, the ADP scheme is used to learn the Nash-optimization solutions of the input-interference nonlinear system such that multiple H∞ performance indices can reach the defined Nash equilibrium. First, the input-interference nonlinear system is given and the Nash equilibrium is defined. An adaptive neural network (NN) observer is introduced to identify the input-interference nonlinear dynamics. Then, the critic NNs are used to learn the multiple H∞ performance indices. A novel adaptive law is designed to update the critic NN weights by minimizing the Hamiltonian-Jacobi-Isaacs (HJI) equation, which can be used to directly calculate the multi-H∞ controls effectively by using input-output data such that the actor structure is avoided. Moreover, the control system stability and updated parameter convergence are proved. Finally, two numerical examples are simulated to verify the proposed ADP scheme for the input-interference nonlinear system.

Tracking Control for a Lower Extremity Exoskeleton Based on Adaptive Dynamic Programing.

The utilization of lower extremity exoskeletons has witnessed a growing presence across diverse domains such as the military, medical treatment, and rehabilitation. This paper introduces a novel design of a lower extremity exoskeleton specifically tailored for individuals engaged in heavy object carrying tasks. The exoskeleton incorporates an impressive 12 degrees of freedom (DOF), with four of them being effectively controlled through hydraulic cylinders. To achieve optimal control of this intricate lower extremity exoskeleton system, the authors propose an adaptive dynamic programming (ADP) algorithm. Several crucial components are established to implement this control scheme. These include the formulation of the state equation for the lower extremity exoskeleton system, which is well-suited for the ADP algorithm. Additionally, a corresponding performance index function based on the tracking error is devised, along with the game algebraic Riccati equation. By employing the value iteration ADP scheme, the lower extremity exoskeleton demonstrates highly effective tracking control. This research not only highlights the potential of the proposed control approach but also showcases its ability to enhance the overall performance and functionality of lower extremity exoskeletons, particularly in scenarios involving heavy object carrying. Overall, this study contributes to the advancement of lower extremity exoskeleton technology and offers valuable insights into the application of ADP algorithms for achieving precise and efficient control in demanding tasks.

Finite‐horizon optimal tracking control for constrained‐input nonlinear interconnected system using aperiodic distributed nonzero‐sum games

This paper proposes a distributed adaptive dynamic programming scheme to investigate the optimal tracking control problem for finite-horizon non-linear interconnected systems with constraint inputs under aperiodic sampling. A N-player nonzero-sum differential game system is constructed with the presented non-linear interconnected system and the tracking error system by introducing the augment vectors. To address the problems of constrained-input and finite-horizon control, a non-quadratic utility function and a finite-horizon cost function are utilized which will arise in the time-varying Hamilton–Jacobi (HJ) equation. Then, a periodic event-triggered scheme is designed to realize aperiodic sampling, where the consumption of communication resources is reduced and the Zeno behavior is avoided. Under the designed periodic event-triggered scheme, the time-varying HJ equation is almost impossible to get an analytical solution due to its hybrid properties and non-linearity. Therefore, the critic neural networks are used to estimate the optimal solution of the HJ equation, and the weight update law is constructed to guarantee the uniformly ultimate bounded of approximated errors. Further, the hybrid nonzero-sum differential game is confirmed to be uniformly ultimate bounded by using the Lyapunov theory. Finally, the obtained distributed PET control strategy is successfully applied to dispose the missile-target intercepter problem.

Integral reinforcement learning based event-triggered control with input saturation

In this paper, a novel integral reinforcement learning (IRL)-based event-triggered adaptive dynamic programming scheme is developed for input-saturated continuous-time nonlinear systems. By using the IRL technique, the learning system does not require the knowledge of the drift dynamics. Then, a single critic neural network is designed to approximate the unknown value function and its learning is not subjected to the requirement of an initial admissible control. In order to reduce computational and communication costs, the event-triggered control law is designed. The triggering threshold is given to guarantee the asymptotic stability of the control system. Two examples are employed in the simulation studies, and the results verify the effectiveness of the developed IRL-based event-triggered control method.

An Adaptive Dynamic Programming Scheme for Nonlinear Optimal Control With Unknown Dynamics and Its Application to Turbofan Engines

In this article, a novel adaptive dynamic programming (ADP) approach is proposed for the optimal control problem of nonlinear continuous control systems with unknown dynamics. First, an alternating iteration algorithm based on Hamilton-Jacobi-Bellman equation is proposed for the optimal control of known nonlinear control systems. Then, the convergence results of the alternating iteration algorithm are obtained by using mathematical induction and monotone bounded convergence theorem. Moreover, the global asymptotic stability of the nonlinear closed-loop system is proved. Second, based on the scheme of alternating iteration algorithm, an ADP algorithm for the optimal control problem with unknown nonlinear dynamic model is developed by using the basis function approximation method and Newton-Leibniz formula, which can update the control strategy online by utilizing input and output information of the system. In addition, the convergence analysis of the proposed ADP algorithm is derived. Finally, the feasibility of the established results is verified by two examples, and the ADP method is applied to the optimal tracking fuel control problem of turbofan engines.

Event-Triggered Integral Sliding-Mode Control for Nonlinear Constrained-Input Systems With Disturbances via Adaptive Dynamic Programming

This article proposed a novel event-triggered integral sliding-mode control (ISMC) strategy for nonlinear system with disturbances via robust adaptive dynamic programming (ADP) considering control constrains. A mixed event-triggered ISMC scheme with two different trigger parts is developed and the optimal performance of sliding-mode dynamics with input constrains is ensured. By guaranteeing that the system trajectory converges to the sliding-mode surface with removing the input disturbances, a discontinuous part triggering rule is presented together with the existence analysis of a lower trigger interval time bound. Then, the optimal event-triggered control of sliding-mode dynamics is converted into a discounted factor-based $H_{\infty }$ constrained control problem under continuous part triggering condition. To solve the event-triggered HJI equation, a critic-only neural network (NN)-based ADP scheme is developed by applying a concurrent learning. The NN weight is updated by analyzing the uniformly ultimately bounded (UUB) stability of sliding-mode dynamics considering the event-triggered condition via the Lyapunov technique. Finally, the validity of our control strategy is verified by simulation.

Adaptive Dynamic Programming Using Lyapunov Function Constraints

This letter is concerned with a stabilizing adaptive dynamic programming (ADP) approach to approximate solution of a given infinite-horizon optimal control problem. Since the latter problem cannot, in general, be solved exactly, a parametrized function approximator for the infinite-horizon cost function is introduced in ADP (so called “critic”). This critic is used to adapt the parameters of the function approximator. The so called “actor” in turn derives the optimal input of the system. It is a notoriously hard problem to guarantee closed-loop stability of ADP due to the use of approximation structures in the control scheme. Since at least stabilizability is always assumed in the analyzes of ADP, it is justified to invoke a respective Lyapunov function. The proposed ADP scheme explicitly uses the said Lyapunov function to simultaneously optimize the critic and guarantee closed-loop stability. A Hessian-free optimization routine is utilized for the actor and critic optimization problems. Convergence to prescribed vicinities of the optima is shown. A computational study showed significant performance improvement for the critic-based approach compared a nominal stabilizing controller for a range of initial conditions.

An Overview of Research on Adaptive Dynamic Programming

Adaptive dynamic programming (ADP) is a novel approximate optimal control scheme, which has recently become a hot topic in the field of optimal control. As a standard approach in the field of ADP, a function approximation structure is used to approximate the solution of Hamilton-Jacobi-Bellman (HJB) equation. The approximate optimal control policy is obtained by using the offline iteration algorithm or the online update algorithm. This paper gives a review of ADP in the order of the variation on the structure of ADP scheme, the development of ADP algorithms and applications of ADP scheme, aiming to bring the reader into this novel field of optimization technology. Furthermore, the future studies are pointed out.

Optimal control for discrete-time affine non-linear systems using general value iteration

In this study, the authors propose a novel adaptive dynamic programming scheme based on general value iteration (VI) to obtain near optimal control for discrete-time affine non-linear systems with continuous state and control spaces. First, the selection of initial value function is different from the traditional VI, and a new method is introduced to demonstrate the convergence property and convergence speed of value function. Then, the control law obtained at each iteration can stabilise the system under some conditions. At last, an error-bound-based condition is derived considering the approximation errors of neural networks, and then the error between the optimal and approximated value functions can also be estimated. To facilitate the implementation of the iterative scheme, three neural networks with Levenberg-Marquardt training algorithm are used to approximate the unknown system, the value function and the control law. Two simulation examples are presented to demonstrate the effectiveness of the proposed scheme.