Adaptive Dynamic Programming Technique Research Articles

Event-Triggered Near-Optimal Control for Unknown Discrete-Time Nonlinear Systems Using Parallel Control.

This article uses parallel control to investigate the problem of event-triggered near-optimal control (ETNOC) for unknown discrete-time (DT) nonlinear systems. First, to achieve parallel control, an augmented nonlinear system (ANS) with an augmented performance index (API) is proposed to introduce the control input into the feedback system. The control stability relationship between the ANS and the original system is analyzed, and it is shown that, by choosing a proper API, optimal control of the ANS with the API can be seen as near-optimal control of the original system with the original performance index (OPI). Second, based on parallel control, a novel event-triggered scheme is proposed, and then a novel ETNOC method is developed using the time-triggered optimal value function of the ANS with the API. The control stability is proved, and an upper bound, which is related to the design parameter, is provided for the actual performance index in advance. Then, to implement the developed ETNOC method for unknown DT nonlinear systems, a novel online learning algorithm is developed without reconstructing unknown systems, and neural network (NN) and adaptive dynamic programming (ADP) techniques are employed in the developed algorithm. The convergence of the signals in the closed-loop system (CLS) is shown using the Lyapunov approach, and the assumption of boundedness of input dynamics is not required. Finally, two simulations justify the theoretical conjectures.

Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming

<abstract><p>The fault-tolerant control problem of a hydraulic servo actuator in the presence of actuator faults is studied utilizing adaptive dynamic programming. This task is challenging because of unknown system dynamics, uncertain disturbances or unmeasurable system states of such highly nonlinear systems in real applications. The aim is to achieve asymptotic tracking and actuator faults compensation by minimizing some predefined performance index. The discrete-time algebraic Riccati equation is iteratively solved by the adaptive dynamic programming approach. For practical reasons, adaptive dynamic programming techniques and fault compensation are integrated to iteratively compute an approximated optimal fault-tolerant control using real-time input/output data without any a priori knowledge of the system dynamics and unmeasurable states. As a result, a fault-tolerant control of hydraulic servo actuator is then designed based on adaptive dynamic programming via output feedback. Also, the convergence analysis of a data-driven fault-tolerant control is theoretically shown as well. Finally, intensive simulation results are given to prove the validity and merits of the developed data-driven fault-tolerant control strategy.</p></abstract>

Learning-Based Cooperative Multiagent Formation Control With Collision Avoidance

This article presents a learning-based controller to solve the cooperative formation control problem for multiagent system (MAS) with collision avoidance. First, the consensus problem of first-order MAS is mostly solved by linear matrix inequality (LMI) without consideration of energy loss. To overcome these difficulties, an adaptive dynamic programming (ADP) technique is fit to solve the consensus problem and similar formation control problem for second-order MAS by the establishment of a performance index function. Besides, we introduce the generalized policy iteration (GPI) algorithm as a kind of ADP technique without the problem of low convergence speed and high computational complexity. Combined with previous works, it can be found that our proposed structure can be extended to high-order cases based on the structure of local neighborhood formation error and algorithm. Afterward, the convergence analysis, optimality analysis, and stability analysis are given. Neural networks (NNs) are also implemented to approximate the iterative control policies and value functions, respectively. Moreover, we realize that many collisions may occur in the formation control problem. Inspired by the idea of the artificial potential field (APF) technique, the concept of the repulsive force field is introduced based on our proposed learning-based structure to avoid collisions simply and efficiently. Finally, a simulation is provided to demonstrate the effectiveness of our proposed method.

Optimal Output Regulation for General Linear Systems via Adaptive Dynamic Programming.

In this article, we consider an adaptive optimal output regulation problem for general linear systems. The purpose of the optimal output regulation problem is to guarantee the stability of the closed-loop system and disturbance rejection, as well by minimizing some predefined performance indices. It can be realized by using an optimal controller, in which both optimal feedback control gain and optimal feedforward control gain are included. First, an adaptive dynamic programming (ADP) technique is used to solve the optimal feedback control gain. Next, the unknown system matrices of the plant are explicitly computed. In addition, based on the property of the minimal polynomial, the coefficient of the exogenous disturbance in the expression of the regulated output can also be calculated. Finally, according to the regulator equation, an extra cost function is given, which aims to obtain the optimal feedforward control gain. The linear vector space optimization methods are used to solve the optimal problem. As a result, the linear optimal output regulation problem can be solved by the approximately optimal feedback and feedforward control gains.

Adaptive neural optimal control via command filter for nonlinear multi‐agent systems including time‐varying output constraints

AbstractIn this article, an optimal command‐filtered backstepping control approach is proposed for uncertain strict‐feedback nonlinear multi‐agent systems (MASs) including output constraints and unmodeled dynamics. One‐to‐one nonlinear mapping (NM) is utilized to recast constrained systems as corresponding unrestricted systems. A dynamical signal is applied to cope with unmodeled dynamics. Based on dynamic surface control (DSC), the feedforward controller is designed by introducing error compensating signals. The optimal feedback controller is produced applying adaptive dynamic programming (ADP) and integral reinforcement learning (IRL) techniques in which neural networks are utilized to approximate the relevant cost functions online with established weight updating laws. Therefore, the entire controller, including feedforward and feedback controllers, not only ensures that all signals in the closed‐loop systems are cooperative semi‐globally uniformly ultimately bounded (SGUUB) and the outputs maintain in the provided time‐varying constraints, but also makes sure that the cost functions achieve minimization. A simulation example is presented to illustrate the feasibility of the proposed control algorithm.

Learning-Based Adaptive Optimal Control for Connected Vehicles in Mixed Traffic: Robustness to Driver Reaction Time.

Through vehicle-to-vehicle (V2V) communication, both human-driven and autonomous vehicles can actively exchange data, such as velocities and bumper-to-bumper distances. Employing the shared data, control laws with improved performance can be designed for connected and autonomous vehicles (CAVs). In this article, taking into account human-vehicle interaction and heterogeneous driver behavior, an adaptive optimal control design method is proposed for a platoon mixed with multiple preceding human-driven vehicles and one CAV at the tail. It is shown that by using reinforcement learning and adaptive dynamic programming techniques, a near-optimal controller can be learned from real-time data for the CAV with V2V communications, but without the precise knowledge of the accurate car-following parameters of any driver in the platoon. The proposed method allows the CAV controller to adapt to different platoon dynamics caused by the unknown and heterogeneous driver-dependent parameters. To improve the safety performance during the learning process, our off-policy learning algorithm can leverage both the historical data and the data collected in real time, which leads to considerably reduced learning time duration. The effectiveness and efficiency of our proposed method is demonstrated by rigorous proofs and microscopic traffic simulations.

Event-Triggered Near-Optimal Control of Discrete-Time Constrained Nonlinear Systems With Application to a Boiler-Turbine System

This article presents a novel event-triggered near-optimal control (ETNOC) method for discrete-time (DT) constrained nonlinear systems. First, the tracking error system is constructed to convert the tracking control problem to the regulation problem. By introducing the tracking error system, the asymmetric control constraints design for the original constrained system can be converted to the symmetric control constraints design for the tracking error system. Second, a novel triggering condition is developed using the time-triggered optimal value function and control law. It is proven that the closed-loop system (CLS) is asymptotically stable under the developed ETNOC method, and there exists a predetermined upper bound for the real performance index. Then, to implement the developed ETNOC method, a parallel control approach with neural networks (NNs) and adaptive dynamic programming techniques is proposed to predict the next state of the system and obtain the optimal value function and control law. The stability analysis of the CLS is provided in the consideration of the estimation errors of the NN weights and state. Finally, the effectiveness of the developed ETNOC method is validated by an application to a boiler-turbine system.

Neurodynamic Programming and Tracking Control for Nonlinear Stochastic Systems by PI Algorithm

In this paper, for nonlinear stochastic systems, the optimal tracking control problem (OTCP) is solved by adaptive dynamic programming (ADP) techniques. Firstly, the complex OTCP is transformed into a stable control optimization problem by reconstructing a new stochastic augmented system. Then, simplifying the actor-critic architecture and reducing the computational load, critic neural networks (NNs) is used in iterative learning. And by using Lyapunov method, the ultimate uniform boundedness (UUB) of the tracking system is proved. To be precise, no literature has been published on the OTCP for nonlinear Itô type stochastic systems via the ADP method. This work is the first attempt in this field. Finally, in simulation, the method is applied to sinusoidal waveform and periodic rectangular step signal, and even unbounded exponential waveform.

Event-Triggered Optimal Parallel Tracking Control for Discrete-Time Nonlinear Systems

A novel event-triggered optimal tracking control (ETOTC) method is developed for discrete-time nonlinear systems in this study. For the time-invariant desired trajectory, we prove that the tracking error is asymptotically stable, and an upper bound of the real performance index can be predetermined by a design parameter. For the time-varying desired trajectory, the developed triggering condition reduces communication costs by relaxing the restriction of the asymptotic stability of the closed-loop system, and we prove that the tracking error is uniformly ultimately bounded (UUB). The developed ETOTC method entails obtaining the next state of the real system. Therefore, a parallel control approach is proposed to predict the next state by constructing a parallel system for the real system. Neural networks (NNs) and adaptive dynamic programming (ADP) techniques are utilized in the parallel control approach. Moreover, the stability analysis of the closed-loop system is shown, and the tracking error and NN weight estimation errors are proved to be UUB using the Lyapunov approach. Finally, we validate the developed ETOTC method through two simulations.

Learning‐based super‐twisting sliding‐mode control for space circumnavigation mission with suboptimal reaching under input constraints

AbstractThe problem of attitude‐orbit synchronous finite‐time control for the space circumnavigation (SCN) mission with input constraints is investigated in this research article. A novel terminal sliding‐mode (TSM) manifold with nonsingular first derivative is developed to ensure that the sliding‐mode reduced‐order system is practical finite‐time stable. Then, the learning‐based adaptive dynamic programming (ADP) technique is adopted to design a super‐twisting sliding‐mode control scheme, so that the proposed TSM is suboptimal reached. Especially, benefited from the introduction of two nonlinear performance indexes, the proposed sliding‐mode control strategy satisfies the saturation constraint of the propulsion system, and the weight update rate of the actor‐critic (AC) network will not be too large, even if the initial value of the system is too large. Finally, the simulation results show that both attitude and orbital tracking errors in the SCN mission are forced to converge to a small neighborhood containing the origin synchronously under input saturation constraints.

Model-Free distributed optimal consensus control of nonlinear multi-agent systems: A graphical game approach

In this paper, the optimal consensus control problem of nonlinear multi-agent systems(MASs) with completely unknown dynamics is considered. The problem is formulated in a differential graphical game approach which can be solved by Hamilton-Jacobi (HJ) equations. The main difficulty in solving the HJ equations lies in the nonlinear coupling between equations. Based on the Adaptive Dynamic Programming (ADP) technique, an VI-PI mixed HDP algorithm is proposed to solve the HJ equations distributedly. With the PI step, a suitable iterative initial value can be obtained according to the initial policies. Then, VI steps are run to get the optimal solution with exponential convergence rate. Neural networks (NNs) are applied to approximate the value functions, which makes the data-driven end-to-end learning possible. A numerical simulation is conducted to show the effectiveness of the proposed algorithm.

Event-triggered optimal control for discrete-time multi-player non-zero-sum games using parallel control

This paper presents a novel event-triggered optimal control (ETOC) method for discrete-time (DT) multi-player non-zero-sum games (NZSGs). First, a novel event-triggered algorithm is developed for DT multi-player NZSGs based on the time-triggered optimal value functions. Therefore, the developed event-triggered algorithm only needs to solve the time-triggered Hamilton–Jacobi-Bellman (HJB) equations. Then, the asymptotic stability of the closed-loop system is proved. Additionally, we show that an upper bound for the sum of the actual performance indices of all the players can be determined in advance. A key step in the implementation of the developed event-triggered algorithm is to obtain the next state of the actual system, which is difficult to implement on the actual system. Thus, a parallel control method is utilized to predict the next state by constructing the parallel system for the actual system and obtain the optimal value functions. The method that combines the developed event-triggered algorithm and parallel control is called the event-triggered optimal parallel control (ETOPC) method. The neural network (NN) technique and the iterative adaptive dynamic programming (IADP) technique are employed in parallel control. Moreover, the control stability is shown further in the consideration of the NN weight approximation errors. Finally, two simulations justify the theoretical conjectures.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

Adaptive dynamic programming (ADP) technique is adopted in this work to investigate the optimal control problem of Markovian jump systems. By utilizing Bellman’s optimality principle, a discrete Hamilton Jacobi Bellman (HJB) equation is established to design the optimal controller for the system under consideration. Then, based on value iteration, a new ADP algorithm is proposed for finding the solution of the established HJB equation. It is proven that the iterative solution sequence generated by the developed ADP iterative approach under zero initial values is monotonically convergent. Neural networks are constructed to accomplish the presented value iteration ADP algorithm. At last, simulation researches for two Markovian jump systems demonstrate the effectiveness of the proposed optimal control method.

Optimal control of unknown continuous time linear periodic systems by adaptive dynamic programming with applications to magnetic attitude control

AbstractThis article studies the data‐driven controller design of continuous‐time linear periodic systems. By treating the optimal control problem as a two‐player zero‐sum game problem, the adaptive dynamic programming technique is utilized. Based on the Newton's method, a one‐loop model‐based policy‐iteration (PI) method is introduced. Since the initial matrix for starting the proposed model‐based PI method is not easy to choose, a data‐driven value iteration method without requiring the knowledge of the initial matrix is then proposed. The convergence of the proposed methods is analyzed and the numerical simulation verifies the effectiveness of the proposed scheme.

Adaptive event-triggered distributed optimal guidance design via adaptive dynamic programming

In this paper, the multi-missile cooperative guidance system is formulated as a general nonlinear multi-agent system. To save the limited communication resources, an adaptive event-triggered optimal guidance law is proposed by designing a synchronization-error-driven triggering condition, which brings together the consensus control with Adaptive Dynamic Programming (ADP) technique. Then, the developed event-triggered distributed control law can be employed by finding an approximate solution of event-triggered coupled Hamilton-Jacobi-Bellman (HJB) equation. To address this issue, the critic network architecture is constructed, in which an adaptive weight updating law is designed for estimating the cooperative optimal cost function online. Therefore, the event-triggered closed-loop system is decomposed into two subsystems: the system with flow dynamics and the system with jump dynamics. By using Lyapunov method, the stability of this closed-loop system is guaranteed and all signals are ensured to be Uniformly Ultimately Bounded (UUB). Furthermore, the Zeno behavior is avoided. Simulation results are finally provided to demonstrate the effectiveness of the proposed method.

The Adaptive Dynamic Programming Toolbox.

The paper develops the adaptive dynamic programming toolbox (ADPT), which is a MATLAB-based software package and computationally solves optimal control problems for continuous-time control-affine systems. The ADPT produces approximate optimal feedback controls by employing the adaptive dynamic programming technique and solving the Hamilton–Jacobi–Bellman equation approximately. A novel implementation method is derived to optimize the memory consumption by the ADPT throughout its execution. The ADPT supports two working modes: model-based mode and model-free mode. In the former mode, the ADPT computes optimal feedback controls provided the system dynamics. In the latter mode, optimal feedback controls are generated from the measurements of system trajectories, without the requirement of knowledge of the system model. Multiple setting options are provided in the ADPT, such that various customized circumstances can be accommodated. Compared to other popular software toolboxes for optimal control, the ADPT features computational precision and time efficiency, which is illustrated with its applications to a highly non-linear satellite attitude control problem.

Nearly Optimal Integral Sliding-Mode Consensus Control for Multiagent Systems With Disturbances

This article considers integral sliding-mode control (ISMC) for multiagent systems with matched disturbances through adaptive dynamic programming (ADP) method. A distributed disturbance observer (DDO) is designed for each agent to generate the disturbance estimation. Using the disturbance estimation, an ISMC scheme is proposed to reject the input disturbance and obtain the equivalent local neighbor consensus sliding-mode dynamics. Then, a discount performance function is designed for each agent, and ADP technique is utilized to address the optimal consensus control for the equivalent sliding-mode dynamics online. Based on the gradient descent algorithm, we propose the adaptive weight tuning laws for the critic-actor neural networks (NNs) to carry out the ADP method. Furthermore, the local neighbor consensus errors and the weight estimation errors for the critic-actor NNs are proved to be uniformly ultimately bounded (UUB). Finally, the practical example is applied to validate the effectiveness of our results.

Adaptive output-feedback optimal control for continuous-time linear systems based on adaptive dynamic programming approach

This paper investigates the adaptive output-feedback optimal control problem for a class of continuous-time (CT) linear systems with dynamic uncertainties. Data-based controller design algorithm with policy iteration form is developed through adaptive dynamic programming (ADP) technique. In the process of proposed algorithm, only measured input and output information of the system is used to learn optimal control gain, but the exactly knowledge of the system is not required. Different from the existing works, the adaptive controllers learned by the algorithm has certain robustness to the dynamic uncertainties. Three examples are given to test the effectiveness and feasibility of the proposed approach.

Zero-sum differential games-based fast adaptive robust optimal sliding mode control design for uncertain missile autopilot with constrained input

This paper designs a novel adaptive robust optimal sliding mode control (SMC) strategy for constrained input missile system with uncertain input and unknown disturbance. First, a smooth function is employed to map the constrained input. Based on SMC theory, a fast adaptive SMC scheme is designed to reduce the effects of unknown input and approximate error which has two advantages. One is that the proposed scheme is independent of the uncertain information. The other is that the proposed scheme forces the states to converge with exponential rate as they are far away from the sliding mode surface, and the states move with linear rate when they are closed to the sliding mode surface. Then, a zero-sum differential games strategy-based adaptive dynamic programming (ADP) technique is employed to obtain the optimal performance of sliding mode dynamics system. The loop-system and estimation weight error can be proved stable by utilising Lyapunov theory. Finally, the missile autopilot system is used to indicate the effectiveness of the designed approach.

Learning-Based Control of Multiple Connected Vehicles in the Mixed Traffic by Adaptive Dynamic Programming

Abstract The emergence of connected and autonomous vehicles (CAVs) has increased opportunities to mitigate the traffic congestion, improve safety and reduce accidents. In this paper, we consider the mixed traffic case with multiple heterogeneous human-driven vehicles and multiple CAVs on freeways. Under mild conditions, the stabilizability of the overall system is proved. With the tracking errors of relative distance and velocity as the states, we design an input-to-state stabilizing controller that solves a linear quadratic regulator problem by means of reinforcement learning and adaptive dynamic programming techniques. The priori knowledge of the vehicle network model is not needed. For a string of connected human-driven and automated vehicles, we give the sufficient conditions to guarantee the general string stability. The proposed learning-based control methodology is validated by means of simulation results.