Adaptive Critic Learning Research Articles

Adaptive Critic Learning-Based Optimal Bipartite Consensus for Multiagent Systems With Prescribed Performance.

Developing a distributed bipartite optimal consensus scheme while ensuring user-predefined performance is essential in practical applications. Existing approaches to this problem typically require a complex controller structure due to adopting an identifier-actor-critic framework and prescribed performance cannot be guaranteed. In this work, an adaptive critic learning (ACL)-based optimal bipartite consensus scheme is developed to bridge the gap. A newly designed error scaling function, which defines the user-predefined settling time and steady accuracy without relying on the initial conditions, is then integrated into a cost function. The backstepping framework combines the ACL and integral reinforcement learning (IRL) algorithm to develop the adaptive optimal bipartite consensus scheme, which contributes a critic-only controller structure by removing the identifier and actor networks in the existing methods. The adaptive law of the critic network is derived by the gradient descent algorithm and experience replay to minimize the IRL-based residual error. It is shown that a compute-saving learning mechanism can achieve the optimal consensus, and the error variables of the closed-loop system are uniformly ultimately bounded (UUB). Besides, in any bounded initial condition, the evolution of bipartite consensus is limited to a user-prescribed boundary under bounded initial conditions. The illustrative simulation results validate the efficacy of the approach.

Adaptive fuzzy decentralized optimal control for interconnected nonlinear systems with unmodeled dynamics via mixed data and event driven method

In this work, based on the adaptive critic learning algorithms and fuzzy logic systems, the decentralized optimal control problem for interconnected nonlinear systems with unmodeled dynamics is investigated by applying a mixed data and event driven method. By introducing a modified performance index function for each auxiliary subsystem, the considered control problem of the whole interconnected system can be transformed into a group of event-driven optimal control problems of the auxiliary subsystems. Then, a three-layer neural network (NN) identifier is employed to reconstruct the completely unknown functions. Based on the obtained data-driven model, a single-critic NN is developed to solve the modified HJB equation. Meanwhile, a novel adaptive controller is designed by fuzzy logic systems to deal with the dynamical uncertainties cased by the unmodeled dynamics. Finally, a robot arm example is given to verify the effectiveness of the proposed control method.

Adaptive critic learning for approximate optimal event-triggered tracking control of nonlinear systems with prescribed performances

This paper studies the approximate optimal event-triggered tracking control problem for a class of second-order nonlinear systems with prescribed performances. By employing a prescribed performances function, restrictions on tracking errors are removed for prescribed performances including convergence rate, maximum overshoot, and steady-state error. On this basis, a complex optimal tracking control problem can be transformed into an optimal stabilising control problem by constructing an augmented system. Subsequently, an event-triggered mechanism is introduced to reduce communication resources. Then, an adaptive critic learning algorithm is used to solve the Hamilton-Jacobi-Bellman equation, where the weights in the critic networks are tuned through the gradient descent approach and experience replay technology. Through the developed learning method, the persistence of excitation condition is released, and the data efficiency is improved. The Lyapunov stability theory is introduced to verify the stability of the close-loop system and the tracking errors are uniformly ultimately bounded. Finally, simulation results are presented to show the effectiveness of the proposed approach.

Adaptive critic learning for event‐triggered safe control of nonlinear safety‐critical systems

AbstractIn this paper, an event‐triggered safe control method based on adaptive critic learning (ACL) is proposed for a class of nonlinear safety‐critical systems. First, a safe cost function is constructed by adding a control barrier function (CBF) to the traditional quadratic cost function; the optimization problem with safety constraints that is difficult to deal with by classical ACL methods is solved. Subsequently, the event‐triggered scheme is introduced to reduce the amount of computation. Further, combining the properties of CBF with the ACL‐based event‐triggering mechanism, the event‐triggered safe Hamilton–Jacobi–Bellman (HJB) equation is derived, and a single critic neural network (NN) framework is constructed to approximate the solution of the event‐triggered safe HJB equation. In addition, the concurrent learning method is applied to the NN learning process, so that the persistence of excitation (PE) condition is not required. The weight approximation error of the NN and the states of the system are proven to be uniformly ultimately bounded (UUB) in the safe set with the Lyapunov theory. Finally, the availability of the presented method can be validated through the simulation.

Event‐based robust performance guarantee for nonaffine plants via system identification

AbstractIn this paper, by making use of recurrent neural networks (RNNs), a novel event‐based robust optimal controller is designed for a category of continuous‐time (CT) nonlinear nonaffine systems with disturbances. For tackling the optimal‐robust problem transformation and the performance guarantee, a system identifier is developed to reconstruct the system, and an event‐triggered critic‐only strategy is investigated to replace the traditional time‐based actor‐critic structure. The RNN is adopted to reconstruct the nonaffine system as an identifier for obtaining the affine model. It simplifies the system structure and reduces computational burdens. In addition, the stability of the unknown nominal system learned by the RNN is proved. Then, using the obtained system identifier and the adaptive critic learning structure, the proposed event‐triggered robust control with performance guarantee is realized. At last, the effectiveness of the given design method is verified by two simulation examples.

Adaptive critic optimization to decentralized event‐triggered control of continuous‐time nonlinear interconnected systems

AbstractIn this article, a decentralized event‐triggered control (ETC) strategy is developed for nonlinear large‐scale systems subjects to matched interconnections. First, the large‐scale interconnected systems are transformed to several nominal isolated subsystems. Then, the optimal control problem of nominal isolated subsystems is solved via an event‐triggered method based on adaptive critic learning. Note that the ETC scheme is adopted to reduce the computational burden and communication resources. Moreover, a single network adaptive critic structure is constructed to approximate the optimal cost and the control policy. By employing a stabilizing term in the weight updating law of the critic neural network, there is no requirement for adopting initial admissible control in the proposed algorithm. Furthermore, we use the Lyapunov method to demonstrate that estimated weight vectors used in the critic networks are uniformly ultimately bounded. Finally, we provide an example to validate the proposed decentralized event‐triggered control strategy.

Adaptive Critic Learning for Constrained Optimal Event-Triggered Control With Discounted Cost.

This article studies an optimal event-triggered control (ETC) problem of nonlinear continuous-time systems subject to asymmetric control constraints. The present nonlinear plant differs from many studied systems in that its equilibrium point is nonzero. First, we introduce a discounted cost for such a system in order to obtain the optimal ETC without making coordinate transformations. Then, we present an event-triggered Hamilton-Jacobi-Bellman equation (ET-HJBE) arising in the discounted-cost constrained optimal ETC problem. After that, we propose an event-triggering condition guaranteeing a positive lower bound for the minimal intersample time. To solve the ET-HJBE, we construct a critic network under the framework of adaptive critic learning. The critic network weight vector is tuned through a modified gradient descent method, which simultaneously uses historical and instantaneous state data. By employing the Lyapunov method, we prove that the uniform ultimate boundedness of all signals in the closed-loop system is guaranteed. Finally, we provide simulations of a pendulum system and an oscillator system to validate the obtained optimal ETC strategy.

Distributed synchronisation control of unknown nonlinear systems with an active leader

This paper investigates the optimal synchronisation problem for unknown nonlinear heterogeneous multi-agent systems (MASs) with an active leader and disturbances. Due to the external input signal of the active leader, the resulted uncooperative behaviours and unpredictability increase difficulty to derive the analytical solution of the coupled Bellman equation. To this end, we develop a distributed model reference adaptive critic learning algorithm with a hierarchical and distributed synchronisation control framework, which transforms the synchronisation problem of unknown nonlinear MASs into that of defined reference ones. Moreover, the algorithm extends our previous work to the distributed MASs, so that it is guaranteed that all the agents reach an agreement with the minimised transient errors and robustness against the disturbances and the neural network (NN) approximation errors. It is proved that the local neighbour tracking errors and the NN weight estimation errors are uniformly ultimately bounded via Lyapunov stability theory. A simulation example for multi-manipulator systems demonstrates the effectiveness of the algorithm.

Self-Learning Robust Control Synthesis and Trajectory Tracking of Uncertain Dynamics.

In this article, we investigate the self-learning robust control synthesis and tracking design of general uncertain dynamical systems. Based on the adaptive critic learning, the robust stabilization method is developed with the help of conducting problem transformation. In addition, by considering the optimal control solution with a discounted cost function, the established method is extended to address the robust trajectory tracking design problem. The Lyapunov stability analysis is also conducted for proving the robustness of the related control plants. Finally, the simulation verification with the three case studies is provided in terms of robust stabilization and trajectory tracking, respectively.

Event-Driven H∞-Constrained Control Using Adaptive Critic Learning.

This article considers an event-driven H∞ control problem of continuous-time nonlinear systems with asymmetric input constraints. Initially, the H∞ -constrained control problem is converted into a two-person zero-sum game with the discounted nonquadratic cost function. Then, we present the event-driven Hamilton-Jacobi-Isaacs equation (HJIE) associated with the two-person zero-sum game. Meanwhile, we develop a novel event-triggering condition making Zeno behavior excluded. The present event-triggering condition differs from the existing literature in that it can make the triggering threshold non-negative without the requirement of properly selecting the prescribed level of disturbance attenuation. After that, under the framework of adaptive critic learning, we use a single critic network to solve the event-driven HJIE and tune its weight parameters by using historical and instantaneous state data simultaneously. Based on the Lyapunov approach, we demonstrate that the uniform ultimate boundedness of all the signals in the closed-loop system is guaranteed. Finally, simulations of a nonlinear plant are presented to validate the developed event-driven H∞ control strategy.

Online Adaptive Critic Learning Control of Unknown Dynamics With Application to Deep Submergence Rescue Vehicle

As a powerful tool for nonlinear systems robust controller design, robust adaptive dynamic programming (RADP) methods require initial admissible control and prior knowledge of disturbance to be effective. As the most effective approach to provide robustness to uncertainties, active disturbance attenuation (ADA), was rarely considered in RADP literatures. To combine ADA with RADP, a neural-network identifier was developed initially to approximate the plant dynamics and the imposed external disturbance. System states was extended with the approximated disturbance to establish ADA actor-critic learning. To relax the initial admissible control constraint, a novel auxiliary system were created based on the identifier dynamics. Theoretical analysis and simulations on unstable nonlinear system show that the approximated control law with respect to the auxiliary system and a newly proposed cost function itself could guarantee asymptotic stability of the original system. Simulations and comparison with other model-free control techniques demonstrated the excellent performance and robustness of the proposed method. Applicability of the proposed method was validated by applying it to trajectory tracking control of a deep submergence rescue vehicle.

Neural critic learning toward robust dynamic stabilization

SummaryIn this article, we focus on developing a neural‐network‐based critic learning strategy toward robust dynamic stabilization for a class of uncertain nonlinear systems. A type of general uncertainties involved both in the internal dynamics and in the input matrix is considered. An auxiliary system with actual action and auxiliary signal is constructed after dynamics decomposition and combination for the original plant. The reasonability of the control problem transformation from robust stabilization to optimal feedback design is also provided theoretically. After that, the adaptive critic learning method based on a neural network is established to derive the approximate optimal solution of the transformed control problem. The critic weight can be initialized to a zero vector, which apparently facilitates the learning process. Numerical simulation is finally presented to illustrate the effectiveness of the critic learning approach for neural robust stabilization.

MRAC for unknown discrete-time nonlinear systems based on supervised neural dynamic programming

This paper investigates the model reference adaptive control (MRAC) problem for unknown nonlinear discrete-time systems, which is how to guarantee adaptation to the variable reference input and robustness to perturbation. A supervised neural dynamic programming (SNDP) approach is developed to solve this MRAC problem, which includes a learning mode and a control mode. In the learning mode, a data-based adaptive critic learning algorithm is proposed, which guarantees that the controlled objective adaptively tracks the reference model on behavior. Such an algorithm also ensures flexible switching from the learning mode to the control mode under which the robustness of the closed-loop control systems is further improved. By employing a newly defined mode scheduler that regulates the learning mode and the control mode, the adaptation and the robustness of the systems are both achieved by the developed SNDP approach. Its uniformly ultimately bounded property is proved by using Lyapunov method. Simulation results verify that the developed SNDP approach ensures the adaptation to the variable reference input, and has superiority over some traditional MRAC methods in improving the robustness.

Adaptive Critic Learning and Experience Replay for Decentralized Event-Triggered Control of Nonlinear Interconnected Systems

In this paper, we develop a decentralized event-triggered control (ETC) strategy for a class of nonlinear systems with uncertain interconnections. To begin with, we show that the decentralized ETC policy for the whole system can be represented by a group of optimal ETC laws of auxiliary subsystems. Then, under the framework of adaptive critic learning, we construct the critic networks to solve the event-triggered Hamilton–Jacobi–Bellman equations related to these optimal ETC laws. The weight vectors used in the critic networks are updated by using the gradient descent approach and the experience replay (ER) technique together. With the aid of the ER technique, we can conquer the difficulty arising in the persistence of excitation condition. Meanwhile, by using classic Lyapunov approaches, we prove that the estimated weight vectors used in the critic networks are uniformly ultimately bounded. Moreover, we demonstrate that the obtained decentralized ETC can force the overall system to be asymptotically stable. Finally, we present an interconnected nonlinear plant to validate the proposed decentralized ETC scheme.

Neural robust stabilization via event-triggering mechanism and adaptive learning technique

The robust control synthesis of continuous-time nonlinear systems with uncertain term is investigated via event-triggering mechanism and adaptive critic learning technique. We mainly focus on combining the event-triggering mechanism with adaptive critic designs, so as to solve the nonlinear robust control problem. This can not only make better use of computation and communication resources, but also conduct controller design from the view of intelligent optimization. Through theoretical analysis, the nonlinear robust stabilization can be achieved by obtaining an event-triggered optimal control law of the nominal system with a newly defined cost function and a certain triggering condition. The adaptive critic technique is employed to facilitate the event-triggered control design, where a neural network is introduced as an approximator of the learning phase. The performance of the event-triggered robust control scheme is validated via simulation studies and comparisons. The present method extends the application domain of both event-triggered control and adaptive critic control to nonlinear systems possessing dynamical uncertainties.

On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $H_{\infty}$ Control.

In this paper, based on the adaptive critic learning technique, the control for a class of unknown nonlinear dynamic systems is investigated by adopting a mixed data and event driven design approach. The nonlinear control problem is formulated as a two-player zero-sum differential game and the adaptive critic method is employed to cope with the data-based optimization. The novelty lies in that the data driven learning identifier is combined with the event driven design formulation, in order to develop the adaptive critic controller, thereby accomplishing the nonlinear control. The event driven optimal control law and the time driven worst case disturbance law are approximated by constructing and tuning a critic neural network. Applying the event driven feedback control, the closed-loop system is built with stability analysis. Simulation studies are conducted to verify the theoretical results and illustrate the control performance. It is significant to observe that the present research provides a new avenue of integrating data-based control and event-triggering mechanism into establishing advanced adaptive critic systems.

Event-based input-constrained nonlinear H∞ state feedback with adaptive critic and neural implementation

In this paper, the continuous-time input-constrained nonlinear H∞ state feedback control under event-based environment is investigated with adaptive critic designs and neural network implementation. The nonlinear H∞ control issue is regarded as a two-player zero-sum game that requires solving the Hamilton–Jacobi–Isaacs equation and the adaptive critic learning (ACL) method is adopted toward the event-based constrained optimal regulation. The novelty lies in that the event-based design framework is combined with the ACL technique, thereby carrying out the input-constrained nonlinear H∞ state feedback via adopting a non-quadratic utility function. The event-based optimal control law and the time-based worst-case disturbance law are derived approximately, by training an artificial neural network called a critic and eventually learning the optimal weight vector. Under the action of the event-based state feedback controller, the closed-loop system is constructed with uniformly ultimately bounded stability analysis. Simulation studies are included to verify the theoretical results as well as to illustrate the event-based H∞ control performance.

Decentralized guaranteed cost control of interconnected systems with uncertainties: A learning-based optimal control strategy

A novel learning-based optimal control approach is constructed to attain the decentralized guaranteed cost controller design for a class of continuous-time complex nonlinear systems with dynamical uncertainties and interconnections. This is performed by combining robust decentralized control formulation with adaptive critic learning technique. By expressing the interconnected subsystems as a whole system and introducing a new cost function for the overall plant, the decentralized guaranteed cost control problem is formulated as an optimal control problem for the nominal overall system. Then, a policy iteration based learning control algorithm is employed to solve the modified Hamilton–Jacobi–Bellman equation with respect to the nominal plant iteratively. A critic neural network is constructed to approximate the optimal state feedback control law and then the uniform ultimate boundedness stability issue is analyzed. Meanwhile, a simulation experiment is conducted to verify the good performance of the control approach.

Adaptive Critic Learning Techniques for Engine Torque and Air–Fuel Ratio Control

A new approach for engine calibration and control is proposed. In this paper, we present our research results on the implementation of adaptive critic designs for self-learning control of automotive engines. A class of adaptive critic designs that can be classified as (model-free) action-dependent heuristic dynamic programming is used in this research project. The goals of the present learning control design for automotive engines include improved performance, reduced emissions, and maintained optimum performance under various operating conditions. Using the data from a test vehicle with a V8 engine, we developed a neural network model of the engine and neural network controllers based on the idea of approximate dynamic programming to achieve optimal control. We have developed and simulated self-learning neural network controllers for both engine torque (TRQ) and exhaust air-fuel ratio (AFR) control. The goal of TRQ control and AFR control is to track the commanded values. For both control problems, excellent neural network controller transient performance has been achieved.

Adaptive Critic Autopilot Design of Bank-to-Turn Missiles Using Fuzzy Basis Function Networks

A new adaptive critic autopilot design for bank-to-turn missiles is presented. In this paper, the architecture of adaptive critic learning scheme contains a fuzzy-basis-function-network based associative search element (ASE), which is employed to approximate nonlinear and complex functions of bank-to-turn missiles, and an adaptive critic element (ACE) generating the reinforcement signal to tune the associative search element. In the design of the adaptive critic autopilot, the control law receives signals from a fixed gain controller, an ASE and an adaptive robust element, which can eliminate approximation errors and disturbances. Traditional adaptive critic reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error interactions with a dynamic environment, however, the proposed tuning algorithm can significantly shorten the learning time by online tuning all parameters of fuzzy basis functions and weights of ASE and ACE. Moreover, the weight updating law derived from the Lyapunov stability theory is capable of guaranteeing both tracking performance and stability. Computer simulation results confirm the effectiveness of the proposed adaptive critic autopilot.