Single Network Adaptive Critic Research Articles

Game-Based Backstepping Design for Strict-Feedback Nonlinear Multi-Agent Systems Based on Reinforcement Learning.

In this article, the game-based backstepping control method is proposed for the high-order nonlinear multi-agent system with unknown dynamic and input saturation. Reinforcement learning (RL) is employed to get the saddle point solution of the tracking game between each agent and the reference signal for achieving robust control. Specifically, the approximate optimal solution of the established Hamilton-Jacobi-Isaacs (HJI) equation is obtained by policy iteration for each subsystem, and the single network adaptive critic (SNAC) architecture is used to reduce the computational burden. In addition, based on the separation operation of the error term from the derivative of the value function, we achieve the different proportions of the two agents in the game to realize the regulation of the final equilibrium point. Different from the general use of the neural network for system identification, the unknown nonlinear dynamic term is approximated based on the state difference obtained by the command filter. Furthermore, a sufficient condition is established to guarantee that the whole system and each subsystem included are uniformly ultimately bounded. Finally, simulation results are given to show the effectiveness of the proposed method.

Self-learning control of model uncertain active suspension systems with observer–critic structure

This paper presents a self-learning control algorithm for model uncertain suspension systems using single network adaptive critic (SNAC) approach. First, a differential neural network (DNN) observer in conjunction with the weight updating law is established to observe the uncertain dynamic. Then, the nominal optimal value function is approximated by a critic NN whose weight is updated by a novel design learning law driven by the filtered parameter error. The online self-learning control policy is thus derived by approximately solving the Hamilton–Jacobi–Bellman (HJB) equation based on SNAC technique. The Lyapunov approach is synthesized to ensure the convergent characteristics of the entire closed-loop system composed of the DNN observer and the self-learning control policy. Computer simulation of a quarter car suspension system is established to verify the effectiveness of the proposed approach. Simulation results illustrated that the designed method can ensure the good performance in terms with the road hold and ride quality. In addition, independent of model and online self-learning characteristics make it possible to design a high-performance vehicle active suspension controller.

Model-free adaptive optimal design for trajectory tracking control of rocket-powered vehicle

An adaptive optimal trajectory tracking controller is presented for the Solid-Rocket-Powered Vehicle (SRPV) with uncertain nonlinear non-affine dynamics in the framework of adaptive dynamic programming. First, considering that the ascent model of the SRPV is non-affine, a model-free Single Network Adaptive Critic (SNAC) method is developed based on the dynamic neural network and the traditional SNAC method. This developed model-free SNAC method overcomes the limitation of the traditional SNAC method that can only be applied to affine systems. Then, a closed-form adaptive optimal controller is designed for the non-affine dynamics of SRPVs. This controller can adjust its parameters under different flight conditions and converge to the approximate optimal controller through online self-learning. Finally, the convergence to the approximate optimal controller is proved. The theoretical analysis of the uniformly ultimate boundedness of the tracking error is also presented. Simulation results demonstrate the effectiveness of the proposed controller.

Optimal and robust control of a class of nonlinear systems using dynamically re-optimised single network adaptive critic design

ABSTRACTFollowing the philosophy of adaptive optimal control, a neural network-based state feedback optimal control synthesis approach is presented in this paper. First, accounting for a nominal system model, a single network adaptive critic (SNAC) based multi-layered neural network (called as NN1) is synthesised offline. However, another linear-in-weight neural network (called as NN2) is trained online and augmented to NN1 in such a manner that their combined output represent the desired optimal costate for the actual plant. To do this, the nominal model needs to be updated online to adapt to the actual plant, which is done by synthesising yet another linear-in-weight neural network (called as NN3) online. Training of NN3 is done by utilising the error information between the nominal and actual states and carrying out the necessary Lyapunov stability analysis using a Sobolev norm based Lyapunov function. This helps in training NN2 successfully to capture the required optimal relationship. The overall architecture is named as ‘Dynamically Re-optimised single network adaptive critic (DR-SNAC)’. Numerical results for two motivating illustrative problems are presented, including comparison studies with closed form solution for one problem, which clearly demonstrate the effectiveness and benefit of the proposed approach.

Robust Regulation of a Class of Nonlinear Systems Using DR-SNAC

Following the philosophy of adaptive optimal control, a new technique is presented for robust optimal regulation of a class of nonlinear systems. First, accounting for a nominal system model, a single network adaptive critic (SNAC) based multi-layered neural network (called as NN 1 ) is synthesized offline for optimal regulation of the nominal system. However, another linear-in-weight neural network (called as NN 2 ) is trained online and augmented to NN1 in such a manner that their combined output represent the desired optimal costate for the actual plant. To do this, the nominal model needs to be updated online to adapt to the actual plant, which done by synthesizing yet another linear-in-weight neural network (called as NN3) online. Training of NN3 is done so that both the unmodelled part of the dynamics as well as its partial derivatives with respect to the states are captured. The overall architecture is named as 'Dynamically re-optimized single network adaptive critic (DR-SNAC)'. Numerical results for Van der Pol problem are presented, which demonstrate its effectiveness, and clearly bring out the potential of the proposed approach.

Variable Time Impulse System Optimization with Continuous Control and Impulse Control

AbstractOptimal control of a variable time impulse system is considered in this study. Besides impulse control, the system is assumed to operate with continuous control also. Necessary conditions for the optimization of such system are derived using the calculus of variations. A reinforcement based solution technique called a single network adaptive critic (SNAC) method is developed in this paper to obtain an optimal solution. Details of the SNAC‐based algorithm are presented. Simulation examples are also presented for illustrative purposes.

Single network adaptive critic aided dynamic inversion for optimal regulation and command tracking with online adaptation for enhanced robustness

SUMMARYTo combine the advantages of both stability and optimality‐based designs, a single network adaptive critic (SNAC) aided nonlinear dynamic inversion approach is presented in this paper. Here, the gains of a dynamic inversion controller are selected in such a way that the resulting controller behaves very close to a pre‐synthesized SNAC controller in the output regulation sense. Because SNAC is based on optimal control theory, it makes the dynamic inversion controller operate nearly optimal. More important, it retains the two major benefits of dynamic inversion, namely (i) a closed‐form expression of the controller and (ii) easy scalability to command tracking applications without knowing the reference commands a priori. An extended architecture is also presented in this paper that adapts online to system modeling and inversion errors, as well as reduced control effectiveness, thereby leading to enhanced robustness. The strengths of this hybrid method of applying SNAC to optimize an nonlinear dynamic inversion controller is demonstrated by considering a benchmark problem in robotics, that is, a two‐link robotic manipulator system. Copyright © 2013 John Wiley & Sons, Ltd.

Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics.

To synthesize fixed-final-time control-constrained optimal controllers for discrete-time nonlinear control-affine systems, a single neural network (NN)-based controller called the Finite-horizon Single Network Adaptive Critic is developed in this paper. Inputs to the NN are the current system states and the time-to-go, and the network outputs are the costates that are used to compute optimal feedback control. Control constraints are handled through a nonquadratic cost function. Convergence proofs of: 1) the reinforcement learning-based training method to the optimal solution; 2) the training error; and 3) the network weights are provided. The resulting controller is shown to solve the associated time-varying Hamilton-Jacobi-Bellman equation and provide the fixed-final-time optimal solution. Performance of the new synthesis technique is demonstrated through different examples including an attitude control problem wherein a rigid spacecraft performs a finite-time attitude maneuver subject to control bounds. The new formulation has great potential for implementation since it consists of only one NN with single set of weights and it provides comprehensive feedback solutions online, though it is trained offline.

Intelligent Constrained Optimal Control of Aerospace Vehicles with Model Uncertainties

Approximate dynamic programming formulation implemented with an “adaptive critic-based” neural network structure has been shown to be a powerful technique to solve theHamilton–Jacobi–Bellman equations. As interest in this technique grows, it is important to consider the enabling factors for their possible implementations. A typical adaptive critic structure consists of two interacting neural networks; in this paper, a new architecture, called the “cost function-based single network adaptive critic” is presented that eliminates one of the networks. This approach is applicable to a wide class of nonlinear systems in engineering where the optimal control equation can be explicitly expressed in terms of the state and cost-related variables. After the “training,” the output of the neural network represents the optimal cost. Optimal control is obtained by finding the derivatives of the output of the network with respect to its input and using it in the expression for optimal control. In practical applications, there usually exist uncertainties in modeling and in system parameters. Furthermore, the controllers have operational limits. The first concern is taken care of through an “approximated system” that contains an estimate of the uncertainties from an online neural network andhelps calculate the optimal control for the changed plant. Regarding the second concern, a nonquadratic term that incorporates the control constraints is used in the performance index. Necessary conditions for optimal control are derived and an algorithm to solve the constrained-control problem with cost function-based single network adaptive critic is developed. Two aerospace systems are used to illustrate the working of the proposed technique.

Optimal blood glucose regulation of diabetic patients using single network adaptive critics

AbstractDiabetes is a long‐term disease during which the body's production and use of insulin are impaired, causing glucose concentration level to increase in the bloodstream. Regulating blood glucose levels as close to normal as possible leads to a substantial decrease in long‐term complications of diabetes. In this paper, an intelligent online feedback‐treatment strategy is presented for the control of blood glucose levels in diabetic patients using single network adaptive critic (SNAC) neural networks (which is based on nonlinear optimal control theory). A recently developed mathematical model of the nonlinear dynamics of glucose and insulin interaction in the blood system has been revised and considered for synthesizing the neural network for feedback control. The idea is to replicate the function of pancreatic insulin, i.e. to have a fairly continuous measurement of blood glucose and a situation‐dependent insulin injection to the body using an external device. Detailed studies are carried out to analyze the effectiveness of this adaptive critic‐based feedback medication strategy. A comparison study with linear quadratic regulator (LQR) theory shows that the proposed nonlinear approach offers some important advantages such as quicker response, avoidance of hypoglycemia problems, etc. Robustness of the proposed approach is also demonstrated from a large number of simulations considering random initial conditions and parametric uncertainties. Copyright © 2009 John Wiley & Sons, Ltd.

Single network adaptive critic design for power system stabilisers

The recently developed single network adaptive critic (SNAC) design has been used in this study to design a power system stabiliser (PSS) for enhancing the small-signal stability of power systems over a wide range of operating conditions. PSS design is formulated as a discrete non-linear quadratic regulator problem. SNAC is then used to solve the resulting discrete-time optimal control problem. SNAC uses only a single critic neural network instead of the action-critic dual network architecture of typical adaptive critic designs. SNAC eliminates the iterative training loops between the action and critic networks and greatly simplifies the training procedure. The performance of the proposed PSS has been tested on a single machine infinite bus test system for various system and loading conditions. The proposed stabiliser, which is relatively easier to synthesise, consistently outperformed stabilisers based on conventional lead-lag and linear quadratic regulator designs.

Continuous-Time Single Network Adaptive Critic for Regulator Design of Nonlinear Control Affine Systems

An optimal control law for a general nonlinear system can be obtained by solving Hamilton-Jacobi-Bellman equation. However, it is difficult to obtain an analytical solution of this equation even for a moderately complex system. In this paper, we propose a continuous-time single network adaptive critic scheme for nonlinear control affine systems where the optimal cost-to-go function is approximated using a parametric positive semi-definite function. Unlike earlier approaches, a continuous-time weight update law is derived from the HJB equation. The stability of the system is analysed during the evolution of weights using Lyapunov theory. The effectiveness of the scheme is demonstrated through simulation examples.

Robust/Optimal Temperature Profile Control of a High-Speed Aerospace Vehicle Using Neural Networks

An approximate dynamic programming (ADP)-based suboptimal neurocontroller to obtain desired temperature for a high-speed aerospace vehicle is synthesized in this paper. A 1-D distributed parameter model of a fin is developed from basic thermal physics principles. "Snapshot" solutions of the dynamics are generated with a simple dynamic inversion-based feedback controller. Empirical basis functions are designed using the "proper orthogonal decomposition" (POD) technique and the snapshot solutions. A low-order nonlinear lumped parameter system to characterize the infinite dimensional system is obtained by carrying out a Galerkin projection. An ADP-based neurocontroller with a dual heuristic programming (DHP) formulation is obtained with a single-network-adaptive-critic (SNAC) controller for this approximate nonlinear model. Actual control in the original domain is calculated with the same POD basis functions through a reverse mapping. Further contribution of this paper includes development of an online robust neurocontroller to account for unmodeled dynamics and parametric uncertainties inherent in such a complex dynamic system. A neural network (NN) weight update rule that guarantees boundedness of the weights and relaxes the need for persistence of excitation (PE) condition is presented. Simulation studies show that in a fairly extensive but compact domain, any desired temperature profile can be achieved starting from any initial temperature profile. Therefore, the ADP and NN-based controllers appear to have the potential to become controller synthesis tools for nonlinear distributed parameter systems.

A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems

Even though dynamic programming offers an optimal control solution in a state feedback form, the method is overwhelmed by computational and storage requirements. Approximate dynamic programming implemented with an Adaptive Critic (AC) neural network structure has evolved as a powerful alternative technique that obviates the need for excessive computations and storage requirements in solving optimal control problems. In this paper, an improvement to the AC architecture, called the "Single Network Adaptive Critic (SNAC)" is presented. This approach is applicable to a wide class of nonlinear systems where the optimal control (stationary) equation can be explicitly expressed in terms of the state and costate variables. The selection of this terminology is guided by the fact that it eliminates the use of one neural network (namely the action network) that is part of a typical dual network AC setup. As a consequence, the SNAC architecture offers three potential advantages: a simpler architecture, lesser computational load and elimination of the approximation error associated with the eliminated network. In order to demonstrate these benefits and the control synthesis technique using SNAC, two problems have been solved with the AC and SNAC approaches and their computational performances are compared. One of these problems is a real-life Micro-Electro-Mechanical-system (MEMS) problem, which demonstrates that the SNAC technique is applicable to complex engineering systems.

Nonlinear control concepts for a UA

A reconfigurable flight control method is developed to be implemented on an Unmanned Aircraft (VA), a thirty percent scale model of the Cessna 150. This paper presents the details of the UA platform, system identification, reconfigurable controller design, development, and implementation on the UA to analyze the performance metrics. A Crossbow Inertial Measurement Unit provides the roll, pitch, and yaw accelerations and rates along with the roll and pitch. The 100-400 mini-air data boom from SpaceAge Control provides the airspeed, altitude, angle of attack, and the side slip angles. System identification is accomplished by commanding preprogrammed inputs to the control surfaces and correlating the corresponding variations at the outputs. A Single Network Adaptive Critic, which is a neural network-based optimal controller, is developed as part of a nonlinear flight control system. An online learning neural network is augmented to form an outer loop to reconfigure and supplement the optimal controller to guarantee a "practical stability" for the airplane. This paper also presents some simulations from the hardware-in-the-loop testing and concludes with an analysis of the flight performance metrics for the controller under investigation.