Adaptive Critic Method Research Articles

Multilayer adaptive critic design with digital twin for data-driven optimal tracking control and industrial applications

In this paper, an optimal trajectory tracking control problem for general nonlinear systems is investigated. An adaptive critic control method with the digital twin (DT) theory is developed. Divergent from the existing tracking control methods, the advantages of adaptive dynamic programming (ADP) and the theory of DT are combined in this paper, and the novel multilayer artificial system structure is constructed. The action-critic structure is employed by each artificial system to obtain an approximate optimal control policy. The model network (MN) is built by using the actual input and output data sets of the controlled system, which means the dependence on the dynamics of the system is overcame. Then, the weights of the trained action network (AN) and MN are passed to the real system to realize the optimal tracking control. The feasibility of the algorithm is proved by theoretical analysis. Finally, the algorithm is applied to a simple nonlinear torsional pendulum system and an industrial wastewater treatment system (WWTS), and the effectiveness of the algorithm is verified. The algorithm effectively realizes the tracking control of nonlinear systems.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this paper, a novel adaptive critic control method is designed to solve an optimal H∞ tracking control problem for continuous nonlinear systems with nonzero equilibrium based on adaptive dynamic programming (ADP). To guarantee the finiteness of a cost function, traditional methods generally assume that the controlled system has a zero equilibrium point, which is not true in practical systems. In order to overcome such obstacle and realize H∞ optimal tracking control, this paper proposes a novel cost function design with respect to disturbance, tracking error and the derivative of tracking error. Based on the designed cost function, the H∞ control problem is formulated as two-player zero-sum differential games, and then a policy iteration (PI) algorithm is proposed to solve the corresponding Hamilton–Jacobi–Isaacs (HJI) equation. In order to obtain the online solution to the HJI equation, a single-critic neural network structure based on PI algorithm is established to learn the optimal control policy and the worst-case disturbance law. It is worth mentioning that the proposed adaptive critic control method can simplify the controller design process when the equilibrium of the systems is not zero. Finally, simulations are conducted to evaluate the tracking performance of the proposed control methods.

Optimal Control and Adaptive Learning for Stabilization of a Quadrotor-type Unmanned Aerial Vehicle via Approximate Dynamic Programming

The development of an optimal controller for stabilization of a quadrotor system using an adaptive critic structure based on policy iteration schemes is proposed in this paper. This approach is inserted in the context of Approximate Dynamic Programming and it is used to solve optimal decision problems on-line, without requiring complete knowledge of the system dynamics model to be controlled. The main feature of the adaptive critic design method that allows for on-line implementation is that it solves the Bellman optimality equation in a forward-in-time fashion, whereas traditional dynamic programming requires a backward-in-time procedure. This feedback control design technique is able to tune the controller parameters on-line in the presence of variations in plant dynamics and external disturbances using data measured along the system trajectories. Computational simulation results based on a quadrotor model demonstrate the effectiveness of the proposed control scheme.

Neural critic learning for tracking control design of constrained nonlinear multi-person zero-sum games

In this paper, an adaptive critic method based on neural networks is established to solve the tracking control problem for multi-person zero-sum games with constrained nonlinear dynamics. First, an augmented system is constructed with the tracking error system and the reference system, an appropriate function is introduced to handle the constrained problem, and a constrained tracking Hamilton–Jacobi-Isaacs (HJI) equation is derived for the augmented system. Then, a constrained tracking design with neural critic learning for multi-person zero-sum games is developed to approximately solve the tracking HJI equation with input constraints. A new updating rule is given and only one critic network is employed during neural critic learning. In addition, we prove that the tracking error in the augmented system is uniformly ultimately bounded by using Lyapunov’s direct method. Finally, an example is given to verify the effectiveness of the proposed method. In this example, we make the number of control inputs less than the number of disturbance inputs.

Adaptive neural tracking control of constrained waverider vehicles via single-network adaptive dynamic programming

The approximate optimization is considered though adaptive dynamic programming (ADP) method for waverider vehicles (WVs). This method can achieve the stability, robustness, and near optimization of the tracking problem for WVs. First, baseline controllers are designed for both velocity subsystem and altitude subsystem via neural network approximation. Meantime, novel auxiliary systems are implemented to compensate the desired control law with actuator saturation. Then, transient optimal controllers are developed using single-network adaptive critic method. Moreover, the stability of closed-loop systems, and the convergence of both the tracking error and the auxiliary system are theoretically proved. Finally, the superiority of the proposed method is proved via compared simulation results.

TCSA: A Traffic Congestion Situation Assessment Scheme Based on Multi-Index Fuzzy Comprehensive Evaluation in 5G-IoV

The traffic congestion situation is an important reference indicator for the orderly control and management of traffic systems. As intelligent transport systems (ITS) become increasingly popular, the challenge of realizing real-time traffic congestion situation assessments (TCSAs) in the post-traffic era is particularly important. In this study, we propose a TCSA scheme for multi-metric fuzzy integrated evaluation based on three predicted vehicle traffic parameters for the 5G Internet of Vehicles (5G-IoV) environment, which is dedicated to accelerating the development of ITS. Firstly, the scheme uses dynamic multi-model adaptive exponential smoothing (DMMAES), which can calculate the optimal smoothing coefficients and weight of each model based on historical prediction errors to predict the average speed and traffic volume and then calculate the predicted traffic speed, traffic flow density, and road saturation of the three traffic congestion indicators. Secondly, the predicted values of the three traffic congestion indicators are used as fuzzy comprehensive evaluation, taking into account the vagueness of the traffic congestion levels, the uncertainty of the indicators, and the conflict among the indicators, using a trapezoidal affiliation function to determine the degree of affiliation of each indicator through the adaptive CRITIC method to determine the weights. Finally, the predicted traffic congestion situations are classified into five levels. The effectiveness of the scheme was verified by the measured data of Yanta North Road in Xi’an. The results showed that the traffic congestion level predicted by TCSA was basically consistent with the actual situation and had a high prediction accuracy.

Decentralized Event-Triggered Online Adaptive Control of Unknown Large-Scale Systems Over Wireless Communication Networks.

In this article, a novel online decentralized event-triggered control scheme is proposed for a class of nonlinear interconnected large-scale systems subject to unknown internal system dynamics and interconnected terms. First, by designing a neural network-based identifier, the unknown internal dynamics of the interconnected systems is reconstructed. Then, the adaptive critic design method is used to learn the approximate optimal control policies in the context of event-triggered mechanism. Specifically, the event-based control processes of different subsystems are independent, asynchronous, and decentralized. That is, the decentralized event-triggering conditions and the controllers only rely on the local state information of the corresponding subsystems, which avoids the transmissions of the state information between the subsystems over the wireless communication networks. Then, with the help of Lyapunov's theorem, the states of the developed closed-loop control system and the critic weight estimation errors are proved to be uniformly ultimately bounded. Finally, the effectiveness and applicability of the event-based control method are verified by an illustrative numerical example and a practical example.

Model-free adaptive optimal design for trajectory tracking control of rocket-powered vehicle

An adaptive optimal trajectory tracking controller is presented for the Solid-Rocket-Powered Vehicle (SRPV) with uncertain nonlinear non-affine dynamics in the framework of adaptive dynamic programming. First, considering that the ascent model of the SRPV is non-affine, a model-free Single Network Adaptive Critic (SNAC) method is developed based on the dynamic neural network and the traditional SNAC method. This developed model-free SNAC method overcomes the limitation of the traditional SNAC method that can only be applied to affine systems. Then, a closed-form adaptive optimal controller is designed for the non-affine dynamics of SRPVs. This controller can adjust its parameters under different flight conditions and converge to the approximate optimal controller through online self-learning. Finally, the convergence to the approximate optimal controller is proved. The theoretical analysis of the uniformly ultimate boundedness of the tracking error is also presented. Simulation results demonstrate the effectiveness of the proposed controller.

Mixed Driven Iterative Adaptive Critic Control Design Towards Nonaffine Discrete-Time Plants

Abstract In this paper, an effective mixed driven framework is constructed involving both data and event considerations. The primary purpose lies in that the mixed driven iterative adaptive critic method is established to address approximate optimal control towards discrete-time nonlinear dynamics. The neural dynamic programming technique is inventively integrated with the mixed driven architecture, such that the knowledge of the controlled plant is needless and the number for updating control inputs is prominently reduced. A triggering threshold is also designed with theoretical guarantee, which renders that the control signals can be updated conditionally. Through carrying out simulation studies with comparisons, the superiority of the present near-optimal regulation approach is confirmed at last.

Neural critic learning toward robust dynamic stabilization

SummaryIn this article, we focus on developing a neural‐network‐based critic learning strategy toward robust dynamic stabilization for a class of uncertain nonlinear systems. A type of general uncertainties involved both in the internal dynamics and in the input matrix is considered. An auxiliary system with actual action and auxiliary signal is constructed after dynamics decomposition and combination for the original plant. The reasonability of the control problem transformation from robust stabilization to optimal feedback design is also provided theoretically. After that, the adaptive critic learning method based on a neural network is established to derive the approximate optimal solution of the transformed control problem. The critic weight can be initialized to a zero vector, which apparently facilitates the learning process. Numerical simulation is finally presented to illustrate the effectiveness of the critic learning approach for neural robust stabilization.

Event-Triggered Adaptive Dynamic Programming for Non-Zero-Sum Games of Unknown Nonlinear Systems via Generalized Fuzzy Hyperbolic Models

In this paper, by incorporating the event-triggered mechanism and the adaptive dynamic programming algorithm, a novel near-optimal control scheme for a class of unknown nonlinear continuous-time non-zero-sum (NZS) differential games is investigated. First, a generalized fuzzy hyperbolic model based identifier is established, using only the input-output data, to relax the requirement for the complete system dynamics. Then, under the event-based framework, the coupled Hamilton-Jacobi equations are derived for the multiplayer NZS games. Then, the adaptive critic design method is employed to approximate the optimal control policies; thus, an identifier-critic architecture is developed to obtain the event-triggered controller. By the virtue of the Lyapunov theory, a state-dependent triggering condition, which is different from the existing works, is developed to achieve the stability of the closed-loop control system both for the continuous and jump dynamics. Finally, two numerical examples are simulated to substantiate the feasibility of the analytical design.

Tracking control optimization scheme of continuous-time nonlinear system via online single network adaptive critic design method

In this paper, the optimal tracking control problem (OTCP) for a class of continuous-time nonlinear systems with infinite horizon cost is discussed. An online adaptive critic design method is proposed to learn the solution of OTCP by constructing an augmented system associated with a discounted performance function, which is composed of the tracking errors and reference trajectory dynamics. Only one neural network (NN) is used as critic module for approximating the performance function in the solution procedure, and thus the architecture is simpler than the typical action-critic structure, which needs more computational load from neural networks. Therefore, by the means of the approximate policy iteration, the tracking errors get converged to a region near zero and the parameters of critic module get converged to the optimal ones based on our proposed method. Both the convergence of the NN weights and the stability of the tracking error dynamics are guaranteed by the Lyapunov theory. Two simulation examples are proposed to verify the effectiveness of the proposed method.

On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $H_{\infty}$ Control.

In this paper, based on the adaptive critic learning technique, the control for a class of unknown nonlinear dynamic systems is investigated by adopting a mixed data and event driven design approach. The nonlinear control problem is formulated as a two-player zero-sum differential game and the adaptive critic method is employed to cope with the data-based optimization. The novelty lies in that the data driven learning identifier is combined with the event driven design formulation, in order to develop the adaptive critic controller, thereby accomplishing the nonlinear control. The event driven optimal control law and the time driven worst case disturbance law are approximated by constructing and tuning a critic neural network. Applying the event driven feedback control, the closed-loop system is built with stability analysis. Simulation studies are conducted to verify the theoretical results and illustrate the control performance. It is significant to observe that the present research provides a new avenue of integrating data-based control and event-triggering mechanism into establishing advanced adaptive critic systems.

Self-evolution Algorithm with Multi-objective for Scheduling a Job-shop-like Knowledgeable Manufacturing Cell

It is difficult to find the global optimal solution for the multi-objective job-shop scheduling problem and the solutions obtained usually belong to the Pareto optimal ones. A multi-objective scheduling optimization model is given for the Job-shop-like knowledgeable manufacturing cell and the relationship between the multi objectives is analyzed. The properties of key arcs of tasks are presented through the analysis of its disjunctive graph and the conclusion is obtained that it is helpless to improve the function value by changing the direction of the middle key arc of job. A simplified neighborhood is proposed based on the conclusion which can reduce greatly the number of feasible solutions to be searched. A self-evolution algorithm for multi-objective scheduling problem is proposed based on the properties of the simplified neighborhood by the use of adaptive heuristic critic method whose associate search module can find the best action for the concurrent solution to obtain a better solution by learning and training, and such ability of the module will be improved continuously with the training increasing. The numerical simulation results show that the algorithm proposed has the excellent ability to search the optimal solution for the proposed scheduling problem and possesses obvious evolution capacities through learning.

Variable Time Impulse System Optimization with Continuous Control and Impulse Control

AbstractOptimal control of a variable time impulse system is considered in this study. Besides impulse control, the system is assumed to operate with continuous control also. Necessary conditions for the optimization of such system are derived using the calculus of variations. A reinforcement based solution technique called a single network adaptive critic (SNAC) method is developed in this paper to obtain an optimal solution. Details of the SNAC‐based algorithm are presented. Simulation examples are also presented for illustrative purposes.

Probabilistic DHP adaptive critic for nonlinear stochastic control systems

Following the recently developed algorithms for fully probabilistic control design for general dynamic stochastic systems (Herzallah & Káarnáy, 2011; Kárný, 1996), this paper presents the solution to the probabilistic dual heuristic programming (DHP) adaptive critic method (Herzallah & Káarnáy, 2011) and randomized control algorithm for stochastic nonlinear dynamical systems. The purpose of the randomized control input design is to make the joint probability density function of the closed loop system as close as possible to a predetermined ideal joint probability density function. This paper completes the previous work (Herzallah & Káarnáy, 2011; Kárný, 1996) by formulating and solving the fully probabilistic control design problem on the more general case of nonlinear stochastic discrete time systems. A simulated example is used to demonstrate the use of the algorithm and encouraging results have been obtained.

Adaptive critic motion control design of autonomous wheeled mobile robot by dual heuristic programming

Autonomous wheeled mobile robot (WMR) needs implementing velocity and path tracking control subject to complex dynamical constraints. Conventionally, this control design is obtained by analysis and synthesis or by domain expert to build control rules. This paper presents an adaptive critic motion control design, which enables WMR to autonomously generate the control ability by learning through trials. The design consists of an adaptive critic velocity control loop and a self-learning posture control loop. The neural networks in the velocity neuro-controller (VNC) are corrected with the dual heuristic programming (DHP) adaptive critic method. Designer simply expresses the control objective by specifying the primary utility function then VNC will attempt to fulfill it through incremental optimization. The posture neuro-controller (PNC) learns by approximating the specialized inverse velocity model of WMR so as to map planned positions to suitable velocity commands. Supervised drive supplies variant velocity commands for PNC and VNC to set up their neural weights. During autonomous drive, while PNC halts learning VNC keeps on correcting its neural weights to optimize the control performance. The proposed design is evaluated on an experimental WMR. The results show that the DHP adaptive critic design is a useful base of autonomous control.

Adaptive critic methods for stochastic systems with input-dependent noise

In this paper, a novel dual heuristic programming (DHP) adaptive-critic-based cautious controller is proposed. The proposed controller avoids the pre-identification training phase of the forward model by taking into consideration model uncertainty when calculating the control law. It is suitable for linear and nonlinear, deterministic and stochastic control systems that are characterized by functional uncertainty. Convergence of the proposed DHP adaptive critic method to the correct value of the cost function is proven by evaluating analytically the correct value of the cost function, which satisfies the Bellman equation, and compares it to that calculated by the proposed method in a simple linear quadratic example. Moreover, the performance of the proposed cautious controller is demonstrated on linear one-dimensional and multidimensional examples.

Robust Neurocontrollers for Systems with Model Uncertainties: A Helicopter Application

A two-neural network approach to solving nonlinear optimal control problems is described. This approach, called the adaptive critic method, consists of one neural network, called the supervisor or the critic, and a second network, called an action network or a controller. The inputs to both these networks are the current states of the system to be controlled. Targets for each network updates are obtained with outputs of the other network, state propagation equations, and the conditions for optimal control. When their outputs are mutually consistent, the controller network output is optimal. The optimality is, however, limited by the underlying system model. Hence, a Lyapunov theory-based analysis for robust stability of the system under model uncertainties is developed and an extra control is developed. This extra control added with the basic control effort from the adaptive critic method guarantees good system performance and stablity under model uncertainties. This approach is demonstrated through a helicopter problem.