Adaptive Critic Design Research Articles

Asymptotical Cooperative Tracking Control for Unknown High-Order Multi-Agent Systems via Distributed Adaptive Critic Design

In this paper, a distributed adaptive critic design-based asymptotical cooperative tracking control scheme is established for a class of unknown high-order multi-agent systems with disturbances on undirected graphs. The cooperative tracking problem of high-order multi-agent systems is first transformed to the stabilizing problem of filtered tracking errors dynamic systems, which only depend on the information from neighboring agents. Then, a novel distributed adaptive critic design-based controller is developed for the newly obtained filtered tracking errors dynamic systems. The special local filtered tracking errors and local critic signals are defined for facilitating the implementation of distributed adaptive critic design. Novel distributed weights updating laws for critic neural networks, actor neural networks, and adaptive parameters are proposed. Furthermore, the RISE technique is introduced to suppress the external disturbances and neural network approximation errors. The asymptotical stability of a closed-loop system is demonstrated by using Lyapunov stability theory. Finally, a simulation example justifying the effectiveness of the proposed control scheme is given.

Optimal and robust control of a class of nonlinear systems using dynamically re-optimised single network adaptive critic design

ABSTRACTFollowing the philosophy of adaptive optimal control, a neural network-based state feedback optimal control synthesis approach is presented in this paper. First, accounting for a nominal system model, a single network adaptive critic (SNAC) based multi-layered neural network (called as NN1) is synthesised offline. However, another linear-in-weight neural network (called as NN2) is trained online and augmented to NN1 in such a manner that their combined output represent the desired optimal costate for the actual plant. To do this, the nominal model needs to be updated online to adapt to the actual plant, which is done by synthesising yet another linear-in-weight neural network (called as NN3) online. Training of NN3 is done by utilising the error information between the nominal and actual states and carrying out the necessary Lyapunov stability analysis using a Sobolev norm based Lyapunov function. This helps in training NN2 successfully to capture the required optimal relationship. The overall architecture is named as ‘Dynamically Re-optimised single network adaptive critic (DR-SNAC)’. Numerical results for two motivating illustrative problems are presented, including comparison studies with closed form solution for one problem, which clearly demonstrate the effectiveness and benefit of the proposed approach.

Robust adaptive critic control design with network-based event-triggered formulation

In this paper, by incorporating the network-based event-triggered formulation, the robust adaptive critic control design for a class of nonlinear continuous-time systems is investigated to fulfill disturbance rejection. First, the designed problem with output information is formulated as a two-player zero-sum differential game and the adaptive critic mechanism is employed toward the event-based minimax optimization involving a suitable triggering condition. Then, the event-based optimal control law and the time-based worst-case disturbance law are learned by training the critic neural network. Besides, the closed-loop system is constructed with stability proof of the critic error dynamics and the sampled-data plant. The theoretical analysis has demonstrated that the infamous Zeno behavior of the proposed event-based adaptive critic design has been avoided. Finally, the developed method is applied toward the robot arm plant, as a mechanical component of the complex robot system, so as to substantiate the performance of disturbance rejection.

Tracking control optimization scheme of continuous-time nonlinear system via online single network adaptive critic design method

In this paper, the optimal tracking control problem (OTCP) for a class of continuous-time nonlinear systems with infinite horizon cost is discussed. An online adaptive critic design method is proposed to learn the solution of OTCP by constructing an augmented system associated with a discounted performance function, which is composed of the tracking errors and reference trajectory dynamics. Only one neural network (NN) is used as critic module for approximating the performance function in the solution procedure, and thus the architecture is simpler than the typical action-critic structure, which needs more computational load from neural networks. Therefore, by the means of the approximate policy iteration, the tracking errors get converged to a region near zero and the parameters of critic module get converged to the optimal ones based on our proposed method. Both the convergence of the NN weights and the stability of the tracking error dynamics are guaranteed by the Lyapunov theory. Two simulation examples are proposed to verify the effectiveness of the proposed method.

Neural Network-Based Solutions for Stochastic Optimal Control Using Path Integrals.

In this paper, an offline approximate dynamic programming approach using neural networks is proposed for solving a class of finite horizon stochastic optimal control problems. There are two approaches available in the literature, one based on stochastic maximum principle (SMP) formalism and the other based on solving the stochastic Hamilton-Jacobi-Bellman (HJB) equation. However, in the presence of noise, the SMP formalism becomes complex and results in having to solve a couple of backward stochastic differential equations. Hence, current solution methodologies typically ignore the noise effect. On the other hand, the inclusion of noise in the HJB framework is very straightforward. Furthermore, the stochastic HJB equation of a control-affine nonlinear stochastic system with a quadratic control cost function and an arbitrary state cost function can be formulated as a path integral (PI) problem. However, due to curse of dimensionality, it might not be possible to utilize the PI formulation for obtaining comprehensive solutions over the entire operating domain. A neural network structure called the adaptive critic design paradigm is used to effectively handle this difficulty. In this paper, a novel adaptive critic approach using the PI formulation is proposed for solving stochastic optimal control problems. The potential of the algorithm is demonstrated through simulation results from a couple of benchmark problems.

Neural-Network-Based Robust Optimal Tracking Control for MIMO Discrete-Time Systems With Unknown Uncertainty Using Adaptive Critic Design.

This paper is concerned with the robust optimal tracking control strategy for a class of nonlinear multi-input multi-output discrete-time systems with unknown uncertainty via adaptive critic design (ACD) scheme. The main purpose is to establish an adaptive actor-critic control method, so that the cost function in the procedure of dealing with uncertainty is minimum and the closed-loop system is stable. Based on the neural network approximator, an action network is applied to generate the optimal control signal and a critic network is used to approximate the cost function, respectively. In contrast to the previous methods, the main features of this paper are: 1) the ACD scheme is integrated into the controllers to cope with the uncertainty and 2) a novel cost function, which is not in quadric form, is proposed so that the total cost in the design procedure is reduced. It is proved that the optimal control signals and the tracking errors are uniformly ultimately bounded even when the uncertainty exists. Finally, a numerical simulation is developed to show the effectiveness of the present approach.

Adaptive Critic Design with Local Gaussian Process Models

In this paper, local Gaussian process (GP) approximation is introduced to build the critic network of adaptive dynamic programming (ADP). The sample data are partitioned into local regions, and for each region, an individual GP model is utilized. The nearest local model is used to predict a given state-action point. With the two-phase value iteration method for a Gaussian-kernel (GK)-based critic network which realizes the update of the hyper-parameters and value functions simultaneously, fast value function approximation can be achieved. Combining this critic network with an actor network, we present a local GK-based ADP approach. Simulations were carried out to demonstrate the feasibility of the proposed approach.

Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics

In this paper, the infinite-horizon robust optimal control problem for a class of continuous-time uncertain nonlinear systems is investigated by using data-based adaptive critic designs. The neural network identification scheme is combined with the traditional adaptive critic technique, in order to design the nonlinear robust optimal control under uncertain environment. First, the robust optimal controller of the original uncertain system with a specified cost function is established by adding a feedback gain to the optimal controller of the nominal system. Then, a neural network identifier is employed to reconstruct the unknown dynamics of the nominal system with stability analysis. Hence, the data-based adaptive critic designs can be developed to solve the Hamilton–Jacobi–Bellman equation corresponding to the transformed optimal control problem. The uniform ultimate boundedness of the closed-loop system is also proved by using the Lyapunov approach. Finally, two simulation examples are presented to illustrate the effectiveness of the developed control strategy.

Adaptive critic design based cooperative control for pulsed power loads accommodation in shipboard power system

Since a pulsed power load (PPL) consumes a huge amount of energy within very short period of time, its deployment might cause large disturbances even instability to a power system that has limited generation, small inertia, and small ramp rate. To mitigate the strike during PPL deployment, energy storage system (ESS) is usually installed in shipboard power system (SPS) to serve as the sole power supply for PPL. To realise fast charging of the ESS and minimise disturbance during charging, generation control and charging control of SPS should be well coordinated. For this important but not well-studied problem, this study presents an adaptive critic design based control algorithm for a non-linear model integrating the basic dynamics of synchronous generator and supercapacitor, which is a popular type of ESS for PPL application. Through interactive learning of two neural networks for cost-to-go function approximation and optimal control approximation, respectively, near optimal control can be realised even under disturbance and model impreciseness. The algorithm is tested with both detailed single- and multiple-generator SPS models and tested through both real-time digital and power hardware-in-the-loop simulations. Simulation results demonstrate the effectiveness of the developed model and control algorithm.

Event-based input-constrained nonlinear H∞ state feedback with adaptive critic and neural implementation

In this paper, the continuous-time input-constrained nonlinear H∞ state feedback control under event-based environment is investigated with adaptive critic designs and neural network implementation. The nonlinear H∞ control issue is regarded as a two-player zero-sum game that requires solving the Hamilton–Jacobi–Isaacs equation and the adaptive critic learning (ACL) method is adopted toward the event-based constrained optimal regulation. The novelty lies in that the event-based design framework is combined with the ACL technique, thereby carrying out the input-constrained nonlinear H∞ state feedback via adopting a non-quadratic utility function. The event-based optimal control law and the time-based worst-case disturbance law are derived approximately, by training an artificial neural network called a critic and eventually learning the optimal weight vector. Under the action of the event-based state feedback controller, the closed-loop system is constructed with uniformly ultimately bounded stability analysis. Simulation studies are included to verify the theoretical results as well as to illustrate the event-based H∞ control performance.

Learning control of fermentation process with an improved DHP algorithm

Control of the fed-batch ethanol fermentation processes to produce maximum product ethanol is one of the key issues in the bioreactor system. However, ethanol fermentation processes exhibit complex behavior and nonlinear dynamics with respect to the cell mass, substrate, feed-rate, etc. An improved dual heuristic programming algorithm based on the least squares temporal difference with gradient correction (LSTDC) algorithm (LSTDC-DHP) is proposed to solve the learning control problem of a fed-batch ethanol fermentation process. As a new algorithm of adaptive critic designs, LSTDC-DHP is used to realize online learning control of chemical dynamical plants, where LSTDC is commonly employed to approximate the value functions. Application of the LSTDC-DHP algorithm to ethanol fermentation process can realize efficient online learning control in continuous spaces. Simulation results demonstrate the effectiveness of LSTDC-DHP, and show that LSTDC-DHP can obtain the near-optimal feed rate trajectory faster than other-based algorithms.

A Lyapunov function based optimal hybrid power system controller for improved transient stability

In this paper, an intelligent power system stabilizer based on a stable and optimal hybrid learning-based adaptive control architecture is proposed which is evolved from approximate dynamic programming technique. The hybrid learning controller is based on two algorithms: (a) A reinforcement learning neural network controller implemented by adaptive critic design (ACD) and (b) A model reference adaptive controller (MRAC). The proposed method uses a novel value priority architecture for integrating these two algorithms. The designed ACD approximates nonlinear functional dynamics of the power system and interacts with the MRAC that adapts based on parametric changes. The value priority scheme developed based on a softmax network generates a hybrid control signal derived from a Lyapunov stability function that evaluates identification, performance, and stability of each controller. The overall hybrid control algorithm ensures convergence to an optimal control solution without using an explicit model of the system and at the same time ensures overall system stability. Theoretical results are validated by simulation studies for electric-generator stabilization on a 2-area 5-generator power system and as a wide-area controller on IEEE 68-bus system.

Adaptive Critic Design Based Control of Tunnel Ventilation System with Variable Jet Speed

In this paper, an adaptive critic design (ACD) based control algorithm is proposed for road tunnel ventilation systems with jet fans. Firstly, a comprehensive dynamic model is established for control purpose, which considers both the static and dynamic nonlinearities between the pollutant concentration and the jet speed or the air flow inside the tunnel. Vehicle speed and density are also integrated into the model as system external disturbances. To cope with the time-varying model parameters and unknown system uncertainties, one artificial neural network (NN) unit is employed in controlling the jet fans to regulate the pollutant concentration to a desired level, even in the presence of varying working conditions. An additional NN is utilized to approximate the cost-to-go function required for the performance optimization. The proposed scheme is also verified and compared with traditional control design in simulation environment. The simulation results substantiate the conclusion that the adaptive controller can achieve a better regulation performance along with less energy consumption.

Design and Real-Time Implementation of Optimal Power System Wide-Area System-Centric Controller Based on Temporal Difference Learning

In this paper, a novel framework for designing and implementing a coordinated wide-area controller architecture for improved power system dynamic stability is presented and tested. The algorithm is an optimal wide-area system-centric controller and observer based on a hybrid reinforcement learning and temporal difference framework. It allows the system to deal with major concerns of wide-area monitoring problem: delays in signal transmission, the uncertainty of the communication network, and data traffic. The main advantage of this design is its ability to learn from the past using eligibility traces and predict the optimal trajectory of cost function through temporal difference method. The control algorithm is evolved from adaptive critic design (ACD) and performed online at a finite horizon through backward and forward view. The ACD controller's training and testing are implemented on the Innovative Integration Picolo card integrated to TMS320C28335 processor. Results on a real experimental test bed using a real power system feeder shows that this architecture provides better stability compared with conventional schemes.

Adaptive Critic Design-Based Dynamic Stochastic Optimal Control Design for a Microgrid With Multiple Renewable Resources

This paper proposes a three-layer optimization and an intelligent control algorithm for a microgrid with multiple renewable resources. A dual heuristic dynamic programming-based system control layer is used to ensure the dynamic performance and voltage dynamics of the microgrid as the system operation conditions change. A local layer maximizes the capability of the photovoltaic (PV) wind power generators and battery systems, and a model predictive control-based device layer increases the tracking accuracy of the converter control. The proposed control scheme, system wide adaptive predictive supervisory control (SWAPSC) smooths the output of PV and wind generators under intermittencies, maintains bus voltage by providing dynamic reactive power support to the grid, and reduces the total system losses while minimizing degradation of battery life span. Performance comparisons are made with and without SWAPSC for an IEEE 13 node test system with a PV farm, a wind farm, and two battery-based energy storage systems.

Adaptive Critic Design Using Policy Iteration Technique for LTI Systems: A Comprehensive Performance Analysis

This paper presents adaptive critic design (ACD) using policy iteration technique for adaptive optimal control of continuous-time linear time-invariant (LTI) dynamical systems. A comprehensive performance analysis of control scheme is presented in this paper. ACD using online policy iteration technique provides an adaptive optimal control solution for an infinite horizon problem subject to the real-time dynamics of a continuous-time system. In general, ACD involves a critic and an actor, where the critic evaluates the performance of present control policy and generates a critic signal to update the control action for performance improvement, and the actor provides the control input to the system being controlled. Policy iteration technique which is based on actor–critic structure consists of two-step iteration: policy evaluation and policy improvement. In this paper, the application of control scheme is implemented for a practical example of LTI systems–load frequency control of power system. The modeling, simulation results, and analysis are presented for both of system models without and with integral control. The comparative performance investigation of adaptive critic control scheme and linear quadratic regulator is also presented.

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

Adaptive Dynamic Programming (ADP) with critic-actor architecture is an effective way to perform online learning control. To avoid the subjectivity in the design of a neural network that serves as a critic network, kernel-based adaptive critic design (ACD) was developed recently. There are two essential issues for a static kernel-based model: how to determine proper hyperparameters in advance and how to select right samples to describe the value function. They all rely on the assessment of sample values. Based on the theoretical analysis, this paper presents a two-phase simultaneous learning method for a Gaussian-kernel-based critic network. It is able to estimate the values of samples without infinitively revisiting them. And the hyperparameters of the kernel model are optimized simultaneously. Based on the estimated sample values, the sample set can be refined by adding alternatives or deleting redundances. Combining this critic design with actor network, we present a Gaussian-kernel-based Adaptive Dynamic Programming (GK-ADP) approach. Simulations are used to verify its feasibility, particularly the necessity of two-phase learning, the convergence characteristics, and the improvement of the system performance by using a varying sample set.

Web Service QoS Prediction Based on Adaptive Dynamic Programming Using Fuzzy Neural Networks for Cloud Services

Recently, more and more traditional services are being migrated into a cloud computing environment that makes the quality of service (QoS) becomes an important factor for service selection and optimal service composition while forming cross-cloud service applications. Considering the nonlinear and dynamic property of QoS data, it is so difficult to achieve dynamic prediction while designing a QoS prediction method with unsatisfactory prediction accuracy. It is thus desirable to explore how to design an effective approach by incorporating some intelligent techniques into the QoS prediction method to improve prediction performance. In this paper, motivated by the adaptive critic design and Q-learning technique, we propose a novel QoS prediction approach to serve this purpose through the combination of fuzzy neural networks and adaptive dynamic programming (ADP), i.e., an online learning scheme. This approach extracts fuzzy rules from QoS data and employs the ADP method to parameter learning of the fuzzy rules. Moreover, we provide a convergence boundedness result for our proposed approach to guarantee the stability. Experimental results on a large-scale QoS service data set verify the prediction accuracy of our proposed approach.

Adaptive critic design and Hopfield neural network based simulation of time delayed photosynthetic production and prey–predator model

This paper presents a neural networks simulation of two complex adaptive ecological systems with a discrete time delay. The first system is related to the photosynthetic production of phytoplankton, the second with a time delayed prey–predator system incorporating a prey refuge. The iterative adaptive critic design is developed to simulate the maximal photosynthetic production of the mechanistic model of phytoplankton photosynthesis with discrete time delays. The prey–predator system is simulated using back-propagation learning of infinite-dimensional dynamical systems. The proposed simulation method is based on the time-dependent recurrent learning of continuous-time Hopfield neural network with a discrete time delay with the prey–predator system as a teacher signal. Furthermore, numerical calculations are included to demonstrate the proposed simulation algorithms.

Direct back propagation neural dynamic programming-based particle swarm optimisation

In this paper, we introduce direct back propagation (BP) neural dynamic programming (NDP) into particle swarm optimisation (PSO). Thus, a direct BP NDP inspired PSO algorithm, which we call NDPSO, is proposed. In NDPSO, since direct BP NDP belongs to the class of heuristic dynamic programming algorithms based on model-based adaptive critic designs and often serves as an online learning control paradigm, critic BP neural network is trained to optimise a total reward-to-go objective, namely to balance Bellman's equation, while action BP neural network is used to train the inertia weight, cognitive, and social coefficients so that the critic BP network output can approach an ultimate reward-to-go objective of success. With the collective aid of action-critic BP neural networks, inertia weight, cognitive, and social coefficients become more adaptive. Besides, the NDPSO's mutation mechanism also has greatly improved the dynamic performance of the standard PSO. Empirical experiments are conducted on both unimodal and multimodal benchmark functions. The experimental results demonstrate NDPSO's effectiveness and superiority to many other PSO variants on solving most multimodal problems.