Event-triggered Adaptive Dynamic Programming Research Articles

ADP-Based Event-Triggered Constrained Optimal Control on Spatiotemporal Process: Application to Temperature Field in Roller Kiln.

The precise control of the spatiotemporal process in a roller kiln is crucial in the production of Ni-Co-Mn layered cathode material of lithium-ion batteries. Since the product is extremely sensitive to temperature distribution, temperature field control is of great significance. In this article, an event-triggered optimal control (ETOC) method with input constraints for the temperature field is proposed, which takes up an important position in reducing the communication and computation costs. A nonquadratic cost function is adopted to describe the system performance with input constraints. First, we present the problem description of the temperature field event-triggered control, where this field is described by a partial differential equation (PDE). Then, the event-triggered condition is designed according to the information of system states and control inputs. On this basis, a framework of the event-triggered adaptive dynamic programming (ETADP) method that is based on the model reduction technology is proposed for the PDE system. A critic network is used to approach the optimal performance index by a neural network (NN) together with that an actor network is used to optimize the control strategy. Furthermore, an upper bound of the performance index and a lower bound of interexecution times, as well as the stabilities of the impulsive dynamic system and the closed-loop PDE system, are also proved. Simulation verification demonstrates the effectiveness of the proposed method.

Data-Driven Event-Triggered Adaptive Dynamic Programming Control for Nonlinear Systems With Input Saturation.

This article is devoted to data-driven event-triggered adaptive dynamic programming (ADP) control for nonlinear systems under input saturation. A global optimal data-driven control law is established by the ADP method with a modified index. Compared with the existing constant penalty factor, a dynamic version is constructed to accelerate error convergence. A new triggering mechanism covering existing results as special cases is set up to reduce redundant triggering events caused by emergent factors. The uniformly ultimate boundedness of error system is established by the Lyapunov method. The validity of the presented scheme is verified by two examples.

Nonzero-sum games using actor-critic neural networks: A dynamic event-triggered adaptive dynamic programming

This paper mainly investigates the nonzero-sum games of nonlinear systems with unmatched uncertainty by using actor-critic neural networks. To handle the unmatched components, an auxiliary system with a modified value function is constructed, which transforms the robust stabilization issue into the optimal control issue. Then, a novel dynamic event-triggering condition is designed to further save bandwidth via introducing a dynamic variable. In addition, the actor-critic algorithm is employed in adaptive dynamic programming to achieve Nash equilibrium, which is tuned together with the control policy. By constructing appropriate Lyapunov functions, a criterion is established to ensure that the considered system is uniformly ultimately bounded. Finally, the effectiveness of the developed strategy is demonstrated by an example.

Guaranteed cost tracking control of constrained input nonlinear uncertain systems using event-based ADP

This article investigates the guaranteed cost robust tracking control problem of nonlinear systems subjected to input constraint and unmatched uncertainty. The event-based adaptive dynamic programming (ADP) approach is utilized to address this problem. First, the tracking error and reference trajectory are combined to form an augmented uncertain system. Then, by decomposing the uncertainty into the matched and unmatched parts, the original tracking problem is converted into the optimal regulation problem of an auxiliary system. The cost function for the auxiliary system is defined, and the associated Hamilton–Jacobi–Bellman (HJB) equation is solved using a single critic neural network (NN). Moreover, a novel event-triggering rule is formulated, and it is shown that the designed event-based controller guarantees that the tracking error is uniformly ultimately bounded. The derivation of event-based guaranteed cost and its relation with the time-based counterpart is presented. The exclusion of the infamous Zeno behavior is guaranteed. The uniform ultimate boundedness of the critic weight estimation error is shown. Finally, the effectiveness of the proposed event-triggered ADP method is illustrated through simulations of the spring–mass–damper system and Van der Pol’s oscillator with unmatched uncertainty.

Event-Triggered Adaptive Dynamic Programming for Hierarchical Sliding-Mode Surface-Based Optimal Control of Switched Nonlinear Systems

Event-Triggered Adaptive Dynamic Programming for Hierarchical Sliding-Mode Surface-Based Optimal Control of Switched Nonlinear Systems

Model-free frequency regulation in islanded microgrids: An event-triggered adaptive dynamic programming approach

This paper is concerned with frequency regulation in islanded microgrids via an event-triggered adaptive dynamic programming method. Comparatively to the existing works, we relax the typical limitations on the system, namely, exactly known models, constant load impedances, and single operation mode. The adopted frequency regulation scheme can reach frequency synchronization within the acceptable bounds in the presence of the model uncertainties, operation mode switching, and time-varying loads with no need for any prior information on the system parameters. By employing an event-triggered scheme, the secondary controller is updated only at the aperiodic triggering instants which contributes to less transmission cost. Besides, the use of measurements can help the adaptive weights to be adjusted online in the approximation process, which allows the proposed controller to adapt to the disturbances in real time. Finally, tests including the islanding process and load variation are performed in MATLAB/SimPowerSystem to show the effectiveness of the proposed method.

Optimal tracking control for unknown nonlinear systems with uncertain input saturation: A dynamic event-triggered ADP algorithm

This paper is devoted to proposing a novel dynamic event-triggered adaptive dynamic programming (ADP) algorithm to tackle the tracking control problem of unknown nonlinear systems with uncertain input saturation. Initially, an augmented system is established to address the investigated optimal tracking control issue with a discounted cost function. An online identifier based on neural networks (NNs) is designed to recover the unknown system dynamics. Furthermore, the controller design consists of two parts, including the optimal control policy for the nominal system and the NN-based feedforward compensator. In particular, the latter leverages adaptive NN techniques to cope with uncertain input constraints which can be deemed as uncertain saturation nonlinearities. A critic NN is constructed to obtain the approximate optimal control policy for the nominal system. Moreover, rather than traditional time-triggered and static event-triggered control schemes, an adaptive dynamic event-triggering condition is designed to determine the system sampling as well as the controller execution to further reduce transmission and computation burden. The introduction of an internal dynamic variable in possession of a positive constant lower bound can realize the dynamic adjustment of triggering threshold and effectively exclude Zeno behavior. Notice that the weights of identifier NN and critic NN are updated simultaneously, only at the event-triggering instants. Subsequently, the Lyapunov-based theoretical analysis is conducted to confirm the uniform ultimate boundedness of the tracking error and all the NN weight estimation errors. Eventually, the feasibility and effectiveness of the developed algorithm are validated by means of two simulation examples.

Guaranteed cost fault-tolerant control for uncertain stochastic systems via dynamic event-triggered adaptive dynamic programming

It is difficult to apply the event-triggered adaptive dynamic programming (ET-ADP) algorithm to Itô-type stochastic systems due to stochastic disturbances. Therefore, few studies have addressed the ET-ADP algorithm for Itô-type stochastic systems. To fill this gap, this paper investigates a guaranteed cost fault-tolerant control problem for uncertain stochastic systems via a dynamic ET-ADP algorithm. First, the original control problem is transformed into the optimal control problem of a nominal system, and sufficient criteria are presented to verify the feasibility of the problem transformation. Second, a dynamic event-triggered condition is designed. Then, the optimal control of the nominal system is achieved via an ET-ADP algorithm. Finally, a numerical simulation and a comparative experiment are presented to illustrate the effectiveness of the developed control scheme.

Event-Triggered Learning Robust Tracking Control of Robotic Systems With Unknown Uncertainties

In this article, a tracking control problem for robotic systems with unknown uncertainties is addressed by using an event-triggered adaptive dynamic programming (ADP) method. First, the tracking control of a n-degree of freedom (DOF) robotic system is transformed to the optimal control of an auxiliary system such that the robust control design of the original system is feasible based on the ADP framework. To reduce the computational burden, an event-triggering mechanism is introduced. The cost function and the optimal control are approximated by a critic neural network (NN), where new weight updating laws are designed to relax the persistence of excitation condition and the requirement of initial stabilizing control. In addition, the stability analysis is rigorously given to prove that the closed-loop system is asymptotically stable while the NNs’ weight approximation error is uniformly ultimately bounded. Finally, a simulation case based on a 2-DOF robotic manipulator is given to verify the effectiveness of the designed control methods.

Event-triggered optimal decentralized control for stochastic interconnected nonlinear systems via adaptive dynamic programming

This paper presents an event-triggered adaptive dynamic programming (ETADP) algorithm to study the optimal decentralized control issue of interconnected nonlinear systems subject to stochastic dynamics. By developing a performance index function for augmented auxiliary subsystem, the primordial control issue is converted into deriving an array of optimal control policies sampling in an aperiodic pattern. Then, an ETADP algorithm is introduced under an identifier-actor-critic network framework, where the identifier aims to determine the stochastic dynamic, the critic aims to assess the system performance and the actor aims to implement the control action. A remarkable feature is that the actor-critic updating laws are constructed through the negative gradient method of a positive function, which is designed in the light of the partial derivative of a Hamilton–Jacobi-Bellman equation. Under the provided weight tuning rule, the traditional ADP algorithm can be significantly simplified. The stability of the close-loop system is verified through direct Lyapunov theory, and a numerical simulation is given to confirm the effectiveness of the optimal control scheme.

Event-triggered adaptive dynamic programming for optimal tracking control of unmanned surface vessel with input constraints

In this paper, an event-triggered control mechanism scheme based on dynamic surface control and adaptive dynamic programming (DSC-ETCADP) is proposed for optimal trajectory tracking of the unmanned surface vessel (USV) with input constraints. First, the control structure is designed using dynamic surface control (DSC) as feedforward control and adaptive dynamic programming as feedback control, which not only ensures the tracking of reference trajectory but also improves the control accuracy. Second, due to the data-driven nature of adaptive dynamic programming (ADP) and the ability to adjust parameters online, it can be solved by USV with unknown disturbance. Third, the event-triggered mechanism can save computation and reduce the number of executions of the controller, in addition to avoiding Zeno behavior. Finally, according to the Lyapunov stability theory, all signals in the closed-loop system are globally uniformly and ultimately bounded (UUB). The DSC-ETCADP scheme is compared with DSC method. Simulation results demonstrate the effectiveness of the proposed method.

Observer-based optimal control method combination with event-triggered strategy for hypersonic morphing vehicle

A morphing-attitude coupling control method based on event-triggered adaptive dynamic programming (ETADP) and a finite-time fuzzy disturbance observer (FTFDO) is proposed for a class of hypersonic morphing vehicle (HMV). Firstly, an attitude-morphing coupled dynamic model is established, and the morphing variable is treated as the state for integrated control with attitude. The control law is updated by an event-triggered mechanism, which is designed in two parts: feedforward control and feedback control. Based on the proposed FTFDO, the matched and mismatched uncertainties are estimated and compensated in finite time to solve the feedforward control input. Based on an adaptive dynamic programming algorithm, Critic-Actor dual networks are constructed to implement policy iteration for an approximate optimal feedback control policy by offline training with exploration. The Critic-Actor dual networks then adopt the ETADP algorithm to tune the weight of the networks online when triggering criteria for policy improvement are met. Event-triggered conditions are developed such that the feedforward control and feedback control are both updated by aperiodic sampling, reducing the number of controller updates required. Zeno behavior is avoided by determining the minimum sampling interval time. The stability of the closed-loop system is demonstrated theoretically. Simulation results verify the control efficiency and robustness of the proposed method.

Event-Triggered Single-Network ADP for Zero-Sum Game of Unknown Nonlinear Systems with Constrained Input

In this paper, an event-triggered adaptive dynamic programming (ADP) method is proposed to deal with the H∞ problem with unknown dynamic and constrained input. Firstly, the H∞-constrained problem is regarded as the two-player zero-sum game with the nonquadratic value function. Secondly, we develop the event-triggered Hamilton–Jacobi–Isaacs(HJI) equation, and an event-triggered ADP method is proposed to solve the HJI equation, which is equivalent to solving the Nash saddle point of the zero-sum game. An event-based single-critic neural network (NN) is applied to obtain the optimal value function, which reduces the communication resource and computational cost of algorithm implementation. For the event-triggered control, a triggering condition with the level of disturbance attenuation is developed to limit the number of sampling states, and the condition avoids Zeno behavior by proving the existence of events with minimum triggering interval. It is proved theoretically that the closed-loop system is asymptotically stable, and the critic NN weight error is uniformly ultimately boundedness (UUB). The learning performance of the proposed algorithm is verified by two examples.

An Event-Triggered Adaptive Dynamic Programming Method for Large-Scale HVAC Systems*

In this paper, an event-triggered adaptive dynamic programming (ADP) algorithm is developed for the temperature control of large-scale heating, ventilation, and air conditioning (HVAC) systems. First, the dynamic model of the system is established by using the conservation of mass and energy, which involves the dynamics of the fan and the cooling coil. Second, the multi-zone temperature control problem is treated as a non-zero-sum game, which requires solving the coupled Hamilton-Jacobian (HJ) equations. Then, the ADP algorithm is employed to solve the HJ equation. Importantly, in order to reduce the network transmission burden of large-scale HVAC systems, the ADP algorithm developed in this paper is based on event-triggering mechanism. Theoretical analysis proves that the tracking error and the neural network weight estimation error are uniformly ultimately bounded. Simulation results verify the effectiveness of the algorithm.

Online Optimal Event-Triggered H∞ Control for Nonlinear Systems With Constrained State and Input

Taking safety and performance into consideration, the state and control input of the actual engineering system are often constrained. For this kind of problem, this article puts forward an online dual event-triggered (ET) adaptive dynamic programming (ADP) optimal control algorithm for a class of nonlinear systems with constrained state and input. First, the original system is transformed into another system through the barrier function, after that, a suitable value function with a nonquadratic utility function is designed to obtain the optimal control pair. In addition, on the premise of the asymptotic stability of the system, the trigger condition is devised, and the intersampling time analysis is proved that the algorithm can avoid the Zeno phenomenon. What is more, the critic, action, and disturbance neural networks (NNs) are trained to approximate value function and control sequences, subsequently, the approximation error is proved to be uniformly ultimately boundedness (UUB). Finally, two comparative experiments based on the robot arm model are simulated to verify that the algorithm can make control policies update only when the system has the requirement and keep satisfactory control effect, which can effectively decrease the number of data transfers and reduce the calculation burden.

Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach.

Hydraulic servo actuators (HSAs) are often used in the industry in tasks that request great power, high accuracy and dynamic motion. It is well known that an HSA is a highly complex nonlinear system, and that the system parameters cannot be accurately determined due to various uncertainties, an inability to measure some parameters and disturbances. This paper considers an event-triggered learning control problem of the HSA with unknown dynamics based on adaptive dynamic programming (ADP) via output feedback. Due to increasing practical application of the control algorithm, a linear discrete model of HSA is considered and an online learning data driven controller is used, which is based on measured input and output data instead of unmeasurable states and unknown system parameters. Hence, the ADP-based data driven controller in this paper requires neither the knowledge of the HSA dynamics nor exosystem dynamics. Then, an event-based feedback strategy is introduced to the closed-loop system to save the communication resources and reduce the number of control updates. The convergence of the ADP-based control algorithm is also theoretically shown. Simulation results verify the feasibility and effectiveness of the proposed approach in solving the optimal control problem of HSAs.

Event-triggered adaptive dynamic programming for decentralized tracking control of input constrained unknown nonlinear interconnected systems

This paper addresses decentralized tracking control (DTC) problems for input constrained unknown nonlinear interconnected systems via event-triggered adaptive dynamic programming. To reconstruct the system dynamics, a neural-network-based local observer is established by using local input–output data and the desired trajectories of all other subsystems. By employing a nonquadratic value function, the DTC problem of the input constrained nonlinear interconnected system is transformed into an optimal control problem. By using the observer-critic architecture, the DTC policy is obtained by solving the local Hamilton–Jacobi–Bellman equation through the local critic neural network, whose weights are tuned by the experience replay technique to relax the persistence of excitation condition. Under the event-triggering mechanism, the DTC policy is updated at the event-triggering instants only. Then, the computational resource and the communication bandwidth are saved. The stability of the closed-loop system is guaranteed by implementing event-triggered DTC policy via Lyapunov’s direct method. Finally, simulation examples are provided to demonstrate the effectiveness of the proposed scheme.

Event-Triggered ADP for Tracking Control of Partially Unknown Constrained Uncertain Systems.

An event-triggered adaptive dynamic programming (ADP) algorithm is developed in this article to solve the tracking control problem for partially unknown constrained uncertain systems. First, an augmented system is constructed, and the solution of the optimal tracking control problem of the uncertain system is transformed into an optimal regulation of the nominal augmented system with a discounted value function. The integral reinforcement learning is employed to avoid the requirement of augmented drift dynamics. Second, the event-triggered ADP is adopted for its implementation, where the learning of neural network weights not only relaxes the initial admissible control but also executes only when the predefined execution rule is violated. Third, the tracking error and the weight estimation error prove to be uniformly ultimately bounded, and the existence of a lower bound for the interexecution times is analyzed. Finally, simulation results demonstrate the effectiveness of the present event-triggered ADP method.

Event-Triggered ADP for Nonzero-Sum Games of Unknown Nonlinear Systems.

For nonzero-sum (NZS) games of nonlinear systems, reinforcement learning (RL) or adaptive dynamic programming (ADP) has shown its capability of approximating the desired index performance and the optimal input policy iteratively. In this article, an event-triggered ADP is proposed for NZS games of continuous-time nonlinear systems with completely unknown system dynamics. To achieve the Nash equilibrium solution approximately, the critic neural networks and actor neural networks are utilized to estimate the value functions and the control policies, respectively. Compared with the traditional time-triggered mechanism, the proposed algorithm updates the neural network weights as well as the inputs of players only when a state-based event-triggered condition is violated. It is shown that the system stability and the weights' convergence are still guaranteed under mild assumptions, while occupation of communication and computation resources is considerably reduced. Meanwhile, the infamous Zeno behavior is excluded by proving the existence of a minimum inter-event time (MIET) to ensure the feasibility of the closed-loop event-triggered continuous-time system. Finally, a numerical example is simulated to illustrate the effectiveness of the proposed approach.

Event-Triggered Optimal Control for the Continuous Stirred Tank Reactor System

Continuous stirred tank reactor (CSTR) is essential equipment found in chemical processing industries. Control of CSTR process has long been a challenging issue due to the high complexity and strong nonlinearity of the chemical process. In this paper, a novel event-triggered (ET) adaptive dynamic programming (ADP) optimal control (OC) algorithm is developed for continuous stirred tank reactor (CSTR) system. In order to reduce the computational load and the communication data between the controller and the actuator, a new event-triggering condition based on Taylor series expansion is designed for the ET controller. The control policy does not update unless the event-triggering condition is not satisfied. Compared with the time-triggering mechanism, the update frequency of the controller can be obviously reduced. Then, an identifer-actor-critic structure is developed to implement the ET ADP controller. In particular, to overcome the challenge of establishing an exact dynamic for CSTR system, an identifier neural network (NN) is employed to reconstruct the unknown system dynamic based on offline data. Furthermore, the actor-critic structure is developed to obtain the ET control law and the value function. In actor and critic NNs, weights are turned just at the triggered instant and remained constant during the inter event times. Finally, the developed ET ADP controller is applied to the CSTR system. Experimental results show that the developed ET approach can cut down the update frequency to 66\%, which is very significant for the real CSTR system.