Adaptive Dynamic Programming Method Research Articles

Integral reinforcement learning-based event-triggered optimal tracking control for modular robot manipulators via non-zero-sum game

Abstract Under an event-triggered mechanism, a non-zero-sum (NZS) game optimal tracking control method for modular robot manipulator (MRM) systems with input constraints is proposed using the adaptive dynamic programming (ADP) method based on integral reinforcement learning (IRL). First, a dynamic model of the MRM system is developed based on joint torque feedback technology, consisting of an n-joint subsystem related to interconnected dynamic coupling (IDC). Second, we design a robust compensation controller to handle the known model term and an optimal compensation controller to deal with the uncertainty term caused by the IDC and friction, respectively. In addition, a nonlinear disturbance observer is established to dispose of the negative effects caused by the uncertain sensor output disturbance. Third, based on differential game theory, we transform the optimal tracking control problem of the MRM system into an n-player NZS game problem. Then, the IRL-based ADP method is adopted, which relaxes the need for system partial unknown dynamic information, and only a critic neural network is used to solve the coupled Hamilton–Jacobi equation, so as to obtain the optimal control policy. Then, using Lyapunov theory, the tracking error of the MRM system is demonstrated to be uniformly ultimately bounded. Finally, the effectiveness and superiority of the proposed algorithm are verified through experiments.

Robust optimal control for uncertain wheeled mobile robot based on reinforcement learning: ADP approach

This paper presents a robust optimal control approach for the wheel mobile robot system, which considers the effects of external disturbances, uncertainties, and wheel slipping. The proposed method utilizes an adaptive dynamic programming (ADP) technique in conjunction with a disturbance observer. Initially, the system's state space model is formulated through the utilization of kinematic and dynamic models. Subsequently, the ADP method is employed to establish an online adaptive optimal controller, which solely relies on a single neural network for the purpose of function approximation. The utilization of the disturbance observer in conjunction with the compensation controller serves to alleviate the effects of disturbances. The Lyapunov theorem establishes the stability of the complete closed-loop system and the convergence of the weights of the neural network. The proposed approach has been shown to be effective through simulation under the effect of the disturbances and the change of the desired trajectory.

Robust Policy Iteration of Uncertain Interconnected Systems With Imperfect Data

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Adaptive Virotherapy Strategy for Organism With Constrained Input Using Medicine Dosage Regulation Mechanism.

In this article, the constrained adaptive control strategy based on virotherapy is investigated for organism using the medicine dosage regulation mechanism (MDRM). First, the tumor-virus-immune interaction dynamics is established to model the relations among the tumor cells (TCs), virus particles, and the immune response. The adaptive dynamic programming (ADP) method is extended to approximately obtain the optimal strategy for the interaction system to reduce the populations of TCs. Due to the consideration of asymmetric control constraints, the nonquadratic functions are proposed to formulate the value function such that the corresponding Hamilton-Jacobi-Bellman equation (HJBE) is derived which can be deemed as the cornerstone of ADP algorithms. Then, the ADP method of a single-critic network architecture which integrates MDRM is proposed to obtain the approximate solutions of HJBE and eventually derive the optimal strategy. The design of MDRM makes it possible for the dosage of the agentia containing oncolytic virus particles to be regulated timely and necessarily. Furthermore, the uniform ultimate boundedness of the system states and critic weight estimation errors is validated by Lyapunov stability analysis. Finally, simulation results are given to show the effectiveness of the derived therapeutic strategy.

Sliding-mode surface-based adaptive optimal nonzero-sum games for saturated nonlinear multi-player systems with identifier-critic networks

This paper addresses the sliding-mode surface-based adaptive optimal nonzero-sum games control problem for continuous-time nonlinear systems with input constraints. By constructing a value function associated with the sliding-mode surface, the original nonzero-sum games problem can be transformed into constructing multiple optimal control policies. Then, an adaptive dynamic programming method under identifier-critic networks is introduced to deal with the Hamilton–Jacobi equation, where the identifier networks are applied to approximate the unknown dynamics. A remarkable feature is that the critic updating laws are designed through experience replay and gradient descent methodologies, such that the persistence of excitation condition can be excluded. Based on the Lyapunov stability theory, all signals of the close-loop system are verified to be uniformly ultimately bounded. Finally, simulation results are given to illustrate the effectiveness of the proposed optimal nonzero-sum games scheme.

Multi-Home Energy Coordination Scheduling Based on Adaptive Dynamic Programming

Implementing cooperative scheduling of multi-home microgrid energy and reducing the dependence on the main grid has become the focus of microgrid energy management research. This paper proposes a new multi-agent adaptive dynamic programming (MAADP) method for the cooperative control of distributed home energy. Each home is defined as a learning agent that needs to reasonably schedule the energy storage system to meet the respective load demand while accomplishing cooperative scheduling among the individual homes. In addition, an energy clearing center (ECC) is introduced to complete the energy exchange between each microgrid to protect the benefits of all parties. The proposed method adopts the learning strategy of ”centralized learning and decentralized execution”to avoid the leakage of private information. The experimental comparison with the benchmark method verifies that the method can realize the cooperative scheduling of each home and reduce the dependence on the main grid.

Model-Free Q-Learning for the Tracking Problem of Linear Discrete-Time Systems.

In this article, a model-free Q-learning algorithm is proposed to solve the tracking problem of linear discrete-time systems with completely unknown system dynamics. To eliminate tracking errors, a performance index of the Q-learning approach is formulated, which can transform the tracking problem into a regulation one. Compared with the existing adaptive dynamic programming (ADP) methods and Q-learning approaches, the proposed performance index adds a product term composed of a gain matrix and the reference tracking trajectory to the control input quadratic form. In addition, without requiring any prior knowledge of the dynamics of the original controlled system and command generator, the control policy obtained by the proposed approach can be deduced by an iterative technique relying on the online information of the system state, the control input, and the reference tracking trajectory. In each iteration of the proposed method, the desired control input can be updated by the iterative criteria derived from a precondition of the controlled system and the reference tracking trajectory, which ensures that the obtained control policy can eliminate tracking errors in theory. Moreover, to effectively use less data to obtain the optimal control policy, the off-policy approach is introduced into the proposed algorithm. Finally, the effectiveness of the proposed algorithm is verified by a numerical simulation.

ADP-Based Event-Triggered Constrained Optimal Control on Spatiotemporal Process: Application to Temperature Field in Roller Kiln.

The precise control of the spatiotemporal process in a roller kiln is crucial in the production of Ni-Co-Mn layered cathode material of lithium-ion batteries. Since the product is extremely sensitive to temperature distribution, temperature field control is of great significance. In this article, an event-triggered optimal control (ETOC) method with input constraints for the temperature field is proposed, which takes up an important position in reducing the communication and computation costs. A nonquadratic cost function is adopted to describe the system performance with input constraints. First, we present the problem description of the temperature field event-triggered control, where this field is described by a partial differential equation (PDE). Then, the event-triggered condition is designed according to the information of system states and control inputs. On this basis, a framework of the event-triggered adaptive dynamic programming (ETADP) method that is based on the model reduction technology is proposed for the PDE system. A critic network is used to approach the optimal performance index by a neural network (NN) together with that an actor network is used to optimize the control strategy. Furthermore, an upper bound of the performance index and a lower bound of interexecution times, as well as the stabilities of the impulsive dynamic system and the closed-loop PDE system, are also proved. Simulation verification demonstrates the effectiveness of the proposed method.

Time-varying gain extended state observer-based adaptive optimal control for disturbed unmanned helicopter

In this paper, the robust adaptive optimal tracking control problem is addressed for the disturbed unmanned helicopter based on the time-varying gain extended state observer (TVGESO) and adaptive dynamic programming (ADP) methods. Firstly, a novel TVGESO is developed to tackle the unknown disturbance, which can overcome the drawback of initial peaking phenomenon in the traditional linear ESO method. Meanwhile, compared with the nonlinear ESO, the proposed TVGESO possesses easier and rigorous stability analysis process. Subsequently, the optimal tracking control issue for the original unmanned helicopter system is transformed into an optimization stabilization problem. By means of the ADP and neural network techniques, the feedforward controller and optimal feedback controller are skillfully designed. Compared with the conventional backstepping approach, the designed anti-disturbance optimal controller can make the unmanned helicopter accomplish the tracking task with less energy. Finally, simulation comparisons demonstrate the validity of the developed control scheme.

Robust Approximate Optimal Trajectory Tracking Control for Quadrotors

This paper uses the adaptive dynamic programming (ADP) method to achieve optimal trajectory tracking control for quadrotors. Relying on an established mathematical model of a quadrotor, the approximate optimal trajectory tracking control, which consists of the steady-state control input and the approximate optimal feedback control input, is designed for a nominal system. Considering the compound disturbances in position and attitude dynamic models, disturbance observers are introduced. The estimated values are used to design robust compensation inputs to suppress the effect of the compound disturbances for good trajectory tracking performance. Theoretically, the Lyapunov theorem demonstrates the stability of a closed-loop system. The robustness and effectiveness of the proposed controller are confirmed by the simulation results.

Data-Driven Event-Triggered Adaptive Dynamic Programming Control for Nonlinear Systems With Input Saturation.

This article is devoted to data-driven event-triggered adaptive dynamic programming (ADP) control for nonlinear systems under input saturation. A global optimal data-driven control law is established by the ADP method with a modified index. Compared with the existing constant penalty factor, a dynamic version is constructed to accelerate error convergence. A new triggering mechanism covering existing results as special cases is set up to reduce redundant triggering events caused by emergent factors. The uniformly ultimate boundedness of error system is established by the Lyapunov method. The validity of the presented scheme is verified by two examples.

Zero-Sum Differential Game-Based Fault-Tolerant Control for a Class of Affine Nonlinear Systems.

This article proposes a zero-sum differential game-based control scheme for a class of affine nonlinear control systems with actuator faults, including loss of control, loss of effectiveness, and bias faults. In the control design, the bias faults and the control signal are chosen as the two opposite sides. The Nash equilibrium is achieved while the optimal control signal and the upper bound of bias faults can be derived from the Hamilton-Jacobi-Isaacs (HJI) equation. An adaptive dynamic programming (ADP) method is employed to estimate the weight of the critic network. The developed zero-sum differential game-based control scheme can ensure the system stability and optimal performance. Two simulation results on a numerical system and a rigid spacecraft model illustrate the effectiveness of the proposed control scheme.

An adaptive dynamic programming method for observer‐based sliding mode control of connected vehicles subject to deception attacks

AbstractThis article investigates the problem of optimal observer‐based sliding mode control (SMC) of connected vehicles subject to deception attacks and disturbances with adaptive dynamic programming (ADP) method. For a group of vehicles with unknown nonlinear dynamics term and disturbance, this article aims to give a control methodology to achieve secure tracking of the desired spacing, velocity and acceleration. A neural network (NN) and an observer are constructed to estimate the unknown nonlinear term and the states, respectively. Then, a SMC scheme incorporating NN approximation is developed and an off‐policy ADP method is used to implement the optimal control of sliding mode dynamics. The proposed method can ensure individual stability and string stability of the set of vehicles. Numerical simulations are conducted to demonstrate the validity of the proposed controller.

Guaranteed cost tracking control of constrained input nonlinear uncertain systems using event-based ADP

This article investigates the guaranteed cost robust tracking control problem of nonlinear systems subjected to input constraint and unmatched uncertainty. The event-based adaptive dynamic programming (ADP) approach is utilized to address this problem. First, the tracking error and reference trajectory are combined to form an augmented uncertain system. Then, by decomposing the uncertainty into the matched and unmatched parts, the original tracking problem is converted into the optimal regulation problem of an auxiliary system. The cost function for the auxiliary system is defined, and the associated Hamilton–Jacobi–Bellman (HJB) equation is solved using a single critic neural network (NN). Moreover, a novel event-triggering rule is formulated, and it is shown that the designed event-based controller guarantees that the tracking error is uniformly ultimately bounded. The derivation of event-based guaranteed cost and its relation with the time-based counterpart is presented. The exclusion of the infamous Zeno behavior is guaranteed. The uniform ultimate boundedness of the critic weight estimation error is shown. Finally, the effectiveness of the proposed event-triggered ADP method is illustrated through simulations of the spring–mass–damper system and Van der Pol’s oscillator with unmatched uncertainty.

Event-Triggering-Learning-Based ADP Control for Post-Stall Pitching Maneuver of Aircraft.

In this article, an improved event-triggering-learning (ETL)-based adaptive dynamic programming (ADP) method for the post-stall pitching maneuver of aircraft is proposed to achieve the robust optimal control and reduce the computational cost. First, a feedforward control with the nonlinear disturbance observer (NDO) technique is designed to attenuate the adverse effects caused by the unsteady aerodynamic disturbances. Subsequently, the ADP method with a critic neural network which is constructed to approximate the value function in the Hamilton-Jacobi-Bellman equation is employed to conduct the optimal control of aircraft. In addition, to reduce the computational cost of learning, the event-triggering (ET) mechanism with an improved ET condition is applied. The Lyapunov stability theory is utilized to prove that all signals in the closed-loop control system are uniformly ultimately bounded. Finally, simulation results are presented to illustrate the effectiveness of the proposed ETL-based ADP method.

Dynamic Event-Triggered Safe Control for Nonlinear Game Systems With Asymmetric Input Saturation.

This article focuses on the Pareto optimal issues of nonlinear game systems with asymmetric input saturation under dynamic event-triggered mechanism (DETM). First, the safe control is guaranteed by transforming the system with safety constraints into the one without state constraints utilizing barrier function. The united cost function integrating nonquadratic utility function is constructed to provide the foundation to achieve the Pareto optimal solutions. Then, the adaptive dynamic programming method with concurrent learning is proposed to approximate the Pareto optimal strategies wherein both current and historical data are utilized. To further lessen the consumptions of computation/communication resources, the DETM is integrated into the adaptive algorithm framework which can avoid Zeno phenomena. All the signals of the closed-loop system are proved to be uniformly ultimately bounded. Finally, the simulation results are given to validate the effectiveness of the proposed method from several aspects.

Decentralized Control for Large-Scale Systems With Actuator Faults and External Disturbances: A Data-Driven Method.

This article investigates optimal control for a class of large-scale systems using a data-driven method. The existing control methods for large-scale systems in this context separately consider disturbances, actuator faults, and uncertainties. In this article, we build on such methods by proposing an architecture that accommodates simultaneous consideration of all of these effects, and an optimization index is designed for the control problem. This diversifies the class of large-scale systems amenable to optimal control. We first establish a min-max optimization index based on the zero-sum differential game theory. Then, by integrating all the Nash equilibrium solutions of the isolated subsystems, the decentralized zero-sum differential game strategy is obtained to stabilize the large-scale system. Meanwhile, by designing adaptive parameters, the impact of actuator failure on the system performance is eliminated. Afterward, an adaptive dynamic programming (ADP) method is utilized to learn the solution of the Hamilton-Jacobi-Isaac (HJI) equation, which does not need the prior knowledge of system dynamics. A rigorous stability analysis shows that the proposed controller asymptotically stabilizes the large-scale system. Finally, a multipower system example is adopted to illustrate the effectiveness of the proposed protocols.

Combination Therapy-Based Adaptive Control for Organism Using Medicine Dosage Regulation Mechanism.

In this article, the optimal control strategy for organism is investigated by using the adaptive dynamic programming (ADP) method under the architecture of nonzero-sum games (NZSGs). First, a tumor model is established to formulate the interaction relationships among normal cells, tumor cells, endothelial cells, and the concentrations of drugs. Then, the ADP-based method of single-critic network architecture is proposed to approximate the coupled Hamilton-Jacobi equations (HJEs) under the medicine dosage regulation mechanism (MDRM). According to the game theory, the approximate MDRM-based optimal strategy can be derived, which is of great practical significance. Owing to the proposed mechanism, the dosages of the chemotherapy and anti-angiogenic drugs can be regulated timely and necessarily. Furthermore, the stability of the closed-loop system with the obtained strategy is analyzed via the Lyapunov theory. Finally, a simulation experiment is conducted to verify the effectiveness of the proposed method.

Online Adaptive Dynamic Programming for Optimal Self-Learning Control of VTOL Aircraft Systems With Disturbances

In this paper, a novel online adaptive dynamic programming control algorithm is developed to solve optimal control problems for Vertical Take Off and Landing (VTOL) aircraft systems. Considering the input disturbances of the VTOL systems and adding perturbation interference term to the value function, a robust policy iteration-based ADP method will be introduced to obtain the optimal controller with an approximate optimal control approach. The convergence property is developed to ensure the value function converges to a finite neighborhood of the optimal one. The uniform ultimate boundedness (UUB) of the iterative errors will be strictly proved by Lyapunov stability theory. Finally, we will give mathematical simulation results with the developed method. Compared with sliding mode control (SMC) and linear quadratic regulator (LQR) methods, the simulation results illustrate better overall performance of the novel method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —As an effective tool to solve the optimal control problem of complex nonlinear systems, adaptive dynamic programming (ADP) can effectively overcome the computational problems caused by “curse of dimensionality”. VTOL aircraft system is a typical strongly coupled, underactuate, non-minimum phase system, and the sudden random disturbances will make the control task more challenging. Aiming at the particularity of VTOL system, a new online policy iteration-based ADP algorithm is developed to obtain optimal control with an approximate optimal control strategy in this paper. Decoupling and disturbances can be effectively avoided. convergence will be analyzed to make sure the value function converge to a finite neighborhood of the optimal value function. The UUB property of the system will be proved by the Lyapunov approach.

Prescribed performance event-triggered fuzzy optimal tracking control for strict-feedback nonlinear systems

The prescribed performance event-triggered optimal tracking control problem is considered for a class of uncertain strict-feedback nonlinear systems with external disturbances. First, the disturbance observers are constructed to estimate external disturbances. Then, the controller contains an adaptive fuzzy controller and an optimal compensation term, in which the unknown nonlinearities and cost function are approximated by the fuzzy logic systems. An adaptive fuzzy controller is established by the dynamic surface control (DSC) technique, which is addressed to handle “computation complexity” issue occurred in conventional backstepping approach. Based on adaptive dynamic programming (ADP) method, an optimal compensation term is developed by minimizing the cost function. Furthermore, communication load is reduced via embedding event-triggered mechanism. In addition, the tracking error can be restricted in the prescribed region with the aid of the prescribed performance control. Thus, the whole control scheme can not only ensure that the tracking error converges to a boundary but also achieve optimization, reduce the communication burden and avoid “computation complexity” issue. The boundedness of all signals in the closed-loop is proved. Finally, the simulation examples illustrate the validity of the designed control scheme.