Integral Reinforcement Learning Method Research Articles

Integral reinforcement learning-based angular acceleration autopilot for high dynamic flight vehicles

During the synthesis of acceleration autopilots for high dynamic flight vehicles (HDFV), autopilots with feedback of angular acceleration (AFAA) have become more perspective with stringent requirements on response speed and high maneuverability, compared with autopilots with feedback of angular rate (AFAR). Integral reinforcement learning (IRL) method has now proved to be an effective technique for adaptive optimal control of partially unknown nonlinear systems. In this paper, a novel data-driven IRL algorithm with “actor-critic” structure is proposed for HDFV utilizing AFAA. As an advanced model-free approach, actor-critic based IRL algorithm learns optimal behaviors by observing the real-time responses from the environment under the action of nonoptimal control policies. Instead of solving algebraic Riccati equation directly, the control policy updates online via the solution of proposed IRL Bellman equation with sensed quantities. Numerical simulation is carried out to validate the effectiveness of proposed online IRL-based angular acceleration autopilot for HDFVs. Besides, the tracking performance under different wave commands, the robustness against parameter uncertainties and the noise attenuation capacity between classical optimal tracking approach and proposed IRL method are analyzed for AFAR and AFAA, respectively. Simulation results show that, angular acceleration autopilot with proposed integral RL algorithm possesses better tracking performance against various disturbances.

An efficient model‐free adaptive optimal control of continuous‐time nonlinear non‐zero‐sum games based on integral reinforcement learning with exploration

AbstractTo reduce the learning time and space occupation, this study presents a novel model‐free algorithm for obtaining the Nash equilibrium solution of continuous‐time nonlinear non‐zero‐sum games. Based on the integral reinforcement learning method, a new integral HJ equation that can quickly and cooperatively determine the Nash equilibrium strategies of all players is proposed. By leveraging the neural network approximation and gradient descent method, simultaneous continuous‐time adaptive tuning laws are provided for both critic and actor neural network weights. These laws facilitate the estimation of the optimal value function and optimal policy without requiring knowledge or identification of the system's dynamics. The closed‐loop system stability and convergence of weights are guaranteed through the Lyapunov analysis. Additionally, the algorithm is enhanced to reduce the number of auxiliary NNs used in the critic. The simulation results for a two‐player non‐zero‐sum game validate the effectiveness of the proposed algorithm.

Suboptimal Leader-to-Coordination Control for Nonlinear Systems With Switching Topologies: A Learning-Based Method.

In the cooperative control for multiagent systems (MASs), the key issues of distributed interaction, nonlinear characteristics, and optimization should be considered simultaneously, which, however, remain intractable theoretically even to this day. Considering these factors, this article investigates leader-to-formation control and optimization for nonlinear MASs using a learning-based method. Under time-varying switching topology, a fully distributed state observer based on neural networks is designed to reconstruct the dynamics and the state trajectory of the leader signal with arbitrary precision under jointly connected topology assumption. Benefitted from the observers, formation for MASs under switching topologies is transformed into tracking control for each subsystem with continuous state generated by the observers. An augmented system with discounted infinite LQR performance index is considered to optimize the control effect. Due to the complexity of solving the Hamilton-Jacobi-Bellman equation, the optimal value function is approximated by a critic network via the integral reinforcement learning method without the knowledge of drift dynamics. Meanwhile, an actor network is also presented to assure stability. The tracking errors and estimation weighted matrices are proven to be uniformly ultimately bounded. Finally, two illustrative examples are given to show the effectiveness of this method.

Integral Reinforcement-Learning-Based Optimal Containment Control for Partially Unknown Nonlinear Multiagent Systems.

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton-Jacobi-Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input-output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

Event-triggered-based online integral reinforcement learning for optimal control of unknown constrained nonlinear systems

For unknown nonlinear systems with actuator saturation, an online policy iteration-based algorithm is employed to solve the optimal event-triggered control problem. To learn the system dynamics, a novel identifier is proposed to make the estimation error converge quickly and the experience replay technique is employed to release the persistence of the excitation condition. To approximate the cost function and the event-triggered control law, we present event-triggered-based critic and actor networks, whose weights are updated only at triggered instants. During the policy iteration process, an event-triggered-based integral reinforcement learning method is proposed to solve the Hamilton–Jacobi–Bellman equation. By utilising the integral reinforcement learning, the network resource is saved and learning efficiency is improved. Based on the Lyapunov method, stability for the closed-loop system and estimation errors for the three networks are analysed. At last, simulation results of two numerical examples are used to show the effectiveness of the proposed method.

Model‐free adaptive optimal control of continuous‐time nonlinear non‐zero‐sum games based on reinforcement learning

In this paper, two novel algorithms to find the Nash equilibrium solution of the non-zero-sum games for continuous-time input-affine nonlinear systems are presented. Based on integral reinforcement learning method, the integral-exploration-coupled Hamilton-Jacobi (HJ) equations are derived, which does not contain any information of the system dynamics. Then, based on neural networks approximation, two different adaptive tuning law of weights are given to estimate the approximate solution of the coupled HJ equations. Both two algorithms can estimate the value function and the policy without knowing or identifying the system dynamics. The closed-loop system stability and the convergence of weights are guaranteed based on Lyapunov analysis. Finally, the simulation results of a two-player non-zero-sum game demonstrate the effectiveness of our algorithms.

Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics

Existing data-driven and feedback traffic control strategies do not consider the heterogeneity of real-time data measurements. Besides, traditional reinforcement learning (RL) methods for traffic control usually converge slowly for lacking data efficiency. Moreover, conventional optimal perimeter control schemes require exact knowledge of the system dynamics and thus they would be fragile to endogenous uncertainties. To handle these challenges, this work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. This work makes the following primary contributions to the transportation literature: (a) A continuous-time control is developed with discrete gain updates to adapt to the discrete-time sensor data. Different from the conventional RL approaches, the reinforcement interval of the proposed IRL method can be varying with respect to the real-time resolution of data measurements. Approximate optimization methods are carried out to address the curse of dimensionality of the optimal control problem with consideration on the resolution of data measurement. (b) To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. (c) The proposed method relaxes the requirement on model calibration in a “model-free” manner that enables robustness against modeling uncertainty and enhances the real-time performance via a data-driven RL algorithm. (d) The convergence of the IRL based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory. The optimal control law is parameterized and then approximated by neural networks (NN), which moderates the computational complexity. Both state and input constraints are considered while no model linearization is required. Numerical examples and simulation experiments are presented to verify the effectiveness and efficiency of the proposed method.

Adaptive model-free fault-tolerant control based on integral reinforcement learning for a highly flexible aircraft with actuator faults

This article presents an adaptive model-free fault-tolerant control scheme based on integral reinforcement learning (IRL) technique for tracking control of a highly flexible aircraft (HFA) with actuator faults. To begin, the integral of the tracking error is introduced as a new state to construct an augmented system for the control design. Following that, the off-policy IRL method is applied to obtain the optimal feedback control law online in order to solve the tracking problem without system knowledge. Furthermore, in order to effectively handle system uncertainties, external disturbances, and actuator faults, an adaptive model-free fault-tolerant controller with an observer-like reference model is developed. The designed controller can guarantee that the closed-loop system is uniformly bounded and the tracking error asymptotically approaches zero by choosing design parameters appropriately. Finally, numerical simulations demonstrate the desirable fault accommodation capability of the proposed fault-tolerant strategy.

Cooperative Robust H-∞ Output Consensus in Continuous-Time Heterogeneous Multi-Agent Systems Using Integral Reinforcement Learning Method

Cooperative Robust H-∞ Output Consensus in Continuous-Time Heterogeneous Multi-Agent Systems Using Integral Reinforcement Learning Method

Value Iteration-Based H∞ Controller Design for Continuous-Time Nonlinear Systems Subject to Input Constraints

In this paper, a novel integral reinforcement learning method is proposed based on value iteration (VI) to design the $H_{\infty }$ controller for continuous-time nonlinear systems subject to input constraints. To confront the control constraints, a nonquadratic function is introduced to reconstruct the ${L_{2}}$ -gain condition for the $H_{\infty }$ control problem. Then, the VI method is proposed to solve the corresponding Hamilton–Jacobi–Isaacs equation initialized with an arbitrary positive semi-definite value function. Compared with most existing works developed based on policy iteration, the initial admissible control policy is no longer required which results in a more free initial condition. The iterative process of the proposed VI method is analyzed and the convergence to the saddle point solution is proved in a general way. For the implementation of the proposed method, only one neural network is introduced to approximate the iterative value function, which results in a simpler architecture with less computational load compared with utilizing three neural networks. To verify the effectiveness of the VI-based method, two nonlinear cases are presented, respectively.

A learning-based optimal tracking controller for continuous linear systems with unknown dynamics: Theory and case study

In this article, a novel continuous-time optimal tracking controller is proposed for the single-input-single-output linear system with completely unknown dynamics. Unlike those existing solutions to the optimal tracking control problem, the proposed controller introduces an integral compensation to reduce the steady-state error and regulates the feedforward part simultaneously with the feedback part. An augmented system composed of the integral compensation, error dynamics, and desired trajectory is established to formulate the optimal tracking control problem. The input energy and tracking error of the optimal controller are minimized according to the objective function in the infinite horizon. With the application of reinforcement learning techniques, the proposed controller does not require any prior knowledge of the system drift or input dynamics. The integral reinforcement learning method is employed to approximate the Q-function and update the critic network on-line. And the actor network is updated with the deterministic learning method. The Lyapunov stability is proved under the persistence of excitation condition. A case study on a hydraulic loading system has shown the effectiveness of the proposed controller by simulation and experiment.

Model-free Optimal Tracking Control for an Aircraft Skin Inspection Robot with Constrained-input and Input Time-delay via Integral Reinforcement Learning

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

Off-policy synchronous iteration IRL method for multi-player zero-sum games with input constraints

In this paper, a novel synchronous off-policy method is given to solve multi-player zero-sum (ZS) game under the condition that the knowledge of system data are completely unknown, the actuators of controls are constrained and the disturbances are bounded simultaneously. The cost functions are built by nonquadratic functions to reflect the constrained properties of inputs. The integral reinforcement learning (IRL) technology is employed to solve Hamilton–Jacobi–Bellman equation, so that the system dynamics are not necessary anymore. The obtained value function is proved to converge to the optimal game values. And the equivalent of traditional policy iteration (PI) algorithm and the proposed algorithm is given in solving the multi-player ZS game with constrained inputs. Three neural networks in this paper are utilized, the critic neural network (CNN) to approach the cost function, the action neural network (ANN) to approach the control policies and the disturbance neural networks (DNN) to approach the disturbances are utilized. Finally, a simulation example is given to demonstrate the convergence and performance of the proposed algorithm.

Integral reinforcement learning-based online adaptive event-triggered control for non-zero-sum games of partially unknown nonlinear systems

This paper develops an integral reinforcement learning (IRL)-based adaptive control method for the multi-player non-zero-sum (NZS) games of the nonlinear continuous-time systems with partially unknown dynamics, in the context of event-triggered mechanism. With the principle of IRL method, the requirement for the system drift dynamics is relaxed in the controller design. Moreover, different from the conventional iteration computation methods, the algorithm developed in this work is implemented in an online adaptive fashion, which provides a new way to combine the IRL algorithm and the event-triggered control framework in solving the NZS game issues. In the event-based algorithm, a state-dependent triggering condition is presented, which not only guarantees the closed-loop system stability, but also reduces the computation and communication loads of the controlled plant. By means of Lyapunov theorem, the uniform ultimate boundedness (UUB) properties of the system states and the critic weight estimation errors have been proved. Finally, two numerical examples are utilized to demonstrate the efficacy of the proposed method.

Data-Based Reinforcement Learning for Nonzero-Sum Games With Unknown Drift Dynamics.

This paper is concerned about the nonlinear optimization problem of nonzero-sum (NZS) games with unknown drift dynamics. The data-based integral reinforcement learning (IRL) method is proposed to approximate the Nash equilibrium of NZS games iteratively. Furthermore, we prove that the data-based IRL method is equivalent to the model-based policy iteration algorithm, which guarantees the convergence of the proposed method. For the implementation purpose, a single-critic neural network structure for the NZS games is given. To enhance the application capability of the data-based IRL method, we design the updating laws of critic weights based on the offline and online iterative learning methods, respectively. Note that the experience replay technique is introduced in the online iterative learning, which can improve the convergence rate of critic weights during the learning process. The uniform ultimate boundedness of the critic weights are guaranteed using the Lyapunov method. Finally, the numerical results demonstrate the effectiveness of the data-based IRL algorithm for nonlinear NZS games with unknown drift dynamics.