Integral Reinforcement Learning Algorithm Research Articles

Communication-Efficient and Collision-Free Motion Planning of Underwater Vehicles via Integral Reinforcement Learning.

Motion planning of underwater vehicles is regarded as a promising technique to make up the flexibility deficiency of underwater sensor networks (USNs). Nonetheless, the unique characteristics of underwater channel and environment make it challenging to achieve the above mission. This article is concerned with a communication-efficient and collision-free motion planning issue for underwater vehicles in fading channel and obstacle environment. We first develop a model-based integral reinforcement learning (IRL) estimator to predict the stochastic signal-to-noise ratio (SNR). With the estimated SNR, an integrated optimization problem for the codesign of communication efficiency and motion planning is constructed, in which the underwater vehicle dynamics, communication capacity, collision avoidance, and position control are all considered. In order to tackle this problem, a model-free IRL algorithm is designed to drive underwater vehicles to the desired position points while maximizing the communication capacity and avoiding the collision. It is worth mentioning that, the proposed motion planning solution in this article considers a realistic underwater communication channel, as well as a realistic dynamic model for underwater vehicles. Finally, simulation and experimental results are demonstrated to verify the effectiveness of the proposed approach.

Event-Triggered Guarantee Cost Control for Partially Unknown Stochastic Systems via Explorized Integral Reinforcement Learning Strategy.

In this article, an integral reinforcement learning (IRL)-based event-triggered guarantee cost control (GCC) approach is proposed for stochastic systems which are modulated by randomly time-varying parameters. First, with the aid of the RL algorithm, the optimal GCC (OGCC) problem is converted into an optimal zero-sum game by solving a modified Hamilton-Jacobin-Isaac (HJI) equation of the auxiliary system. Moreover, in order to address the stochastic zero-sum game, we propose an on-policy IRL-based control approach involved by the multivariate probabilistic collocation method (MPCM), which can accurately predict the mean value of uncertain functions with randomly time-varying parameters. Furthermore, a novel GCC method, which combines the explorized IRL algorithm and MPCM, is designed to relax the restriction of knowing the system dynamics for the class of stochastic systems. On this foundation, for the purpose of reducing computation cost and avoiding the waste of resources, we propose an event-triggered GCC approach involved with explorized IRL and MPCM by utilizing critic-actor-disturbance neural networks (NNs). Meanwhile, the weight vectors of three NNs are updated simultaneously and aperiodically according to the designed triggering condition. The ultimate boundedness (UB) properties of the controlled systems have been proved by means of the Lyapunov theorem. Finally, the effectiveness of the developed GCC algorithms is illustrated via two simulation examples.

An online model-free adaptive learning control solution for robotic arms

This paper focuses on the online control of a class of nonlinear dynamical systems, specifically robotic manipulators. Solutions utilizing Proportional–Integral–Derivative (PID) control schemes are employed to control the joints of robotic manipulators. However, the existing control strategies utilize fixed gains, which do not fully account for the inherent nonlinearity of the dynamical structure or the dynamics of reference-tracking error. Additionally, the individual joint’s dynamic performance is optimized independently from the performance of other joints. This work introduces an adaptive integral Reinforcement Learning algorithm to control a four-DoF robotic arm in real time. This is done using a model-free Value Iteration process implemented in a continuous-time mode. The solution does not assume any knowledge of the dynamics of the robot arm and does not require any initial admissible control strategy to proceed with the adaptive learning solution. The self-learning algorithm provides adaptable strategies to control the turntable, forearm, bicep, and wrist joints of the robotic arm. The performance of the adaptive learning solution is compared with those of Proportional–Integral–Derivative and high-order model-free adaptive control schemes to highlight its effectiveness.

Integral reinforcement learning‐based optimal tracking control for uncertain nonlinear systems under input constraint and specified performance constraints

AbstractThis article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

Integral reinforcement learning-based angular acceleration autopilot for high dynamic flight vehicles

During the synthesis of acceleration autopilots for high dynamic flight vehicles (HDFV), autopilots with feedback of angular acceleration (AFAA) have become more perspective with stringent requirements on response speed and high maneuverability, compared with autopilots with feedback of angular rate (AFAR). Integral reinforcement learning (IRL) method has now proved to be an effective technique for adaptive optimal control of partially unknown nonlinear systems. In this paper, a novel data-driven IRL algorithm with “actor–critic” structure is proposed for HDFV utilizing AFAA. As an advanced model-free approach, “actor–critic” based IRL algorithm learns optimal behaviors by observing the real-time responses from the environment under the action of nonoptimal control policies. Instead of solving algebraic Riccati equation directly, the control policy updates online via the solution of proposed IRL Bellman equation with sensed quantities. Numerical simulation is carried out to validate the effectiveness of proposed online IRL-based angular acceleration autopilot for HDFVs. Besides, the tracking performance under different wave commands, the robustness against parameter uncertainties and the noise attenuation capacity between classical optimal tracking approach and proposed IRL method are analyzed for AFAR and AFAA, respectively. Simulation results show that, angular acceleration autopilot with proposed integral RL algorithm possesses better tracking performance against various disturbances.

Two-player nonlinear Stackelberg differential game via off-policy integral reinforcement learning

For the nonzero-sum games, players have taken different strategies to achieve the Nash equilibrium. However, regarding hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with a leader–follower strategy to cope with this problem. This paper solved two-player continuous-time nonlinear Stackelberg differential games with unknown system dynamics using an off-policy integral reinforcement learning (IRL) technique. A two-level hierarchy optimal control problem is the Stackelberg differential game, and locating the open-loop Stackelberg equilibrium is equivalent to solving the coupled Hamiltonian-Jacobi (HJ) equations. First, the optimal strategies for the leader and the follower are derived. Then, the closed-loop system’s asymptotic stability of optimal control solution is proved. Based on the policy iteration (PI) algorithm, an off-policy IRL algorithm is developed to solve the Stackelberg game for completely unknown system dynamics. The control strategies and cost functions of the leader and follower are approximated using neural networks (NNs). It is demonstrated that NNs weight errors of off-policy algorithm are uniformly ultimately bounded (UUB). Lastly, two simulation examples demonstrate that the proposed learning algorithm is capable of obtaining two-player Stackelberg equilibrium strategies.

Reinforcement learning based sliding mode control for two-time-scale multi-agent systems with malicious attacks

The optimal consensus problem for linear two-time-scale multi-agent systems under malicious attacks is studied in this paper. Firstly, an integral sliding mode function is devised to guide the system trajectory towards the sliding mode surface and the impact of attacks can be eliminated. Then, the optional consensus problem is reformulated as a zero-sum game problem between each agent and its neighbouring agents. Thus, the game algebraic Riccati equation with singularly perturbed parameter is formulated. Furthermore, to avoid the requirement of the system dynamics information, an integral reinforcement learning algorithm is presented to obtain the optimal control policy for multi-agent systems. Compared with existing learning methods, the obtained reinforcement learning algorithm is devoid of potential calculation error issues from singularly perturbed parameter. Meanwhile, the convergence of the proposed algorithm is verified. Finally, a simulation example is provided to demonstrate the efficacy of the proposed control method.

Model-free optimal robust tracking control for the joint actuated by pneumatic artificial muscles with saturated air pressure input

At present, positioning and tracking the joint angle of the PAM-actuated system is still a challenging issue, due to the inherent hysteresis, high nonlinearities, and time-varying characteristics. Considering the problem of uncertain parameters, external disturbance, and saturated input, a model-free optimal robust tracking control scheme is proposed for the PAMs system in this paper. Also, a nonquadratic cost function is introduced to address the saturation air pressure input. Moreover, an integral reinforcement learning(IRL) algorithm is developed to solve the H∞ tracking Hamilton–Jacobi–Isaacs(HJI) equation. And the IRL algorithm is implemented on a critic–actor-disturbance neural network (NN) structure. The presented control scheme avoids the complete system dynamic and simultaneously compensates for the uncertain parameters, and external disturbance. Then, the effectiveness of the optimal robust tracking controller is verified via computational simulation and keeps the system stable. In addition, the tracking experiment is implemented on a hardware platform, indicating that the presented approach achieves satisfactory tracking performance.

Dynamic event-triggered-based online IRL algorithm for the decentralized control of the input and state constrained large-scale unmatched interconnected system

This article proposed a novel adaptive decentralized control (ADC) method for the continuous-time state-constrained and input-constrained large-scale unmatched interconnection system by the means of the adaptive critic design in the edge dynamic event-triggered (EDET) mechanism. The barrier function is used to transform the state-constrained system to the common system without constrained states. To overcome the influence of the unmatched interconnection terms, the auxiliary systems are devised for each subsystem of the large-scale system. Moreover, the non-quadratic utility functions are introduced to constrain the input of the auxiliary systems. The decentralized control scheme of the large-scale unmatched interconnection system can be realized by solving the optimal control schemes of the auxiliary systems. Then with the help of the single critic neural network (NN), the approximate optimal control policies are acquired by designing the approximate cost functions. After that, an integral reinforcement learning (IRL)-based scheme is proposed to solve the integral Bellman equation rather than the complex coupled Hamilton–Jacobi-Bellman equation (HJBE). Simultaneously, the EDET mechanism is introduced to reduce the computation efforts and communication loads. The asymptotic stability of the constrained large-scale unmatched interconnection system is proved. In addition, the infamous Zeno behavior is effectively avoided. Finally, two simulation cases are given to verify the effectiveness of the proposed algorithm.

A novel learning-based data-driven [formula omitted] control strategy for vanadium redox flow battery in DC microgrids

Vanadium redox flow battery (VRB) is one of the most promising batteries at present. In order to enhance the stability and anti-interference ability of VRB in microgrids, a novel learning-based data-driven H∞ control approach is proposed for the VRB, which uses a new integral reinforcement learning algorithm to produce excellent steady-state and dynamic responses only by measurements. Compared to the model-based control methods, it is insensitive to model parameter variations. Furthermore, compared to most of the existing artificial intelligent control approaches that require large amounts of experimental data for offline neural network (NN) training, the proposed control strategy contributes to eliminate the offline training process and therefore, does not need the costly and tedious training data acquisition process. More importantly, the proposed control offers guaranteed closed-loop control stability, which cannot be achieved by nearly all the control methods that purely rely on the offline trained NNs. In this paper, the rigorous proof of stability is given, and the validity of the proposed method is verified by simulation results.

A learning-based approach to event-triggered guaranteed cost control for completely unknown nonlinear systems

This paper develops a novel guaranteed cost control (GCC) approach under the event-triggered mechanism for completely unknown systems using integral reinforcement learning (IRL) algorithm. First, based on the adaptive dynamic programming (ADP) method, the GCC problem is addressed by transforming it into the optimal control problem. Second, without using the accurate information of system dynamics, a model-free data-based GCC approach is designed via IRL algorithm. Moreover, for the purpose of reducing the waste of communication resources, a GCC algorithm is presented under the event-triggered mechanism for completely unknown system by utilizing the explorized IRL algorithm. The critic–actor–disturbance neural networks (NNs) are applied to approximate near optimal solution. In addition, the weight estimations of NNs are tuned synchronously according to the designed novel triggering condition. Furthermore, the stability analysis of the controlled system is given by utilizing the Lyapunov principle. Finally, the simulation results are presented to verify the feasibility of the designed approach.

Dynamic Event-Triggered-Based Integral Reinforcement Learning Algorithm for Frequency Control of Microgrid With Stochastic Uncertainty

The intermittency of renewable energy and the uncertainty of load put forward higher robustness requirements for the frequency recovery of microgrid (MG). The energy storage equipment provides an idea for frequency control because of its fast response and strong controllability. In this paper, an on-line adaptive frequency control method is proposed to control the governor and energy storage to realize the frequency recovery of MG under stochastic uncertainty. First, the MG system with external disturbances is constructed as a zero-sum differential game model to obtain a robust optimal control scheme. Then, considering the system parameter uncertainty, an improved integral reinforcement learning (IRL) algorithm is designed, in which the reinforcement signal contains a non-quadratic function to solve the energy storage control constraints. Furthermore, a novel dynamic event-triggered control (DETC) is developed to reduce control update times of energy storage. The dynamic variable in DETC coupled with static trigger includes not only the past triggering information but also the disturbances. DETC has a larger trigger threshold than static event-triggered control (SETC). Meanwhile, the proposed algorithm is implemented by an action-critic network structure, in which the action network of energy storage is updated aperiodically. Finally, simulation results show that the proposed control algorithm can realize the frequency recovery of MG, and it has good robustness compared with other algorithms.

Reinforcement Learning-Based Near Optimization for Continuous-Time Markov Jump Singularly Perturbed Systems

The design of a suboptimal controller for continuous-time Markov jump singularly perturbed systems with partially unknown dynamics is studied in this paper. With fast and slow decomposition technique, the original Markov jump singularly perturbed systems are decomposed into fast and slow subsystems as a new attempt. On this basis, an offline parallel Kleinman algorithm and an online parallel integral reinforcement learning algorithm are presented to cope with the different subsystems, respectively. Meanwhile, the controllers obtained by the above two algorithms are used to design the suboptimal controllers for original systems. Furthermore, the suboptimality of the proposed controllers is also discussed. Finally, an example of the electric circuit model is shown to illustrate the applicability of the proposed method.

Space manipulator optimal impedance control using integral reinforcement learning

This paper examines the optimal impedance control problem for large-scale space manipulator operational tasks with unknown contact dynamics and partial measurements. More specifically, by quantifying the interaction performance using a discounted value function, the optimal impedance control problem is tactfully transformed into a linear quadratic tracking problem. By resorting to the historical inputs and outputs, an improved state reconstruction method is presented, which obviates the velocity measurement. Unlike the existing state reconstruction method for continuous-time systems, the estimation bias caused by probing noise is completely eliminated under the improved state reconstruction method. Based on this, a novel model-free value iteration integral reinforcement learning algorithm is developed to approximate optimal impedance parameters. Compared with the earlier integral reinforcement learning algorithms, the proposed algorithm not only averts any prior contact dynamics knowledge and full-state measurements, but also eliminates the heavy dependence on the specific initial stabilization control. In addition, the implementation and convergence of the proposed algorithm are discussed successively. Finally, numerical simulations verify the effectiveness of the theoretical results.

A New Robust Integral Reinforcement Learning Based Control Algorithm for Interleaved DC/DC Boost Converter

This article proposes a novel online integral reinforcement learning (IRL) based data-driven control algorithm for interleaved dc/dc boost converter. The proposed algorithm is independent of system model due to the usage of a three-layer neural network (NN). Furthermore, its controller gains are autonomously adjusted online through the value function based NN weights updating mechanism, which simplifies the controller gain tuning process. Compared to the conventional model-dependent control approaches, it provides superior control performance. Additionally, the proposed method contributes to significantly reduce the computational burden of classical IRL algorithm by removing the disturbance updating process. Experimental results are presented to verify the efficacy of the proposed algorithm.

Integral Reinforcement-Learning-Based Optimal Containment Control for Partially Unknown Nonlinear Multiagent Systems.

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton-Jacobi-Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input-output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

Neural network-based event-triggered integral reinforcement learning for constrained [formula omitted] tracking control with experience replay

Since input constraints and external disturbances are unavoidable in tracking control problems, how to obtain a controller in this case to save communication and data resources at the same time is very challenging. Aiming at these challenges, this paper develops a novel neural network (NN)-based event-triggered integral reinforcement learning (IRL) algorithm for constrained H∞ tracking control problems. First, the constrained H∞ tracking control problem is transformed into a regulation problem. Second, an event-triggered optimal controller is designed to reduce network transmission burden and improve resource utilization, where a novel threshold is proposed and its non-negativity can be guaranteed. Third, for implementation purpose, a novel NN-based event-triggered IRL algorithm is developed. In order to improve data utilization, the experience replay technique with an easy-to-verify condition is employed in the learning process. Theoretical analysis proves that the tracking error and weight estimation error are uniformly ultimately bounded. Finally, simulation verification shows the effectiveness of the present method.

Trajectory Tracking of Underactuated VTOL Aerial Vehicles With Unknown System Parameters Via IRL

This article studies the optimal control policy learning for underactuated vertical take-off and landing (VTOL) aerial vehicles subject to the unknown mass and inertia matrix. A novel off-policy integral reinforcement learning (IRL) scheme is presented for simultaneously unknown parameter identification and optimal trajectory tracking. In the outer loop of the VTOL vehicles, a novel off-policy IRL scheme is proposed, where the fixed control policy for data generation is chosen to be different from the iterated control policy and the feedforward term with an unknown mass can be learned along with the optimal control policy. In the inner loop, a hybrid off-policy IRL algorithm is developed to tackle the optimal attitude control policy learning and inertia matrix identification under the hybrid control scheme introduced by the employed inner–outer loop control strategy. A simulation study is finally provided to demonstrate the effectiveness of the proposed algorithm.

Event-triggered integral reinforcement learning for nonzero-sum games with asymmetric input saturation

In this paper, an event-triggered integral reinforcement learning (IRL) algorithm is developed for the nonzero-sum game problem with asymmetric input saturation. First, for each player, a novel non-quadratic value function with a discount factor is designed, and the coupled Hamilton–Jacobi equation that does not require a complete knowledge of the game is derived by using the idea of IRL. Second, the execution of each player is based on the event-triggered mechanism. In the implementation, an adaptive dynamic programming based learning scheme using a single critic neural network (NN) is developed. Experience replay technique is introduced into the classical gradient descent method to tune the weights of the critic NN. The stability of the system and the elimination of Zeno behavior are proved. Finally, simulation experiments verify the effectiveness of the event-triggered IRL algorithm.

Online reinforcement learning multiplayer non-zero sum games of continuous-time Markov jump linear systems

In this paper, a novel online mode-free integral reinforcement learning algorithm is proposed to solve the multiplayer non-zero sum games. We first collect and learn the subsystems information of states and inputs; then we use the online learning to compute the corresponding N coupled algebraic Riccati equations. The policy iterative algorithm proposed in this paper can solve the coupled algebraic Riccati equations corresponding to the multiplayer non-zero sum games. Finally, the effectiveness and feasibility of the design method of this paper is proved by simulation example with three players.