Off-policy Integral Reinforcement Learning Research Articles

Two-player nonlinear Stackelberg differential game via off-policy integral reinforcement learning

For the nonzero-sum games, players have taken different strategies to achieve the Nash equilibrium. However, regarding hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with a leader–follower strategy to cope with this problem. This paper solved two-player continuous-time nonlinear Stackelberg differential games with unknown system dynamics using an off-policy integral reinforcement learning (IRL) technique. A two-level hierarchy optimal control problem is the Stackelberg differential game, and locating the open-loop Stackelberg equilibrium is equivalent to solving the coupled Hamiltonian-Jacobi (HJ) equations. First, the optimal strategies for the leader and the follower are derived. Then, the closed-loop system’s asymptotic stability of optimal control solution is proved. Based on the policy iteration (PI) algorithm, an off-policy IRL algorithm is developed to solve the Stackelberg game for completely unknown system dynamics. The control strategies and cost functions of the leader and follower are approximated using neural networks (NNs). It is demonstrated that NNs weight errors of off-policy algorithm are uniformly ultimately bounded (UUB). Lastly, two simulation examples demonstrate that the proposed learning algorithm is capable of obtaining two-player Stackelberg equilibrium strategies.

Nearly Optimal Control for Mixed Zero-Sum Game Based on Off-Policy Integral Reinforcement Learning.

In this article, we solve a class of mixed zero-sum game with unknown dynamic information of nonlinear system. A policy iterative algorithm that adopts integral reinforcement learning (IRL), which does not depend on system information, is proposed to obtain the optimal control of competitor and collaborators. An adaptive update law that combines critic-actor structure with experience replay is proposed. The actor function not only approximates optimal control of every player but also estimates auxiliary control, which does not participate in the actual control process and only exists in theory. The parameters of the actor-critic structure are simultaneously updated. Then, it is proven that the parameter errors of the polynomial approximation are uniformly ultimately bounded. Finally, the effectiveness of the proposed algorithm is verified by two given simulations.

Data-Based Optimal Synchronization of Heterogeneous Multiagent Systems in Graphical Games via Reinforcement Learning.

This article studies the optimal synchronization of linear heterogeneous multiagent systems (MASs) with partial unknown knowledge of the system dynamics. The object is to realize system synchronization as well as minimize the performance index of each agent. A framework of heterogeneous multiagent graphical games is formulated first. In the graphical games, it is proved that the optimal control policy relying on the solution of the Hamilton-Jacobian-Bellmen (HJB) equation is not only in Nash equilibrium, but also the best response to fixed control policies of its neighbors. To solve the optimal control policy and the minimum value of the performance index, a model-based policy iteration (PI) algorithm is proposed. Then, according to the model-based algorithm, a data-based off-policy integral reinforcement learning (IRL) algorithm is put forward to handle the partially unknown system dynamics. Furthermore, a single-critic neural network (NN) structure is used to implement the data-based algorithm. Based on the data collected by the behavior policy of the data-based off-policy algorithm, the gradient descent method is used to train NNs to approach the ideal weights. In addition, it is proved that all the proposed algorithms are convergent, and the weight-tuning law of the single-critic NNs can promote optimal synchronization. Finally, a numerical example is proposed to show the effectiveness of the theoretical analysis.

Off-Policy Model-Free Learning for Multi-Player Non-Zero-Sum Games With Constrained Inputs

In this paper, multi-player non-zero-sum games with control constraints are studied by utilizing a novel model-free approach based on adaptive dynamic programming framework. First, the model-based policy iteration (PI) method is provided, which requires the system dynamics, and the convergence is demonstrated. Then, aiming to eliminate the need for the system dynamics, a model-free iterative method is obtained by using the off-policy integral reinforcement learning (IRL) scheme based on the PI approach. Moreover, the system data is collected in order to construct the model-free approach. Besides, we analyze the convergence of the off-policy IRL approach by proving the equivalence between the model-free iterative approach and the model-based iterative approach. Remarkably, in the implementation of the scheme, the control policy and cost function are approximated by utilizing the actor-critic networks. The least square algorithm is utilized to learn the actor-critic networks weights depended on the collected data sets. Finally, two cases are provided to demonstrate the effectiveness of the established framework.

Off‐policy integral reinforcement learning‐based optimal tracking control for a class of nonzero‐sum game systems with unknown dynamics

AbstractThis article studies the optimal tracking control problem of a class of multi‐input nonlinear system with unknown dynamics based on reinforcement learning (RL) and nonzero‐sum game theory. First of all, an augmented system composed of the tracking error dynamics and the command generator dynamics is constructed. Then, a tracking coupled Hamilton–Jacobi (HJ) equations associated with discounted cost function is derived, which gives the Nash equilibrium solution. The existence of Nash equilibrium is proved. To approximate the Nash equilibrium solution of tracking coupled HJ equations, we give two model‐based policy iteration (PI) algorithms, and analyze their equivalence and convergence. Further, to get rid of the prior knowledge of system dynamics, an off‐policy integral reinforcement learning (OP‐IRL) algorithm implemented by neural networks (NNs) is proposed. The weights of critic NNs and actor NNs are updated simultaneously by the gradient descent method. The convergence of the NNs weights and the stability of the closed‐loop error systems are proved. Finally, numerical simulation results are provided to demonstrate the effectiveness of the proposed OP‐IRL method.

Trajectory Tracking of Underactuated VTOL Aerial Vehicles With Unknown System Parameters Via IRL

This article studies the optimal control policy learning for underactuated vertical take-off and landing (VTOL) aerial vehicles subject to the unknown mass and inertia matrix. A novel off-policy integral reinforcement learning (IRL) scheme is presented for simultaneously unknown parameter identification and optimal trajectory tracking. In the outer loop of the VTOL vehicles, a novel off-policy IRL scheme is proposed, where the fixed control policy for data generation is chosen to be different from the iterated control policy and the feedforward term with an unknown mass can be learned along with the optimal control policy. In the inner loop, a hybrid off-policy IRL algorithm is developed to tackle the optimal attitude control policy learning and inertia matrix identification under the hybrid control scheme introduced by the employed inner–outer loop control strategy. A simulation study is finally provided to demonstrate the effectiveness of the proposed algorithm.

Data-based decentralized learning scheme for nonlinear systems with mismatched interconnections

In this paper, the decentralized learning scheme for nonlinear systems with mismatched interconnections is developed by using the off-policy integral reinforcement leaning algorithm. First, the decentralized control of the overall system is transformed into the optimal control of each subsystem by introducing an auxiliary control. In order to relax the knowledge of system dynamics, a model-free policy iteration algorithm is derived based on the off-policy integral reinforcement learning. Then, the model-free policy iteration algorithm is used to solve the related Hamilton–Jacobi-Bellman equations, where only the collected system data is required. For implementation purpose, neural networks are employed to approximate the optimal cost functions and the optimal control policies, respectively. Moreover, the least squares method and the experience replay technique are combined to learn neural network weights. Finally, a mismatched interconnected system and a photovoltaic power system are presented to verify the effectiveness of the proposed algorithm.

Adaptive model-free fault-tolerant control based on integral reinforcement learning for a highly flexible aircraft with actuator faults

This article presents an adaptive model-free fault-tolerant control scheme based on integral reinforcement learning (IRL) technique for tracking control of a highly flexible aircraft (HFA) with actuator faults. To begin, the integral of the tracking error is introduced as a new state to construct an augmented system for the control design. Following that, the off-policy IRL method is applied to obtain the optimal feedback control law online in order to solve the tracking problem without system knowledge. Furthermore, in order to effectively handle system uncertainties, external disturbances, and actuator faults, an adaptive model-free fault-tolerant controller with an observer-like reference model is developed. The designed controller can guarantee that the closed-loop system is uniformly bounded and the tracking error asymptotically approaches zero by choosing design parameters appropriately. Finally, numerical simulations demonstrate the desirable fault accommodation capability of the proposed fault-tolerant strategy.

Data-driven game-based control of microsatellites for attitude takeover of target spacecraft with disturbance

This paper investigates the problem of using multiple microsatellites to control the attitude of a target spacecraft losing control ability. Considering external disturbance and unknown system dynamics, a data-driven robust control method based on game theory is proposed. Firstly, the attitude takeover control of the target using multiple microsatellites is modeled as a robust differential game among disturbance and multiple microsatellites, in which microsatellites can obtain the worst-case control policies. Subsequently, policy iteration algorithm is put forward to acquire the robust Nash equilibrium control policies of microsatellites with known dynamics, which is a basis of data-driven algorithm. Then, by employing off-policy integral reinforcement learning, a data-driven online controller without information about system dynamics is developed to get the feedback gain matrices of microsatellites by learning robust Nash equilibrium solution from online input-state data. To validate the effectiveness of the proposed control method, numerical simulations are provided.

Off-Policy Reinforcement Learning for Tracking in Continuous-Time Systems on Two Time Scales

This article applies a singular perturbation theory to solve an optimal linear quadratic tracker problem for a continuous-time two-time-scale process. Previously, singular perturbation was applied for system regulation. It is shown that the two-time-scale tracking problem can be separated into a linear-quadratic tracker (LQT) problem for the slow system and a linear-quadratic regulator (LQR) problem for the fast system. We prove that the solutions to these two reduced-order control problems can approximate the LQT solution of the original control problem. The reduced-order slow LQT and fast LQR control problems are solved by off-policy integral reinforcement learning (IRL) using only measured data from the system. To test the effectiveness of the proposed method, we use an industrial thickening process as a simulation example and compare our method to a method with the known system model and a method without time-scale separation.

Data-Based H∞Control for the Constrained-Input Nonlinear Systems and its Applications in Chaotic Circuit Systems

In this paper, $H_\infty $ control problem is investigated by off-policy integral reinforcement learning (IRL) method for the nonlinear systems with completely unknown dynamics, disturbances, and constrained-input. Firstly, according to a model-based policy iteration (PI) algorithm, a model-free algorithm is proposed based on the derived iterative equation, and the equivalence of model-based PI algorithm and model-free algorithm is proven. Then, the model-free algorithm is implemented by off-policy IRL technology to solve the Hamilton-Jacobi-Isaacs (HJI) equation with the collected system data by the least-square approach, where three neural networks (NNs) are constructed to approximate the value function, control and the disturbance. Finally, our proposed methods are applied to stabilize an autonomous third-order Chua’s chaotic circuit system and a non-autonomous second-order memristive chaotic circuit system to illustrate the efficiency of the proposed method.

Event-Driven Off-Policy Reinforcement Learning for Control of Interconnected Systems.

In this article, we introduce a novel approximate optimal decentralized control scheme for uncertain input-affine nonlinear-interconnected systems. In the proposed scheme, we design a controller and an event-triggering mechanism (ETM) at each subsystem to optimize a local performance index and reduce redundant control updates, respectively. To this end, we formulate a noncooperative dynamic game at every subsystem in which we collectively model the interconnection inputs and the event-triggering error as adversarial players that deteriorate the subsystem performance and model the control policy as the performance optimizer, competing against these adversarial players. To obtain a solution to this game, one has to solve the associated Hamilton-Jacobi-Isaac (HJI) equation, which does not have a closed-form solution even when the subsystem dynamics are accurately known. In this context, we introduce an event-driven off-policy integral reinforcement learning (OIRL) approach to learn an approximate solution to this HJI equation using artificial neural networks (NNs). We then use this NN approximated solution to design the control policy and event-triggering threshold at each subsystem. In the learning framework, we guarantee the Zeno-free behavior of the ETMs at each subsystem using the exploration policies. Finally, we derive sufficient conditions to guarantee uniform ultimate bounded regulation of the controlled system states and demonstrate the efficacy of the proposed framework with numerical examples.

Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

Adaptive Optimal Control for Stochastic Multiplayer Differential Games Using On-Policy and Off-Policy Reinforcement Learning.

Control-theoretic differential games have been used to solve optimal control problems in multiplayer systems. Most existing studies on differential games either assume deterministic dynamics or dynamics corrupted with additive noise. In realistic environments, multidimensional environmental uncertainties often modulate system dynamics in a more complicated fashion. In this article, we study stochastic multiplayer differential games, where the players' dynamics are modulated by randomly time-varying parameters. We first formulate two differential games for systems of general uncertain linear dynamics, including the two-player zero-sum and multiplayer nonzero-sum games. We then show that optimal control policies, which constitute the Nash equilibrium solutions, can be derived from the corresponding Hamiltonian functions. Stability is proven using the Lyapunov type of analysis. In order to solve the stochastic differential games online, we integrate reinforcement learning (RL) and an effective uncertainty sampling method called the multivariate probabilistic collocation method (MPCM). Two learning algorithms, including the on-policy integral RL (IRL) and off-policy IRL, are designed for the formulated games, respectively. We show that the proposed learning algorithms can effectively find the Nash equilibrium solutions for the stochastic multiplayer differential games.

Mix-zero-sum differential games for linear systems with unknown dynamics based on off-policy IRL

This paper discusses a multi-player mixed-zero-sum (MZS) differential games with completely unknown dynamics. Based on off-policy integral reinforcement learning (IRL), a novel algorithm is proposed to obtain the optimal control. First, a policy iteration algorithm is put forward to obtain the optimal solution for deterministic system. Next, the case that the system dynamics is completely unknown is considered. And an IRL-based off-policy algorithm is presented. Meanwhile, the convergence of the presented algorithms is proved in this paper. At the end, the effectiveness of the proposed algorithm is shown by a simulation.

Power Scheduling in More Electric Aircraft Based on an Optimal Adaptive Control Strategy

In this article, an optimal control problem (OCP) for the power management system is proposed for the more electric aircraft (MEA) with consideration of the dynamics in the generators and the energy storage systems. Electrical power components included in the electrical power system (EPS) of MEA such as the generator, the bus, the rectifier, and the battery system are modeled in this OCP. Based on this formulation, a three-stage optimization architecture is introduced to obtain the optimal power references and the optimal control policies. In the first stage, an optimization problem for the power scheduling and allocation is solved as a mixed integer quadratic programming problem and the optimal power allocation schemes (power references) are obtained. In the second stage, an OCP to minimize the fluctuations in power generation systems is proposed and solved via an off-policy integral reinforcement learning approach. In the third stage, the voltage restoration is applied to maintain the main-bus voltage in the onboard high voltage dc system. Extensive case studies are carried out to illustrate the effectiveness of our proposed methodology. Simulations show that the optimal power references are obtained based on various requirements, and the optimal power control policy is achieved based on off-policy iterations.

Synchronous optimal control method for nonlinear systems with saturating actuators and unknown dynamics using off-policy integral reinforcement learning

The present study establishes an approximate optimal critic learning algorithm, based on the single-network integral reinforcement learning (IRL) algorithm and intends to solve the optimal control problem for an unknown nonlinear system with saturating actuators. The value function is formulated through building generalized nonquadratic functions. In order to solve the Hamilton–Jacobi–Bellman (HJB) equation, a novel optimal scheme for the control approximation, based on the off-policy iteration is presented. Moreover, the single-neural network implementation procedure is introduced to complete the iteration algorithm. The synchronous IRL policy iteration is proposed to update the weight of the critic neural network. Finally, reasonable simulation results are provided for confirming the effectiveness of the proposed optimal approximation control technique in solving equations for a linear and oscillating systems.

Optimal Robust Output Containment of Unknown Heterogeneous Multiagent System Using Off-Policy Reinforcement Learning.

This paper investigates optimal robust output containment problem of general linear heterogeneous multiagent systems (MAS) with completely unknown dynamics. A model-based algorithm using offline policy iteration (PI) is first developed, where the -copy internal model principle is utilized to address the system parameter variations. This offline PI algorithm requires the nominal model of each agent, which may not be available in most real-world applications. To address this issue, a discounted performance function is introduced to express the optimal robust output containment problem as an optimal output-feedback design problem with bounded -gain. To solve this problem online in real time, a Bellman equation is first developed to evaluate a certain control policy and find the updated control policies, simultaneously, using only the state/output information measured online. Then, using this Bellman equation, a model-free off-policy integral reinforcement learning algorithm is proposed to solve the optimal robust output containment problem of heterogeneous MAS, in real time, without requiring any knowledge of the system dynamics. Simulation results are provided to verify the effectiveness of the proposed method.

Off-Policy Integral Reinforcement Learning Method to Solve Nonlinear Continuous-Time Multiplayer Nonzero-Sum Games.

This paper establishes an off-policy integral reinforcement learning (IRL) method to solve nonlinear continuous-time (CT) nonzero-sum (NZS) games with unknown system dynamics. The IRL algorithm is presented to obtain the iterative control and off-policy learning is used to allow the dynamics to be completely unknown. Off-policy IRL is designed to do policy evaluation and policy improvement in the policy iteration algorithm. Critic and action networks are used to obtain the performance index and control for each player. The gradient descent algorithm makes the update of critic and action weights simultaneously. The convergence analysis of the weights is given. The asymptotic stability of the closed-loop system and the existence of Nash equilibrium are proved. The simulation study demonstrates the effectiveness of the developed method for nonlinear CT NZS games with unknown system dynamics.

Off-policy integral reinforcement learning optimal tracking control for continuous-time chaotic systems**Project supported by the National Natural Science Foundation of China (Grant Nos. 61304079 and 61374105), the Beijing Natural Science Foundation, China (Grant Nos. 4132078 and 4143065), the China Postdoctoral Science Foundation (Grant No. 2013M530527), the Fundamental Research Funds for the Central Universities, China (Grant No. FRF-TP-14-119A2), and the Open Research

This paper estimates an off-policy integral reinforcement learning (IRL) algorithm to obtain the optimal tracking control of unknown chaotic systems. Off-policy IRL can learn the solution of the HJB equation from the system data generated by an arbitrary control. Moreover, off-policy IRL can be regarded as a direct learning method, which avoids the identification of system dynamics. In this paper, the performance index function is first given based on the system tracking error and control error. For solving the Hamilton–Jacobi–Bellman (HJB) equation, an off-policy IRL algorithm is proposed. It is proven that the iterative control makes the tracking error system asymptotically stable, and the iterative performance index function is convergent. Simulation study demonstrates the effectiveness of the developed tracking control method.