Discounted Performance Function Research Articles

Optimal Tracking Control of Heterogeneous MASs Using Event-Driven Adaptive Observer and Reinforcement Learning.

This article considers the output tracking control problem of nonidentical linear multiagent systems (MASs) using a model-free reinforcement learning (RL) algorithm, where partial followers have no prior knowledge of the leader's information. To lower the communication and computing burden among agents, an event-driven adaptive distributed observer is proposed to predict the leader's system matrix and state, which consists of the estimated value of relative states governed by an edge-based predictor. Meanwhile, the integral input-based triggering condition is exploited to decide whether to transmit its private control input to its neighbors. Then, an RL-based state feedback controller for each agent is developed to solve the output tracking control problem, which is further converted into the optimal control problem by introducing a discounted performance function. Inhomogeneous algebraic Riccati equations (AREs) are derived to obtain the optimal solution of AREs. An off-policy RL algorithm is used to learn the solution of inhomogeneous AREs online without requiring any knowledge of the system dynamics. Rigorous analysis shows that under the proposed event-driven adaptive observer mechanism and RL algorithm, all followers are able to synchronize the leader's output asymptotically. Finally, a numerical simulation is demonstrated to verify the proposed approach in theory.

The Adaptive Optimal Output Feedback Tracking Control of Unknown Discrete-Time Linear Systems Using a Multistep Q-Learning Approach

This paper investigates the output feedback (OPFB) tracking control problem for discrete-time linear (DTL) systems with unknown dynamics. To solve this problem, we use an augmented system approach, which first transforms the tracking control problem into a regulation problem with a discounted performance function. The solution to this problem is derived using a Bellman equation, based on the Q-function. In order to overcome the challenges of unmeasurable system state variables, we employ a multistep Q-learning algorithm that surpasses the advantages of the policy iteration (PI) and value iteration (VI) techniques and state reconstruction methods for output feedback control. As such, the requirement for an initial stabilizing control policy for the PI method is removed and the convergence speed of the learning algorithm is improved. Finally, we demonstrate the effectiveness of the proposed scheme using a simulation example.

Off-Policy Learning-Based Following Control of Cooperative Autonomous Vehicles Under Distributed Attacks

This paper investigates the resilient distributed secure output path following control problem of heterogeneous autonomous ground vehicles (AGVs) subject to cyber attacks based on reinforcement learning algorithm. Most existing results are subject to the same attack models for all communication channels, however multiple channels launched by different attackers are considered in this paper. First, a predictor-acknowledgement clock algorithm for each vehicle is proposed to judge whether the communication channel among neighboring vehicles is attacked or not by receiving or transmitting an acknowledgement. Then, a resilient distributed predictor is proposed to predict the pinning vehicle’s state for each vehicle. In addition, a resilient local control protocol consisting of the feedforward state provided by the predictor and the local feedback state of each vehicle is developed for the output path following problem, which is further converted to the optimal control problem by designing a discounted performance function. Discounted algebraic Riccati equations (AREs) are derived to address the optimal control problem. An off-policy reinforcement learning (RL) algorithm is put forward to learn the solution of discounted AREs online without any prior knowledge of vehicles’ dynamics. It is shown that the RL-based output path following control problem of AGVs imposed by cyber attacks can be achieved in an optimal manner. Finally, a numerical example is provided to verify the effectiveness of theoretical analysis.

Attitude-Tracking Control for Over-Actuated Tailless UAVs at Cruise Using Adaptive Dynamic Programming

Using adaptive dynamic programming (ADP), this paper presents a novel attitude-tracking scheme for over-actuated tailless unmanned aerial vehicles (UAVs) that integrates control and control allocation while accounting for nonlinearity and nonaffine control inputs. The proposed method uses the idea of nonlinear dynamic inversion to create an augmented system and converts the optimal tracking problem into an optimal regulation problem using a discounted performance function. Drawing inspiration from incremental control, this method achieves optimal tracking control for the nonaffine system by simply using a critic-only structure. Moreover, the unique design of the performance function ensures robustness against model uncertainties and external disturbances. The ADP method was found to outperform traditional control architectures that separate control and control allocation, achieving the same level of attitude-tracking performance through a more optimized approach. Furthermore, unlike many recent optimal controllers for nonaffine systems, our method does not require any model identifiers and demonstrates robustness. The superiority of the ADP-based approach is verified through two simulated scenarios, and its internal mechanism is further discussed. The theoretical analysis of robustness and stability is also provided.

Reinforcement Q-Learning Based Current Control for Grid-Connected Three-Phase Inverters with Unknown System Dynamics

This paper proposes a Q-learning-based state feedback suboptimal controller to solve the current control problem of three-phase grid-connected LCL coupled inverters with unknown circuit parameters. In practice, the circuit parameters of the inverters will change obviously for reasons such as calculation errors, external environment and operation aging, which makes the dynamics of the inverter system become unknown. With the circuit’s model of the LCL inverters and the reference current dynamic, an augmented system is constructed and a discounted performance function is formulated as the optimal objective of current control which transform it into a H∞ tracking control. Using Q-learning algorithm with model-free characteristics, a current controller is proposed in which an iterative reinforcement learning (RL) algorithm is embedded. Simulations are presented to verify the valid of the proposed control scheme, where especially the results shows that it can keep excellent control performance under the condition of inverter parameter mutation and grid voltage distortion.

Single‐network ADP for solving optimal event‐triggered tracking control problem of completely unknown nonlinear systems

In this paper, we propose an optimal event-triggered tracking control scheme for completely unknown nonlinear systems under the adaptive dynamic programming (ADP) framework. A data-driven model based on recurrent neural networks (RNNs) is first constructed to model the system uncertainties including the drift dynamics and the input gain matrix, and the modeling error caused by NN approximation is well eliminated through adding a compensation term in the data-driven model such that the model state can asymptotically track the system state. Apart from the traditional construction of optimal tracking controllers, in this paper, an augmented system is developed and a discounted performance function is considered to achieve the optimality. By employing the Bellman optimal principle, an event-triggered tracking Hamilton–Jacobi–Bellman (HJB) equation is then formulated. The approximate solution of the HJB equation can be obtained by virtue of a critic NN, which significantly simplifies the implementation architecture of ADP. Both the historical state data and the current state data are incorporated into the updating of the weight vector in the critic NN, in this circumstance, the persistence of excitation assumption is not needed anymore. It is strictly proven via Lyapunov stability theory that the tracking error state and the critic NN weight are uniformly ultimately bounded. Simulation results examine the validity of the design scheme.

Event‐driven adaptive near‐optimal tracking control of the robot in aircraft skin inspection

AbstractIn this article, we discuss a near‐optimal tracking control problem (NOTCP) of robots used for inspecting aircraft skin with partially unknown systems, unmeasurable states, unknown disturbances, and unknown output delay. A novel observer based on an augmented neural network is designed to overcome the unknown disturbances, unknown output delay, and unknown internal states. An augmented system state, composed of the tracking error and reference system state, is proposed to introduce a new nonquadratic discounted performance function for the NOTCP. Due to the complexity in solving the Hamilton–Jacobi–Bellman equation, an online policy iteration is presented under the adaptive dynamic programming (ADP) framework. Unlike the traditional ADP, the event‐driven algorithm updates the control input only when the event is triggered, which reduces the computational cost and transmission load. Both the control policy and the observer are updated according to the developed triggering condition. Convergence to a near‐optimal control solution and the stability analysis of the proposed algorithm are shown through the Lyapunov candidate function for both the continuous and jump dynamics. The performance of the proposed algorithm is demonstrated by simulation.

Heterogeneous formation control of multiple rotorcrafts with unknown dynamics by reinforcement learning

In this paper, a distributed model-free solution based on reinforcement learning is proposed for the leader–follower formation control problem of heterogeneous multi-agent systems. The multi-agent system consists of multiple rotorcrafts involving a virtual leader and multiple followers, where the dynamics of leaders and followers is unknown. The formation control problem is firstly formulated as an optimal output regulation problem. A discounted performance function is then introduced to guarantee that the tracking error asymptotically converges to zero, and an online off-policy reinforcement learning algorithm is proposed to solve the optimal output problem online using the data generated along the trajectories of the agents. A simulation example is provided to validate the effectiveness of the proposed control method.

Reinforcement Q-Learning Algorithm for H∞ Tracking Control of Unknown Discrete-Time Linear Systems

This article addresses the online reinforcement $Q$ -learning algorithms to design $H_{\infty }$ tracking controller for unknown discrete-time linear systems. An augmented system composed of the original system and the command generator is constructed, and a discounted performance function is introduced to establish a discounted game algebraic Riccati equation (GARE). The existence conditions of a solution to the GARE are proposed and a lower bound is found for the discount factor to assure the stability of the $H_{\infty }$ tracking control solution. The $Q$ -function Bellman equation is then derived, based on which the reinforcement $Q$ -learning algorithm is developed to learn the solution to $H_{\infty }$ tracking control problem without knowing the system dynamics. Both state-data-driven and output-data-driven reinforcement $Q$ -learning algorithms toward finding the control policies are proposed. Unlike the value function approximation (VFA)-based approach, it is proved that the $Q$ -learning scheme brings out no bias of solution to the $Q$ -function Bellman equation under the probing noise satisfying the persistent excitation (PE) condition, and therefore, converges to the nominal discounted GARE solution. Moreover, the proposed output-data-driven method is more powerful than the state-data-driven method as it may not be available to completely measure the full system states in practical applications. A simulation example with a single-phase voltage-source UPS inverter is used to verify the effectiveness of the proposed $Q$ -learning algorithms.

Reinforcement Q-Learning Incorporated With Internal Model Method for Output Feedback Tracking Control of Unknown Linear Systems

This paper investigates the output feedback (OPFB) tracking control problem for discrete-time linear (DTL) systems with unknown dynamics. With the approach of augmented system, the tracking control problem is first turned into a regulation problem with a discounted performance function, the solution of which relies on the Q-function based Bellman equation. Then, a novel value iteration (VI) scheme based on reinforcement Q-learning mechanism is proposed for solving the Q-function Bellman equation without knowing the system dynamics. Moreover, the convergence of the VI based Q-learning is proved by indicating that it converges to the Q-function Bellman equation and it brings out no bias of solution even under the probing noise satisfying the persistent excitation (PE) condition. As a result, the OPFB tracking controller can be learned online by using the past input, output, and reference trajectory data of the augmented system. The proposed scheme removes the requirement of initial admissible policy in the policy iteration (PI) method. Finally, effectiveness of the proposed scheme is demonstrated through a simulation example.

Optimal Synchronization of Heterogeneous Nonlinear Systems With Unknown Dynamics

Optimal output synchronization of multi-agent leader–follower systems with unknown nonlinear dynamics is considered. The agents are assumed heterogeneous so that the dynamics may be nonidentical. A distributed observer is designed to estimate the leader state for each agent. A discounted performance function is defined for each agent, and an augmented Hamilton–Jacobi–Bellman (HJB) equation is derived to find its minimal value. The HJB solution depends on the trajectories of the local state and the distributed observer state. A control protocol based on the HJB solution assures that the synchronization error goes to zero locally asymptotically fast for all agents. The proposed approach has two main advantages compared to standard output synchronization methods. First, it is optimal in the sense that it not only makes the steady-state synchronization error zero, but also minimizes the transient error. Second, it does not require the explicit solution to the output regulator equations, though the HJB solutions implicitly provide optimal solutions to them. Finally, a reinforcement learning technique is used to learn the optimal control protocol for each agent without requiring any knowledge of the agents or the leader dynamics. Simulation studies on a notional multi-agent system validate the proposed approach.

Optimal Robust Output Containment of Unknown Heterogeneous Multiagent System Using Off-Policy Reinforcement Learning.

This paper investigates optimal robust output containment problem of general linear heterogeneous multiagent systems (MAS) with completely unknown dynamics. A model-based algorithm using offline policy iteration (PI) is first developed, where the -copy internal model principle is utilized to address the system parameter variations. This offline PI algorithm requires the nominal model of each agent, which may not be available in most real-world applications. To address this issue, a discounted performance function is introduced to express the optimal robust output containment problem as an optimal output-feedback design problem with bounded -gain. To solve this problem online in real time, a Bellman equation is first developed to evaluate a certain control policy and find the updated control policies, simultaneously, using only the state/output information measured online. Then, using this Bellman equation, a model-free off-policy integral reinforcement learning algorithm is proposed to solve the optimal robust output containment problem of heterogeneous MAS, in real time, without requiring any knowledge of the system dynamics. Simulation results are provided to verify the effectiveness of the proposed method.

Output‐feedback H∞quadratic tracking control of linear systems using reinforcement learning

SUMMARYThis paper presents an online learning algorithm based on integral reinforcement learning (IRL) to design an output‐feedback (OPFB) H∞tracking controller for partially unknown linear continuous‐time systems. Although reinforcement learning techniques have been successfully applied to find optimal state‐feedback controllers, in most control applications, it is not practical to measure the full system states. Therefore, it is desired to design OPFB controllers. To this end, a general boundedL2‐gain tracking problem with a discounted performance function is used for the OPFB H∞tracking. A tracking game algebraic Riccati equation is then developed that gives a Nash equilibrium solution to the associated min‐max optimization problem. An IRL algorithm is then developed to solve the game algebraic Riccati equation online without requiring complete knowledge of the system dynamics. The proposed IRL‐based algorithm solves an IRL Bellman equation in each iteration online in real time to evaluate an OPFB policy and updates the OPFB gain using the information given by the evaluated policy. An adaptive observer is used to provide the knowledge of the full states for the IRL Bellman equation during learning. However, the observer is not needed after the learning process is finished. A simulation example is provided to verify the convergence of the proposed algorithm to a suboptimal OPFB solution and the performance of the proposed method.

Tracking control optimization scheme of continuous-time nonlinear system via online single network adaptive critic design method

In this paper, the optimal tracking control problem (OTCP) for a class of continuous-time nonlinear systems with infinite horizon cost is discussed. An online adaptive critic design method is proposed to learn the solution of OTCP by constructing an augmented system associated with a discounted performance function, which is composed of the tracking errors and reference trajectory dynamics. Only one neural network (NN) is used as critic module for approximating the performance function in the solution procedure, and thus the architecture is simpler than the typical action-critic structure, which needs more computational load from neural networks. Therefore, by the means of the approximate policy iteration, the tracking errors get converged to a region near zero and the parameters of critic module get converged to the optimal ones based on our proposed method. Both the convergence of the NN weights and the stability of the tracking error dynamics are guaranteed by the Lyapunov theory. Two simulation examples are proposed to verify the effectiveness of the proposed method.

Optimal Output-Feedback Control of Unknown Continuous-Time Linear Systems Using Off-policy Reinforcement Learning.

A model-free off-policy reinforcement learning algorithm is developed to learn the optimal output-feedback (OPFB) solution for linear continuous-time systems. The proposed algorithm has the important feature of being applicable to the design of optimal OPFB controllers for both regulation and tracking problems. To provide a unified framework for both optimal regulation and tracking, a discounted performance function is employed and a discounted algebraic Riccati equation (ARE) is derived which gives the solution to the problem. Conditions on the existence of a solution to the discounted ARE are provided and an upper bound for the discount factor is found to assure the stability of the optimal control solution. To develop an optimal OPFB controller, it is first shown that the system state can be constructed using some limited observations on the system output over a period of the history of the system. A Bellman equation is then developed to evaluate a control policy and find an improved policy simultaneously using only some limited observations on the system output. Then, using this Bellman equation, a model-free Off-policy RL-based OPFB controller is developed without requiring the knowledge of the system state or the system dynamics. It is shown that the proposed OPFB method is more powerful than the static OPFB as it is equivalent to a state-feedback control policy. The proposed method is successfully used to solve a regulation and a tracking problem.

H ∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning.

This paper deals with the design of an H ∞ tracking controller for nonlinear continuous-time systems with completely unknown dynamics. A general bounded L2 -gain tracking problem with a discounted performance function is introduced for the H ∞ tracking. A tracking Hamilton-Jacobi-Isaac (HJI) equation is then developed that gives a Nash equilibrium solution to the associated min-max optimization problem. A rigorous analysis of bounded L2 -gain and stability of the control solution obtained by solving the tracking HJI equation is provided. An upper-bound is found for the discount factor to assure local asymptotic stability of the tracking error dynamics. An off-policy reinforcement learning algorithm is used to learn the solution to the tracking HJI equation online without requiring any knowledge of the system dynamics. Convergence of the proposed algorithm to the solution to the tracking HJI equation is shown. Simulation examples are provided to verify the effectiveness of the proposed method.

Actor-critic-based optimal tracking for partially unknown nonlinear discrete-time systems.

This paper presents a partially model-free adaptive optimal control solution to the deterministic nonlinear discrete-time (DT) tracking control problem in the presence of input constraints. The tracking error dynamics and reference trajectory dynamics are first combined to form an augmented system. Then, a new discounted performance function based on the augmented system is presented for the optimal nonlinear tracking problem. In contrast to the standard solution, which finds the feedforward and feedback terms of the control input separately, the minimization of the proposed discounted performance function gives both feedback and feedforward parts of the control input simultaneously. This enables us to encode the input constraints into the optimization problem using a nonquadratic performance function. The DT tracking Bellman equation and tracking Hamilton-Jacobi-Bellman (HJB) are derived. An actor-critic-based reinforcement learning algorithm is used to learn the solution to the tracking HJB equation online without requiring knowledge of the system drift dynamics. That is, two neural networks (NNs), namely, actor NN and critic NN, are tuned online and simultaneously to generate the optimal bounded control policy. A simulation example is given to show the effectiveness of the proposed method.

Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning

In this paper, a new formulation for the optimal tracking control problem (OTCP) of continuous-time nonlinear systems is presented. This formulation extends the integral reinforcement learning (IRL) technique, a method for solving optimal regulation problems, to learn the solution to the OTCP. Unlike existing solutions to the OTCP, the proposed method does not need to have or to identify knowledge of the system drift dynamics, and it also takes into account the input constraints a priori. An augmented system composed of the error system dynamics and the command generator dynamics is used to introduce a new nonquadratic discounted performance function for the OTCP. This encodes the input constrains into the optimization problem. A tracking Hamilton–Jacobi–Bellman (HJB) equation associated with this nonquadratic performance function is derived which gives the optimal control solution. An online IRL algorithm is presented to learn the solution to the tracking HJB equation without knowing the system drift dynamics. Convergence to a near-optimal control solution and stability of the whole system are shown under a persistence of excitation condition. Simulation examples are provided to show the effectiveness of the proposed method.