Optimal Output Regulation Problem Research Articles

Distributed adaptive cooperative optimal output regulation via integral reinforcement learning

This paper studies the optimal cooperative output regulation problem for unknown linear multi-agent systems by the integral reinforcement learning technique. Existing results on this problem were obtained by a non-fully distributed learning process. In contrast, we propose a distributed learning algorithm over the jointly connected switching communication networks. Moreover, by modifying the existing algorithm, we reduce the computational cost and weaken the solvability conditions. Two numerical examples are used to illustrate the effectiveness of our approach.

Incremental reinforcement learning and optimal output regulation under unmeasurable disturbances

In this paper, we propose novel data-driven optimal dynamic controller design frameworks, via both state-feedback and output-feedback, for solving optimal output regulation problems of linear discrete-time systems subject to unknown dynamics and unmeasurable disturbances using reinforcement learning (RL). Fundamentally different from existing research on optimal output regulation problems and RL, the proposed procedures can determine both the optimal control gain and the optimal dynamic compensator simultaneously instead of presetting a non-optimal dynamic compensator. Moreover, we present incremental dataset-based RL algorithms to learn the optimal dynamic controllers that do not require the measurements of the external disturbance and the exostate during learning, which is of great practical importance. Besides, we show that the proposed incremental dataset-based learning methods are more robust to a class of measurement noises with arbitrary magnitudes than routine RL algorithms. Comprehensive simulation results validate the efficacy of our methodologies.

Hybrid Iteration ADP Algorithm to Solve Cooperative, Optimal Output Regulation Problem for Continuous-Time, Linear, Multiagent Systems: Theory and Application in Islanded Modern Microgrids With IBRs

This paper proposes a novel adaptive dynamic programming (ADP) algorithm, named hybrid iteration (HI), to solve the cooperative, optimal output regulation problem (CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP) for continuous-time, linear, multi-agent systems. Unlike traditional ADP algorithms, i.e., policy iteration (PI) and value iteration (VI), HI does not need an initial stabilizing control policy required by PI. At the same time, it maintains a faster convergence rate compared with VI. First, a model-based HI algorithm is proposed to solve the CO <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> RP. Based on the proposed HI algorithm, a data-driven, adaptive, optimal controller is developed to solve the cooperative, adaptive, optimal output regulation problem without using any information about the physics of the system. Instead, the states/input information collected along the trajectories of the dynamic system is employed. The proposed data-driven HI is applied to the adaptive, optimal secondary voltage control (also known as voltage restoration control) of an islanded modern microgrid based on inverter-based resources. Compared with the VI and PI algorithms, comparative simulation results demonstrate that the proposed HI approach is significantly able to save the convergence time of the central processing unit (also known as CPU) deployed, reduce the number of learning iterations, and remove the requirement of the initial stabilizing control policy. Comparative experiments reveal the practicality and superiority of the proposed methodology.

Adaptive optimal output regulation of unknown linear continuous-time systems by dynamic output feedback and value iteration

This paper proposes a value iteration based learning algorithm to solve the optimal output regulation problem for linear continuous-time systems, which aims at achieving disturbance rejection and asymptotic tracking simultaneously. Notably, the state is unmeasurable and the system dynamics is completely unknown, which greatly increases the challenge of solving the output regulation problem. Firstly, we present a dynamic output feedback controller design method by combining the internal model, setting a virtual exo-system, and constructing the augmented internal state without using any knowledge of system. Then, by establishing a novel iterative learning equation which requires no repeated finite window integral operations, an adaptive dynamic programming based learning algorithm with employing value-iteration scheme is proposed to estimate the optimal feedback control gain, which may lead to a reduction of the computational load. The analysis on solvability and convergence shows that the estimated control gain converges to the optimal control gain. Finally, a physical experiment on control of an unmanned quadrotor illustrates the effectiveness of the proposed algorithm.

Data-driven optimal output regulation for unknown linear discrete-time systems based on parameterization approach

The output regulation problem has been studied based on a parameterization approach. Different from existing literature, the proposed method does not rely on prior knowledge of the system dynamics. Instead, it leverages state and input data to address the absence of information regarding unmodeled dynamics. Firstly, the output regulation problem is transformed into a stabilization problem by using coordination transformation. The feedback control gain is computed directly by solving an optimization problem using input and state data. The feedforward control gain is obtained by using data-based solutions of regulator equations. Secondly, the design of a dynamic feedback controller is also explored within the framework of a data-driven strategy. Thirdly, two algorithms corresponding to the optimal and dynamic data-driven output regulation problems are developed to implement the proposed data-driven method. Finally, a simulation example is carried out to illustrate the effectiveness of the developed data-driven approach.

Model‐Free Cooperative Optimal Output Regulation for Linear Discrete‐Time Multi‐Agent Systems Using Reinforcement Learning

In this paper, an off‐policy model‐free algorithm is presented for solving the cooperative optimal output regulation problem for linear discrete‐time multi‐agent systems. First, an adaptive distributed observer is designed for each follower to estimate the leader’s information. Then, a distributed feedback‐feedforward controller is developed for each follower to solve the cooperative optimal output regulation problem utilizing the follower’s state information and the adaptive distributed observer. Based on the reinforcement learning method, an adaptive algorithm is presented to find the optimal feedback gains via online data collection from system trajectory. By designing a Sylvester map, the solution to the regulator equations is calculated via data collected from the optimal feedback gain design steps, and the feedforward control gain is found. Finally, an off‐policy model‐free algorithm is proposed to design the distributed feedback‐feedforward controller for each follower to solve the cooperative optimal output regulation problem. A numerical example is given to verify the effectiveness of this proposed approach.

Observer-Based Adaptive Inverse Optimal Output Regulation for a Class of Uncertain Nonlinear Systems

This paper addresses the adaptive inverse optimal output regulation problem for a class of uncertain nonlinear systems driven by an exosystem. The unknown parameters, internal disturbances, and unmeasured states are contained in the nonlinear system. Firstly, the output regulation problem is decomposed into a feedforward control design problem which can be solved by the internal model based on the output regulation theory, and an adaptive inverse optimal stabilization problem. Then an auxiliary system is designed, and a new state observer related to the auxiliary system is given. By combining adaptive control technology and inverse optimal control method, a novel adaptive output feedback inverse optimal controller is developed to make the output of the system track the reference signal fast. With this control strategy, all the signals of the closed-loop system are uniformly ultimately bounded (UUB), and the newly well-defined cost functional which is connected with the auxiliary system and the controller can be minimized. Finally, a simulation case is put forward to verify the feasibility of the newly raised controller and the state observer.

Optimal containment preview control for continuous-time multi-agent systems using internal model principle

ABSTRACT Optimal containment preview analysis and distributed design for multi-agent systems with general continuous-time linear dynamics and directed acyclic communication topology are considered. Firstly, the state augmentation technique and topology reconstruction method are applied to formulate the original problem as a set of internal-model based local optimal output regulation problem. Secondly, the linear superposition principle is employed to obtain the preview feed-forward compensations corresponding to multiple leaders. Thirdly, by expressing each preview compensation term as a form about the current state of the leader, the solvability of the local optimal output regulation problem, as well as the optimal regulation problems, are proved. On this basis, the sufficient conditions and the distributed optimal control strategy are derived, which ensure the convergence of the containment preview problem. Finally, the effectiveness of the distributed design is illustrated by a numerical experiment.

Reinforcement learning‐based robust optimal output regulation for constrained nonlinear systems with static and dynamic uncertainties

AbstractThis article investigates the robust optimal output regulation problem for the constrained uncertain nonlinear systems. A two‐step design framework is proposed to overcome the difficulties brought by uncertainties, constraint and performance optimization. First, the feedforward controller is designed by the internal model principle, and the robust output regulation problem is transformed into the robust stabilization problem. Then, the optimal control problem with input saturation is further considered in the robust feedback controller design process. With the help of a non‐quadratic cost functional, actor‐critic algorithm and robust redesign technique are brought together to design the constrained robust optimal feedback controller. Finally, stability analysis based on Lyapunov method shows that all the signals of the closed‐loop system remain bounded, and the tracking error is uniformly ultimately bounded with arbitrarily small ultimate bound. Simulation results illustrate the effectiveness of the proposed methodology.

Optimal Output Regulation for General Linear Systems via Adaptive Dynamic Programming.

In this article, we consider an adaptive optimal output regulation problem for general linear systems. The purpose of the optimal output regulation problem is to guarantee the stability of the closed-loop system and disturbance rejection, as well by minimizing some predefined performance indices. It can be realized by using an optimal controller, in which both optimal feedback control gain and optimal feedforward control gain are included. First, an adaptive dynamic programming (ADP) technique is used to solve the optimal feedback control gain. Next, the unknown system matrices of the plant are explicitly computed. In addition, based on the property of the minimal polynomial, the coefficient of the exogenous disturbance in the expression of the regulated output can also be calculated. Finally, according to the regulator equation, an extra cost function is given, which aims to obtain the optimal feedforward control gain. The linear vector space optimization methods are used to solve the optimal problem. As a result, the linear optimal output regulation problem can be solved by the approximately optimal feedback and feedforward control gains.

Observer-Based Finite-Time Inverse Optimal Output Regulation for Uncertain Nonlinear Systems

This paper deals with the finite-time inverse optimal output regulation problem for a class of uncertain nonlinear systems that are subjected to an exosystem. The nonlinear systems not only contain external disturbances in the exosystem but also take the unknown nonlinear functions and unmeasured states into account. The output regulation problem is first converted into a stabilization problem through the internal model. Then, fuzzy logic systems (FLSs) are employed to approximate the unknown nonlinear functions. An auxiliary system is constructed and, based on the auxiliary system, a fuzzy state observer is designed to estimate the unmeasured states. Furthermore, a novel adaptive fuzzy finite-time inverse optimal output feedback controller and adaptive law are designed by combining backstepping technology, adaptive control technology, finite-time stability theory, and the inverse optimal control method. The control algorithm ensures that all the signals of the closed-loop system are semiglobally practical finite-time stable (SGPFS), so the newly well-defined cost function that is connected with the auxiliary system can be minimized. Finally, the validity of the approach is confirmed by virtue of the simulation results.

Reinforcement Learning-Based Cooperative Optimal Output Regulation via Distributed Adaptive Internal Model.

In this article, a data-driven distributed control method is proposed to solve the cooperative optimal output regulation problem of leader-follower multiagent systems. Different from traditional studies on cooperative output regulation, a distributed adaptive internal model is originally developed, which includes a distributed internal model and a distributed observer to estimate the leader's dynamics. Without relying on the dynamics of multiagent systems, we have proposed two reinforcement learning algorithms, policy iteration and value iteration, to learn the optimal controller through online input and state data, and estimated values of the leader's state. By combining these methods, we have established a basis for connecting data-distributed control methods with adaptive dynamic programming approaches in general since these are the theoretical foundation from which they are built.

Optimal output regulation for unknown continuous-time linear systems by internal model and adaptive dynamic programming

This paper addresses an optimal output regulation problem for linear time-invariant systems with unknown dynamics. First, a new augmented virtual system is designed to replace the original system and the internal model. Then, by incorporating the data from the augmented virtual system with adaptive dynamic programming (ADP) method, a new iterative learning equation without requiring integral operations or constructing internal model in advance is proposed for establishing the data-driven learning algorithm. Compared with existing ADP-based learning algorithms for linear continuous-time systems, the proposed learning algorithm relaxes both requirements on recording the complete continuous data and setting an initial stabilizing control policy. Finally, the effectiveness of the proposed algorithm is illustrated by an example.

Data-driven adaptive-critic optimal output regulation towards water level control of boiler-turbine systems

This paper considers a new adaptive-critic optimal output regulation scheme for continuous-time general linear systems with unknown dynamics and unmeasurable disturbance. The objective of researching the optimal output regulation problem is pursuing the stability of the closed-loop system while enabling the output to optimally track the reference signal with disturbance rejection. The problem is solved by employing an approximate optimal regulator that consist of optimal feedback control obtained by a data-driven learning algorithm and optimal feedforward control achieved by solving regulator equation. The missing system matrices of the plant are exactly figured out by employing state/input data. Moreover, by employing the minimal polynomial of the exosystem matrix, the parameter of interference in the output equation is available. Then, a predefined cost function is presented based on the regulator equation that aims to design the optimal control law. The stability analysis demonstrate that the presented control scheme can stabilize the closed-loop system, meanwhile the output asymptotically tracks the reference signal. Finally, the effectiveness of the developed optimal control scheme is validated through an application towards the drum water level control of boiler-turbine systems.

Optimal output tracking control of linear discrete-time systems with unknown dynamics by adaptive dynamic programming and output feedback

This paper discusses the output-feedback-based model-free optimal output tracking control problem of discrete-time systems with completely unknown system models under mild assumptions. The only information that allows utilisation is the system output and the reference output. To overcome these challenges, the paper aims to solve a model-free optimal output regulation problem for achieving optimal output tracking control; solving an optimal output regulation problem is equivalent to solving a linear quadratic regulation (LQR) problem and a constrained static optimisation problem. The state reconstruction is first given to represent the system state in terms of the input and output sequences. A data-driven value iteration (VI) algorithm is then proposed to iteratively approximate the solution to the discrete-time algebraic Riccati equation (ARE) of the corresponding LQR problem on the basis of input and output data. Next, based on the iterative solutions with respect to the ARE, a model-free solution is provided for the corresponding constrained static optimisation problem. Finally, an alternative reference system equivalent to the original reference system is established to avoid the requirement of having the knowledge of the reference system dynamics and the reference state. A numerical example is employed to demonstrate the effectiveness of the proposed control scheme.

Resilient reinforcement learning and robust output regulation under denial-of-service attacks

In this paper, we have proposed a novel resilient reinforcement learning approach for solving robust optimal output regulation problems of a class of partially linear systems under both dynamic uncertainties and denial-of-service attacks. Fundamentally different from existing works on reinforcement learning, the proposed approach rigorously analyzes both the resilience of closed-loop systems against attacks and the robustness against dynamic uncertainties. Moreover, we have proposed an original successive approximation approach, named hybrid iteration, to learn the robust optimal control policy, that converges faster than value iteration, and is independent of an initial admissible controller. Simulation results demonstrate the efficacy of the proposed approach.

H∞optimal preview tracking control problem with disturbance attenuation

AbstractIn this paper, theH∞optimal preview tracking control problem of continuous‐time linear system with external disturbance is studied, where the reference signal is previewed and its amplitude does not decay. First, an augmented system with error vector, original system state derivative, and internal model state derivative is constructed, which transforms the tracking problem into theH∞optimal output regulation problem. Second, theH∞optimal control method is used to design the controller, and the output regulation of the augmented system is achieved by finding the upper bound of the discount factor. Then the optimal controller, including error integral, preview feed‐forward compensation, and internal model compensation, is obtained for the original problem, and meanwhile the sufficient conditions are also provided to guarantee the existence of the controller. Finally, both theoretical and simulation results show that under the proposed controller, the closed‐loop system can achieve asymptotic tracking and has robustness against the disturbance.

Value iteration and adaptive optimal output regulation with assured convergence rate

In this paper, we investigate the learning-based adaptive optimal output regulation problem with convergence rate requirement for disturbed linear continuous-time systems. An adaptive optimal control approach is proposed based on reinforcement learning and adaptive dynamic programming to learn the optimal regulator with assured convergence rate. The above-mentioned problem is successfully solved by tackling a static optimization problem to find the optimal solution to the regulator equations, and a dynamic and constrained optimization problem to obtain the optimal feedback control gain. Without requiring on the accurate system dynamics or a stabilizing feedback control gain, a novel online value iteration algorithm is proposed, which can learn both the optimal feedback control gain and the corresponding feedforward control gain using measurable data. Moreover, the output of the closed-loop system is guaranteed to converge faster or equal to a predefined convergence rate set by user. Finally, the numerical analysis on a LCL coupled inverter-based distributed generation system shows that the proposed approach can achieve desired disturbance rejection and tracking performance.

Adaptive optimal output regulation of linear discrete-time systems based on event-triggered output-feedback

This paper presents novel event-triggered control approaches to solve the adaptive optimal output regulation problem for a class of linear discrete-time systems. Different from most existing research on output regulation problems, the developed adaptive optimal control approaches are based on (1) output-feedback instead of full-state or partial-state feedback, (2) adaptive dynamic programming (ADP) which provides approximate solutions of the optimal control problem without requiring the precise knowledge of the plant dynamics, and (3) an event-triggering mechanism that reduces the communication between the controller and the plant. It is shown that the system in closed-loop with the developed controllers is asymptotically stable at an equilibrium of interest, and the tracking errors asymptotically converge to zero. Moreover, the suboptimality of the closed-loop system is directly determined by the relative threshold, which is a ratio between the triggering threshold and the actual state. A numerical simulation example is employed to verify the effectiveness of the proposed methodologies.

Heterogeneous formation control of multiple rotorcrafts with unknown dynamics by reinforcement learning

In this paper, a distributed model-free solution based on reinforcement learning is proposed for the leader–follower formation control problem of heterogeneous multi-agent systems. The multi-agent system consists of multiple rotorcrafts involving a virtual leader and multiple followers, where the dynamics of leaders and followers is unknown. The formation control problem is firstly formulated as an optimal output regulation problem. A discounted performance function is then introduced to guarantee that the tracking error asymptotically converges to zero, and an online off-policy reinforcement learning algorithm is proposed to solve the optimal output problem online using the data generated along the trajectories of the agents. A simulation example is provided to validate the effectiveness of the proposed control method.