Kleinman Algorithm Research Articles

Solving Infinite-Dimensional Harmonic Lyapunov and Riccati Equations

In this paper, we address the problem of solving infinite-dimensional harmonic algebraic Lyapunov and Riccati equations up to an arbitrary small error. This question is of major practical importance for analysis and stabilization of periodic systems including tracking of periodic trajectories. We first give a closed form of a Floquet factorization in the general setting of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L^{2}$</tex-math></inline-formula> matrix functions and study the spectral properties of infinite-dimensional harmonic matrices and their truncated version. This spectral study allows us to propose a generic and numerically efficient algorithm to solve infinite-dimensional harmonic algebraic Lyapunov equations up to an arbitrary small error. We combine this algorithm with the Kleinman algorithm to solve infinite-dimensional harmonic Riccati equations and we apply the proposed results to the design of a harmonic LQ control with periodic trajectory tracking.

Reinforcement Learning-Based Near Optimization for Continuous-Time Markov Jump Singularly Perturbed Systems

The design of a suboptimal controller for continuous-time Markov jump singularly perturbed systems with partially unknown dynamics is studied in this paper. With fast and slow decomposition technique, the original Markov jump singularly perturbed systems are decomposed into fast and slow subsystems as a new attempt. On this basis, an offline parallel Kleinman algorithm and an online parallel integral reinforcement learning algorithm are presented to cope with the different subsystems, respectively. Meanwhile, the controllers obtained by the above two algorithms are used to design the suboptimal controllers for original systems. Furthermore, the suboptimality of the proposed controllers is also discussed. Finally, an example of the electric circuit model is shown to illustrate the applicability of the proposed method.

Q-Learning for Continuous-Time Linear Systems: A Data-Driven Implementation of the Kleinman Algorithm

A data-driven strategy to estimate the optimal feedback and the value function in an infinite-horizon, continuous-time, linear-quadratic optimal control problem for an unknown system is proposed. The method permits the construction of the optimal policy without any knowledge of the model, without requiring that the time derivatives of the state are available for the design, and without even assuming that an initial stabilizing feedback policy is available. Two alternative architectures are discussed: the first scheme revolves around the periodic computation of some matrix inversions involving the Q-function, whereas the second approach relies on a purely continuous-time implementation of some dynamic systems whose trajectories are uniformly attracted by the solutions to the above algebraic equations. Interestingly, the proposed strategy essentially constitutes a (direct) data-driven implementation of the celebrated Kleinman algorithm, hence subsuming the particularly appealing features of the latter, such as quadratic monotone convergence to the optimal solution. The theory is then validated by the means of practically motivated applications.

Reinforcement Learning Based Optimal Control of Linear Singularly Perturbed Systems

This brief studies the optimal control problem of linear singularly perturbed systems (SPSs) via reinforcement learning (RL). We first present an offline model-based algorithm on the basis of Kleinman algorithm to solve the underlying algebraic Riccati equation with singular perturbation parameter and its convergence is proven. This revised version of Kleinman algorithm is the key to develop the subsequent learning algorithm. Then, by means of two time-scale characteristics of SPSs, we present a novel model-free learning algorithm relating to the online measurement of state and input to design the optimal controller. Compared with the existing learning approaches, the obtained RL algorithm is free of ill-conditioned numerical issues caused by the co-existence of fast and slow modes in SPSs and thus is more robust with respect to the computation and measurement errors. Finally, we verify the effectiveness of the developed results by a simulation example.

Reconfigurability evaluation for disturbance rejection control systems under actuator outages

This work aims to develop a quantitative reconfigurability evaluation method for control systems with disturbances in the case of actuator outages. The basic idea of this method is to investigate and quantify the performance of the nominal system under the condition that the specified faults are recoverable. We propose a reconfigurability index based on H2 performance that simultaneously considers both the normal scenario and various fault scenarios. A modified Newton–Kleinman algorithm is used to calculate the evaluation index. Guided by the result of the reconfigurability evaluation, the system structure is optimized to achieve a tradeoff between the control performance under normal conditions and the reconfiguration performance under the fault conditions. Finally, the effectiveness of the proposed method is illustrated through two application examples, namely, aircraft and satellite attitude control systems.

Policy Iteration Algorithm for Singular Controlled Diffusion Processes

In this paper, the infinite horizon optimal control problems for singular diffusion processes are considered from the viewpoints of Markov decision processes and perturbation analysis, where the singularity of diffusion means that the covariance matrix of the system noise is allowed to be degenerate. A formula of performance difference under two different controls is derived and leads to a comparison theorem. By the comparison theorem, starting from a control, a so-called better control can be selected. Therefore, a control policy iteration algorithm is developed, by which the performance improves step by step and converges to the optimal one. When this applies to the stochastic affine nonlinear regulator and stochastic linear quadratic optimal control problems, better control can be constructed in a closed form. It is also shown that when the considered stochastic systems degenerate to the deterministic ones, the proposed algorithm reduces to the adaptive dynamic programming algorithm [J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, Adaptive dynamic programming, IEEE Trans. Systems Man Cybernet., 32 (2002), pp. 140--153] for the affine nonlinear systems and to the well-known Kleinman algorithm [D. L. Kleinman, On an iterative technique for Riccati equation computation, IEEE Trans. Automat. Control, 13 (1968), pp. 114--115] for the linear quadratic optimal control problem.

Numerical Computation for H∞ State Feedback Control of Large-Scale Systems

In this paper, H∞ state feedback control for large-scale systems is investigated. The attention is focused on the design of the high-order approximate H∞ controller which is based on the numerical solutions. The novel contribution is that a new iterative algorithm is derived by combining the Kleinman algorithm and the fixed point algorithm. As a result, H∞ controller can be constructed in the same dimension of the subsystems. It is newly proved that the proposed algorithm has the quadratic convergence. Moreover, it is shown that the resulting controller achieves O(E2k) approximation of the optimal H∞ cost.

Asymptotic Expansions and a New Numerical Algorithm of the Algebraic Riccati Equation for Multiparameter Singularly Perturbed Systems

In this paper we study a continuous-time multiparameter algebraic Riccati equation (MARE) with an indefinite sign quadratic term. The existence of a unique and bounded solution of the MARE is newly established. We show that the Kleinman algorithm can be used to solve the sign indefinite MARE. The proof of the convergence and the existence of the unique solution of the Kleinman algorithm is done by using the Newton–Kantorovich theorem. Furthermore, we present new algorithms for solving the generalized multiparameter algebraic Lyapunov equation (GMALE) by means of the fixed-point algorithm.

A revised Kleinman algorithm to solve algebraic Riccati equation of singularly perturbed systems

In this paper, we show that the Kleinman algorithm can be used well to solve the algebraic Riccati equation (ARE) of singularly perturbed systems, where the quadratic term of the ARE may be indefinite. The quadratic convergence property of the Kleinman algorithm is proved by using the Newton–Kantorovich theorem when the initial condition is chosen appropriately. In addition, the numerical method to solve the generalized algebraic Lyapunov equation (GALE) appearing in the Kleinman algorithm is given.

The Guaranteed Cost Control Problem of Uncertain Singularly Perturbed Systems

In this paper we study the algebraic Riccati equation corresponding to the guaranteed cost control theory for an uncertain singularly perturbed system. The construction of the controller involves solving the full-order algebraic Riccati equation with small parameter ε. Under control-oriented assumptions, we first provide the sufficient conditions such that the full-order algebraic Riccati equation has a positive semi-definite stabilizing solution. Next we propose an iterative algorithm based on the Kleinman algorithm to solve the algebraic Riccati equation which depends on the parameter ε. Our new idea is to use the solutions of the reduced-order algebraic Riccati equations for the initial condition. By using the iterative algorithm, we can easily obtain a required solution of the algebraic Riccati equation. Moreover, using the initial conditions without ε, we show that there exists an ε̃ such that the proposed algorithm has quadratic convergence. Finally, in order to show the effectiveness of the proposed algorithm, numerical examples are included.

Existence and Numerical Solution for Norm-Type Algebraic Riccati Equations

This paper considers a continuous algebraic Riccati equation with an additional norm-type term. This equation is called a norm-type algebraic Riccati equation and arises in guaranteed cost control on the basis of a norm-type upper bound. It is shown that there exists a solution under the stabilizability condition and the assumption that the additional term is not too large. Also, the well-known Kleinman's algorithm is extended to this case.

The difference periodic Ricati equation for the periodic prediction problem

Gives a comprehensive treatment of several important aspects of the discrete-time periodic Riccati equation (DPRE) arising from the prediction problem for linear discrete-time periodic systems. The authors analyze the symmetric periodic positive semidefinite (SPPS) solution of the DPRE under appropriate assumptions of stabilizability and detectability of the periodic system. Among the results obtained are necessary and sufficient conditions for the existence and uniqueness of the SPPS solution and the stability of the resulting closed-loop system. Some of these results can be seen as extensions of the corresponding results for the time-invariant case; however, a number of them contain contributions to the time-invariant case as well. The paper also gives a numerical algorithm based on an iterative linearization procedure for computing the SPPS solution. The algorithm is a periodic version of Kleinman's algorithm for the time-invariant case. >

A lower bound for the solution of the algebraic Riccati equation of optimal control and a geometric convergence rate for the Kleinman algorithm

A new sharp lower bound for the solution of the algebraic Riccati equation of optimal control is presented for the case when the state cost matrix Q is positive definite. The bound is easier to evaluate than previous sharp bounds. It is then used in the derivation of a geometric convergence rate for the Kleinman algorithm (an iterative method for solving the algebraic Riccati equation).

Iterative techniques for Riccati game equations

We present a monotone iterative technique for the computation of a solution of a Riccati-type equation relevant to the theory of differential games. For this purpose, we show that the Kleinman algorithm for Riccati equation computations converges under extremely general conditions.