Linear Quadratic Problem Research Articles

Space manipulator optimal impedance control using integral reinforcement learning

This paper examines the optimal impedance control problem for large-scale space manipulator operational tasks with unknown contact dynamics and partial measurements. More specifically, by quantifying the interaction performance using a discounted value function, the optimal impedance control problem is tactfully transformed into a linear quadratic tracking problem. By resorting to the historical inputs and outputs, an improved state reconstruction method is presented, which obviates the velocity measurement. Unlike the existing state reconstruction method for continuous-time systems, the estimation bias caused by probing noise is completely eliminated under the improved state reconstruction method. Based on this, a novel model-free value iteration integral reinforcement learning algorithm is developed to approximate optimal impedance parameters. Compared with the earlier integral reinforcement learning algorithms, the proposed algorithm not only averts any prior contact dynamics knowledge and full-state measurements, but also eliminates the heavy dependence on the specific initial stabilization control. In addition, the implementation and convergence of the proposed algorithm are discussed successively. Finally, numerical simulations verify the effectiveness of the theoretical results.

Efficient Learning of a Linear Dynamical System With Stability Guarantees

We propose a principled method for projecting an arbitrary square matrix to the non-convex set of asymptotically stable matrices. Leveraging ideas from large deviations theory, we show that this projection is optimal in an information-theoretic sense and that it simply amounts to shifting the initial matrix by an optimal linear quadratic feedback gain, which can be computed exactly and highly efficiently by solving a standard linear quadratic regulator problem. The proposed approach allows us to learn the system matrix of a stable linear dynamical system from a single trajectory of correlated state observations. The resulting estimator is guaranteed to be stable and offers statistical bounds on the estimation error.

Near-Optimal Design of Safe Output-Feedback Controllers From Noisy Data

As we transition towards the deployment of data-driven controllers for black-box cyberphysical systems, complying with hard safety constraints becomes a primary concern. Two key aspects should be addressed when input-output data are corrupted by noise: how much uncertainty can one tolerate without compromising safety, and to what extent is the control performance affected? By focusing on finite-horizon constrained linear-quadratic problems, we provide an answer to these questions in terms of the model mismatch incurred during a preliminary identification phase. We propose a control design procedure based on a quasiconvex relaxation of the original robust problem and we prove that, if the uncertainty is sufficiently small, the synthesized controller is safe and near-optimal, in the sense that the suboptimality gap increases <i>linearly</i> with the model mismatch level. Since the proposed method is independent of the specific identification procedure, our analysis holds in combination with state-of-the-art behavioral estimators beyond standard least-squares. The main theoretical results are validated by numerical experiments.

An addendum to the problem of zero-sum LQ stochastic mean-field dynamic games

In this paper, we first address a linear quadratic mean-field game problem with a leader–follower structure. By adopting a Riccati-type approach, we show how one can obtain a state-feedback representation of the pairs of strategies which achieve an open-loop Stackelberg equilibrium in terms of the global solutions of a system of coupled matrix differential Riccati-type equations. In the second part of this paper, we obtain necessary and sufficient conditions for the solvability of the involved coupled generalized Riccati equations.

Robust Output Regulation and Reinforcement Learning-Based Output Tracking Design for Unknown Linear Discrete-Time Systems

In this article, we investigate the optimal output tracking problem for linear discrete-time systems with unknown dynamics using reinforcement learning (RL) and robust output regulation theory. This output tracking problem only allows to utilize the outputs of the reference system and the controlled system, rather than their states, and differs from most existing works that depend on the state of the system. The optimal tracking problem is formulated into a linear quadratic regulation problem by proposing a family of dynamic discrete-time controllers. Then, it is shown that solving the output tracking problem is equivalent to solving output regulation equations, whose solution, however, requires the knowledge of the complete and accurate system dynamics. To remove such a requirement, an off-policy RL algorithm is proposed using only the measured output data along the trajectory of the system and the reference output. By introducing reexpression error and analyzing the rank condition of the parameterization matrix, we ensure the uniqueness of the proposed RL-based optimal control via output feedback.

Maximum Entropy Optimal Control of Continuous-Time Dynamical Systems

Maximum entropy reinforcement learning methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of the existing techniques are designed for discrete-time systems although there has been a growing interest to handle physical processes evolving in continuous time. As a first step toward their extension to continuous-time systems, this article aims to study the theory of maximum entropy optimal control in continuous time. Applying the dynamic programming principle, we derive a novel class of Hamilton–Jacobi–Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. We further show that the optimal control is uniquely characterized as Gaussian in the case of control-affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. The results of our numerical experiments demonstrate the performance of our maximum entropy method in continuous-time optimal control and reinforcement learning problems.

Time-Domain Decomposition for Mixed-Integer Optimal Control Problems

We consider mixed-integer optimal control problems, whose optimality conditions involve global combinatorial optimization aspects for the corresponding Hamiltonian pointwise in time. We propose a time-domain decomposition, which makes this problem class accessible for mixed-integer programming using parallel-in-time direct discretizations. The approach is based on a decomposition of the optimality system and the interpretation of the resulting subproblems as suitably chosen mixed-integer optimal control problems on subintervals in time. An iterative procedure then ensures continuity of the states at the boundaries of the subintervals via co-state information encoded in virtual controls. We prove convergence of this iterative scheme for discrete-continuous linear-quadratic problems and present numerical results both for linear-quadratic as well as nonlinear problems.

Optimal control and stabilization for linear mean‐field system with indefinite quadratic cost functional

AbstractIn this paper, we consider an optimal control and stabilization problem of linear mean‐field (MF) system, where the quadratic cost functional is allowed to be indefinite. Inspired by the equivalent cost functional method, we introduce a subset, which helps us to investigate the convergence property of generalized differential Riccati equations arising in indefinite mean‐field linear quadratic (MF‐LQ) problems with finite horizon. A coupled generalized algebraic Riccati equation (GARE) is thus obtained. More importantly, the solution pair of corresponding GARE can be decomposed into a solution pair of a coupled standard algebraic Riccati equation (SARE) and a matrix pair in the subset we introduce. Thus, an equivalent relationship is established between GARE and SARE. With this equivalence, we derive necessary and sufficient conditions to stabilize linear MF‐system with indefinite weighting matrices in mean‐square sense.

Optimal Distributed ADMM-Based Control for Frequency Synchronization in Isolated AC Microgrids

To restrict transient dynamics of each distributed generator (DG) in isolated AC micro-grids (MGs), an optimal distributed alternating direction method of multiplier (ADMM)-based control scheme, will be explored for achieving frequency synchronization within a given finite horizon. By introducing the consensus variable, we re-phrase the frequency synchronization problem as a linear quadratic tracking problem under the framework of multi-agent systems. The objective function of each individual DG is defined as the accumulation of quadratic frequency errors and quadratic inputs. Since the consensus variable is also part of the decision variable, more flexibility can be gained in developing the optimal distributed ADMM-based algorithm within a finite horizon. Since the direct extension of ADMM for static optimization can not be fully distributed, a fully distributed ADMM-based control is developed by exploring the optimal distributed control theory in each DG. To validate the performance of the proposed method, real-time simulations on OPAL-RT are conducted to validate the effectiveness of the proposed optimal distributed ADMM-based control strategy even under large load variations and plug-and-play operations of DGs.

Optimal feedback controls of stochastic linear quadratic control problems in infinite dimensions with random coefficients

It is a longstanding unsolved problem to characterize the optimal feedback controls for general linear quadratic optimal control problem of stochastic evolution equation with random coefficients. A solution to this problem is given in [22] under some assumptions which can be verified for interesting concrete models, such as controlled stochastic wave equations, controlled stochastic Schrödinger equations, etc. More precisely, the authors establish the equivalence between the existence of an optimal feedback operator and the solvability of the corresponding operator-valued, backward stochastic Riccati equations. However, their result cannot cover some important stochastic partial differential equations, such as stochastic heat equations, stochastic stokes equations, etc. A key contribution of the current work is to relax the C0-group assumption of unbounded linear operator A in [22] and using the contraction semigroup assumption instead. Therefore, our result can be well applicable in the linear quadratic problem of stochastic parabolic equations. To this end, we introduce a suitable notion to the aforementioned Riccati equation, and some delicate techniques which are even new in the finite dimensional case.

Least squares parameter estimation and multi-innovation least squares methods for linear fitting problems from noisy data

Least squares is an important method for solving linear fitting problems and quadratic optimization problems. This paper explores the properties of the least squares methods and the multi-innovation least squares methods. It demonstrates lemmas and theorems about the least squares and multi-innovation least squares parameter estimation algorithms after reviewing and surveying some important contributions in the area of system identification, such as the auxiliary model identification idea, the multi-innovation identification theory, the hierarchical identification principle, the coupling identification concept and the filtering identification idea. The results of the least squares and multi-innovation least squares algorithms for linear regressive systems with white noises can be extended to other systems with colored noises.

On the Discrete-Time Minimum Principle in Multiple-Mode Systems

This paper presents a robust version of the discrete-time minimum principle for uncertain systems. The uncertainty enters the dynamics of the system by an unknown value that belongs to a finite set. Each element of such a set represents a possible dynamic realization of the discrete-time trajectory. Following a mathematical programming approach, we reformulate the problem to obtain general necessary conditions that allow finding the worst-case optimality in the form of a weighted sum. These conditions are then applied to the time-varying linear affine quadratic problem with multiple modes and a general cost, which includes tracking signals in both: the state and the control variables. We illustrate our result with an application to a production-inventory control problem with uncertain dynamics.

Linear pencils and quadratic programming problems with a quadratic constraint

Given bounded selfadjoint operators A and B acting on a Hilbert space H, consider the linear pencil P(λ)=A+λB, λ∈R. The set of parameters λ such that P(λ) is a positive (semi)definite operator is characterized. These results are applied to solving a quadratic programming problem with an equality quadratic constraint (or a QP1EQC problem).

Linear Quadratic Control Using Model-Free Reinforcement Learning

In this article, we consider linear quadratic (LQ) control problem with process and measurement noises. We analyze the LQ problem in terms of the average cost and the structure of the value function. We assume that the dynamics of the linear system is unknown and only noisy measurements of the state variable are available. Using noisy measurements of the state variable, we propose two model-free iterative algorithms to solve the LQ problem. The proposed algorithms are variants of policy iteration routine where the policy is greedy with respect to the average of all previous iterations. We rigorously analyze the properties of the proposed algorithms, including stability of the generated controllers and convergence. We analyze the effect of measurement noise on the performance of the proposed algorithms, the classical off-policy, and the classical <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -learning routines. We also investigate a model-building approach, inspired by adaptive control, where a model of the dynamical system is estimated and the optimal control problem is solved assuming that the estimated model is the true model. We use a benchmark to evaluate and compare our proposed algorithms with the classical off-policy, the classical <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -learning, and the policy gradient. We show that our model-building approach performs nearly identical to the analytical solution and our proposed policy iteration-based algorithms outperform the classical off-policy and the classical <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -learning algorithms on this benchmark but do not outperform the model-building approach.

A Game—Theoretic Model for a Stochastic Linear Quadratic Tracking Problem

In this paper, we solve a stochastic linear quadratic tracking problem. The controlled dynamical system is modeled by a system of linear Itô differential equations subject to jump Markov perturbations. We consider the case when there are two decision-makers and each of them wants to minimize the deviation of a preferential output of the controlled dynamical system from a given reference signal. We assume that the two decision-makers do not cooperate. Under these conditions, we state the considered tracking problem as a problem of finding a Nash equilibrium strategy for a stochastic differential game. Explicit formulae of a Nash equilibrium strategy are provided. To this end, we use the solutions of two given terminal value problems (TVPs). The first TVP is associated with a hybrid system formed by two backward nonlinear differential equations coupled by two algebraic nonlinear equations. The second TVP is associated with a hybrid system formed by two backward linear differential equations coupled by two algebraic linear equations.

Closed-form and robust expressions for data-driven LQ control

This article provides an overview of certain direct data-driven control results, where control sequences are computed from (noisy) data collected during offline control experiments without an explicit identification of the system dynamics. For the case of noiseless datasets, we derive several closed-form data-driven expressions that solve a variety of optimal control problems for linear systems with quadratic cost functions of the state and input (including the linear quadratic regulator problem, the minimum energy control problem, and the linear quadratic control problem with terminal constraints), discuss their advantages and drawbacks with respect to alternative data-driven and model-based approaches, and showcase their effectiveness through a number of numerical studies. Interestingly, these results provide an alternative and explicit way of solving classic control problems that, for instance, does not require the solution of an implicit and recursive Riccati equation as in the model-based setting. For the case of noisy datasets, we show how the closed-form expressions derived in the noiseless setting can be modified to compensate for the bias induced by noise, and perform a sensitivity analysis to reveal favorable asymptotic robustness properties of the derived data-driven controls. We conclude the paper with some considerations and a discussion of outstanding questions and directions of future investigation.

Optimal exploration strategies for finite horizon regret minimization in some adaptive control problems

In this work, we consider the problem of regret minimization in adaptive minimum variance and linear quadratic control problems. Regret minimization has been extensively studied in the literature for both types of adaptive control problems. Most of these works give results of the optimal rate of the regret in the asymptotic regime. In the minimum variance case, the optimal asymptotic rate for the regret is log(T) which can be reached without any additional external excitation. On the contrary, for most adaptive linear quadratic problems, it is necessary to add an external excitation in order to get the optimal asymptotic rate of √T. In this paper, we will actually show from a theoretical study, as well as, in simulations that when the control horizon is pre-specified a lower regret can be obtained with either no external excitation or a new exploration type termed immediate.

Решение линейно-квадратичных задач в дискретно-непрерывном формате с внешними воздействиями

Two linear-quadratic problems are considered on the set of piecewise-constant controls. The first problem contains a discrete perturbation on the right side of the controlled system and uncertain parameters in a quadratic functional with sign-indefinite matrices. Its solution is obtained by the guaranteed result rule and is implemented in the form of a finite-dimensional minimax problem. There are obtained conditions for the parameters that convert the objective function to a convex-concave structure and give the possibility of an effective solving of the problem. These are linear inequalities containing extreme eigenvalues of symmetric matrices. The second problem is related to the functional in the discrete variant, which is defined as the deviation of the phase trajectory from consecutive time realizations of the external influence. It gives the opportunity of the step by step searching of the extremmum at each node point of the time interval. Local problems can be effectively solved in a finite number of iterations.

Stochastic linear quadratic optimal tracking control for discrete-time systems with delays based on Q-learning algorithm

&lt;abstract&gt;&lt;p&gt;In this paper, a reinforcement Q-learning method based on value iteration (Ⅵ) is proposed for a class of model-free stochastic linear quadratic (SLQ) optimal tracking problem with time delay. Compared with the traditional reinforcement learning method, Q-learning method avoids the need for accurate system model. Firstly, the delay operator is introduced to construct a novel augmented system composed of the original system and the command generator. Secondly, the SLQ optimal tracking problem is transformed into a deterministic one by system transformation and the corresponding Q function of SLQ optimal tracking control is derived. Based on this, Q-learning algorithm is proposed and its convergence is proved. Finally, a simulation example shows the effectiveness of the proposed algorithm.&lt;/p&gt;&lt;/abstract&gt;

Optimal Control for a Multistage Uncertain Random System

Chance theory is a mathematical methodology for modelling complex systems including both uncertainty and randomness. Based on chance theory, this paper introduces the optimal control model for a multistage uncertain random system. For solving such a model, recurrence equations are presented via Bellman’s principle. With the aid of recurrence equations, the bang-bang problem of optimal control model and indefinite linear quadratic (LQ) problem of optimal control model are studied. Meanwhile, the exact solutions of such problems are presented. Furthermore, numerical examples are provided on the proposed theorems to show the effectiveness of our results.