Linear Quadratic Regulator Problem Research Articles

Inferring Human Control Intent Using Inverse Linear Quadratic Regulator With Output Penalty Versus Gain Penalty: Better Fit but Similar Intent

Abstract To identify the underlying mechanisms of human motor control, parametric models are utilized. One approach of employing these models is the inferring the control intent (estimating motor control strategy). A well-accepted assumption is that human motor control is optimal; thus, the intent is inferred by solving an inverse optimal control (IOC) problem. Linear quadratic regulator (LQR) is a well-established optimal controller, and its inverse LQR (ILQR) problem has been used in the literature to infer the control intent of one subject. This implementation used a cost function with gain penalty, minimizing the error between LQR gain and a preliminary estimated gain. We hypothesize that relying on an estimated gain may limit ILQR optimization capability. In this study, we derive an ILQR optimization with output penalty, minimizing the error between the model output and the measured output. We conducted the test on 30 healthy subjects who sat on a robotic seat capable of rotation. The task utilized a physical human–robot interaction with a perturbation torque as input and lower and upper body angles as output. Our method significantly improved the goodness of fit compared to the gain-penalty ILQR. Moreover, the dominant inferred intent was not statistically different between the two methods. To our knowledge, this work is the first that infers motor control intent for a sample of healthy subjects. This is a step closer to investigating control intent differences between healthy subjects and subjects with altered motor control, e.g., low back pain.

Data-driven estimation of the algebraic Riccati equation for the discrete-time inverse linear quadratic regulator problem

In this paper, we propose a method for estimating the algebraic Riccati equation (ARE) with respect to an unknown discrete-time system from the system state and input observation. The inverse optimal control (IOC) problem asks, “What objective function is optimized by a given control system?” The inverse linear quadratic regulator (ILQR) problem is an IOC problem that assumes a linear system and quadratic objective function. The ILQR problem can be solved by solving a linear matrix inequality that contains the ARE. However, the system model is required to obtain the ARE, and it is often unknown in fields in which the IOC problem occurs, for example, biological system analysis. Our method directly estimates the ARE from the observation data without identifying the system. This feature enables us to economize the observation data using prior information about the objective function. We provide a data condition that is sufficient for our method to estimate the ARE. We conducted a numerical experiment to demonstrate that our method can estimate the ARE with less data than system identification if the prior information is sufficient.

Small-disturbance input-to-state stability of perturbed gradient flows: Applications to LQR problem

This paper studies the effect of perturbations on the gradient flow of a general nonlinear programming problem, where the perturbation may arise from inaccurate gradient estimation in the setting of data-driven optimization. Under suitable conditions on the objective function, the perturbed gradient flow is shown to be small-disturbance input-to-state stable (ISS), which implies that, in the presence of a small-enough perturbation, the trajectories of the perturbed gradient flow must eventually enter a small neighborhood of the optimum. This work was motivated by the question of robustness of direct methods for the linear quadratic regulator problem, and specifically the analysis of the effect of perturbations caused by gradient estimation or round-off errors in policy optimization. We show small-disturbance ISS for three of the most common optimization algorithms: standard gradient flow, natural gradient flow, and Newton gradient flow.

High-performance LQR control for continuous time systems with redundant control inputs

This paper investigates the linear quadratic regulator (LQR) problem for continuous-time linear systems with redundant control inputs. The LQR control problem has been widely studied, but due to technical difficulties, there is little advance in the study of adding input redundancies. To address this problem, a convergent matrix series, the limit of a monotonically decreasing sequence, is first presented as an upper bound of the symmetric positive definite solution of a class of continuous algebraic Riccati equations (CAREs). Compared with the existing studies on this topic, the obtained bounds are more precise. Moreover, by utilising these upper bounds in the LQR problem of adding input redundancies, a class of sufficient conditions is presented to decrease the quadratic performance index. Finally, the corresponding numerical examples are given to illustrate the effectiveness of our results and compare them with the existing results. Meanwhile, some simulation experiments show that systems with redundant control inputs have good control performance.

Policy Algebraic Equation for the Discrete-Time Linear Quadratic Regulator Problem

Policy Algebraic Equation for the Discrete-Time Linear Quadratic Regulator Problem

Structured Output Feedback Control for Linear Quadratic Regulator Using Policy Gradient Method

We consider the static output feedback control for Linear Quadratic Regulator problems with structured constraints under the assumption that system parameters are unknown. To solve the problem in the model free setting, we propose the policy gradient algorithm based on the gradient projection method and show its global convergence to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula> -stationary points. In addition, we introduce a variance reduction technique and show both theoretically and numerically that it significantly reduces the variance in the gradient estimation. We also show in the numerical experiments that the model free approach efficiently solves the problem.

Approximate dynamic programming for constrained linear systems: A piecewise quadratic approximation approach

Approximate dynamic programming (ADP) faces challenges in dealing with constraints in control problems. Model predictive control (MPC) is, in comparison, well-known for its accommodation of constraints and stability guarantees, although its computation is sometimes prohibitive. This paper introduces an approach combining the two methodologies to overcome their individual limitations. The predictive control law for constrained linear quadratic regulation (CLQR) problems has been proven to be piecewise affine (PWA) while the value function is piecewise quadratic. We exploit these formal results from MPC to design an ADP method for CLQR problems with a known model. A novel convex and piecewise quadratic neural network with a local–global architecture is proposed to provide an accurate approximation of the value function, which is used as the cost-to-go function in the online dynamic programming problem. An efficient decomposition algorithm is developed to generate the control policy and speed up the online computation. Rigorous stability analysis of the closed-loop system is conducted for the proposed control scheme under the condition that a good approximation of the value function is achieved. Comparative simulations are carried out to demonstrate the potential of the proposed method in terms of online computation and optimality.

On the Certainty-Equivalence Approach to Direct Data-Driven LQR Design

The linear quadratic regulator (LQR) problem is a cornerstone of automatic control, and it has been widely studied in the data-driven setting. The various data-driven approaches can be classified as indirect (i.e., based on an identified model) versus direct or as robust (i.e., taking uncertainty into account) versus certainty-equivalence. Here we show how to bridge these different formulations and propose a novel, direct, and regularized formulation. We start from indirect certainty-equivalence LQR, i.e., least-square identification of state-space matrices followed by a nominal model-based design, formalized as a bi-level program. We show how to transform this problem into a single-level, regularized, and direct data-driven control formulation, where the regularizer accounts for the least-square data fitting criterion. For this novel formulation we carry out a robustness and performance analysis in presence of noisy data. In a numerical case study we compare regularizers promoting either robustness or certainty-equivalence, and we demonstrate the remarkable performance when blending both of them.

Inheritance properties of the conjugate discrete-time algebraic Riccati equation

In this paper we consider a class of conjugate discrete-time Riccati equations, arising originally from the linear quadratic regulation problem for discrete-time antilinear systems. Under mild and reasonable assumptions, the existence of the maximal solution to the conjugate discrete-time Riccati equation, in which the control weighting matrix is nonsingular and its constant term is Hermitian, will be inherited to a transformed discrete-time algebraic Riccati equation. Based on this inheritance property, an accelerated fixed-point iteration is proposed for finding the maximal solution via the transformed Riccati equation. Numerical examples are shown to illustrate the correctness of our theoretical results and the feasibility of the proposed algorithm.

Suboptimal consensus protocol design for a class of multiagent systems

This article presents a new technique for suboptimal consensus protocol design for a class of multiagent systems. The proposed technique is based upon the extension of newly developed sufficient conditions for suboptimal linear–quadratic optimal control design, which are derived in this paper by an explication of a noniterative solution technique of the infinite-horizon linear quadratic regulation problem in Krotov framework. For suboptimal consensus protocol design, the structural requirements on the overall feedback gain matrix, which are inherently imposed by agents dynamics and their interaction topology, are recast on a specific matrix introduced in a suitably formulated convex optimization problem. As a result, preassigning the identical feedback gain matrices to a network of homogeneous agents, which act on the relative state variables with respect to their neighbors is not required. The suboptimality of the computed control laws is quantified by implicitly deriving an upper bound on the cost in terms of the solution of a convex optimization problem and initial conditions of the agents instead of specifying it a priori. Numerical examples are provided to demonstrate the implementation of proposed approaches and their comparison with existing methods in the literature.

Optimization of Weight Matrices for the Linear Quadratic Regulator Problem Using Algebraic Closed-Form Solutions

This work proposes an analytical gradient-based optimization approach to determine the optimal weight matrices that make the state and control input at the final time close to zero for the linear quadratic regulator problem. Most existing methodologies focused on regulating the diagonal elements using only bio-inspired approaches or analytical approaches. The method proposed, contrarily, optimizes both diagonal and off-diagonal matrix elements based on the gradient. Moreover, by introducing a new variable composed of the steady-state and time-varying terms for the Riccati matrix and using the coordinate transformation for the state, one develops algebraic equationsbased closed-form solutions to generate the required states and numerical partial derivatives for an optimization strategy that does not require the computationally intensive numerical integration process. The authors test the algorithm with one- and two-degrees-of-freedom linear plant models, and it yields the weight matrices that successfully satisfy the pre-defined requirement, which is the norm of the augmented states less than 10−5. The results suggest the broad applicability of the proposed algorithm in science and engineering.

Approximation of the infinite-horizon value function of the switched LQR problem

This paper considers the approximation of the infinite-horizon value function of the continuous-time switched linear quadratic regulator (LQR) problem. We propose a two successive approximation approach to achieving arbitrarily high accuracy. The infinite-horizon value function is first approximated by a finite-horizon value function, and the resulting truncation error is shown to converge to zero exponentially as the length of the control horizon increases. Then, the finite-horizon value function is approximated through a uniform discretization of the switched linear system, and the approximation error is proven to be proportional to the discretization step. These theoretical results facilitate the development of an efficient approximation algorithm, which further results in a near-optimal control policy. Compared with the existing methods, our algorithm obviates numerical issues induced by singular optimal control. Two numerical examples are presented to illustrate the effectiveness of our algorithm.

Policy-Iteration-Based Finite-Horizon Approximate Dynamic Programming for Continuous-Time Nonlinear Optimal Control.

The Hamilton-Jacobi-Bellman (HJB) equation serves as the necessary and sufficient condition for the optimal solution to the continuous-time (CT) optimal control problem (OCP). Compared with the infinite-horizon HJB equation, the solving of the finite-horizon (FH) HJB equation has been a long-standing challenge, because the partial time derivative of the value function is involved as an additional unknown term. To address this problem, this study first-time bridges the link between the partial time derivative and the terminal-time utility function, and thus it facilitates the use of the policy iteration (PI) technique to solve the CT FH OCPs. Based on this key finding, the FH approximate dynamic programming (ADP) algorithm is proposed leveraging an actor-critic framework. It is shown that the algorithm exhibits important properties in terms of convergence and optimality. Rather importantly, with the use of multilayer neural networks (NNs) in the actor-critic architecture, the algorithm is suitable for CT FH OCPs toward more general nonlinear and complex systems. Finally, the effectiveness of the proposed algorithm is demonstrated by conducting a series of simulations on both a linear quadratic regulator (LQR) problem and a nonlinear vehicle tracking problem.

System Stability of Learning-Based Linear Optimal Control With General Discounted Value Iteration.

For discounted optimal regulation design, the stability of the controlled system is affected by the discount factor. If an inappropriate discount factor is employed, the optimal control policy might be unstabilizing. Therefore, in this article, the effect of the discount factor on the stabilization of control strategies is discussed. We develop the system stability criterion and the selection rules of the discount factor with respect to the linear quadratic regulator problem under the general discounted value iteration algorithm. Based on the monotonicity of the value function sequence, the method to judge the stability of the controlled system is established during the iteration process. In addition, once some stability conditions are satisfied at a certain iteration step, all control policies after this iteration step are stabilizing. Furthermore, combined with the undiscounted optimal control problem, the practical rule of how to select an appropriate discount factor is constructed. Finally, several simulation examples with physical backgrounds are conducted to demonstrate the present theoretical results.

A Unified Framework for Data-Driven Optimal Control of Connected Vehicles in Mixed Traffic

This paper presents a unified approach to the problem of learning-based optimal control of connected human-driven and autonomous vehicles in mixed-traffic environments including both the freeway and ring road settings. The stabilizability of a string of connected vehicles including multiple autonomous vehicles (AVs) and heterogeneous human-driven vehicles (HDVs) is studied by a model reduction technique and the Popov-Belevitch-Hautus (PBH) test. For this problem setup, a linear quadratic regulator (LQR) problem is formulated and a solution based on adaptive dynamic programming (ADP) techniques is proposed without a priori knowledge on model parameters. To start the learning process, an initial stabilizing control law is obtained using the small-gain theorem for the ring road case. It is shown that the obtained stabilizing control law can achieve general <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {L}_{p}$</tex-math></inline-formula> string stability under appropriate conditions. Besides, to minimize the impact of external disturbance, a linear quadratic zero-sum game is introduced and solved by an iterative learning-based algorithm. Finally, the simulation results verify the theoretical analysis and the proposed methods achieve desirable performance for control of a mixed-vehicular network.

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an epsilon-optimal solution with at most an order of epsilon to -2.5 sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

Singular Optimal Linear Quadratic Regulator of Switched Systems

In this paper, we will investigate the singular linear quadratic regulator (LQR) problem of switched linear system in finite time horizon. The proposed method is transforming into a switched LQR problem by adopting linear transformation. Next, we adopt an embedding transformation method to convert the switched LQR problem to a traditional optimal control problem, so the bang-bang-type solution of the embedded optimal control problem in finite time is the optimal solution to the switched LQR problem. The switching sequence of modes and the switching instants can be calculated by solving a closed-form optimal switching condition. The optimal state feedback control law is determined simultaneously. Then, by solving a sequence of Riccati equation, we find some conditions that ensure switched LQR problem can be convert to the singular LQR problem. Finally, a numerical example is presented to demonstrate the effectiveness of the proposed method.

Global Convergence of Policy Gradient Primal–Dual Methods for Risk-Constrained LQRs

While the techniques in optimal control theory are often model-based, the policy optimization (PO) approach directly optimizes the performance metric of interest. Even though it has been an essential approach for reinforcement learning problems, there is little theoretical understanding of its performance. In this paper, we focus on the risk-constrained linear quadratic regulator (RC-LQR) problem via the PO approach, which requires addressing a challenging non-convex constrained optimization problem. To solve it, we first build on our earlier result that an optimal policy has a time-invariant affine structure to show that the associated Lagrangian function is coercive, locally gradient dominated, and has local Lipschitz continuous gradient, based on which we establish strong duality. Then, we design policy gradient primal-dual methods with global convergence guarantees in both model-based and sample-based settings. Finally, we use samples of system trajectories in simulations to validate our methods.

Efficient Learning of a Linear Dynamical System With Stability Guarantees

We propose a principled method for projecting an arbitrary square matrix to the non-convex set of asymptotically stable matrices. Leveraging ideas from large deviations theory, we show that this projection is optimal in an information-theoretic sense and that it simply amounts to shifting the initial matrix by an optimal linear quadratic feedback gain, which can be computed exactly and highly efficiently by solving a standard linear quadratic regulator problem. The proposed approach allows us to learn the system matrix of a stable linear dynamical system from a single trajectory of correlated state observations. The resulting estimator is guaranteed to be stable and offers statistical bounds on the estimation error.

Data-driven quadratic stabilization and LQR control of LTI systems

In this paper, we propose a framework to solve the data-driven quadratic stabilization (DDQS) and the data-driven linear quadratic regulator (DDLQR) problems for both continuous and discrete-time systems. Given noisy input/state measurements and a few priors, we aim to find a state feedback controller guaranteed to quadratically stabilize all systems compatible with the a-priori information and the experimental data. In principle, finding such a controller is a non-convex robust optimization problem. Our main result shows that, by exploiting duality, the problem can be recast into a convex, albeit infinite-dimensional, functional Linear Program. To address the computational complexity entailed in solving this problem, we show that a sequence of increasingly tight finite dimensional semi-definite relaxations can be obtained using sum-of-squares and Putinar’s Positivstellensatz arguments. Finally, we show that these arguments can also be used to find controllers that minimize a worst-case (over all plants in the consistency set) closed-loop H2 cost. The effectiveness of the proposed algorithm is illustrated through comparisons against existing data-driven methods that handle ℓ∞ bounded noise.