Quadratic Control Problem Research Articles

Stochastic linear–quadratic control problems with affine constraints

This paper investigates the stochastic linear–quadratic control problems with affine constraints, in which both equality and inequality constraints are involved. With the help of the Pontryagin maximum principle and Lagrangian duality theory, both the dual problem and the state feedback form of the solution are obtained for the primal problem. Under the Slater condition, the strong duality is proved between the dual problem and the primal problem, and the KKT condition is also provided for solving the primal problem. Moreover, a new sufficient condition is given for the invertibility assumption, which ensures the uniqueness of the solutions to the dual problem. Finally, two numerical examples are provided to illustrate our main results.

Linear–quadratic stochastic Volterra controls I: Causal feedback strategies

In this paper, we formulate and investigate the notion of causal feedback strategies arising in linear–quadratic control problems for stochastic Volterra integral equations (SVIEs) with singular and non-convolution-type coefficients. We show that there exists a unique solution, which we call the causal feedback solution, to the closed-loop system of a controlled SVIE associated with a causal feedback strategy. Furthermore, introducing two novel equations named a Type-II extended backward stochastic Volterra integral equation and a Lyapunov–Volterra equation, we prove a duality principle and a representation formula for a quadratic functional of controlled SVIEs in the framework of causal feedback strategies.

Reducing Obizhaeva–Wang-type trade execution problems to LQ stochastic control problems

We start with a stochastic control problem where the control process is of finite variation (possibly with jumps) and acts as integrator both in the state dynamics and in the target functional. Problems of such type arise in the stream of literature on optimal trade execution pioneered by Obizhaeva and Wang (J. Financ. Mark. 16:1–32, 2013) (models with finite resilience). We consider a general framework where the price impact and the resilience are stochastic processes. Both are allowed to have diffusive components. First we continuously extend the problem from processes of finite variation to progressively measurable processes. Then we reduce the extended problem to a linear–quadratic (LQ) stochastic control problem. Using the well-developed theory on LQ problems, we describe the solution to the obtained LQ one and translate it back to the solution for the (extended) initial trade execution problem. Finally, we illustrate our results by several examples. Among other things, the examples discuss the Obizhaeva–Wang model with random (terminal and moving) targets, the necessity to extend the initial trade execution problem to a reasonably large class of progressively measurable processes (even going beyond semimartingales), and the effects of diffusive components in the price impact process and/or the resilience process.

Linear Quadratic Control Problem of Systems With Markov Jumps in Reverse Time and Observation With Anticipation of the Jumps

Linear Quadratic Control Problem of Systems With Markov Jumps in Reverse Time and Observation With Anticipation of the Jumps

A necessary and sufficient condition for the existence of the stabilizing solution of a large class of discrete-time Riccati type equations with periodic coefficients

This paper is devoted to the study of a large class of discrete-time backward Riccati equations arising in several linear quadratic (LQ) type control problems both in the deterministic and in the stochastic frameworks. The periodic time-varying case is considered. We propose existence and uniqueness conditions for a global special solution named stabilizing solution for such equations. Beside the stabilizability condition, the criterion derived in this paper is expressed based on some suitable properties of the characteristic multipliers of a discrete-time, periodic linear equation adequately constructed using the coefficients of the given equation. Our result englobes, as particular cases, several existing results in the literature.

Stochastic controls of fractional Brownian motion

Abstract We consider a stochastic control problem for a non-linear forward-backward stochastic differential equation driven by fractional Brownian motion, with Hurst parameter H ∈ ( 0 , 1 ) {H\in(0,1)} , in the case where the set of the control domain is convex. We provide an estimation of the solution and establish the necessary and sufficient optimality conditions in the form of the stochastic maximum principle. We apply the theory to solve a linear quadratic stochastic control problem.

Global Optimality Guarantees for Policy Gradient Methods

Policy gradient methods, which have powered a lot of recent success in reinforcement learning, search for an optimal policy in a parameterized policy class by performing stochastic gradient descent on the cumulative expected cost-to-go under some initial state distribution. Although widely used, these methods lack theoretical guarantees as the optimization objective is typically nonconvex even for simple control problems, and hence are understood to only converge to a stationary point. In “Global Optimality Guarantees for Policy Gradient Methods,” J. Bhandari and D. Russo identify structural properties of the underlying MDP that guarantee that despite nonconvexity, the optimization objective has no suboptimal stationary points, ensuring asymptotic convergence of policy gradient methods to globally optimal policies. Under stronger conditions, authors show the policy gradient objective to satisfy a Polyak-lojasiewicz (gradient dominance) condition that yields fast convergence rates. In addition, when some of the said conditions are relaxed, authors provide bounds on the suboptimality gap of any stationary point. The results rely on a key connection with policy iteration, a classic dynamic programming algorithm which solves a single period optimization problem at every step. The authors show how structure in the single period optimization problems solved by policy iteration translate into nice properties of the multiperiod policy gradient objective, making it amenable for first-order methods to find globally optimal solutions. The authors also instantiate their framework for several classical control problems including tabular and linear MDPs, linear quadratic control, optimal stopping, and finite-horizon inventor control problems.

A bias-variance perspective of data-driven control

Data-driven control, the task of designing a controller based on process data, finds application in a wide range of disciplines and the topic has been intensively studied over more than half a century. The main purpose of this contribution is to elucidate on the commonalities between data-driven control and parameter estimation. In particular, we discuss the bias-variance trade-of, i.e. rather than aiming for the optimal controller one should aim for a constrained version, that may be characterized by tunable parameters, corresponding to hyperparameters in parameter estimation. As a result we shift attention from indirect vs direct data driven control by highlighting the important role played by (complete) minimal sufficient statistics. To keep technicalities at a minimum, still capturing the essential features of the problem, we consider the problem of minimizing the expected control cost for a quadratic open loop control problem applied to a finite impulse response system. In a Gaussian white noise setting, the maximum-likelihood parameter estimate constitutes a complete minimal sufficient statistic which allows us to focus on controllers that are functions of this model estimate without loss of statistical accuracy. We make a systematic study of three different controller structures and two different tuning techniques and illustrate their behaviours numerically.

Stochastic linear quadratic optimal control problem with terminal state constraints

AbstractThis paper addresses the stochastic linear quadratic (LQ) control problem with first‐ and second‐order moment constraints on the terminal state. The problem is a modified version of the optimal covariance control problem, where the terminal state is steered to a given probability distribution. Studying a multiplicative‐noise stochastic system rather than an additive‐noise system is a salient feature. Unlike the existing ideas in the optimal steering, by using the Lagrange multipliers method and establishing the stochastic maximum principle, our problem is converted into solving forward–backward stochastic difference equations (FBSDEs), which is a special stochastic two‐point boundary‐value problem (TPBVP). We provide the optimal closed‐loop controller and necessary and sufficient solvability conditions by developing a nonhomogeneous relationship between the optimal state and costate in FBSDEs. Finally, numerical examples are given to demonstrate our results.

Decentralized control for discrete‐time mean‐field systems with multiple controllers of delayed information

AbstractIn this paper, the finite horizon asymmetric information linear quadratic (LQ) control problem is investigated for a discrete‐time mean field system. Different from previous works, multiple controllers with different information sets are involved in the mean field system dynamics. The coupling of different controllers makes it quite difficult in finding the optimal control strategy. Fortunately, by applying the Pontryagin's maximum principle, the corresponding decentralized control problem of the finite horizon is investigated. The contributions of this paper can be concluded as follows: For the first time, based on the solution of a group of mean‐field forward and backward stochastic difference equations (MF‐FBSDEs), the necessary and sufficient solvability conditions are derived for the asymmetric information LQ control for the mean field system with multiple controllers. Furthermore, by the use of an innovative orthogonal decomposition approach, the optimal decentralized control strategy is derived, which is based on the solution to a non‐symmetric Riccati‐type equation.

Indefinite LQ optimal control for stochastic Takagi–Sugeno fuzzy system under sensor data scheduling: Finite‐horizon case

AbstractThe present paper considers the finite‐horizon indefinite linear quadratic (LQ) control problem for stochastic Takagi–Sugeno (T‐S) fuzzy systems with input delay. In this paper, we consider the presence of sensor data scheduling, which imposes a communication energy constraint and necessitates optimal state estimation for measurements. Then, by utilizing dynamic programming principles, the stochastic LQ problem under consideration can be solved, while the optimal control policy is developed in terms of the unique solutions to a set of coupled difference Riccati equations (CDREs). Specifically, for simple delay‐free case, the linear matrix inequalities based conditions are also proposed, whose feasibility is shown to be equivalent to the well‐posedness of the indefinite LQ control under consideration. As an application, our theoretic analysis is extended to study the intermittent observation model caused by random denial‐of‐service attack.

The mean‐field linear quadratic optimal control problem for stochastic systems controlled by impulses

AbstractIn the present note, we state and solve the linear quadratic (LQ) control problem for a class of McKean–Vlasov stochastic differential equations for which the control actions are of impulsive nature. After reformulating the original optimization problem using an orthogonal decomposition of the state variables, we introduce an adequately defined system of two coupled matrix Lyapunov‐type differential equations with jumps. Such equations are central for the definition of the optimal feedback gains corresponding to the closed‐loop representation of the optimal LQ control.

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Zero-Order Asymptotics for the Solution of One Type of Singularly Perturbed Linear–Quadratic Control Problems in the Critical Case

Zero-Order Asymptotics for the Solution of One Type of Singularly Perturbed Linear–Quadratic Control Problems in the Critical Case

On the Sample Complexity of Decentralized Linear Quadratic Regulator With Partially Nested Information Structure

We study the problem of control policy design for decentralized state-feedback linear quadratic control with a partially nested information structure, when the system model is unknown. We propose a model-based learning solution, which consists of two steps. First, we estimate the unknown system model from a single system trajectory of finite length, using least squares estimation. Next, based on the estimated system model, we design a decentralized control policy that satisfies the desired information structure. We show that the suboptimality gap between our control policy and the optimal decentralized control policy (designed using accurate knowledge of the system model) scales linearly with the estimation error of the system model. Using this result, we provide an end-to-end sample complexity result for learning decentralized controllers for a linear quadratic control problem with a partially nested information structure.

Mean–variance portfolio selection under no-shorting rules: A BSDE approach

This paper revisits the mean–variance portfolio selection problem in continuous-time within the framework of short-selling of stocks is prohibited via backward stochastic differential equation approach. To relax the strong condition in Li et al. (Li et al. 2002), the above issue is formulated as a stochastic recursive optimal linear–quadratic control problem. Due to no-shorting rules (namely, the portfolio taking non-negative values), the well-known “completion of squares” no longer applies directly. To overcome this difficulty, we study the corresponding Hamilton–Jacobi–Bellman (HJB, for short) equation inherently and derive the two groups of Riccati equations. On one hand, the value function constructed via Riccati equations is shown to be a viscosity solution of the HJB equation mentioned before; On the other hand, by means of these Riccati equations and backward semigroup, we are able to get explicitly the efficient frontier and efficient investment strategies for the recursive utility mean–variance portfolio optimization problem.

Linear quadratic control for multiple time‐delayed uncertain random systems

AbstractThis study focuses on time‐delayed linear quadratic control problems (LQCPs). Firstly, a general time‐delayed control problem for a linear time‐varying system is investigated. This problem is equivalent to an optimization problem without time delays, and corresponding recurrence equations are proposed accordingly. Furthermore, based on recurrence equations and chance theory, two different types of LQCPs are researched, where the control weighting matrices are positive definite and indefinite, respectively. Exact solutions for the optimal results of these LQCPs are provided. Finally, numerical examples are provided to demonstrate the findings.

Temporal Semi-discretizations of a Backward Semilinear Stochastic Evolution Equation

This paper studies the convergence of three temporal semi-discretizations for a backward semilinear stochastic evolution equation. For general terminal value and general coefficient with Lipschitz continuity, the convergence of the first two temporal semi-discretizations is established, and an explicit convergence rate is derived for the third temporal semi-discretization. The third temporal semi-discretization is applied to a general stochastic linear quadratic control problem, and the convergence of a temporally semi-discrete approximation to the optimal control is established.

Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient Method and Global Convergence

Recently, policy optimization has received renewed attention from the control community due to various applications in reinforcement learning tasks. In this article, we investigate the global convergence of the gradient method for quadratic optimal control of discrete-time Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, with static-state feedback controllers and quadratic performance costs. Despite the nonconvexity of the resultant problem, we are still able to identify several useful properties such as coercivity, gradient dominance, and smoothness. Based on these properties, we prove that the gradient method converges to the optimal-state feedback controller for MJLS at a linear rate if initialized at a controller, which is mean-square stabilizing. This article brings new insights for understanding the performance of the policy gradient method on the Markovian jump linear quadratic control problem.

Optimal feedback controls of stochastic linear quadratic control problems in infinite dimensions with random coefficients

It is a longstanding unsolved problem to characterize the optimal feedback controls for general linear quadratic optimal control problem of stochastic evolution equation with random coefficients. A solution to this problem is given in [22] under some assumptions which can be verified for interesting concrete models, such as controlled stochastic wave equations, controlled stochastic Schrödinger equations, etc. More precisely, the authors establish the equivalence between the existence of an optimal feedback operator and the solvability of the corresponding operator-valued, backward stochastic Riccati equations. However, their result cannot cover some important stochastic partial differential equations, such as stochastic heat equations, stochastic stokes equations, etc. A key contribution of the current work is to relax the C0-group assumption of unbounded linear operator A in [22] and using the contraction semigroup assumption instead. Therefore, our result can be well applicable in the linear quadratic problem of stochastic parabolic equations. To this end, we introduce a suitable notion to the aforementioned Riccati equation, and some delicate techniques which are even new in the finite dimensional case.