Stochastic Linear Quadratic Control Problem Research Articles

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Stochastic linear–quadratic control problems with affine constraints

This paper investigates the stochastic linear–quadratic control problems with affine constraints, in which both equality and inequality constraints are involved. With the help of the Pontryagin maximum principle and Lagrangian duality theory, both the dual problem and the state feedback form of the solution are obtained for the primal problem. Under the Slater condition, the strong duality is proved between the dual problem and the primal problem, and the KKT condition is also provided for solving the primal problem. Moreover, a new sufficient condition is given for the invertibility assumption, which ensures the uniqueness of the solutions to the dual problem. Finally, two numerical examples are provided to illustrate our main results.

Stochastic controls of fractional Brownian motion

Abstract We consider a stochastic control problem for a non-linear forward-backward stochastic differential equation driven by fractional Brownian motion, with Hurst parameter H ∈ ( 0 , 1 ) {H\in(0,1)} , in the case where the set of the control domain is convex. We provide an estimation of the solution and establish the necessary and sufficient optimality conditions in the form of the stochastic maximum principle. We apply the theory to solve a linear quadratic stochastic control problem.

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Lagrangian dual method for solving stochastic linear quadratic optimal control problems with terminal state constraints

A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed. The strong convergence of the iterative sequence generated by ALM is proved. In addition, some sufficient conditions for the surjectivity of the linear constraint mapping are obtained.

Constrained mean-variance investment-reinsurance under the Cramér–Lundberg model with random coefficients

In this paper, we study an optimal mean-variance investment-reinsurance problem for an insurer (she) under the Cramér–Lundberg model with random coefficients. At any time, the insurer can purchase reinsurance or acquire new business and invest her surplus in a security market consisting of a risk-free asset and multiple risky assets, subject to a general convex cone investment constraint. We reduce the problem to a constrained stochastic linear-quadratic control problem with jumps whose solution is related to a system of partially coupled stochastic Riccati equations (SREs). Then we devote ourselves to establishing the existence and uniqueness of solutions to the SREs by pure backward stochastic differential equation (BSDE) techniques. We achieve this with the help of approximation procedure, comparison theorems for BSDEs with jumps, log transformation and BMO martingales. The efficient investment-reinsurance strategy and efficient mean-variance frontier are explicitly given through the solutions of the SREs, which are shown to be a linear feedback form of the wealth process and a half-line, respectively.

Stochastic linear quadratic optimal control problem with terminal state constraints

AbstractThis paper addresses the stochastic linear quadratic (LQ) control problem with first‐ and second‐order moment constraints on the terminal state. The problem is a modified version of the optimal covariance control problem, where the terminal state is steered to a given probability distribution. Studying a multiplicative‐noise stochastic system rather than an additive‐noise system is a salient feature. Unlike the existing ideas in the optimal steering, by using the Lagrange multipliers method and establishing the stochastic maximum principle, our problem is converted into solving forward–backward stochastic difference equations (FBSDEs), which is a special stochastic two‐point boundary‐value problem (TPBVP). We provide the optimal closed‐loop controller and necessary and sufficient solvability conditions by developing a nonhomogeneous relationship between the optimal state and costate in FBSDEs. Finally, numerical examples are given to demonstrate our results.

Stochastic optimal control and piecewise parameterization and optimization method for inventory control system improvement

In recent decades, stochastic control systems have been widely used in industrial production, biomedicine, aerospace, military strategy and so on. In this paper, an approximate optimal strategy derived from a stochastic linear quadratic (SLQ) optimal control problem is considered, and a piecewise parameterization and optimization (PPAO) method is proposed. Firstly, using the principle of dynamic programming, the control form of SLQ optimal control problem is relevant to a Riccati differential equation. It is well known that the Riccati differential equation is difficult to be solved analytically. Thus, we present a PPAO method for finding an approximate optimal strategy for stochastic control problems. Here, a piecewise parameter control can be obtained by solving first-order differential equations rather than Riccati differential equations. Finally, the inventory control problems with different dimensions are used to justify the feasibility of PPAO method, and the results compared with the original optimal control are given. The results show that parametric control greatly simplifies the control form, and a small model error is ensured.

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Stochastic linear quadratic optimal control problems with expectation-type linear equality constraints on the terminal states

A stochastic linear quadratic (LQ) optimal control problem with an expectation-type linear equality constraint on the terminal state is considered. Under the solvability condition on a stochastic Riccati equation and a surjectivity condition on the linear constraint mapping, the constrained stochastic LQ problem is solved completely by the Lagrangian duality theory. Some equivalent characterizations of the surjectivity condition are discussed by the controllability theory of linear control systems. Especially, the equivalence between the surjectivity condition and a Kalman-type rank condition is proved under proper conditions.

Optimal regulator for a class of nonlinear stochastic systems with random coefficients

We consider an optimal regulator problem for a class of nonlinear stochastic systems with a square-root nonlinearity and random coefficients, and using the quadratic-linear criterion. This represents a certain nonlinear generalisation of the stochastic linear-quadratic control problem with random coefficients. The solution if found in an explicit closed-form as an affine state-feedback control in terms of a Riccati and linear backward stochastic differential equations. As an application, we give the solution to an optimal investment problem in a market with random coefficients.

Temporal Semi-discretizations of a Backward Semilinear Stochastic Evolution Equation

This paper studies the convergence of three temporal semi-discretizations for a backward semilinear stochastic evolution equation. For general terminal value and general coefficient with Lipschitz continuity, the convergence of the first two temporal semi-discretizations is established, and an explicit convergence rate is derived for the third temporal semi-discretization. The third temporal semi-discretization is applied to a general stochastic linear quadratic control problem, and the convergence of a temporally semi-discrete approximation to the optimal control is established.

On the stochastic linear quadratic optimal control problem by piecewise constant controls: The infinite horizon time case

This paper is devoted to the problem of indefinite stochastic linear quadratic (LQ) optimal control by piecewise constant controls in an infinite horizon case. By restricting the set of admissible controls to the class of piecewise constant stochastic processes, we reformulated the above control problem under the setting of systems modeled by Itô differential equations controlled by impulses. We show that the solution in a state feedback form of the indefinite stochastic LQ control problem is equivalent to the existence of a global stabilizing solution associated to a class of backward matrix linear differential equations with a Riccati type jumping operators.

An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

Optimal feedback controls of stochastic linear quadratic control problems in infinite dimensions with random coefficients

It is a longstanding unsolved problem to characterize the optimal feedback controls for general linear quadratic optimal control problem of stochastic evolution equation with random coefficients. A solution to this problem is given in [22] under some assumptions which can be verified for interesting concrete models, such as controlled stochastic wave equations, controlled stochastic Schrödinger equations, etc. More precisely, the authors establish the equivalence between the existence of an optimal feedback operator and the solvability of the corresponding operator-valued, backward stochastic Riccati equations. However, their result cannot cover some important stochastic partial differential equations, such as stochastic heat equations, stochastic stokes equations, etc. A key contribution of the current work is to relax the C0-group assumption of unbounded linear operator A in [22] and using the contraction semigroup assumption instead. Therefore, our result can be well applicable in the linear quadratic problem of stochastic parabolic equations. To this end, we introduce a suitable notion to the aforementioned Riccati equation, and some delicate techniques which are even new in the finite dimensional case.

Risk-Aware Linear Quadratic Control Using Conditional Value-at-Risk

Stochastic linear quadratic control problems are considered from the viewpoint of risks. In particular, a worst-case conditional value-at-risk (CVaR) of quadratic objective function is minimized subject to additive disturbances whose first two moments of the distribution are known. The study focuses on three problems of finding the optimal feedback gain that minimizes the quadratic cost of: stationary distribution, one-step, and infinite time horizon. For the stationary distribution problem, it is proved that the optimal control gain that minimizes the worst-case CVaR of the quadratic cost is equivalent to that of the standard (stochastic) linear quadratic regulator. For the one-step problem, an approach to an optimal solution as well as analytical suboptimal solutions are presented. For the infinite time horizon problem, two suboptimal solutions that bound the optimal solution and an approach to an optimal solution for a special case are discussed. The presented theorems are illustrated with numerical examples.

Stochastic linear-quadratic control with a jump and regime switching on a random horizon

In this paper, we study a stochastic linear-quadratic control problem with random coefficients and regime switching on a random horizon $ [0,T\wedge\tau] $, where $ \tau $ is a given random jump time for the underlying state process and $ T $ a constant. We obtain the explicit optimal feedback control and explicit optimal value of the problem by solving a system of stochastic Riccati equations (SREs) with jumps on the random horizon $ [0,T\wedge\tau] $. By the decomposition approach stemming from filtration enlargement theory, we express the solution to the system of SREs with jumps in terms of another system of SREs involving only Brownian filtration on the deterministic horizon $ [0,T] $. Solving the latter system is the key theoretical contribution of this paper and we accomplish this under three different conditions, one of which seems to be new in the literature. The above results are then applied to study a mean-variance hedging problem with random parameters that depend on both Brownian motion and Markov chain. The optimal portfolio and optimal value are presented in closed forms with the aid of a system of linear backward stochastic differential equations with jumps and unbounded coefficients in addition to the SREs with jumps.

Optimal order execution under price impact: a hybrid model

In this paper we explore optimal liquidation in a market populated by a number of heterogeneous market makers that have limited inventory-carrying and risk-bearing capacity. We derive a reduced form model for the dynamics of their aggregated inventory considering a proper scaling limit. The resulting price impact profile is shown to depend on the characteristics and relative importance of their inventories. The model is flexible enough to reproduce the empirically documented power law behavior of the price impact function. For any choice of the market makers characteristics, optimal execution within this modeling approach can be recast as a linear-quadratic stochastic control problem. The value function and the associated optimal trading rate can be obtained semi-explicitly subject to solving a differential matrix Riccati equation. Numerical simulations are conducted to illustrate the performance of the resulting optimal liquidation strategy in relation to standard benchmarks. Remarkably, they show that the increase in performance is determined by a substantial reduction of higher order moment risk.