Stochastic Linear Quadratic Optimal Control Research Articles

Stochastic linear quadratic optimal control for model-free discrete-time systems based on Q-learning algorithm

Abstract Solving the stochastic linear quadratic (SLQ) optimal control problem generally needs full information about system dynamics. In this paper, a Q-learning iteration algorithm is adopted to solve the control problem for model-free discrete-time systems. Firstly, the condition of the well-posedness for the SLQ problem is given. In order to solve the SLQ problem, the stochastic problem is transformed into the deterministic one. Secondly, in the iteration process of Q-learning algorithm, the H matrix sequence and control gain matrix sequence are obtained without the knowledge of system parameters, and the convergence proof of two sequences is also given. Lastly, two simulation examples are supplied to explain the effectiveness of the Q-learning algorithm.

Infinite horizon linear quadratic Pareto game of the stochastic singular systems

This paper is concerned with the linear quadratic (LQ) Pareto game of the stochastic singular systems in infinite horizon. Firstly, the optimal control problem of the weighted sum cost functional is discussed. Utilizing the equivalent transformation method, the weighted sum LQ optimal control problem is transformed into a stochastic LQ optimization problem. Based on the classical stochastic LQ optimal control theory, the necessary and sufficient condition for the solvability of the indefinite weighted sum LQ optimal control is put forward. Then, the LQ Pareto game of the stochastic singular systems is studied. By the discussion of the convexity of the cost functionals, a sufficient condition for the existence of the Pareto solutions is obtained via the solvability of the corresponding generalized algebraic Riccati equation (GARE). Moreover, we derive all Pareto solutions based on the solution of a Lyapunov equation. Finally, an example is given to show the effectiveness of the proposed results.

An Indefinite Stochastic Linear Quadratic Optimal Control Problem with Delay and Related Forward–Backward Stochastic Differential Equations

In this paper, we will study an indefinite stochastic linear quadratic optimal control problem, where the controlled system is described by a stochastic differential equation with delay. By introducing the relaxed compensator as a novel method, we obtain the well-posedness of this linear quadratic problem for indefinite case. And then, we discuss the uniqueness and existence of the solutions for a kind of anticipated forward–backward stochastic differential delayed equations. Based on this, we derive the solvability of the corresponding stochastic Hamiltonian systems, and give the explicit representation of the optimal control for the linear quadratic problem with delay in an open-loop form. The theoretical results are validated as well on the control problems of engineering and economics under indefinite condition.

Existence of a Mean-Square Stabilizing Solution to a Modified Algebraic Riccati Equation

In this paper, the existence of a mean-square stabilizing solution to a discrete-time modified algebraic Riccati equation (MARE), which arises in the study of some stochastic linear quadratic optimal control problems, is investigated. The theory of cone-invariant operators is employed as the mathematical tool to tackle this problem. We provide some criteria for the cone stability, cone observability, and cone detectability of a class of cone-invariant systems in terms of the associated distinguished eigenvalues. Then two scenarios concerning the problem of existence in terms of the positive definiteness of the input weighting matrix are considered. We first study the MARE under the assumption that the input weighting matrix in the cost function is positive definite. Then an explicit necessary and sufficient condition is obtained. Such a condition is derived for the very first time and it indicates that the common condition of observability or detectability of certain stochastic systems is unnecessary. Onl...

Linear quadratic optimal control of nonhomogeneous vector differential equations with FBM

In this paper, the solution of stochastic linear quadratic optimal control problem of nonhomogeneous vector differential equation driven by fractional Brownian motion of Hurst parameter \( H(>0.5)\) is established. A stochastic Riccati equation is derived by considering adjoint equations satisfying backward stochastic differential equations. The feedback form of the optimal control and the optimal cost are derived using completion of squares technique. The results established are illustrated with examples.

Information Transfer of Control Strategies: Dualities of Stochastic Optimal Control Theory and Feedback Capacity of Information Theory

The control-coding capacity of stochastic control systems is introduced, and its operational meaning is established using randomized control strategies, which simultaneously control output processes encode information, and communicate information from control processes to output processes. The control-coding capacity is the analog Shannon's coding-capacity of noisy channels. Furthermore, duality relations to stochastic optimal control problems with deterministic and randomized control strategies are identified including the following. First, extremum problems of stochastic optimal control with directed information payoff are equivalent to feedback capacity problems of information theory, in which the control system act as a communication channel. Second, for Gaussian linear decision models with average quadratic constraints, it is shown that optimal randomized strategies are Gaussian, and decompose into a deterministic part and a random part. The deterministic part is precisely the optimal strategy of the linear quadratic Gaussian stochastic optimal control problem, whereas the random part is the solution of an water-filling information transmission problem that encodes information, which is estimated by a decoder.

A Class of Linear Quadratic Gaussian Hybrid Optimal Control Problems with Realization–Independent Riccati Equations

A class of stochastic linear quadratic hybrid optimal control problems is presented for which the Hamiltonian boundary conditions appearing in the associated necessary optimality conditions of the Stochastic Hybrid Minimum Principle and Stochastic Hybrid Dynamic Programming are path-independent. Consequently, the linear quadratic Gaussian regulation problem associated with this class of stochastic hybrid optimal control problems can be solved via (stochastic) Riccati equations which are independent of the realization of stochastic diffusion terms. An analytic example of a scalar hybrid system is provided to illustrate the results, and the relation to the deterministic case is discussed.

Infinite horizon jump-diffusion forward-backward stochastic differential equations and their application to backward linear-quadratic problems

In this paper, we investigate infinite horizon jump-diffusion forward-backward stochastic differential equations under some monotonicity conditions. We establish an existence and uniqueness theorem, two stability results and a comparison theorem for solutions to such kind of equations. Then the theoretical results are applied to study a kind of infinite horizon backward stochastic linear-quadratic optimal control problems, and then differential game problems. The unique optimal controls for the control problems and the unique Nash equilibrium points for the game problems are obtained in closed forms.

Indefinite stochastic linear-quadratic optimal control problems with random jumps and related stochastic Riccati equations

We discuss the stochastic linear-quadratic (LQ) optimal controlproblem with Poisson processes under the indefinite case. Based on thewellposedness of the LQ problem, the main idea is expressed by thedefinition of relax compensator that extends the stochasticHamiltonian system and stochastic Riccati equation with Poissonprocesses (SREP) from the positive definite case to the indefinite case. Wemainly study the existence and uniqueness of the solution for thestochastic Hamiltonian system and obtain the optimal control withopen-loop form. Then, we further investigate the existence anduniqueness of the solution for SREP in some special case and obtainthe optimal control in close-loop form.

Discrete-time mean-field stochastic H 2/H ∞ control

The finite horizon H 2/H ∞ control problem of mean-field type for discrete-time systems is considered in this paper. Firstly, the authors derive a mean-field stochastic bounded real lemma (SBRL). Secondly, a sufficient condition for the solvability of discrete-time mean-field stochastic linearquadratic (LQ) optimal control is presented. Thirdly, based on SBRL and LQ results, this paper establishes a sufficient condition for the existence of discrete-time stochastic H 2/H ∞ control of meanfield type via the solvability of coupled matrix-valued equations.

Numerical solution of the finite horizon stochastic linear quadratic control problem

The treatment of the stochastic linear quadratic optimal control problem with finite time horizon requires the solution of stochastic differential Riccati equations. We propose efficient numerical methods, which exploit the particular structure and can be applied for large-scale systems. They are based on numerical methods for ordinary differential equations such as Rosenbrock methods, backward differentiation formulas, and splitting methods. The performance of our approach is tested in numerical experiments.

Stochastic Linear Quadratic Optimal Control Problems in Infinite Horizon

This paper is concerned with stochastic linear quadratic (LQ, for short) optimal control problems in an infinite horizon with constant coefficients. It is proved that the non-emptiness of the admissible control set for all initial state is equivalent to the $L^2$-stabilizability of the control system, which in turn is equivalent to the existence of a positive solution to an algebraic Riccati equation (ARE, for short). Different from the finite horizon case, it is shown that both the open-loop and closed-loop solvabilities of the LQ problem are equivalent to the existence of a static stabilizing solution to the associated generalized ARE. Moreover, any open-loop optimal control admits a closed-loop representation. Finally, the one-dimensional case is worked out completely to illustrate the developed theory.

The Research of Discrete Mean - Variance Portfolio Problem with Time-Delay

Due to the financial sector complicated variety of events, each financial problems from changes to know its essence, the change rule, from the change of strategy to formulate relevant policy and policy into effect, etc., the process inevitably has a certain lag. Therefore, in order to better reflect the actual situation, we study the portfolio model with delays in this paper. By joining our delay control item, the optimization model was established, the goal is to maximize earnings expectations. In this paper, it studies the continuous time without delay the mean - variance portfolio problems on the basis of existing research. It established auxiliary problem using the stochastic linear quadratic optimal control theory. Using the maximum principle, the solution of the optimal investment strategy are given and it analysis the case, the conclusion is in conformity with the actual. It studies the existing time delay portfolio strategy problem in discrete time case. Based on the stochastic LQ (linear quadratic) optimal control theory, it established the discrete time model with time delay. The paper has carried on the solution and example analysis.

Linear Quadratic Stochastic Optimal Control of Forward Backward Stochastic Control System Associated with Lévy Process

This paper analyzes one kind of linear quadratic (LQ) stochastic control problem of forward backward stochastic control system associated with Lévy process. We obtain the explicit form of the optimal control, then prove it to be unique, and get the linear feedback regulator by introducing one kind of generalized Riccati equation. Finally, we discuss the solvability of the generalized Riccati equation, and its existence and uniqueness of the solutions are proved in a special case.

Indefinite Mean-Field Stochastic Linear-Quadratic Optimal Control: From Finite Horizon to Infinite Horizon

In this paper, the finite-horizon and the infinite-horizon indefinite mean-field stochastic linear-quadratic optimal control problems are studied. Firstly, the open-loop optimal control and the closed-loop optimal strategy for the finite-horizon problem are introduced, and their characterizations, difference and relationship are thoroughly investigated. The open-loop optimal control can be defined for a fixed initial state, whose existence is characterized via the solvability of a linear mean-field forward-backward stochastic difference equation with stationary conditions and a convexity condition. On the other hand, the existence of a closed-loop optimal strategy is shown to be equivalent to any one of the following conditions: the solvability of a couple of generalized difference Riccati equations, the finiteness of the value function for all the initial pairs, and the existence of the open-loop optimal control for all the initial pairs. It is then proved that the solution of the generalized difference Riccati equations converges to a solution of a couple of generalized algebraic Riccati equations. By studying another generalized algebraic Riccati equation, the existence of the maximal solution of the original ones is obtained together with the fact that the stabilizing solution is the maximal solution. Finally, we show that the maximal solution is employed to express the optimal value of the infinite-horizon indefinite mean-field linear-quadratic optimal control. Furthermore, for the question whether the maximal solution is the stabilizing solution, the necessary and the sufficient conditions are presented for several cases.

Gaussian density estimates for the solution of singular stochastic Riccati equations

Stochastic Riccati equation is a backward stochastic differential equation with singular generator which arises naturally in the study of stochastic linear-quadratic optimal control problems. In this paper, we obtain Gaussian density estimates for the solutions to this equation.

Mean-field stochastic linear quadratic optimal control problems: closed-loop solvability

An optimal control problem is studied for a linear mean-field stochastic differential equation with a quadratic cost functional. The coefficients and the weighting matrices in the cost functional are all assumed to be deterministic. Closed-loop strategies are introduced, which require to be independent of initial states; and such a nature makes it very useful and convenient in applications. In this paper, the existence of an optimal closed-loop strategy for the system (also called the closed-loop solvability of the problem) is characterized by the existence of a regular solution to the coupled two (generalized) Riccati equations, together with some constraints on the adapted solution to a linear backward stochastic differential equation and a linear terminal value problem of an ordinary differential equation.

Mean-field stochastic linear–quadratic optimal control with Markov jump parameters

This paper considers a class of mean-field stochastic linear–quadratic optimal control problems with Markov jump parameters. The new feature of these problems is that means of state and control are incorporated into the systems and the cost functional. Based on the modes of Markov chain, the corresponding decomposition technique of augmented state and control is introduced. It is shown that, under some appropriate conditions, there exists a unique optimal control, which can be explicitly given via solutions of two generalized difference Riccati equations. A numerical example sheds light on the theoretical results established.

Continuous-time mean–variance portfolio selection with random horizon in an incomplete market

In this paper, we consider a continuous-time mean–variance portfolio selection problem with random market parameters and random time horizon in an incomplete market. This problem will be formulated as a linearly constrained stochastic linear quadratic (LQ) optimal control problem. The solvability of this LQ problem will be reduced to the global solvability of two backward stochastic differential equations (BSDEs). One is conventionally called a stochastic Riccati equation (SRE), and the other is referred to as an auxiliary BSDE. We shall apply the martingales of bounded mean oscillation, briefly called BMO-martingales, to provide a direct and simplified proof of the solvability of the two BSDEs. We also derive closed-form expressions for both the optimal portfolios and the efficient frontier in terms of the solutions of the two BSDEs.

The stochastic linear quadratic optimal control problem in Hilbert spaces: A polynomial chaos approach

We consider the stochastic linear quadratic optimal control problem for state equations of the Ito-Skorokhod type, where the dynamics are driven by strongly continuous semigroup. We provide a numerical framework for solving the control problem using a polynomial chaos expansion approach in white noise setting. After applying polynomial chaos expansion to the state equation, we obtain a system of infinitely many deterministic partial differential equations in terms of the coefficients of the state and the control variables. We set up a control problem for each equation, which results in a set of deterministic linear quadratic regulator problems. Solving these control problems, we find optimal coefficients for the state and the control. We prove the optimality of the solution expressed in terms of the expansion of these coefficients compared to a direct approach. Moreover, we apply our result to a fully stochastic problem, in which the state, control and observation operators can be random, and we also consider an extension to state equations with memory noise.