Stochastic Linear Quadratic Optimal Control Research Articles

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Decentralized stochastic linear-quadratic optimal control with risk constraint and partial observation

This paper addresses a risk-constrained decentralized stochastic linear-quadratic optimal control problem with one remote controller and one local controller, where the risk constraint is posed on the cumulative state weighted variance in order to reduce the oscillation of system trajectory. In this model, local controller can only partially observe the system’s state, and sends the estimate of state to remote controller through an unreliable channel, whereas the channel from remote controller to local controller is perfect; here, the unreliable channel means the Bernoulli channel, that is, when data is transmitted on this channel, there is a certain probability of failure. For the considered constrained optimization problem with Bernoulli channel, we first punish the risk constraint into cost function through Lagrange multiplier method, and the resulting augmented cost function will include a quadratic mean-field term of state. In the sequel, for any but fixed multiplier, explicit solutions to finite-horizon and infinite-horizon mean-field decentralized linear-quadratic problems are derived together with necessary and sufficient condition on the mean-square stability of optimal system. Then, approach to find the optimal Lagrange multiplier is presented based on bisection method. Finally, two numerical examples are given to show the efficiency of the obtained results.

Stochastic LQ optimal control for Markov jumping systems with multiplicative noise using reinforcement learning

The stochastic linear quadratic (LQ) problem of discrete-time linear Markov jumping systems with multiplicative noise is investigated in this paper. Two reinforcement learning algorithms, one model-based and one model-free, are designed. The algorithms can be readily adapted to handle nonlinear cases by employing an appropriate function approximator. In the linear case, the network structure is designed to enable the problem to be solved without succumbing to modeling errors. For the model-free algorithm, an approach based on moving averages is proposed to reduce estimation variance. A proof of convergence is also provided. Finally, the effectiveness of the proposed algorithms is demonstrated through a numerical example.

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Lagrangian dual method for solving stochastic linear quadratic optimal control problems with terminal state constraints

A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed. The strong convergence of the iterative sequence generated by ALM is proved. In addition, some sufficient conditions for the surjectivity of the linear constraint mapping are obtained.

Stochastic linear quadratic optimal control problem with terminal state constraints

AbstractThis paper addresses the stochastic linear quadratic (LQ) control problem with first‐ and second‐order moment constraints on the terminal state. The problem is a modified version of the optimal covariance control problem, where the terminal state is steered to a given probability distribution. Studying a multiplicative‐noise stochastic system rather than an additive‐noise system is a salient feature. Unlike the existing ideas in the optimal steering, by using the Lagrange multipliers method and establishing the stochastic maximum principle, our problem is converted into solving forward–backward stochastic difference equations (FBSDEs), which is a special stochastic two‐point boundary‐value problem (TPBVP). We provide the optimal closed‐loop controller and necessary and sufficient solvability conditions by developing a nonhomogeneous relationship between the optimal state and costate in FBSDEs. Finally, numerical examples are given to demonstrate our results.

Stochastic optimal control and piecewise parameterization and optimization method for inventory control system improvement

In recent decades, stochastic control systems have been widely used in industrial production, biomedicine, aerospace, military strategy and so on. In this paper, an approximate optimal strategy derived from a stochastic linear quadratic (SLQ) optimal control problem is considered, and a piecewise parameterization and optimization (PPAO) method is proposed. Firstly, using the principle of dynamic programming, the control form of SLQ optimal control problem is relevant to a Riccati differential equation. It is well known that the Riccati differential equation is difficult to be solved analytically. Thus, we present a PPAO method for finding an approximate optimal strategy for stochastic control problems. Here, a piecewise parameter control can be obtained by solving first-order differential equations rather than Riccati differential equations. Finally, the inventory control problems with different dimensions are used to justify the feasibility of PPAO method, and the results compared with the original optimal control are given. The results show that parametric control greatly simplifies the control form, and a small model error is ensured.

Stochastic Linear Quadratic Optimal Control of Speed and Position of Multiple Trains on a Single-Track Line

Stochastic Linear Quadratic Optimal Control of Speed and Position of Multiple Trains on a Single-Track Line

Stochastic linear quadratic optimal control problems with expectation-type linear equality constraints on the terminal states

A stochastic linear quadratic (LQ) optimal control problem with an expectation-type linear equality constraint on the terminal state is considered. Under the solvability condition on a stochastic Riccati equation and a surjectivity condition on the linear constraint mapping, the constrained stochastic LQ problem is solved completely by the Lagrangian duality theory. Some equivalent characterizations of the surjectivity condition are discussed by the controllability theory of linear control systems. Especially, the equivalence between the surjectivity condition and a Kalman-type rank condition is proved under proper conditions.

Stochastic Linear-Quadratic Optimal Control with Partial Observation

This paper studies a class of quadratic optimal control problems for partially observable linear dynamical systems. In contrast to the full information case, the control is required to be adapted to the filtration generated by the observation system, which in turn is influenced by the control. The variation method fails in this case due to the fact that the filtration is not fixed. To overcome the difficulty, we use the orthogonal decomposition of the state process to write the cost functional as the sum of two parts: one is a functional of the control and the filtering process and the other part is independent of the choice of the control. The first part possesses a mathematical structure similar to the full information problem. By completing the square, it is shown that the open-loop optimal control is given by a feedback representation via the filtering process. The optimal value is also obtained explicitly.

Stochastic Linear-Quadratic Optimal Control Problems with Random Coefficients and Markovian Regime Switching System

Stochastic Linear-Quadratic Optimal Control Problems with Random Coefficients and Markovian Regime Switching System

On the stochastic linear quadratic optimal control problem by piecewise constant controls: The infinite horizon time case

This paper is devoted to the problem of indefinite stochastic linear quadratic (LQ) optimal control by piecewise constant controls in an infinite horizon case. By restricting the set of admissible controls to the class of piecewise constant stochastic processes, we reformulated the above control problem under the setting of systems modeled by Itô differential equations controlled by impulses. We show that the solution in a state feedback form of the indefinite stochastic LQ control problem is equivalent to the existence of a global stabilizing solution associated to a class of backward matrix linear differential equations with a Riccati type jumping operators.

An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems

Zero-Sum Stackelberg Stochastic Linear-Quadratic Differential Games

The paper is concerned with a zero-sum Stackelberg stochastic linear-quadratic (LQ) differential game over finite horizons. Under a fairly weak condition, the Stackelberg equilibrium is explicitly obtained by first solving a forward stochastic LQ optimal control problem (SLQ problem) and then a backward SLQ problem. Two Riccati equations are derived for constructing the Stackelberg equilibrium. An interesting finding is that the difference of these two Riccati equations coincides with the Riccati equation associated with the zero-sum Nash stochastic LQ differential game, which implies that under the uniform convexity-concavity condition, the Stackelberg equilibrium and the Nash equilibrium are actually identical. Consequently, the Stackelberg equilibrium admits a linear state feedback representation, and the Nash game can be solved in a leader-follower manner.

Erratum to: Turnpike Properties for Stochastic Linear-Quadratic Optimal Control Problems

Erratum to: Turnpike Properties for Stochastic Linear-Quadratic Optimal Control Problems

Stochastic linear quadratic optimal tracking control for discrete-time systems with delays based on Q-learning algorithm

<abstract><p>In this paper, a reinforcement Q-learning method based on value iteration (Ⅵ) is proposed for a class of model-free stochastic linear quadratic (SLQ) optimal tracking problem with time delay. Compared with the traditional reinforcement learning method, Q-learning method avoids the need for accurate system model. Firstly, the delay operator is introduced to construct a novel augmented system composed of the original system and the command generator. Secondly, the SLQ optimal tracking problem is transformed into a deterministic one by system transformation and the corresponding Q function of SLQ optimal tracking control is derived. Based on this, Q-learning algorithm is proposed and its convergence is proved. Finally, a simulation example shows the effectiveness of the proposed algorithm.</p></abstract>

A stochastic linear-quadratic optimal control problem with jumps in an infinite horizon

<abstract><p>In this paper, a stochastic linear-quadratic (LQ, for short) optimal control problem with jumps in an infinite horizon is studied, where the state system is a controlled linear stochastic differential equation containing affine term driven by a one-dimensional Brownian motion and a Poisson stochastic martingale measure, and the cost functional with respect to the state process and control process is quadratic and contains cross terms. Firstly, in order to ensure the well-posedness of our stochastic optimal control of infinite horizon with jumps, the $ L^2 $-stabilizability of our control system with jump is introduced. Secondly, it is proved that the $ L^2 $-stabilizability of our control system with jump is equivalent to the non-emptiness of the admissible control set for all initial state and is also equivalent to the existence of a positive solution to some integral algebraic Riccati equation (ARE, for short). Thirdly, the equivalence of the open-loop and closed-loop solvability of our infinite horizon optimal control problem with jumps is systematically studied. The corresponding equivalence is established by the existence of a $ stabilizing\ solution $ of the associated generalized algebraic Riccati equation, which is different from the finite horizon case. Moreover, any open-loop optimal control for the initial state $ x $ admiting a closed-loop representation is obatined.</p></abstract>

Indefinite Backward Stochastic Linear-Quadratic Optimal Control Problems

This paper is concerned with a backward stochastic linear-quadratic (LQ, for short) optimal control problem with deterministic coefficients. The weighting matrices are allowed to be indefinite, and cross-product terms in the control and state processes are presented in the cost functional. Based on a Hilbert space method, necessary and sufficient conditions are derived for the solvability of the problem, and a general approach for constructing optimal controls is developed. The crucial step in this construction is to establish the solvability of a Riccati-type equation, which is accomplished under a fairly weak condition by investigating the connection with forward stochastic LQ optimal control problems.

Closed-Loop Solvability of Stochastic Linear-Quadratic Optimal Control Problems with Poisson Jumps

The stochastic linear–quadratic optimal control problem with Poisson jumps is addressed in this paper. The coefficients in the state equation and the weighting matrices in the cost functional are all deterministic but are allowed to be indefinite. The notion of closed-loop strategies is introduced, and the sufficient and necessary conditions for the closed-loop solvability are given. The optimal closed-loop strategy is characterized by a Riccati integral–differential equation and a backward stochastic differential equation with Poisson jumps. A simple example is given to demonstrate the effectiveness of the main result.