Stochastic Linear Quadratic Regulator Research Articles

On the Maximum Principle for Optimal Control Problems of Stochastic Volterra Integral Equations with Delay

In this paper, we prove both necessary and sufficient maximum principles for infinite horizon discounted control problems of stochastic Volterra integral equations with finite delay and a convex control domain. The corresponding adjoint equation is a novel class of infinite horizon anticipated backward stochastic Volterra integral equations. Our results can be applied to discounted control problems of stochastic delay differential equations and fractional stochastic delay differential equations. As an example, we consider a stochastic linear-quadratic regulator problem for a delayed fractional system. Based on the maximum principle, we prove the existence and uniqueness of the optimal control for this concrete example and obtain a new type of explicit Gaussian state-feedback representation formula for the optimal control.

Risk-Aware Linear Quadratic Control Using Conditional Value-at-Risk

Stochastic linear quadratic control problems are considered from the viewpoint of risks. In particular, a worst-case conditional value-at-risk (CVaR) of quadratic objective function is minimized subject to additive disturbances whose first two moments of the distribution are known. The study focuses on three problems of finding the optimal feedback gain that minimizes the quadratic cost of: stationary distribution, one-step, and infinite time horizon. For the stationary distribution problem, it is proved that the optimal control gain that minimizes the worst-case CVaR of the quadratic cost is equivalent to that of the standard (stochastic) linear quadratic regulator. For the one-step problem, an approach to an optimal solution as well as analytical suboptimal solutions are presented. For the infinite time horizon problem, two suboptimal solutions that bound the optimal solution and an approach to an optimal solution for a special case are discussed. The presented theorems are illustrated with numerical examples.

Economic MPC of Markov Decision Processes: Dissipativity in undiscounted infinite-horizon optimal control

Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic MPC applies to problems based on deterministic dynamics or to very specific classes of stochastic problems, and does not readily extend to generic Markov decision processes. In this paper, we clarify the core reason for this difficulty, and propose a generalization of the economic MPC dissipativity theory that circumvents it. This generalization focuses on undiscounted infinite-horizon problems and is based on nonlinear stage cost functionals, allowing one to discuss the Lyapunov asymptotic stability of policies for Markov decision processes in terms of the probability measures underlying their stochastic dynamics. This theory is illustrated for the stochastic linear quadratic regulator with Gaussian process noise, for which a storage functional can be provided explicitly. For the sake of brevity, we limit our discussion to undiscounted Markov decision processes.

Model-free design of stochastic LQR controller from a primal–dual optimization perspective

To further understand the underlying mechanism of various reinforcement learning (RL) algorithms and also to better use the optimization theory to make further progress in RL, many researchers begin to revisit the linear–quadratic regulator (LQR) problem, whose setting is simple and yet captures the characteristics of RL. Inspired by this, this work is concerned with the model-free design of stochastic LQR controller for linear systems subject to Gaussian noises, from the perspective of primal–dual optimization. We first reformulate the stochastic LQR problem as a constrained non-convex optimization problem, which is shown to have strong duality. Then, to solve this non-convex optimization problem, we propose a model-based primal–dual (MB-PD) algorithm based on the properties of the resulting Karush–Kuhn–Tucker (KKT) conditions. We also give a model-free implementation for the MB-PD algorithm by solving a transformed dual feasibility condition. More importantly, we establish the connection between the proposed MB-PD algorithm and classical policy iteration algorithm, which provides a novel primal–dual optimization perspective to understand the common RL algorithms. Finally, we provide a high-dimensional case study to show the performance of the proposed algorithms.

Robust Reinforcement Learning for Stochastic Linear Quadratic Control with Multiplicative Noise

This chapter studies the robustness of reinforcement learning for discrete-time linear stochastic systems with multiplicative noise evolving in continuous state and action spaces. As one of the popular methods in reinforcement learning, the robustness of policy iteration is a longstanding open problem for the stochastic linear quadratic regulator (LQR) problem with multiplicative noise. A solution in the spirit of input-to-state stability is given, guaranteeing that the solutions of the policy iteration algorithm are bounded and enter a small neighborhood of the optimal solution, whenever the error in each iteration is bounded and small. In addition, a novel off-policy multiple-trajectory optimistic least-squares policy iteration algorithm is proposed, to learn a near-optimal solution of the stochastic LQR problem directly from online input/state data, without explicitly identifying the system matrices. The efficacy of the proposed algorithm is supported by rigorous convergence analysis and numerical results on a second-order example.

Stochastic Secondary Frequency Control of Islanded Microgrid Under Uncertainties

As the penetration of renewable energy, generally, uncertainties, increase in microgrids, a more dynamic and complex system is emerging that makes the load frequency control (LFC) of islanded microgrids more challenging due to their stochastic dynamic nature. This stochastic nature can create oscillatory frequency response, which eventually leads to deteriorating control function and instability even under primary and traditional secondary controllers. Due to the uncertainties, the measurements and microgrid states are continually perturbed; hence, they cannot be directly utilized for feedback control. Therefore, it is required to utilize an observer-based stochastic feedback control that correctly recognizes these uncertainties and sieves incorrect states, frequency deviation here, and can properly maintain the balance between the energy sources and the demand. This article develops an optimal observer-based secondary control for LFC, framed in two layers that are the stochastic unknown input observer (SUIO) and linear quadratic regulator. The proposed SUIO not only can address the uncertainties, e.g., renewable energy, load, and measurement noise, with efficient control effort but also performs robust against parameter changes of the microgrid. The simulation results confirm the efficacy of this proposed control as compared to the traditional secondary control method.

A Stochastic LQR Model for Child Order Placement in Algorithmic Trading

Modern Algorithmic Trading (Algo) allows institutional investors and traders to liquidate or establish big security positions in a fully automated or low-touch manner. Most existing academic or industrial Algos focus on how to slice a big parent order into smaller child orders over a given time horizon. Few models rigorously tackle the actual placement of these child orders. Instead, placement is mostly done with a combination of empirical signals and heuristic decision processes. A self-contained, realistic, and fully functional Child Order Placement (COP) model may never exist due to all the inherent complexities, e.g., fragmentation due to multiple venues, dynamics of limit order books, lit vs. dark liquidity, different trading sessions and rules. In this paper, we propose a reductionism COP model that focuses exclusively on the interplay between placing passive limit orders and sniping using aggressive takeout orders. The dynamic programming model assumes the form of a stochastic linear-quadratic regulator (LQR) and allows closed-form solutions under the backward Bellman equations. Explored in detail are model assumptions and general settings, the choice of state and control variables and the cost functions, and the derivation of the closed-form solutions.

Solving quantum stochastic LQR optimal control problem in Fock space and its application in finance

This paper is an attempt for solving operator-valued quantum stochastic optimal control problems, in Fock space. For this purpose, the dynamics of the classical system state is described by Hudson–Parthasarathy type Quantum Stochastic Differential Equation (QSDE) in Fock space and then by associating a quadratic performance criterion with the QSDE, a Quantum Stochastic Linear Quadratic Regulator (QS-LQR) optimal control problem is formulated. Also, an algorithm for solving the QS-LQR optimal control problem is designed. For solving the resulting optimal control problem, a new HJB equation is obtained. Thereby, the operator valued control process is obtained. Two theorems are proved to facilitate the algorithm. In this paper, for the first time, the optimal strategy for trading stock is designed via the presented method. For this purpose, Merton portfolio allocation problem is solved. The simulation results show that portfolio optimal performances, minimum risk and maximum return are achieved via presented method.

Reliability‐based control algorithms for nonlinear hysteretic systems based on enhanced stochastic averaging of energy envelope

SummaryCurrent reliability‐based control techniques have been successfully applied to linear systems; however, incorporation of stochastic nonlinear behavior of systems in such control designs remains a challenge. This paper presents two reliability‐based control algorithms that minimize failure probabilities of nonlinear hysteretic systems subjected to stochastic excitations. The proposed methods include constrained reliability‐based control (CRC) and unconstrained reliability‐based control (URC) algorithms. Accurate probabilistic estimates of nonlinear system responses to stochastic excitations are derived analytically using enhanced stochastic averaging of energy envelope proposed previously by the authors. Convolving these demand estimates with capacity models yields the reliability of nonlinear systems in the control design process. The CRC design employs the first‐level and second‐level optimizations sequentially where the first‐level optimization solves the Hamilton–Jacobi–Bellman equation and the second‐level optimization searches for optimal objective function parameters to minimize the probability of failure. In the URC design, a single optimization minimizes the probability of failure by directly searching for the optimal control gain. Application of the proposed control algorithms to a building on nonlinear foundation has shown noticeable improvements in system performance under various stochastic excitations. The URC design appears to be the most optimal method as it reduced the probability of slight damage to 8.7%, compared with 11.6% and 19.2% for the case of CRC and a stochastic linear quadratic regulator, respectively. Under the Kobe ground motion, the normalized peak drift displacement with respect to stochastic linear quadratic regulator is reduced to 0.78 and 0.81 for the URC and CRC cases, respectively, at comparable control force levels. Copyright © 2017 John Wiley & Sons, Ltd.

Solution to stochastic LQR problem with multiple inputs

This paper considers the stochastic linear quadratic regulation (LQR) problem for Ito stochastic systems with multiple input controllers. The explicit controllers are given in terms of two Riccati equations by introducing one new costate and establishing the homogeneous relationship between the state and the new costate. More importantly, it is more computation saving for the derived Riccati equations than the one derived by augmentation technique.

A Numerical Approximation Framework for the Stochastic Linear Quadratic Regulator on Hilbert Spaces

We present an approximation framework for computing the solution of the stochastic linear quadratic control problem on Hilbert spaces. We focus on the finite horizon case and the related differential Riccati equations (DREs). Our approximation framework is concerned with the so-called "singular estimate control systems" (Lasiecka in Optimal control problems and Riccati equations for systems with unbounded controls and partially analytic generators: applications to boundary and point control problems, 2004) which model certain coupled systems of parabolic/hyperbolic mixed partial differential equations with boundary or point control. We prove that the solutions of the approximate finite-dimensional DREs converge to the solution of the infinite-dimensional DRE. In addition, we prove that the optimal state and control of the approximate finite-dimensional problem converge to the optimal state and control of the corresponding infinite-dimensional problem.

Reliable approximations of probability-constrained stochastic linear-quadratic control

Here we consider a state-constrained stochastic linear-quadratic control problem. This problem has linear dynamics and a quadratic cost, and states are required to satisfy a probabilistic constraint. In this paper, the joint probabilistic constraint in the model is converted to a conservative deterministic constraint using a multi-dimensional Chebyshev bound. A maximum volume inscribed ellipsoid problem is solved to obtain this probability bound. Using the probability bound, we develop a recursive state feedback control algorithm for a special class of state-constrained stochastic linear-quadratic regulator (LQR). The performance of this approach is explored in a numerical example.

Energy-based stochastic control of neural mass models suggests time-varying effective connectivity in the resting state

Several studies posit energy as a constraint on the coding and processing of information in the brain due to the high cost of resting and evoked cortical activity. This suggestion has been addressed theoretically with models of a single neuron and two coupled neurons. Neural mass models (NMMs) address mean-field based modeling of the activity and interactions between populations of neurons rather than a few neurons. NMMs have been widely employed for studying the generation of EEG rhythms, and more recently as frameworks for integrated models of neurophysiology and functional MRI (fMRI) responses. To date, the consequences of energy constraints on the activity and interactions of ensembles of neurons have not been addressed. Here we aim to study the impact of constraining energy consumption during the resting-state on NMM parameters. To this end, we first linearized the model, then used stochastic control theory by introducing a quadratic cost function, which transforms the NMM into a stochastic linear quadratic regulator (LQR). Solving the LQR problem introduces a regime in which the NMM parameters, specifically the effective connectivities between neuronal populations, must vary with time. This is in contrast to current NMMs, which assume a constant parameter set for a given condition or task. We further simulated energy-constrained stochastic control of a specific NMM, the Wilson and Cowan model of two coupled neuronal populations, one of which is excitatory and the other inhibitory. These simulations demonstrate that with varying weights of the energy-cost function, the NMM parameters show different time-varying behavior. We conclude that constraining NMMs according to energy consumption may create more realistic models. We further propose to employ linear NMMs with time-varying parameters as an alternative to traditional nonlinear NMMs with constant parameters.

Networked control approaches for a nuclear power plant pressurizer subsystem

This paper extends former results about the networked reference tracking control of a pressurizer in a pressurized water nuclear power plant. An L2-gain based method is used in a deterministic framework to compute the maximum allowable transmit intervals (MATIs) for the actually implemented dynamic inversion based controller that still guarantee the required performance. Furthermore, a stochastic linear quadratic regulator with a state estimator is also designed that can take packet losses into consideration. It is shown through the comparison that both methods are applicable for networked control but the LQ controller combined with a Kalman filter has better output variance properties.

Stochastic linear quadratic regulation for discrete-time linear systems with input delay

This paper considers the stochastic LQR problem for systems with input delay and stochastic parameter uncertainties in the state and input matrices. The problem is known to be difficult due to the presence of interactions among the delayed input channels and the stochastic parameter uncertainties in the channels. The key to our approach is to convert the LQR control problem into an optimization one in a Hilbert space for an associated backward stochastic model and then obtain the optimal solution to the stochastic LQR problem by exploiting the dynamic programming approach. Our solution is given in terms of two generalized Riccati difference equations (RDEs) of the same dimension as that of the plant.

Generalized Lyapunov Equation Approach to State-Dependent Stochastic Stabilization/Detectability Criterion

In this paper, the generalized Lyapunov equation approach is used to study stochastic stabilization/detectability with state-multiplicative noise. Some practical test criteria for stochastic stabilization and detectability, such as stochastic Popov-Belevitch-Hautus criterion for exact detectability, are obtained. Moreover, useful properties of the generalized Lyapunov equation are derived based on critical stability and exact detectability introduced in this paper. As applications, first, the stochastic linear quadratic regulator as well as the related generalized algebraic Riccati equation are discussed extensively. Second, the infinite horizon stochastic <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> / <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">infin</sub> control with state- and control-dependent noise is also investigated, which extends and improves the recently published results.

Joint Predictive Control of Power and Rate for Wireless Networks

To mitigate the loop delay in distributed wireless networks, a predictive power and rate control scheme is proposed for the system model that also accounts for the congestion levels and input delay instead of state-delayed in a network. A measurement feedback control problem with input delay is formulated by minimizing the energy of the difference between the actual and the desired signal-to-interference-plus-noise ratio (SNR) levels, as well as the energy of the control sequence. To solve this problem, we present two Riccati equations for the control and the estimation for the time delay systems. A complete analytical optimal controller is obtained by using the separation principle and solving two Riccati equations, where one is backward equation for stochastic linear quadratic regulation and the other is the standard filtering Riccati equation. Simulation results illustrate the performance of the proposed power and the rate control scheme.

An iterative method for solving stochastic Riccati differential equations for the stochastic LQR problem

This paper develops an iterative method for solving stochastic Riccati differential equations for stochastic linear quadratic regulator (LQR) problems some of which may be even with indefinite control weight costs. We obtain a sequence of solutions of linear matrix differential equations which converges uniformly to the solution of a stochastic Riccati differential equation.

Almost sure optimality and optimality in probability for stochastic linear-quadratic regulator with partial information

Partially observed linear-quadratic regulator is considered over an infinite time horizon. A limiting per unit time inequality is proved for the random difference between the cost corresponding to the feedback control based on Kalman filter estimates and the cost corresponding to an alternative control. Under suitable assumptions of admissibility for a control, it is shown that the feedback control mentioned above is asymptotically optimal almost surely and in probability

Stochastic optimal LQR control with integral quadratic constraints and indefinite control weights

A standard assumption in traditional (deterministic and stochastic) optimal (minimizing) linear quadratic regulator (LQR) theory is that the control weighting matrix in the cost functional is strictly positive definite. In the deterministic case, this assumption is in fact necessary for the problem to be well-posed because positive definiteness is required to make it a convex optimization problem. However, it has recently been shown that in the stochastic case, when the diffusion term is dependent on the control, the control weighting matrix may have negative eigenvalues but the problem remains well-posed. In this paper, the completely observed stochastic LQR problem with integral quadratic constraints is studied. Sufficient conditions for the well-posedness of this problem are given. Indeed, we show that in certain cases, these conditions may be satisfied, even when the control weighting matrices in the cost and all of the constraint functionals have negative eigenvalues. It is revealed that the seemingly nonconvex problem (with indefinite control weights) can actually be a convex one by virtue of the uncertainty in the system. Finally, when these conditions are satisfied, the optimal control is explicitly derived using results from duality theory.