Linear Quadratic Control Problem Research Articles

Approximation of optimal feedback controls for stochastic reaction-diffusion equations

In this paper we present a method to approximate optimal feedback controls for stochastic reaction-diffusion equations. We derive two approximation results providing the theoretical foundation of our approach and allowing for explicit error estimates. The approximation of optimal feedback controls by neural networks is discussed as an explicit application of our method. We illustrate our findings in the case of a linear quadratic control problem with a numerical example.

Stochastic linear–quadratic control problems with affine constraints

This paper investigates the stochastic linear–quadratic control problems with affine constraints, in which both equality and inequality constraints are involved. With the help of the Pontryagin maximum principle and Lagrangian duality theory, both the dual problem and the state feedback form of the solution are obtained for the primal problem. Under the Slater condition, the strong duality is proved between the dual problem and the primal problem, and the KKT condition is also provided for solving the primal problem. Moreover, a new sufficient condition is given for the invertibility assumption, which ensures the uniqueness of the solutions to the dual problem. Finally, two numerical examples are provided to illustrate our main results.

Hybrid coati–grey wolf optimization with application to tuning linear quadratic regulator controller of active suspension systems

Vehicle suspension systems have become increasingly crucial for both driving safety and comfort. Active suspension systems can dynamically adjust suspension characteristics in real-time by introducing force into the system. Designing a controller for the real-time adjustment of the control force in active suspension systems is essential to meet challenging control objectives, including body acceleration, suspension deflection, and tire deflection. This article proposes a hybrid optimization approach named Coati–Grey Wolf Optimization (COAGWO), which combines the strengths of the coati optimization algorithm and grey wolf optimization to tune the gains of linear quadratic control applied to vehicle suspension systems. The COAGWO algorithm incorporates a unique strategy inspired by the Coati Optimization Algorithm, allowing wolves to climb trees. This enhancement significantly improves the wolves’ ability to explore the global search space and reduces the likelihood of being trapped in local optima. Initially, we conduct extensive experiments using a suite of challenging optimization problems from the CEC2019 benchmark to evaluate the effectiveness of the COAGWO algorithm. The effectiveness of COAGWO is compared against several state-of-the-art algorithms, including grey wolf, coati, aquila-grey wolf, whale, reptile search, tunicate swarm, and seagull optimization algorithms. The experimental results demonstrate that COAGWO consistently outperforms these algorithms in terms of solution quality and convergence speed. For the optimal weight selection problem of linear quadratic control applied to the control of vehicle suspension systems, the excellent performance of the proposed method is illustrated through comparative simulation studies under various road disturbance conditions. The results indicate that the COAGWO algorithm achieves a more efficient active suspension system compared to competitor algorithms by reducing the overall acceleration of the driver’s body, thereby enhancing ride comfort.

Linear–quadratic stochastic Volterra controls I: Causal feedback strategies

In this paper, we formulate and investigate the notion of causal feedback strategies arising in linear–quadratic control problems for stochastic Volterra integral equations (SVIEs) with singular and non-convolution-type coefficients. We show that there exists a unique solution, which we call the causal feedback solution, to the closed-loop system of a controlled SVIE associated with a causal feedback strategy. Furthermore, introducing two novel equations named a Type-II extended backward stochastic Volterra integral equation and a Lyapunov–Volterra equation, we prove a duality principle and a representation formula for a quadratic functional of controlled SVIEs in the framework of causal feedback strategies.

Adaptive temperature control of a reverse flow process by using reinforcement learning approach

This work focuses on the design of an optimal adaptive control system for temperature regulation in a catalytic flow reversal reactor (CFRR), utilizing a reinforcement learning (RL) approach. First, a policy iteration algorithm is introduced to learn the optimal solution of the associated linear-quadratic control problem online. It should be mentioned that this approach is not reliant on the internal dynamics of the CFRR system, which is a complex process and is most effectively modeled using Partial Differential Equations (PDEs). The convergence of the iteration algorithm is established, assuming the initial policy is stabilizing. Additionally, a second algorithm is presented to enhance the implementability of the reinforcement learning algorithm from a practical perspective. Numerical simulations are carried out to illustrate the efficacy of the proposed algorithm.

Stochastic maximum principle for partially observed optimal control problem of McKean–Vlasov FBSDEs with Teugels martingales

Abstract In this paper, we study the stochastic maximum principle for a partially observed optimal control problem of forward-backward stochastic differential equations (FBSDEs for short) of McKean–Vlasov type driven by both a family of Teugels martingales and an independent Brownian motion. The coefficients of the system and the cost functional depending on the state of the solution process as well as its probability law and the control variable. We establish partially observed necessary conditions of optimality for this system under assumption that the control domain is supposed to be convex. Our main result is based on Girsavov’s theorem and the derivatives with respect to probability law. As an application of the general theory, a partially observed linear-quadratic control problem of McKean–Vlasov type is studied in terms of stochastic filtering.

Linear Quadratic Control Problem of Systems With Markov Jumps in Reverse Time and Observation With Anticipation of the Jumps

Linear Quadratic Control Problem of Systems With Markov Jumps in Reverse Time and Observation With Anticipation of the Jumps

A necessary and sufficient condition for the existence of the stabilizing solution of a large class of discrete-time Riccati type equations with periodic coefficients

This paper is devoted to the study of a large class of discrete-time backward Riccati equations arising in several linear quadratic (LQ) type control problems both in the deterministic and in the stochastic frameworks. The periodic time-varying case is considered. We propose existence and uniqueness conditions for a global special solution named stabilizing solution for such equations. Beside the stabilizability condition, the criterion derived in this paper is expressed based on some suitable properties of the characteristic multipliers of a discrete-time, periodic linear equation adequately constructed using the coefficients of the given equation. Our result englobes, as particular cases, several existing results in the literature.

Stochastic controls of fractional Brownian motion

Abstract We consider a stochastic control problem for a non-linear forward-backward stochastic differential equation driven by fractional Brownian motion, with Hurst parameter H ∈ ( 0 , 1 ) {H\in(0,1)} , in the case where the set of the control domain is convex. We provide an estimation of the solution and establish the necessary and sufficient optimality conditions in the form of the stochastic maximum principle. We apply the theory to solve a linear quadratic stochastic control problem.

Global Optimality Guarantees for Policy Gradient Methods

Policy gradient methods, which have powered a lot of recent success in reinforcement learning, search for an optimal policy in a parameterized policy class by performing stochastic gradient descent on the cumulative expected cost-to-go under some initial state distribution. Although widely used, these methods lack theoretical guarantees as the optimization objective is typically nonconvex even for simple control problems, and hence are understood to only converge to a stationary point. In “Global Optimality Guarantees for Policy Gradient Methods,” J. Bhandari and D. Russo identify structural properties of the underlying MDP that guarantee that despite nonconvexity, the optimization objective has no suboptimal stationary points, ensuring asymptotic convergence of policy gradient methods to globally optimal policies. Under stronger conditions, authors show the policy gradient objective to satisfy a Polyak-lojasiewicz (gradient dominance) condition that yields fast convergence rates. In addition, when some of the said conditions are relaxed, authors provide bounds on the suboptimality gap of any stationary point. The results rely on a key connection with policy iteration, a classic dynamic programming algorithm which solves a single period optimization problem at every step. The authors show how structure in the single period optimization problems solved by policy iteration translate into nice properties of the multiperiod policy gradient objective, making it amenable for first-order methods to find globally optimal solutions. The authors also instantiate their framework for several classical control problems including tabular and linear MDPs, linear quadratic control, optimal stopping, and finite-horizon inventor control problems.

Douglas–Rachford algorithm for control- and state-constrained optimal control problems

<abstract><p>We consider the application of the Douglas–Rachford (DR) algorithm to solve linear-quadratic (LQ) control problems with box constraints on the state and control variables. We have split the constraints of the optimal control problem into two sets: one involving the ordinary differential equation with boundary conditions, which is affine, and the other, a box. We have rewritten the LQ control problems as the minimization of the sum of two convex functions. We have found the proximal mappings of these functions, which we then employ for the projections in the DR iterations. We propose a numerical algorithm for computing the projection onto the affine set. We present a conjecture for finding the costates and the state constraint multipliers of the optimal control problem, which can, in turn, be used to verify the optimality conditions. We conducted numerical experiments with two constrained optimal control problems to illustrate the performance and the efficiency of the DR algorithm in comparison with the traditional approach of direct discretization.</p></abstract>

Linear-quadratic stochastic volterra controls II. Optimal strategies and riccati-volterra equations

In this paper, we study linear-quadratic control problems for stochastic Volterra integral equations with singular and non-convolution-type coefficients. The weighting matrices in the cost functional are not assumed to be non-negative definite. From a new viewpoint, we formulate a framework of causal feedback strategies. The existence and the uniqueness of a causal feedback optimal strategy are characterized by means of the corresponding Riccati-Volterra equation. The causal feedback optimal strategy is explicitly written by a finite dimensional (matrix-valued) function which solves the Riccati-Volterra equation.

Constrained mean-variance investment-reinsurance under the Cramér–Lundberg model with random coefficients

In this paper, we study an optimal mean-variance investment-reinsurance problem for an insurer (she) under the Cramér–Lundberg model with random coefficients. At any time, the insurer can purchase reinsurance or acquire new business and invest her surplus in a security market consisting of a risk-free asset and multiple risky assets, subject to a general convex cone investment constraint. We reduce the problem to a constrained stochastic linear-quadratic control problem with jumps whose solution is related to a system of partially coupled stochastic Riccati equations (SREs). Then we devote ourselves to establishing the existence and uniqueness of solutions to the SREs by pure backward stochastic differential equation (BSDE) techniques. We achieve this with the help of approximation procedure, comparison theorems for BSDEs with jumps, log transformation and BMO martingales. The efficient investment-reinsurance strategy and efficient mean-variance frontier are explicitly given through the solutions of the SREs, which are shown to be a linear feedback form of the wealth process and a half-line, respectively.

Stochastic linear quadratic optimal control problem with terminal state constraints

AbstractThis paper addresses the stochastic linear quadratic (LQ) control problem with first‐ and second‐order moment constraints on the terminal state. The problem is a modified version of the optimal covariance control problem, where the terminal state is steered to a given probability distribution. Studying a multiplicative‐noise stochastic system rather than an additive‐noise system is a salient feature. Unlike the existing ideas in the optimal steering, by using the Lagrange multipliers method and establishing the stochastic maximum principle, our problem is converted into solving forward–backward stochastic difference equations (FBSDEs), which is a special stochastic two‐point boundary‐value problem (TPBVP). We provide the optimal closed‐loop controller and necessary and sufficient solvability conditions by developing a nonhomogeneous relationship between the optimal state and costate in FBSDEs. Finally, numerical examples are given to demonstrate our results.

Decentralized control for discrete‐time mean‐field systems with multiple controllers of delayed information

AbstractIn this paper, the finite horizon asymmetric information linear quadratic (LQ) control problem is investigated for a discrete‐time mean field system. Different from previous works, multiple controllers with different information sets are involved in the mean field system dynamics. The coupling of different controllers makes it quite difficult in finding the optimal control strategy. Fortunately, by applying the Pontryagin's maximum principle, the corresponding decentralized control problem of the finite horizon is investigated. The contributions of this paper can be concluded as follows: For the first time, based on the solution of a group of mean‐field forward and backward stochastic difference equations (MF‐FBSDEs), the necessary and sufficient solvability conditions are derived for the asymmetric information LQ control for the mean field system with multiple controllers. Furthermore, by the use of an innovative orthogonal decomposition approach, the optimal decentralized control strategy is derived, which is based on the solution to a non‐symmetric Riccati‐type equation.

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Indefinite LQ optimal control for stochastic Takagi–Sugeno fuzzy system under sensor data scheduling: Finite‐horizon case

AbstractThe present paper considers the finite‐horizon indefinite linear quadratic (LQ) control problem for stochastic Takagi–Sugeno (T‐S) fuzzy systems with input delay. In this paper, we consider the presence of sensor data scheduling, which imposes a communication energy constraint and necessitates optimal state estimation for measurements. Then, by utilizing dynamic programming principles, the stochastic LQ problem under consideration can be solved, while the optimal control policy is developed in terms of the unique solutions to a set of coupled difference Riccati equations (CDREs). Specifically, for simple delay‐free case, the linear matrix inequalities based conditions are also proposed, whose feasibility is shown to be equivalent to the well‐posedness of the indefinite LQ control under consideration. As an application, our theoretic analysis is extended to study the intermittent observation model caused by random denial‐of‐service attack.

The mean‐field linear quadratic optimal control problem for stochastic systems controlled by impulses

AbstractIn the present note, we state and solve the linear quadratic (LQ) control problem for a class of McKean–Vlasov stochastic differential equations for which the control actions are of impulsive nature. After reformulating the original optimization problem using an orthogonal decomposition of the state variables, we introduce an adequately defined system of two coupled matrix Lyapunov‐type differential equations with jumps. Such equations are central for the definition of the optimal feedback gains corresponding to the closed‐loop representation of the optimal LQ control.

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Error Analysis of the Feedback Controls Arising in the Stochastic Linear Quadratic Control Problems

Zero-Order Asymptotics for the Solution of One Type of Singularly Perturbed Linear–Quadratic Control Problems in the Critical Case

Zero-Order Asymptotics for the Solution of One Type of Singularly Perturbed Linear–Quadratic Control Problems in the Critical Case