Linear Quadratic Optimal Control Problem Research Articles

Be greedy and learn: efficient and certified algorithms for parametrized optimal control problems

We consider parametrized linear-quadratic optimal control problems and provide their online-efficient solutions by combining greedy reduced basis methods and machine learning algorithms. To this end, we first extend the greedy control algorithm, which builds a reduced basis for the manifold of optimal final time adjoint states, to the setting where the objective functional consists of a penalty term measuring the deviation from a desired state and a term describing the control energy. Afterwards, we apply machine learning surrogates to accelerate the online evaluation of the reduced model. The error estimates proven for the greedy procedure are further transferred to the machine learning models and thus allow for efficient a posteriori error certification. We discuss the computational costs of all considered methods in detail and show by means of two numerical examples the tremendous potential of the proposed methodology.

Synthesis of a Regulator for a Linear-Quadratic Optimal Control Problem

Synthesis of a Regulator for a Linear-Quadratic Optimal Control Problem

Linear quadratic optimal control of stochastic 2-D Roesser models

This paper investigates the linear quadratic (LQ) optimal control problem for the stochastic two-dimensional (2-D) systems governed by Roesser models with multiplicative noise. The main contribution is to give the necessary and sufficient optimality condition by proposing a set of novel forward and backward stochastic partial difference equations (FBSPDE), and to further present the explicitly optimal feedback control laws on the finite horizon and on the infinite horizon based on the Riccati-like difference equations and the algebraic equation, respectively. Several numerical simulations are provided to illustrate the performance of the designed controllers.

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Finite-horizon and infinite-horizon linear quadratic optimal control problems: A data-driven Euler scheme

In this paper, we devote our focus to studying finite-horizon and infinite-horizon linear quadratic optimal control (LQOC) problems in continuous time. We focus on algorithms for approximating optimal controls of these two problems without the knowledge of all system coefficients. Firstly, we transform the backward differential Riccati equation (DRE) corresponding to the finite-horizon LQOC problem into a forward ordinary differential equation. Then, we discretize the forward equation by the Euler method. Subsequently, we use the collected state and input data to calculate all discretized points of the forward equation. At the same time, we obtain the optimal control of the finite-horizon LQOC problem. Moreover, we approximate the optimal control of the infinite-horizon LQOC problem by virtue of an asymptotic behavior between the DRE and an algebraic Riccati equation (ARE) arising in the infinite-horizon LQOC problem. Finally, we validate the obtained results via two practically motivated examples.

Infinite Horizon Mean-Field Linear Quadratic Optimal Control Problems with Jumps and the Related Hamiltonian Systems

Infinite Horizon Mean-Field Linear Quadratic Optimal Control Problems with Jumps and the Related Hamiltonian Systems

Linear quadratic optimal control problems of infinite‐dimensional mean‐field type with jumps

AbstractIn this paper, we formulate and investigate a framework for the theory of the linear quadratic optimal control problem (LQ problem) for infinite‐dimensional mean‐field stochastic evolution systems with jumps. We ensure the well‐posedness of the investigated problems by establishing the existence, uniqueness, and a priori estimates for mild solutions to general infinite‐dimensional mean‐field forward stochastic evolution equations (MF‐SEE) and mean‐field backward stochastic evolution equations (MF‐BSEE) with jumps. Leveraging the Yosida approximation theory, we establish a dual theory between MF‐SEE and MF‐BSEE with jumps, overcoming the challenge posed by the inapplicability of Itô's formula in the context of mild solutions. Our main results regarding the existence and uniqueness of the optimal control, along with corresponding dual characterizations and state feedback representations, are obtained through convex analysis techniques, our established dual theory, and decoupling methods.

Linear quadratic optimal control turnpike in finite and infinite dimension: Two-term expansion of the value function

In this paper, we consider a linear quadratic (LQ) optimal control problem in both finite and infinite dimensions. We derive an asymptotic expansion of the value function as the fixed time horizon T tends to infinity. The leading term in this expansion, proportional to T, corresponds to the optimal value attained through the classical turnpike theory in the associated static problem. The remaining terms are associated with optimal stabilization problems towards the turnpike.

Decentralized stochastic linear-quadratic optimal control with risk constraint and partial observation

This paper addresses a risk-constrained decentralized stochastic linear-quadratic optimal control problem with one remote controller and one local controller, where the risk constraint is posed on the cumulative state weighted variance in order to reduce the oscillation of system trajectory. In this model, local controller can only partially observe the system’s state, and sends the estimate of state to remote controller through an unreliable channel, whereas the channel from remote controller to local controller is perfect; here, the unreliable channel means the Bernoulli channel, that is, when data is transmitted on this channel, there is a certain probability of failure. For the considered constrained optimization problem with Bernoulli channel, we first punish the risk constraint into cost function through Lagrange multiplier method, and the resulting augmented cost function will include a quadratic mean-field term of state. In the sequel, for any but fixed multiplier, explicit solutions to finite-horizon and infinite-horizon mean-field decentralized linear-quadratic problems are derived together with necessary and sufficient condition on the mean-square stability of optimal system. Then, approach to find the optimal Lagrange multiplier is presented based on bisection method. Finally, two numerical examples are given to show the efficiency of the obtained results.

Optimal control for both forward and backward discrete-time systems

Forward discrete-time systems use past information to update the current state, while backward discrete-time systems use future information to update the current state. This study focuses on optimal control problems within the context of forward and backward discrete-time systems. We begin by investigating a general optimal control problem for both forward and backward discrete-time systems. Leveraging the inherent properties of these systems and the Bellman optimality principle, we derive recursive equations as a means to solve such optimal control problems. Using these recursive equations, we obtain analytical expressions for both the optimal controls and optimal values of bang–bang and linear quadratic optimal control problems. Finally, we present a numerical example and an industrial wastewater treatment problem to illustrate and demonstrate our findings.

Pointwise Error Estimates of Numerical Solutions to Linear Quadratic Optimal Control Problems

An auxiliary optimal control problem is formulated that provides with its unique solution, a continuous representation of the global error of a numerical approximation to the solution of a linear quadratic optimal control problem. The resulting error functions are characterized as the unique solutions to an optimality system that is reformulated as a boundary value problem. With this formulation, reliable pointwise error estimates can be generated utilizing well-established techniques of defect control. A novel algorithm based on defect correction and defect control is presented that generates pointwise approximations to the global error of numerical optimal control solutions on a uniform grid. It is proven and numerically validated that this algorithm can generate pointwise estimates that approximate the true global error with a prescribed accuracy.

Turnpike Properties for Mean-Field Linear-Quadratic Optimal Control Problems

Turnpike Properties for Mean-Field Linear-Quadratic Optimal Control Problems

Minimization of integral quadratic estimate of controlled variable in systems with distributed parameters

A constructive method for solving the linear-quadratic problem of optimal control of a parabolic-type system with distributed parameters is proposed under the condition of uniform estimation of target sets. The optimality criterion takes the form of an integral quadratic estimate of the controlled state function in the spatio-temporal domain of its definition. A parameterized representation of control inputs is given with the required accuracy within special intervals of the optimal process, where control inputs cannot be determined using first-order analytical optimality conditions. The suggested approach is based on a previously developed alternance method for constructing parameterized algorithms of programmed control, which heavily relies on fundamental regularities of the subject area. It is demonstrated that the equations of the optimal regulators within the special intervals are reduced to the linear feedback algorithms based on the measured states of the objects. These algorithms are supplemented with switches at boundary points to apply admissible control inputs corresponding to the calculated values of the controlled variable.

Forward–backward stochastic evolution equations in infinite dimensions and application to LQ optimal control problems

This paper focuses on the study of forward–backward stochastic evolution equations (FBSEEs), which are a class of nonlinear fully coupled forward–backward stochastic differential equations (FBSDEs), in infinite dimensions. Drawing inspiration from various linear-quadratic (LQ) optimal control problems, we apply a set of domination-monotonicity conditions that are more relaxed compared to general conditions. Within this framework, we employ the method of continuation to establish the unique solvability result and provide a pair of solution estimates. Notably, to address the challenges posed by the infinite-dimensional setting, we introduce a class of approximating equations, as Itô’s formula is not directly applicable. Conversely, these results find application in various LQ problems, where the stochastic Hamiltonian systems precisely correspond to the FBSEEs satisfying the aforementioned domination-monotonicity conditions. Consequently, by solving the corresponding stochastic Hamiltonian systems, we can obtain explicit expressions for the optimal controls.

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration for LQR control of unknown stochastic-parameter linear systems

This paper focuses on the linear-quadratic optimal control problem for unknown stochastic-parameter linear systems using reinforcement learning methods. Based on the second moments of random system matrices, a model-based value iteration algorithm is proposed to solve the problem and is proved to be convergent by using the contraction mapping theorem. For the case without knowing any information about the random system matrices, a normalized model-free value iteration algorithm is presented to learn the optimal control law by estimating the data for the next time. The collected data are normalized first in our model-free algorithm to reduce the error of the least squares method. It is proved that our algorithm can obtain an approximate optimal solution. Our algorithm is applicable for any distribution of the random system matrices and does not require an initial mean-square stabilizing control policy. Finally, an example illustrates that our algorithm converges to an approximate optimal control policy and that the normalization step can significantly reduce convergence errors.

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Lagrangian dual method for solving stochastic linear quadratic optimal control problems with terminal state constraints

A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed. The strong convergence of the iterative sequence generated by ALM is proved. In addition, some sufficient conditions for the surjectivity of the linear constraint mapping are obtained.

Time-inconsistent linear quadratic optimal control problem for forward–backward stochastic differential equations

We study the time-inconsistent linear quadratic optimal control problem for forward–backward stochastic differential equations with potentially indefinite cost weighting matrices for both the state and the control variables. Our research makes two contributions. Firstly, we introduce a novel type of Riccati equation system with parameters and constraint conditions, known as the generalized equilibrium Riccati equation. This equation system offers a comprehensive solution for the closedloop equilibrium strategy of the problem at hand. Secondly, we establish the well-posedness of the generalized equilibrium Riccati equation for the one-dimensional case, provided certain conditions are met.

Linear quadratic optimal control problems for stochastic evolution equations in infinite horizon

Linear quadratic optimal control problems for stochastic evolution equations in infinite horizon