Stochastic Optimal Control Problem Research Articles

Computationally Tractable Gaussian Process-based Stochastic Predictive Control Using Backoffs

Current stochastic nonlinear model predictive control (SNMPC) hinges on the lack of high-fidelity models that describe the system behavior and the lack of tractable solution methods that handle chance constraints. Motivated by this, a model-and data-driven predictive control approach using Gaussian processes (GP-MDPC) is synthesized in this paper. It exploits GPs to learn the unknown dynamics and apply Taylor expansion for uncertainty propagation through probabilistic modeling. A backoff approximation method is explored to reformulate the chance constraints into tractable expressions. Finally, a finite-horizon stochastic optimal control problem (FH-SOCP) is formulated.

Sufficient maximum principle for partially observed mean-field stochastic optimal control problems with delays

Sufficient maximum principle for partially observed mean-field stochastic optimal control problems with delays

Well-posedness for a transmission problem connecting first and second-order operators

We establish the existence and uniqueness of viscosity solutions within a domain Ω ⊆ ℝn for a class of equations governed by elliptic and eikonal type equations in disjoint regions. Our primary motivation stems from the Hamilton-Jacobi equation that arises in the context of a stochastic optimal control problem.

Optimal Control of Infinite-Dimensional Differential Systems with Randomness and Path-Dependence and Stochastic Path-Dependent Hamilton–Jacobi Equations

This paper is devoted to the stochastic optimal control problem of infinite-dimensional differential systems allowing for both path-dependence and measurable randomness. As opposed to the deterministic path-dependent cases studied by Bayraktar and Keller [J. Funct. Anal. 275 (2018) 2096-2161], the value function turns out to be a random field on the path space and it is characterized by a stochastic path-dependent Hamilton-Jacobi (SPHJ) equation. A notion of viscosity solution is proposed and the value function is proved to be the unique viscosity solution to the associated SPHJ equation.

First- and Second-Order Maximum Principles for Discrete-Time Stochastic Optimal Control With Recursive Utilities

This paper deals with the discrete-time stochastic optimal control problems with recursive utilities under weakened convexity assumption. A new stochastic maximum principle is established. Moreover, by constructing two new adjoint equations and two new variational equations for the backward stochastic difference equation (BS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Delta$</tex-math></inline-formula> E), we obtain the second-order necessary optimality condition of quasi-singular control. Finally, as an illustration, a discrete-time mean-variance portfolio selection mixed with a recursive utility functional optimization problem is solved.

A dam management problem with energy production as an optimal switching problem

AbstractWe consider an optimal stochastic control problem for a dam. Electrical power production is operating under an uncertain setting for electricity market prices and water level which has to be kept under control. Indeed, the water level inside the basin cannot exceed a certain threshold for safety reasons, and at the same time cannot decrease below another threshold in order to keep power production active. We model this situation as a mixed control problem with regular and switching controls under constraints. We characterize the value function as solution of an HJB equation and provide some numerical approximating methods. We shall illustrate by numerical examples the main achievements of the present approach.

Second‐order necessary optimality conditions for discrete‐time stochastic systems

SummaryThis paper deals with the second‐order necessary optimality conditions for discrete‐time stochastic optimal control problems under weakened convexity assumptions. Using a special variation of the control, and by virtue of a new discrete‐time backward stochastic equation, we establish a more general and constructive first‐order necessary optimality condition in the form of a global stochastic maximum principle. Moreover, by introducing a new discrete‐time backward stochastic matrix equation, the second‐order multipoint necessary optimality conditions of singular controls are derived, which covers and improves the classical second‐order necessary optimality conditions of discrete‐time stochastic systems.

Stochastic linear quadratic optimal control problem with terminal state constraints

AbstractThis paper addresses the stochastic linear quadratic (LQ) control problem with first‐ and second‐order moment constraints on the terminal state. The problem is a modified version of the optimal covariance control problem, where the terminal state is steered to a given probability distribution. Studying a multiplicative‐noise stochastic system rather than an additive‐noise system is a salient feature. Unlike the existing ideas in the optimal steering, by using the Lagrange multipliers method and establishing the stochastic maximum principle, our problem is converted into solving forward–backward stochastic difference equations (FBSDEs), which is a special stochastic two‐point boundary‐value problem (TPBVP). We provide the optimal closed‐loop controller and necessary and sufficient solvability conditions by developing a nonhomogeneous relationship between the optimal state and costate in FBSDEs. Finally, numerical examples are given to demonstrate our results.

Dynamic programming principle for backward doubly stochastic recursive optimal control problem and sobolev weak solution of the stochastic Hamilton-Jacobi-Bellman equation

In this paper, we investigate a backward doubly stochastic recursive optimal control problem wherein the cost function is expressed as the solution to a backward doubly stochastic differential equation. We present the dynamical programming principle for this type of optimal control problem and establish that the value function is the unique Sobolev weak solution to the associated stochastic Hamilton-Jacobi-Bellman equation.

Numerical solution of Hamilton–Jacobi–Bellman PDEs in stochastic optimal control problems using fractional-order Legendre collocation method

The collocation method is one of the most powerful and effective techniques to solve nonlinear differential equations. This method reduces the original problem to a system of algebraic equations. Awareness of the nature of the problem and the analytical behavior of the solution has an important influence on the choice of type and number of basis functions required by the collocation method for solving the different classes of differential equations. In general, the polynomial basis functions such as the Legendre polynomials are not always an appropriate choice to approximate the functions that can be expanded based on monomial basis functions with real powers. In such cases, a combination of polynomial basis functions with some maps to replace the natural powers with arbitrary real powers can be a suitable alternative. This paper presents a collocation method based on the shifted fractional-order Legendre functions to solve the linear and nonlinear Hamilton–Jacobi–Bellman PDEs in stochastic optimal control problems. The convergence analysis of the proposed method has also been investigated. Moreover, the method is used to solve some practical problems with different solution structures such as resource extraction and Merton’s portfolio selection models. Numerical results for four different examples indicated that the collocation method based on the fractional-order Legendre functions are provided. Due to the non differentiability of the solution of HJB PDEs at x = 0 for resource extraction and Merton’s portfolio selection models, the Legendre collocation method cannot produce solutions with the desired accuracy. In fact, it is observed that the collocation method based on the fractional-order Legendre functions can be more efficient than the standard Legendre collocation method. Also, the sample paths of the control and state processes are obtained together with the estimate given by the Monte Carlo simulation.

Weak Second-Order Conditions of Runge–Kutta Method for Stochastic Optimal Control Problems

Weak Second-Order Conditions of Runge–Kutta Method for Stochastic Optimal Control Problems

A maximum principle for discrete-time stochastic optimal control problem with delay

A maximum principle for discrete-time stochastic optimal control problem with delay

Applying real options with reinforcement learning to assess commercial CCU deployment

Carbon capture and utilization (CCU), which emerged as a means to reduce anthropogenic carbon emissions, has been highlighted to close the carbon cycle and combat climate change. CCU involves utilizing or converting captured CO2 to create value-added products that can replace or supplement fossil fuel-derived products. In order to meet climate goals, commercial-scale CCU facilities need to be built and their capacities increased, but barriers to large-scale CCU deployment still exist, primarily caused by large uncertainties in product/energy prices, markets, technologies, and policies. Conventional techno-economic analysis (TEA) methods cannot appropriately assess viability of CCU deployment projects which include dynamic uncertainties and deployment strategies. More flexible methods that can account for both time-varying uncertainties and dynamic capacity building are needed. We propose a systematic framework for the evaluation of commercial CCU deployment using the theory of real options and reinforcement learning (RL). RL is needed as considering options of adding capacities or delaying/abandoning the project at multiple time points under dynamic uncertainties lead to a large-scale stochastic optimal control problem which cannot be solved using conventional optimization methods like stochastic programming. The framework consists of three steps: surrogate modeling, uncertainty modeling, and assessment modeling. The framework is dependent on the type of CCU technology, uncertainties, real options and RL algorithm. We demonstrate the application of the framework through a case study of CO2 hydrogenation-to-methanol process in Europe, which is a late-stage, i.e., high technology readiness level (TRL), technology.

Modified dynamic programming algorithms for GLOSA systems with stochastic signal switching times

A discrete-time stochastic optimal control problem was recently proposed to address the GLOSA (Green Light Optimal Speed Advisory) problem in cases where the next signal switching time is decided in real time and is therefore uncertain in advance. The corresponding numerical solution via SDP (Stochastic Dynamic Programming) calls for substantial computation time, which excludes problem solution in the vehicle’s on-board computer in real time. In this context, the present paper concentrates on the challenge of developing numerical algorithms to solve efficiently the stochastic GLOSA problem. As a first attempt to overcome the computation time bottleneck, a modified version of Dynamic Programming, known as Discrete Differential Dynamic Programming (DDDP) was recently employed for the numerical solution of the stochastic optimal control problem and was demonstrated to achieve results equivalent to those obtained with the ordinary SDP algorithm, albeit with significantly reduced computation times. After an outline of the DDDP approach, the present work considers a second modified version of Dynamic Programming, known as Differential Dynamic Programming (DDP). For the stochastic GLOSA problem, it is demonstrated that DDP achieves quasi-instantaneous (extremely fast) solutions in terms of CPU times, which allows for the proposed approach to be readily executable online, in an MPC (Model Predictive Control) framework, in the vehicle’s on-board computer. The novel numerical approach is tested and demonstrated by use of realistic examples and is compared to the SDP and DDDP solutions. It should be noted that DDP does not require discretization of variables, hence the obtained solutions may be slightly superior to the standard SDP solutions.

Stochastic maximum principle for moving average control system

AbstractIn this paper, we consider the stochastic optimal control problem for moving average control system. The corresponding moving average stochastic differential equation is a kind of integral differential equations. We prove the existence and uniqueness of the solution of the moving average stochastic differential equations. We obtain the stochastic maximum principle of the moving average optimal control system by introducing a kind of generalized anticipated backward stochastic differential equations. We prove the existence and uniqueness of the solution of this adjoint equation, which is singular at 0. As an application, the linear quadratic moving average control problem is investigated to illustrate the main results.

Master Bellman equation in the Wasserstein space: Uniqueness of viscosity solutions

We study the Bellman equation in the Wasserstein space arising in the study of mean field control problems, namely stochastic optimal control problems for McKean-Vlasov diffusion processes. Using the standard notion of viscosity solution à la Crandall-Lions extended to our Wasserstein setting, we prove a comparison result under general conditions on the drift and reward coefficients, which coupled with the dynamic programming principle, implies that the value function is the unique viscosity solution of the Master Bellman equation. This is the first uniqueness result in such a second-order context. The classical arguments used in the standard cases of equations in finite-dimensional spaces or in infinite-dimensional separable Hilbert spaces do not extend to the present framework, due to the awkward nature of the underlying Wasserstein space. The adopted strategy is based on finite-dimensional approximations of the value function obtained in terms of the related cooperative n n -player game, and on the construction of a smooth gauge-type function, built starting from a regularization of a sharp estimate of the Wasserstein metric; such a gauge-type function is used to generate maxima/minima through a suitable extension of the Borwein-Preiss generalization of Ekeland’s variational principle on the Wasserstein space.

Calibration in the “real world” of a partially specified stochastic volatility model

AbstractWe study the “real world” calibration of a partially specified stochastic volatility model, where the analytic expressions of the asset price drift rate and of the stochastic variance drift are not specified. The model is calibrated matching the observed asset log returns and the priors assigned by the investor. No option price data are used in the calibration. The priors chosen for the asset price drift rate and for the stochastic variance drift are those suggested by the Heston model. For this reason, the model presented can be considered as an “enhanced” Heston model. The calibration problem is formulated as a stochastic optimal control problem and solved using the dynamic programming principle. The model presented and the Heston model are calibrated using synthetic and Standard &amp; Poor 500 (S&amp;P500) data. The calibrated models are used to produce 6, 12, and 24 months in the future synthetic and S&amp;P500 forecasts.

Massively parallelizable proximal algorithms for large‐scale stochastic optimal control problems

AbstractScenario‐based stochastic optimal control problems suffer from the curse of dimensionality as they can easily grow to six and seven figure sizes. First‐order methods are suitable as they can deal with such large‐scale problems, but may perform poorly and fail to converge within a reasonable number of iterations. To achieve a fast rate of convergence and high solution speeds, in this article, we propose the use of two proximal quasi‐Newtonian limited‐memory algorithms—minfbe applied to the dual problem and the Newton‐type alternating minimization algorithm (nama)—which can be massively parallelized on lockstep hardware such as graphics processing units. In particular, we use minfbe and nama to solve scenario‐based stochastic optimal control problems with affine dynamics, convex quadratic cost functions (with the stage cost functions being strongly convex in the control variable) and joint state‐input convex constraints. We demonstrate the performance of these methods, in terms of convergence speed and parallelizability, on large‐scale problems involving millions of variables.

On a Stochastic Fundamental Lemma and Its Use for Data-Driven Optimal Control

Data-driven control based on the fundamental lemma by Willems et al. is frequently considered for deterministic LTI systems subject to measurement noise. However, besides measurement noise, stochastic disturbances might also directly affect the dynamics. In this paper, we leverage Polynomial Chaos Expansions (PCE) to extend the deterministic fundamental lemma towards stochastic systems. This extension allows to predict future statistical distributions of the inputs and outputs for stochastic LTI systems in data-driven fashion, i.e., based on the knowledge of previously recorded input-output-disturbance data and of the disturbance distribution we perform data-driven uncertainty propagation. Finally, we analyze data-driven stochastic optimal control problems and we propose a conceptual framework for data-driven stochastic predictive control. Numerical examples illustrate the efficacy of the proposed concepts.

Optimal stopping and impulse control in the presence of an anticipated regime switch

We consider a class of stochastic optimal stopping and impulse control problems where the agent solving the problem anticipates that a regime switch will happen at a random time in the future. We assume that there are only two regimes, the regime switching time is exponentially distributed, the underlying stochastic process is a linear, regular, time-homogeneous diffusion in both regimes and the payoff may be regime-dependent. This is in contrast with most existing literature on the topic, where regime switching is modulated by a continuous-time Markov chain and the underlying process and payoff belong to the same parametric family in all regimes. We state a set of easily verifiable sufficient conditions under which the solutions to these problems are given by one-sided threshold strategies. We prove uniqueness of the thresholds and characterize them as solutions to certain algebraic equations. We also study how anticipation affects optimal policies i.e. we present various comparison results for problems with and without regime switching. It may happen that the anticipative value functions and optimal policies coincide with the usual ones even if the regime switching structure is non-trivial. We illustrate our results with practical examples.