Stochastic Control Problem Research Articles

The general maximum principle for discrete-time stochastic control problems

In this paper, we discuss the problem of discrete-time stochastic control with multiplicative noise, where the control domains may not be convex. We propose a novel approach inspired by the classical spike variation in continuous-time cases, but adapted to the discrete-time setting by perturbing in a random scale instead of time scale. Our approach allows us to obtain the maximum principle recursively without the need for variation equations or adjoint equations, which are typically required in previous methods. Moreover, we provide two maximum principles, one for the general case and the other when feedback forms exist. Our approach covers the previous results when the control domains are convex. Finally, we demonstrate the effectiveness of our new results through an example.

A mixed singular/switching control problem with terminal cost for modulated diffusion processes

In this paper, we study the regularity of the value function associated with a stochastic control problem involving two controls that act simultaneously on a modulated multidimensional diffusion process. The first is a switching control modelling a random clock. Whenever the random clock rings, the generator matrix is replaced by another, leading to a different dynamic for the finite state Markov chain of the modulated diffusion process. The second is a singular stochastic control that is executed on the process within each regime.

Modified dynamic programming algorithms for GLOSA systems with stochastic signal switching times

A discrete-time stochastic optimal control problem was recently proposed to address the GLOSA (Green Light Optimal Speed Advisory) problem in cases where the next signal switching time is decided in real time and is therefore uncertain in advance. The corresponding numerical solution via SDP (Stochastic Dynamic Programming) calls for substantial computation time, which excludes problem solution in the vehicle’s on-board computer in real time. In this context, the present paper concentrates on the challenge of developing numerical algorithms to solve efficiently the stochastic GLOSA problem. As a first attempt to overcome the computation time bottleneck, a modified version of Dynamic Programming, known as Discrete Differential Dynamic Programming (DDDP) was recently employed for the numerical solution of the stochastic optimal control problem and was demonstrated to achieve results equivalent to those obtained with the ordinary SDP algorithm, albeit with significantly reduced computation times. After an outline of the DDDP approach, the present work considers a second modified version of Dynamic Programming, known as Differential Dynamic Programming (DDP). For the stochastic GLOSA problem, it is demonstrated that DDP achieves quasi-instantaneous (extremely fast) solutions in terms of CPU times, which allows for the proposed approach to be readily executable online, in an MPC (Model Predictive Control) framework, in the vehicle’s on-board computer. The novel numerical approach is tested and demonstrated by use of realistic examples and is compared to the SDP and DDDP solutions. It should be noted that DDP does not require discretization of variables, hence the obtained solutions may be slightly superior to the standard SDP solutions.

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Time-inconsistent stochastic linear-quadratic control problem with indefinite control weight costs

Stochastic maximum principle for moving average control system

AbstractIn this paper, we consider the stochastic optimal control problem for moving average control system. The corresponding moving average stochastic differential equation is a kind of integral differential equations. We prove the existence and uniqueness of the solution of the moving average stochastic differential equations. We obtain the stochastic maximum principle of the moving average optimal control system by introducing a kind of generalized anticipated backward stochastic differential equations. We prove the existence and uniqueness of the solution of this adjoint equation, which is singular at 0. As an application, the linear quadratic moving average control problem is investigated to illustrate the main results.

Master Bellman equation in the Wasserstein space: Uniqueness of viscosity solutions

We study the Bellman equation in the Wasserstein space arising in the study of mean field control problems, namely stochastic optimal control problems for McKean-Vlasov diffusion processes. Using the standard notion of viscosity solution à la Crandall-Lions extended to our Wasserstein setting, we prove a comparison result under general conditions on the drift and reward coefficients, which coupled with the dynamic programming principle, implies that the value function is the unique viscosity solution of the Master Bellman equation. This is the first uniqueness result in such a second-order context. The classical arguments used in the standard cases of equations in finite-dimensional spaces or in infinite-dimensional separable Hilbert spaces do not extend to the present framework, due to the awkward nature of the underlying Wasserstein space. The adopted strategy is based on finite-dimensional approximations of the value function obtained in terms of the related cooperative n n -player game, and on the construction of a smooth gauge-type function, built starting from a regularization of a sharp estimate of the Wasserstein metric; such a gauge-type function is used to generate maxima/minima through a suitable extension of the Borwein-Preiss generalization of Ekeland’s variational principle on the Wasserstein space.

Massively parallelizable proximal algorithms for large‐scale stochastic optimal control problems

AbstractScenario‐based stochastic optimal control problems suffer from the curse of dimensionality as they can easily grow to six and seven figure sizes. First‐order methods are suitable as they can deal with such large‐scale problems, but may perform poorly and fail to converge within a reasonable number of iterations. To achieve a fast rate of convergence and high solution speeds, in this article, we propose the use of two proximal quasi‐Newtonian limited‐memory algorithms—minfbe applied to the dual problem and the Newton‐type alternating minimization algorithm (nama)—which can be massively parallelized on lockstep hardware such as graphics processing units. In particular, we use minfbe and nama to solve scenario‐based stochastic optimal control problems with affine dynamics, convex quadratic cost functions (with the stage cost functions being strongly convex in the control variable) and joint state‐input convex constraints. We demonstrate the performance of these methods, in terms of convergence speed and parallelizability, on large‐scale problems involving millions of variables.

Calibration in the “real world” of a partially specified stochastic volatility model

AbstractWe study the “real world” calibration of a partially specified stochastic volatility model, where the analytic expressions of the asset price drift rate and of the stochastic variance drift are not specified. The model is calibrated matching the observed asset log returns and the priors assigned by the investor. No option price data are used in the calibration. The priors chosen for the asset price drift rate and for the stochastic variance drift are those suggested by the Heston model. For this reason, the model presented can be considered as an “enhanced” Heston model. The calibration problem is formulated as a stochastic optimal control problem and solved using the dynamic programming principle. The model presented and the Heston model are calibrated using synthetic and Standard & Poor 500 (S&P500) data. The calibrated models are used to produce 6, 12, and 24 months in the future synthetic and S&P500 forecasts.

A singular stochastic control problem with direction switching cost

We introduce a new class of singular stochastic control problems in which the process controller not only chooses the push intensity, at a price proportional to the displacement caused by his action, but he can also change the allowable control direction, paying a fixed cost for each such switching. Singular control of the one-dimensional Brownian motion with quadratic instantaneous cost function and costly direction switching on the infinite time horizon is analyzed in detail, leading to a closed-form solution. This example is used as an illustration of qualitative differences between the class of problems considered here and classic singular stochastic control.

On a Stochastic Fundamental Lemma and Its Use for Data-Driven Optimal Control

Data-driven control based on the fundamental lemma by Willems et al. is frequently considered for deterministic LTI systems subject to measurement noise. However, besides measurement noise, stochastic disturbances might also directly affect the dynamics. In this paper, we leverage Polynomial Chaos Expansions (PCE) to extend the deterministic fundamental lemma towards stochastic systems. This extension allows to predict future statistical distributions of the inputs and outputs for stochastic LTI systems in data-driven fashion, i.e., based on the knowledge of previously recorded input-output-disturbance data and of the disturbance distribution we perform data-driven uncertainty propagation. Finally, we analyze data-driven stochastic optimal control problems and we propose a conceptual framework for data-driven stochastic predictive control. Numerical examples illustrate the efficacy of the proposed concepts.

Optimal stopping and impulse control in the presence of an anticipated regime switch

We consider a class of stochastic optimal stopping and impulse control problems where the agent solving the problem anticipates that a regime switch will happen at a random time in the future. We assume that there are only two regimes, the regime switching time is exponentially distributed, the underlying stochastic process is a linear, regular, time-homogeneous diffusion in both regimes and the payoff may be regime-dependent. This is in contrast with most existing literature on the topic, where regime switching is modulated by a continuous-time Markov chain and the underlying process and payoff belong to the same parametric family in all regimes. We state a set of easily verifiable sufficient conditions under which the solutions to these problems are given by one-sided threshold strategies. We prove uniqueness of the thresholds and characterize them as solutions to certain algebraic equations. We also study how anticipation affects optimal policies i.e. we present various comparison results for problems with and without regime switching. It may happen that the anticipative value functions and optimal policies coincide with the usual ones even if the regime switching structure is non-trivial. We illustrate our results with practical examples.

A stochastic control perspective on term structure models with roll-over risk

In this paper, we consider a generic interest rate market in the presence of roll-over risk, which generates spreads in spot/forward term rates. We do not require classical absence of arbitrage and rely instead on a minimal market viability assumption, which enables us to work in the context of the benchmark approach. In a Markovian setting, we extend the control-theoretic approach of Gombani and Runggaldier (Math. Finance23 (2013) 659–686) and derive representations of spot/forward spreads as value functions of suitable stochastic optimal control problems, formulated under the real-world probability and with power-type objective functionals. We determine endogenously the funding–liquidity spread by relating it to the risk-sensitive optimisation problem of a representative investor.

The Maximum Principle for Stochastic Control Problem with Jumps in Progressive Structure

The Maximum Principle for Stochastic Control Problem with Jumps in Progressive Structure

Stochastic optimal control of mesostructure of supramolecular assemblies using dissipative particle dynamics and dynamic programming with experimental validation

The self-assembly process, where molecules form complex structures through interaction forces, has broad applications in various fields. However, controlling the dynamics of self-assembling molecules presents challenges, such as kinetic trapping, and the stochastic and nonlinear nature of governing equations. Recent studies have used Markov decision processes to control self-assemblies, but this approach may not be suitable for more complex systems and may result in disassembly or irreversible changes. To address these limitations, a novel integrated framework for the systematic development of control schemes for self-assembling systems is proposed. The framework uses a dissipative particle dynamics model to accurately capture self-assembly and generate a high-fidelity representation of relevant nanostructures. The probabilities of transitioning between states are calculated, which enables the formulation of a stochastic optimal control problem. To solve this problem, dynamic programming is employed. The framework is applied to a case study involving dynamic binary complexes with multiple unwanted metastable states, which serves to experimentally demonstrate the efficacy of the controller in driving the system towards desired morphologies. Finally, the closed-loop simulation results highlight that the proposed framework can drive the system of thermosensitive DBCs towards target morphologies while avoiding kinetic traps.

Malliavin derivative of Teugels martingales and mean-field type stochastic maximum principle

We study the mean-field type stochastic control problem where the dynamics is governed by a general Lévy process with moments of all orders. For this, we introduce the power jump processes and the related Teugels martingales and give the Malliavin derivative with respect to Teugels martingales. We derive necessary and sufficient conditions for optimality of our control problem in the form of a mean-field stochastic maximum principle.

Research on Practical Path Tracking Control of Autonomous Underwater Vehicle Based on Constructive Dynamic Gain Controller

In this study, the stochastic nonlinear system-based trajectory tracking control problem of an autonomous underwater vehicle (AUV) is studied. We investigate the time-varying gain adaptive control method to find possible approaches to reduce the excessive computational burden. Enhanced adaptive algorithms are devised by considering the dynamic characteristics of AUV motion. By transforming the original controller design problems into parameter selection problems and subsequently solving them using the functional time-varying observer technical theorem, we can achieve optimal control performance. The control system is shown to constrain system state error due to stochastic disturbances within arbitrarily small domains. A coordinate transformation is proposed for all system states to meet boundedness conditions. We show that the closed-loop stability is confirmed, the system is asymptotically probabilistically stable, and contraction limits given in the stability analysis may be used to certify the convergence of the AUV trajectory errors. A large number of simulation studies using an underwater vehicle model have proved the effectiveness and robustness of the proposed approach. A real-time, time-varying gain constructive control strategy is further developed for the hardware-in-the-loop simulation; the effectiveness of the controller design is confirmed by introducing the controller into the AUV actuator model.

Value function estimators for Feynman–Kac forward–backward SDEs in stochastic optimal control

Two novel numerical estimators are proposed for solving forward–backward stochastic differential equations (FBSDEs) appearing in the Feynman–Kac representation of the value function in stochastic optimal control problems. In contrast to the current numerical approaches, which are based on the discretization of the continuous-time FBSDE, we propose a converse approach, namely, we obtain a discrete-time approximation of the value function, and then we derive a discrete-time estimator that resembles the continuous-time counterpart. The proposed approach allows for the construction of higher accuracy estimators along with an error analysis. The approach is applied to the policy improvement step in a reinforcement learning framework. Numerical results, along with the corresponding error analysis, demonstrate that the proposed estimators show significant improvement in terms of accuracy over classical Euler–Maruyama-based estimators. In the case of LQ problems, we demonstrate that our estimators result in near machine-precision level accuracy, in contrast to previously proposed methods that can potentially diverge on the same problems.

Stochastic Optimal Control for Multivariable Dynamical Systems Using Expectation Maximization.

Trajectory optimization is a fundamental stochastic optimal control (SOC) problem. This article deals with a trajectory optimization approach for dynamical systems subject to measurement noise that can be fitted into linear time-varying stochastic models. Exact/complete solutions to these kind of control problems have been deemed analytically intractable in literature because they come under the category of partially observable Markov decision processes (MDPs). Therefore, effective solutions with reasonable approximations are widely sought for. We propose a reformulation of stochastic control in a reinforcement learning setting. This type of formulation assimilates the benefits of conventional optimal control procedure, with the advantages of maximum likelihood approaches. Finally, an iterative trajectory optimization paradigm called as SOC-expectation maximization (SOC-EM) is put forth. This trajectory optimization procedure exhibits better performance in terms of reduction in cumulative cost-to-go which is proven both theoretically and empirically. Furthermore, we also provide novel theoretical work which is related to uniqueness of control parameter estimates. Analysis of the control covariance matrix is presented, which handles stochasticity through efficiently balancing exploration and exploitation.

Neural network approximation of optimal controls for stochastic reaction-diffusion equations.

We present a numerical algorithm that allows the approximation of optimal controls for stochastic reaction-diffusion equations with additive noise by first reducing the problem to controls of feedback form and then approximating the feedback function using finitely based approximations. Using structural assumptions on the finitely based approximations, rates for the approximation error of the cost can be obtained. Our algorithm significantly reduces the computational complexity of finding controls with asymptotically optimal cost. Numerical experiments using artificial neural networks as well as radial basis function networks illustrate the performance of our algorithm. Our approach can also be applied to stochastic control problems for high dimensional stochastic differential equations and more general stochastic partial differential equations.

Maximum principle for conditional mean-field FBSDEs systems with regime-switching involving impulse controls

In this work, a stochastic control problem for a forward-backward regime-switching system with mean-field interactions involving impulse controls is studied. One feature of this study lies in the conditional mean-field term used in the model. Necessary conditions for both regular and impulse controls are established via convex variational technique, and the verification theorems are obtained under appropriate convexity assumptions as well. A mean-variance investment and consumption example is discussed to test the validity of the theoretical results, and numerical simulations are also carried out to make the results clearer.