Continuous-time Stochastic Control Research Articles

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Theoretical Foundations and Implications of Neural Ordinary Differential Equations (Nodes) For Real-Time Portfolio Optimization

This paper embarks on a comprehensive exploration of the theoretical landscape surrounding the integration of Neural Ordinary Differential Equations (NODEs) into the domain of real-time portfolio optimization. The study commences by establishing the background and motivation for this research, shedding light on the challenges encountered in real-time portfolio management and the potential transformative role NODEs can play in addressing these challenges. The theoretical framework unfolds in a structured manner, encompassing critical facets of portfolio optimization theory. It delves into classical portfolio optimization methodologies, including the mean- variance framework and continuous-time stochastic control techniques. This solid theoretical foundation provides the basis for understanding the nuances of optimizing portfolio weights, expected returns, and risk measures. The heart of the research lies in the integration of NODEs, an innovative fusion of deep learning and differential equations, into the fabric of portfolio optimization. NODEs, with their adaptability and ability to model continuous- time dynamics, emerge as a potent tool for real-time portfolio rebalancing and decision-making. The study provides an in-depth overview of NODEs, elucidating their architecture and their application in modeling financial time series data. This theoretical journey leads to the exploration of practical implications. The study highlights the potential benefits of incorporating NODEs into portfolio management, including improved risk management, enhanced returns, and the capacity for adaptive asset allocation strategies. However, it also addresses the limitations and challenges associated with this integration, such as data quality issues and computational requirements. In conclusion, this research presents a theoretical framework that bridges the gap between deep learning and continuous-time financial models, offering a promising avenue for real-time portfolio optimization. The insights derived from this study serve as a foundation for future research and practical applications in navigating the intricate landscape of financial markets.

Me, myself and I: A general theory of non-Markovian time-inconsistent stochastic control for sophisticated agents

We develop a theory for continuous-time non-Markovian stochastic control problems which are inherently time-inconsistent. Their distinguishing feature is that the classical Bellman optimality principle no longer holds. Our formulation is cast within the framework of a controlled non-Markovian forward stochastic differential equation, and a general objective functional setting. We adopt a game-theoretic approach to study such problems, meaning that we seek for sub-game perfect Nash equilibrium points. As a first novelty of this work, we introduce and motivate a refinement of the definition of equilibrium that allows us to establish a direct and rigorous proof of an extended dynamic programming principle, in the same spirit as in the classical theory. This in turn allows us to introduce a system consisting of an infinite family of backward stochastic differential equations analogous to the classical HJB equation. We prove that this system is fundamental, in the sense that its well-posedness is both necessary and sufficient to characterise the value function and equilibria. As a final step we provide an existence and uniqueness result. Some examples and extensions of our results are also presented.

Trajectory Optimization of Chance-Constrained Nonlinear Stochastic Systems for Motion Planning Under Uncertainty

In this article, we present generalized polynomial chaos-based sequential convex programming (gPC-SCP) to compute a suboptimal solution for a continuous-time chance-constrained stochastic nonlinear optimal control (SNOC) problem. The approach enables motion planning for robotic systems under uncertainty. The gPC-SCP method involves two steps. The first step is to derive a surrogate problem of deterministic nonlinear optimal control (DNOC) with convex constraints by using gPC expansion and the distributionally robust convex subset of the chance constraints. The second step is to solve the DNOC problem using sequential convex programming for trajectory generation and control. We prove that in the unconstrained case, the optimal value of the DNOC converges to that of SNOC asymptotically and that any feasible solution of the constrained DNOC is a feasible solution of the chance-constrained SNOC. We also present the predictor–corrector extension (gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> ) for real-time motion trajectory generation in the presence of stochastic uncertainty. In the gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> method, we first predict the uncertainty using the gPC method and then optimize the motion plan to accommodate the uncertainty. We empirically demonstrate the efficacy of the gPC-SCP and the gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> methods for the following two test cases: first, collision checking under uncertainty in actuation and physical parameters and second, collision checking with stochastic obstacle model for 3DOF and 6DOF robotic systems. We validate the effectiveness of the gPC-SCP method on the 3DOF robotic spacecraft testbed.

Linear quadratic Gaussian control with advanced continuous-time disturbance models for building thermal regulation

This paper introduces a linear quadratic control scheme for a continuous-time system with observations taken at discrete times. Particular attention is given to the derivation of the disturbance terms in the model. Control performance may depend critical on accurate disturbance forecasts. This is the case for building climate control, where solar rays pass through e.g. windows and deliver significant amounts of energy and the dynamics can be very fast, fluctuating, and spontaneous. We thus argue that it is critical for control performance to sufficiently describe and include disturbances in the control description to obtain satisfactory control accuracy. We suggest and derive in details a control framework based on continuous-time stochastic differential equations (SDEs) and linear quadratic Gaussian control using an advanced continuous-time disturbance model to supply disturbance forecasts. The numerical simulation results suggest that control with embedded forecasts handles uncertainties well and provides up to 26% performance improvements compared to standard disturbance mitigation techniques. Furthermore, we demonstrate that the quadratic controller has a useful trade-off between variability in the control signal, economic cost, and variability around the reference point.

Double Deep Q-Learning for Optimal Execution

ABSTRACT Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.

Compositional Construction of Finite MDPs for Continuous-Time Stochastic Systems: A Dissipativity Approach

Abstract This paper provides a compositional scheme based on dissipativity approaches for constructing finite abstractions of continuous-time continuous-space stochastic control systems. The proposed framework enjoys the structure of the interconnection topology and employs a notion of stochastic storage functions, that describe joint dissipativity-type properties of subsystems and their abstractions. By utilizing those stochastic storage functions, one can establish a relation between continuous-time continuous-space stochastic systems and their finite counterparts while quantifying probabilistic distances between their output trajectories. Consequently, one can employ the finite system as a suitable substitution of the continuous-time one in the controller design process with a guaranteed error bound. In this respect, we first leverage dissipativity-type compositional conditions for the compositional quantification of the distance between the interconnection of continuous-time continuous-space stochastic systems and that of their discrete-time (finite or infinite) abstractions. We then consider a specific class of stochastic affine systems and construct their finite abstractions together with their corresponding stochastic storage functions. We illustrate the effectiveness of the proposed techniques by applying them to a physical case study.

Optimal control on graphs: existence, uniqueness, and long-term behavior

The literature on continuous-time stochastic optimal control seldom deals with the case of discrete state spaces. In this paper, we provide a general framework for the optimal control of continuous-time Markov chains on finite graphs. In particular, we provide results on the long-term behavior of value functions and optimal controls, along with results on the associated ergodic Hamilton-Jacobi equation.

Compositional Construction of Control Barrier Functions for Networks of Continuous-Time Stochastic Systems

Abstract In this paper, we propose a compositional framework for the construction of control barrier functions for networks of continuous-time stochastic control systems. The proposed scheme is based on a notion of so-called pseudo-barrier functions computed for subsystems, using which one can synthesize state-feedback controllers for interconnected systems enforcing safety specifications over a finite-time horizon. Particularly, we first leverage sufficient small-gain type conditions to compositionally construct control barrier functions for interconnected systems based on the corresponding pseudo-barrier functions computed for subsystems. Then, using the constructed control barrier functions, we quantify upper bounds on exit probabilities - the probability that an interconnected system reaches certain unsafe regions - in a finite-time horizon. We employ a systematic technique based on the sum-of-squares optimization program to search for pseudo-barrier functions of subsystems while synthesizing safety controllers. We demonstrate our proposed results by applying them to a temperature regulation in a network of 1000 rooms.

Dynamic pricing in retail with diffusion process demand

When randomness in demand affects the sales of a product, retailers use dynamic pricing strategies to maximize their profits. In this article, we formulate the pricing problem as a continuous-time stochastic optimal control problem and find the optimal policy by solving the associated Hamilton-Jacobi-Bellman (HJB) equation. We propose a new approach to modelling the randomness in the dynamics of sales based on diffusion processes. The model assumes a continuum approximation to the stock levels of the retailer which should scale much better to large-inventory problems than the existing Poisson process models in the revenue management literature. The diffusion process approach also enables modelling of the demand volatility, whereas Poisson process models do not. We present closed-form solutions to the HJB equation when there is no randomness in the system. It turns out that the deterministic pricing policy is near-optimal for systems with demand uncertainty. Numerical errors in calculating the optimal pricing policy may, in fact, result in a lower profit on average than with the heuristic pricing policy.

Optimal selection and release problem in software testing process: A continuous time stochastic control approach

This paper studies a joint selection of test cases and release problem for a software under test with predetermined classes of test cases and release time deadline. The software test manager can make three alternative choices dynamically during software testing progress before the deadline: continue testing and select a class of test cases, release the software, or scrap the software, with the objective of minimizing the cumulative testing cost plus penalty cost after releasing or scrapping the software. We formulate the problem as a continuous time stochastic control model and provide a mathematically rigorous method to establish the concavity of the optimal cost function. Based on this property, we are able to characterize that the optimal release policy has a threshold structure. Moreover, the thresholds are founded to be monotone in the residual time length in the case of homogeneous release cost. Besides, we put forward a method based on low convex envelope and discover that the optimal selection policy also has a threshold or other simple structure, if the running cost or the removal cost is the same for all classes. Finally, we present an approximation algorithm of computing the optimal cost function, by which some numerical examples are studied to justify our theoretical results and the robustness of our policy. We also conduct a case study to compare our dynamic selection and release testing policy with two other commonly used testing policies and find that our policy is the best in most instances.

An HJB approach to a general continuous-time mean-variance stochastic control problem

Abstract A general continuous mean-variance problem is considered for a diffusion controlled process where the reward functional has an integral and a terminal-time component. The problem is transformed into a superposition of a static and a dynamic optimization problem. The value function of the latter can be considered as the solution to a degenerate HJB equation either in the viscosity or in the Sobolev sense (after a regularization) under suitable assumptions and with implications with regards to the optimality of strategies. There is a useful interplay between the two approaches – viscosity and Sobolev.

Robust time-inconsistent stochastic control problems

This paper establishes a general analytical framework for continuous-time stochastic control problems for an ambiguity-averse agent (AAA) with time-inconsistent preference, where the control problems do not satisfy Bellman’s principle of optimality. The AAA is concerned about model uncertainty in the sense that she is not completely confident in the reference model of the controlled Markov state process and rather considers some similar alternative models. The problems of interest are studied within a set of dominated models and the AAA seeks for an optimal decision that is robust with respect to model risks. We adopt a game-theoretic framework and the concept of subgame perfect Nash equilibrium to derive an extended dynamic programming equation and extended Hamilton–Jacobi–Bellman–Isaacs equations for characterizing the robust dynamically optimal control of the problem. We also prove a verification theorem to theoretically support our construction of robust control. To illustrate the tractability of the proposed framework, we study an example of robust dynamic mean–variance portfolio selection under two cases: 1. constant risk aversion; and 2. state-dependent risk aversion.

Constrained Quadratic Risk Minimization via Forward and Backward Stochastic Differential Equations

In this paper we study a continuous-time stochastic linear quadratic control problem arising from mathematical finance. We model the asset dynamics with random market coefficients and portfolio strategies with convex constraints. Following the convex duality approach, we show that the necessary and sufficient optimality conditions for both the primal and dual problems can be written in terms of processes satisfying a system of forward and backward stochastic differential equations (FBSDEs) together with other conditions. We characterize explicitly the optimal wealth and portfolio processes as functions of adjoint processes from the dual FBSDEs in a dynamic fashion, and vice versa. We apply the results to solve quadratic risk minimization problems with cone constraints and derive the explicit representations of solutions to the extended stochastic Riccati equations for such problems.

Valuation of power plants

In this paper we develop continuous-time stochastic control models for valuation and operation of three stylised types of power plants in an electricity market: a renewable plant, a conventional plant and a storage plant. Examples of these types of power plants are respectively wind turbines, gas-fired generation units and hydroelectric facilities. We demonstrate how to derive analytical or quasi-analytical solutions in spite of many details in modeling. In particular, we model uncertainty in electricity prices and in production input/output when it is relevant for the technology considered. Input/output is assumed to follow a diffusion process, whereas the price process may include jumps. Our models account for special characteristics of the technologies, including a non-Normal distribution of wind speeds as well as start-up and shut-down costs of thermal units. We use our models to assess the impact of conjectured future market conditions such as increasing average price level, price volatility and correlation between renewable production and electricity prices.

Robust Time-Inconsistent Stochastic Control Problems

This paper establishes a general analytical framework for continuous-time stochastic control problems for an ambiguity-averse agent (AAA) with time-inconsistent preference, where the control problems do not satisfy Bellman's principle of optimality. The AAA is concerned about model uncertainty in the sense that she is not completely confident in the reference model of the controlled Markov state process and rather considers some similar alternative models. The problems of interest are studied within a set of dominated models and the AAA seeks for an optimal decision that is robust with respect to model risks. We adopt a game-theoretic framework and the concept of subgame perfect Nash equilibrium to derive an extended dynamic programming equation and extended Hamilton -- Jacobi -- Bellman -- Isaacs equations for characterizing the robust dynamically optimal control of the problem. We also prove a verification theorem to theoretically support our construction of robust control. To illustrate the tractability of the proposed framework, we study an example of robust dynamic mean-variance portfolio selection under two cases: 1. constant risk aversion; and 2. state-dependent risk aversion.

Optimal Dynamic Risk Taking

We analyze a continuous-time stochastic control problem that arises in the study of several important issues in financial economics. An agent controls the drift and volatility of a diffusion output process by dynamically selecting one of an arbitrary (but finite) number of projects and the termination time. The optimal policy depends on the projects’ risk-adjusted drifts that are determined by their drifts, volatilities, and the curvature (or relative risk aversion) of the agent’s payoff function. We prove that the optimal policy only selects projects in the spanning subset. Furthermore, if the projects’ risk-adjusted drifts are consistently ordered for all output values, then the optimal policy is characterized by at most K − 1 switching triggers, where K is the number of projects in the spanning subset. We also characterize the optimal policy when the consistent ordering condition does not hold, and we outline a general and tractable computational algorithm to derive the optimal policies.

On time-inconsistent stochastic control in continuous time

In this paper, which is a continuation of the discrete-time paper (Björk and Murgoci in Finance Stoch. 18:545–592, 2004), we study a class of continuous-time stochastic control problems which, in various ways, are time-inconsistent in the sense that they do not admit a Bellman optimality principle. We study these problems within a game-theoretic framework, and we look for Nash subgame perfect equilibrium points. For a general controlled continuous-time Markov process and a fairly general objective functional, we derive an extension of the standard Hamilton–Jacobi–Bellman equation, in the form of a system of nonlinear equations, for the determination of the equilibrium strategy as well as the equilibrium value function. The main theoretical result is a verification theorem. As an application of the general theory, we study a time-inconsistent linear-quadratic regulator. We also present a study of time-inconsistency within the framework of a general equilibrium production economy of Cox–Ingersoll–Ross type (Cox et al. in Econometrica 53:363–384, 1985).

Optimal Control of Conditional Value-at-Risk in Continuous Time

We consider continuous-time stochastic optimal control problems featuring conditional value-at-risk (CVaR) in the objective. The major difficulty in these problems arises from time inconsistency, which prevents us from directly using dynamic programming. To resolve this challenge, we convert to an equivalent bilevel optimization problem in which the inner optimization problem is standard stochastic control. Furthermore, we provide conditions under which the outer objective function is convex and differentiable. We compute the outer objective's value via a Hamilton--Jacobi--Bellman equation and its gradient via the viscosity solution of a linear parabolic equation, which allows us to perform gradient descent. The significance of this result is that we provide an efficient dynamic-programming-based algorithm for optimal control of CVaR without lifting the state space. To broaden the applicability of the proposed algorithm, we propose convergent approximation schemes in cases where our key assumptions do not hold and characterize relevant suboptimality bounds. In addition, we extend our method to a more general class of risk metrics, which includes mean variance and median deviation. We also demonstrate a concrete application to portfolio optimization under CVaR constraints. Our results contribute an efficient framework for solving time-inconsistent CVaR-based sequential optimization.