Continuous-time Stochastic Optimal Control Research Articles

On the adaptation of the Lagrange formalism to continuous time stochastic optimal control: A Lagrange-Chow redux

We show how the classical Lagrangian approach to solving constrained optimization problems from standard calculus can be extended to solve continuous time stochastic optimal control problems. Connections to mainstream approaches such as the Hamilton-Jacobi-Bellman equation and the stochastic maximum principle are drawn. Our approach is linked to the stochastic maximum principle, but more direct and tied to the classical Lagrangian principle, avoiding the use of backward stochastic differential equations in its formulation. Using infinite dimensional functional analysis, we formalize and extend the approach first outlined in Chow (1992) within a rigorous mathematical setting using infinite dimensional functional analysis. We provide examples that demonstrate the usefulness and effectiveness of our approach in practice. Further, we demonstrate the potential for numerical applications facilitating some of our key equations in combination with Monte Carlo backward simulation and linear regression, therefore illustrating a completely different and new avenue for the numerical application of Chow's methods.

Duality and Approximation of Stochastic Optimal Control Problems under Expectation Constraints

We consider a continuous time stochastic optimal control problem under both equality and inequality constraints on the expectation of some functionals of the controlled process. Under a qualification condition, we show that the problem is in duality with an optimization problem involving the Lagrange multiplier associated with the constraints. Then by convex analysis techniques, we provide a general existence result and some a priori estimation of the dual optimizers. We further provide a necessary and sufficient optimality condition for the initial constrained control problem. The same results are also obtained for a discrete time constrained control problem. Moreover, under additional regularity conditions, it is proved that the discrete time control problem converges to the continuous time problem, possibly with a convergence rate. This convergence result can be used to obtain numerical algorithms to approximate the continuous time control problem, which we illustrate by two simple numerical examples.

Optimal control on graphs: existence, uniqueness, and long-term behavior

The literature on continuous-time stochastic optimal control seldom deals with the case of discrete state spaces. In this paper, we provide a general framework for the optimal control of continuous-time Markov chains on finite graphs. In particular, we provide results on the long-term behavior of value functions and optimal controls, along with results on the associated ergodic Hamilton-Jacobi equation.

Optimal execution with dynamic risk adjustment

This article considers the problem of optimal liquidation of a position in a risky security quoted in a financial market, where price evolution are risky and trades have an impact on price as well as uncertainty in the filling orders. The problem is formulated as a continuous time stochastic optimal control problem aiming at maximising a generalised risk-adjusted profit and loss function. The expression of the risk adjustment is derived from the general theory of dynamic risk measures and is selected in the class of g-conditional risk measures. The resulting theoretical framework is nonclassical since the target function depends on backward components. We show that, under a quadratic specification of the driver of a backward stochastic differential equation, it is possible to find a closed form solution and an explicit expression of the optimal liquidation policies. In this way, it is immediate to quantify the impact of risk adjustment on the profit and loss and on the expression of the optimal liquidation policies.

Dynamic pricing in retail with diffusion process demand

When randomness in demand affects the sales of a product, retailers use dynamic pricing strategies to maximize their profits. In this article, we formulate the pricing problem as a continuous-time stochastic optimal control problem and find the optimal policy by solving the associated Hamilton-Jacobi-Bellman (HJB) equation. We propose a new approach to modelling the randomness in the dynamics of sales based on diffusion processes. The model assumes a continuum approximation to the stock levels of the retailer which should scale much better to large-inventory problems than the existing Poisson process models in the revenue management literature. The diffusion process approach also enables modelling of the demand volatility, whereas Poisson process models do not. We present closed-form solutions to the HJB equation when there is no randomness in the system. It turns out that the deterministic pricing policy is near-optimal for systems with demand uncertainty. Numerical errors in calculating the optimal pricing policy may, in fact, result in a lower profit on average than with the heuristic pricing policy.

Optimal Control of Conditional Value-at-Risk in Continuous Time

We consider continuous-time stochastic optimal control problems featuring conditional value-at-risk (CVaR) in the objective. The major difficulty in these problems arises from time inconsistency, which prevents us from directly using dynamic programming. To resolve this challenge, we convert to an equivalent bilevel optimization problem in which the inner optimization problem is standard stochastic control. Furthermore, we provide conditions under which the outer objective function is convex and differentiable. We compute the outer objective's value via a Hamilton--Jacobi--Bellman equation and its gradient via the viscosity solution of a linear parabolic equation, which allows us to perform gradient descent. The significance of this result is that we provide an efficient dynamic-programming-based algorithm for optimal control of CVaR without lifting the state space. To broaden the applicability of the proposed algorithm, we propose convergent approximation schemes in cases where our key assumptions do not hold and characterize relevant suboptimality bounds. In addition, we extend our method to a more general class of risk metrics, which includes mean variance and median deviation. We also demonstrate a concrete application to portfolio optimization under CVaR constraints. Our results contribute an efficient framework for solving time-inconsistent CVaR-based sequential optimization.

Optimal dynamic asset allocation strategy for ELA scheme of DC pension plan during the distribution phase

In this paper, we study the optimal dynamic asset allocation strategy for the ELA scheme of DC pension plan during the distribution phase. In an ELA scheme of DC pension plan, the assets are invested in equities and bonds, and are distributed to the plan participants by an actuarial method. The survived participant can also obtain a survival credit from the mortality risk-sharing implicit in the pension plan. The goal of the scheme is to maintain the stable purchasing power of the plan participants, i.e., to minimize the square deviations of the distribution and a predetermined level by choosing the optimal dynamic asset allocation proportions. We formalize the problem into a continuous-time stochastic optimal control problem and establish the optimal dynamic asset allocation strategy by stochastic dynamic programming method. We obtain the optimal dynamic asset allocation proportions invested in the equities and bonds, and give an economical explanation of the key factors influencing the strategy.

On the Equilibrium Behavior of a Supply Chain Market for Capacity

Acapacity market is a business-to-business exchange in which equally capable suppliers compete with one another to satisfy generic orders from diverse buyers. The market is asymmetric because the buyers can carry inventory of the products ordered but the suppliers cannot store their capacity. In such a market, we might expect to see something like a price for capacity emerge to equilibrate demand and supply. The financial risk of participating in such a market will be driven by the volatility of the capacity price. In this paper we develop a model to explore the behavior of such a market and demonstrate, for example, that volatility of the price for capacity increases, to a point, when inflexibility of the capacity increases. We can also make statements about how the resolution of price uncertainty in the capacity market is related to the resolution of demand uncertainty faced by the buyers. Another contribution of the paper is to explain the role of market characteristics in how the market acts to minimize shortages caused by consumer demand uncertainty. We use continuous time stochastic optimal control techniques and numerical experiments to demonstrate these insights.

STOCHASTIC OPTIMAL CONTROL OF PARTIALLY OBSERVABLE NONLINEAR SYSTEMS

This paper presents a new theory for solving the continuous-time stochastic optimal control problem for a very general class of nonlinear (nonautonomous and nonaffine controlled) systems with partial state information. The proposed theory transforms the nonlinear problem into a sequence of linear-quadratic Gaussian (LQG) and time-varying problems, which converge (uniformly in time) under very mild conditions of local Lipschitz continuity. These results have been previously presented for deterministic nonlinear systems under perfect state measurements for finite horizons, but the present study shows how an additional class of nonlinear problems, involving partially observable stochastic systems, can be handled with the same theory. The method introduces an “approximating sequence of Riccati equations” (ASRE) to explicitly find the error covariance matrix and nonlinear time-varying optimal feedback controllers for such nonlinear systems, which is achieved using the framework of Kalman-Bucy filtering, separation principle and LQR theory. The paper shows a practical way of designing optimal feedback control systems for complex nonlinear stochastic problems using a combination of modern LQG estimation and LQ control-design methodologies.

A Matlab Package for Approximating the Solution to a Continuous-Time Stochastic Optimal Control Problem

Computing the solution to a stochastic optimal control problem is difficult. A method of approximating a solution to a given stochatic optimal problem was developed in [1]. This paper describes a suite of Matlab functions implementing this method of approximating a solution to a given continuous stochastic optimal control problem.

An Approximated Solution to Continuous-Time Stochastic Optimal Control Problems Through Markov Decision Chains

Strategies for constructing a Markov decision chain approximating a continuous-time finite-horizon optimal control problem are investigated. Some simple, analytically soluble, examples are treated and low computational complexity is reported. Extensions to the method and implementation are discussed. In particular, relevance of the approximated solution to a stochastic renewable resource valuation problem is examined.

Optimal Linear Preview Control of Active Vehicle Suspension

SUMMARY The problem of linear preview control of vehicle suspension is considered as a continuous time stochastic optimal control problem. In the proposed approach minimal a priori information about the road irregularities is assumed and measurement errors are taken into account. It is shown that estimation and control issues can be decoupled. The problem formulation and the analytical solution are given in a general form and hence they apply to other problems in which the system disturbances are unknown a priori, even in a stochastic sense, but some preview information is possible. The solution is applied to a two-degree-of-freedom (2-DOF) vehicle model. The effects of preview information on ride comfort, road holding, working space of the suspension and power requirements are examined in time and frequency domains. The results show that the greatest potential is for improving road holding properties. This effect could not have been observed in previous studies based on a 1-DOF vehicle model. It is also ...

Necessary and Sufficient Dynamic Programming Conditions for Continuous Time Stochastic Optimal Control

Necessary and Sufficient Dynamic Programming Conditions for Continuous Time Stochastic Optimal Control