Stochastic Optimal Control Problem Research Articles

Optimal annuitisation, housing and reverse mortgage in retirement in the presence of a means-tested public pension

AbstractIn this paper we develop a model to find optimal decisions in retirement with respect to the consumption, risky asset allocation, access to annuities, reverse mortgage and the option to scale housing in the presence of a means-tested public pension. To solve the corresponding high-dimensional optimal stochastic control problem, we use the Least-Squares Monte Carlo simulation method. The model is applied in the context of the Australian retirement system. Few retirees in Australia utilise financial products in retirement, such as annuities or reverse mortgages. Since the government-provided means-tested Age Pension in Australia is an indirect annuity stream which is typically higher than the consumption floor, it can be argued that this could be the reason why many Australians do not annuitise. In addition, in Australia where assets allocated to the family home are not included in the means tests of Age Pension, the incentive to over-allocate wealth into housing assets is high. This raises the question whether a retiree is really better off over-allocating into the family home, while accessing home equity later on either via downsizing housing or by taking out a reverse mortgage. Our findings confirm that means-tested pension crowds out voluntary annuitisation in retirement, and that annuitisation is optimal sooner rather than later once retired. We find that it is never optimal to downscale housing when the means-tested pension and a reverse mortgage are available; only when there is no other way to access equity then downsizing is the only option.

Approximate propagation of normal distributions for stochastic optimal control of nonsmooth systems

We present a method for the approximate propagation of mean and covariance of a probability distribution through ordinary differential equations (ODE) with discontinuous right-hand side. For piecewise affine systems, a normalization of the propagated probability distribution at every time step allows us to analytically compute the expectation integrals of the mean and covariance dynamics while explicitly taking into account the discontinuity. This leads to a natural smoothing of the discontinuity such that for relevant levels of uncertainty the resulting ODE can be integrated directly with standard schemes and it is neither necessary to prespecify the switching sequence nor to use a switch detection method. We then show how this result can be employed in the more general case of piecewise smooth functions based on a structure preserving linearization scheme. The resulting dynamics can be straightforwardly used within standard formulations of stochastic optimal control problems with chance constraints.

Nonparametric statistical inference for stochastic optimal control problems and its applications for financial investment

Against a backdrop of financial decision-making, this study examines nonparametric statistical inference problems in stochastic optimal control problems. First, we construct a nonparametric estimation model using the wavelet estimation method for drift and diffusion coefficients. In this study, we focus on stochastic linear–quadratic (SLQ) problems, solved using Riccati equations, and apply this to a cash management problem within the linear–quadratic framework. The results show the effectiveness of the wavelet method using a numerical case.

On the adaptation of the Lagrange formalism to continuous time stochastic optimal control: A Lagrange-Chow redux

We show how the classical Lagrangian approach to solving constrained optimization problems from standard calculus can be extended to solve continuous time stochastic optimal control problems. Connections to mainstream approaches such as the Hamilton-Jacobi-Bellman equation and the stochastic maximum principle are drawn. Our approach is linked to the stochastic maximum principle, but more direct and tied to the classical Lagrangian principle, avoiding the use of backward stochastic differential equations in its formulation. Using infinite dimensional functional analysis, we formalize and extend the approach first outlined in Chow (1992) within a rigorous mathematical setting using infinite dimensional functional analysis. We provide examples that demonstrate the usefulness and effectiveness of our approach in practice. Further, we demonstrate the potential for numerical applications facilitating some of our key equations in combination with Monte Carlo backward simulation and linear regression, therefore illustrating a completely different and new avenue for the numerical application of Chow's methods.

Constrained minimum variance and covariance steering based on affine disturbance feedback control parameterization

AbstractThis paper deals with finite‐horizon minimum‐variance and covariance steering problems subject to constraints. The goal of the minimum variance problem is to steer the state mean of an uncertain system to a prescribed vector while minimizing the trace of its terminal state covariance whereas the goal in the covariance steering problem is to steer the covariance matrix of the terminal state to a prescribed positive definite matrix. The paper proposes a solution approach that relies on a stochastic version of the affine disturbance feedback control parametrization. In this control policy parametrization, the control input at each stage is expressed as an affine function of the history of disturbances that have acted upon the system. It is shown that this particular parametrization reduces the stochastic optimal control problems considered in this paper into tractable convex programs or difference of convex functions programs with essentially the same decision variables. In addition, the paper proposes a variation of this control parametrization that relies on truncated histories of past disturbances, which allows for sub‐optimal controllers to be designed that strike a balance between performance and computational cost. The suboptimality of the truncated policies is formally analyzed and closed form expressions are provided for the performance loss due to the use of the truncation scheme. Finally, the paper concludes with a comparative analysis of the truncated versions of the proposed policy parametrization and other standard policy parametrizations through numerical simulations.

Decentralized stochastic linear-quadratic optimal control with risk constraint and partial observation

This paper addresses a risk-constrained decentralized stochastic linear-quadratic optimal control problem with one remote controller and one local controller, where the risk constraint is posed on the cumulative state weighted variance in order to reduce the oscillation of system trajectory. In this model, local controller can only partially observe the system’s state, and sends the estimate of state to remote controller through an unreliable channel, whereas the channel from remote controller to local controller is perfect; here, the unreliable channel means the Bernoulli channel, that is, when data is transmitted on this channel, there is a certain probability of failure. For the considered constrained optimization problem with Bernoulli channel, we first punish the risk constraint into cost function through Lagrange multiplier method, and the resulting augmented cost function will include a quadratic mean-field term of state. In the sequel, for any but fixed multiplier, explicit solutions to finite-horizon and infinite-horizon mean-field decentralized linear-quadratic problems are derived together with necessary and sufficient condition on the mean-square stability of optimal system. Then, approach to find the optimal Lagrange multiplier is presented based on bisection method. Finally, two numerical examples are given to show the efficiency of the obtained results.

Data-Driven Stochastic Optimal Control With Safety Constraints Using Linear Transfer Operators

We provide a data-driven framework for optimal control of a continuous-time control-affine stochastic dynamical system. The proposed framework relies on the linear operator theory involving Perron-Frobenius (P-F) and Koopman operators. Our first result involving the P-F operator provides a convex formulation to the optimal control problem in the dual space of densities. This convex formulation of the stochastic optimal control problem leads to an infinite-dimensional convex program. The finite-dimensional approximation of the convex program is obtained using a data-driven approximation of the linear operators. We provide rigorous error bounds and convergence rate for the data-driven approximation of the linear operators and comment on the convergence rate of the optimal control w.r.t. data size. Our second results demonstrate using the Koopman operator, dual to the P-F operator, for the stochastic optimal control design. We show that the Hamilton Jacobi Bellman (HJB) equation can be expressed using the Koopman operator. We provide an iterative procedure along the lines of a popular policy iteration algorithm based on the data-driven approximation of the Koopman operator for solving the HJB equation. The two formulations, namely the convex formulation involving the P-F operator and the Koopman-based formulation using the HJB equation, can be viewed as dual to each other where the duality follows due to the dual nature of the P-F and Koopman operators. Finally, we present examples to demonstrate the efficacy of the developed framework and verify the convergence rates for the operator and optimal control numerically.

Controlled Reflected McKean–Vlasov SDEs and Neumann Problem for Backward SPDEs

This paper is concerned with the stochastic optimal control problem of a 1-dimensional McKean–Vlasov stochastic differential equation (SDE) with reflection, of which the drift coefficient and diffusion coefficient can be both dependent on the state of the solution process along with its law and control. One backward stochastic partial differential equation (BSPDE) with the Neumann boundary condition can represent the value function of this control problem. Existence and uniqueness of the solution to the above equation are obtained. Finally, the optimal feedback control can be constructed by the BSPDE.

Uncertainty-resilient constrained rendezvous trajectory optimization via stochastic feedback control and unscented transformation

This paper proposes an innovative approach for constrained rendezvous trajectory optimization (CRTO) using stochastic optimal feedback control and unscented transform (UT) uncertainty quantification. The method overcomes limitations of traditional deterministic CRTO solutions that suffer from uncontrolled terminal state error growth and severely deteriorated path constraint satisfaction when initial state uncertainty, dynamics uncertainty, and navigation uncertainty are considered. The approach involves constructing a stochastic optimal feedback control (SOFC) problem with chance constraints and introducing linear feedback control to regulate both the mean and variance of terminal state errors. UT is employed to approximately quantify the state errors and their propagation, transforming the SOFC problem into an unconstrained deterministic optimization (UDO) problem. Differential dynamic programming (DDP) is then used to solve the UDO problem. The obtained stochastic optimal solution provides a robust rendezvous trajectory and a corresponding explicit closed-loop guidance law, improving terminal accuracy and path constraint satisfaction in the presence of various considered uncertainties. The influence of different term weights in the objective function on terminal accuracy and path constraint satisfaction is also studied. The effectiveness of the proposed approach is verified through Monte Carlo simulation, demonstrating the robustness of the closed-loop control strategy. The results highlight the potential of the method in enhancing terminal accuracy and path constraint satisfaction in uncertain rendezvous scenarios.

Optimal Control of Probability on a Target Set for Continuous-Time Markov Chains

In this paper, a stochastic optimal control problem is considered for a continuous-time Markov chain taking values in a denumerable state space over a fixed finite horizon. The optimality criterion is the probability that the process remains in a target set before and at a certain time. The optimal value is a superadditive capacity of target sets. Under some minor assumptions for the controlled Markov process, we establish the dynamic programming principle, based on which we prove that the value function is a classical solution of the Hamilton-Jacobi-Bellman equation on a discrete lattice space. We then prove that there exists an optimal deterministic Markov control under the compactness assumption of control domain. We further prove that the value function is the unique solution of the HJB equation. We also consider the case starting from the outside of target set and give the corresponding results. Finally, we apply our results to two examples.

Convexified Open-Loop Stochastic Optimal Control for Linear Systems With Log-Concave Disturbances

We consider open-loop solutions to the stochastic optimal control of a linear dynamical system with an additive non-Gaussian, log-concave disturbance. We propose a novel, sampling-free approach, based on characteristic functions and convex optimization, to cast the stochastic optimal control problem as a difference-of-convex program. Our method invokes higher moments, resulting in less conservatism compared to moment-based approaches. We employ piecewise affine approximations and the convex-concave procedure for efficient solution via standard conic solvers. We demonstrate that the proposed solution is competitive with sampling and moment based approaches, without compromising probabilistic constraints.

Duality for Nonlinear Filtering II: Optimal Control

This paper is concerned with the development and use of duality theory for a nonlinear filtering model with white noise observations. The main contribution of this paper is to introduce a stochastic optimal control problem as a dual to the nonlinear filtering problem. The mathematical statement of the dual relationship between the two problems is given in the form of a duality principle. The constraint for the optimal control problem is the backward stochastic differential equation (BSDE) introduced in the companion paper. The optimal control solution is obtained from an application of the maximum principle, and subsequently used to derive the equation of the nonlinear filter. The proposed duality is shown to be an exact extension of the classical Kalman-Bucy duality, and different from other types of optimal control and variational formulations given in literature.

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

Linear quadratic stochastic optimal control with state- and control-dependent noises: A deterministic data approach

This paper investigates an infinite-horizon linear quadratic stochastic optimal control (LQSOC) problem with state- and control-dependent noises, in which two system matrices are unknown. First, a deterministic system is introduced and a relationship among its system trajectory, its control and the matrices to be solved is formulated. Subsequently, based on the deterministic system, an adaptive dynamic programming (ADP) algorithm is established to tackle the LQSOC problem. Then, the convergence analysis is presented by proving the equivalence between the proposed algorithm and an existing policy iteration algorithm. Finally, the effectiveness of the obtained algorithm is verified by a perturbed turbocharged diesel engine optimal control design example.

Convergence with rates for a Riccati-based discretization of SLQ problems with SPDEs

Abstract We consider a new discretization in space (parameter $h>0$) and time (parameter $\tau>0$) of a stochastic optimal control problem, where a quadratic functional is minimized subject to a linear stochastic heat equation with linear noise. Its construction is based on the perturbation of a generalized difference Riccati equation to approximate the related feedback law. We prove a convergence rate of almost ${\mathcal O}(h^{2}+\tau )$ for its solution, and conclude from it a rate of almost ${\mathcal O}(h^{2}+\tau )$ resp. ${\mathcal O}(h^{2}+\tau ^{1/2})$ for computable approximations of the optimal state and control with additive resp. multiplicative noise.

Controlling a generalized Fokker–Planck equation via inputs with nonlocal action

This paper concerns an optimal control problem (P) related to a generalized Fokker–Planck equation. Basic properties of the solutions to the generalized FP equation are derived via a semigroup approach in the space H−1(Rd). Problem (P) is proven to be deeply related to a stochastic optimal control problem (PS) for a McKean–Vlasov equation. The existence of an optimal control is obtained for the deterministic problem (P). The existence of an optimal control is established for an approximating optimal control problem (Ph) related to a backward Euler approximation of the generalized FP equation (with a constant discretization step h). One proves under additional hypotheses that “(Ph) converges to (P)” in a certain sense. First order necessary conditions for (Ph) are derived as well.

Stochastic Optimal Linear Control for Generalized Cost Functions With Time-Invariant Stochastic Parameters.

This study presents the design of feedback controllers for generalized cost functions to deal with stochastic optimal control problems. Target linear systems contain time-invariant stochastic parameters that describe system uncertainty. The cost functions involve nonlinear mappings and polynomial forms of the system states and inputs to express various performance metrics. Unfortunately, these properties cause difficulties in solving the problems. Conventional methods, such as the principle of optimality, are not employed to solve such problems owing to the time-invariant parameters. As opposed to the well-known quadratic functions, handling the generalized cost functions is a complicated task. This study overcomes these challenges by deriving an explicit relation between the cost function and the linear feedback gain of a controller. The derived relation enables the feedback gain to be optimized via a gradient method. A theoretical analysis ensures the convergence of the proposed gradient method. A suboptimal feedback controller is obtained to solve the problem, even for the generalized cost. Furthermore, the controller guarantees robust stability of the feedback system even with the stochastic parameters. It is demonstrated that the proposed cost function can express an expectation of a quadratic cost, risk-sensitive cost, polynomial cost, and input-to-state gain. A numerical simulation shows the effectiveness of the proposed method.

Lagrangian dual method for solving stochastic linear quadratic optimal control problems with terminal state constraints

A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed. The strong convergence of the iterative sequence generated by ALM is proved. In addition, some sufficient conditions for the surjectivity of the linear constraint mapping are obtained.

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

Dynamic programming principle for one kind of stochastic recursive optimal control problem with Markovian switching

Optimal Control of Infinite-Dimensional Differential Systems with Randomness and Path-Dependence and Stochastic Path-Dependent Hamilton–Jacobi Equations

This paper is devoted to the stochastic optimal control problem of infinite-dimensional differential systems allowing for both path-dependence and measurable randomness. As opposed to the deterministic path-dependent cases studied by Bayraktar and Keller [J. Funct. Anal. 275 (2018) 2096-2161], the value function turns out to be a random field on the path space and it is characterized by a stochastic path-dependent Hamilton-Jacobi (SPHJ) equation. A notion of viscosity solution is proposed and the value function is proved to be the unique viscosity solution to the associated SPHJ equation.