Time recursive control of stochastic dynamical systems using forward dynamics and applications

Abstract

The solution of a stochastic optimal control problem may be associated with that of the Hamilton–Jacobi–Bellman (HJB) equation, which is a second order partial differential equation subject to a terminal condition. When this equation is semilinear and satisfies certain other constraints, it can be solved via a nonlinear version of the Feynman–Kac formula. According to this approach, the solution to the HJB equation can be obtained by simulating an associated pair of partly coupled forward–backward stochastic differential equations. Although an elegant way to interpret and solve a partial differential equation, simulating the system of forward–backward equations can be computationally inefficient. In this work, the HJB equation pertaining to the optimal control problem is reformulated such that instead of the given terminal condition, it is now subject to an appropriate initial condition. In the process, while the total cost associated with the control problem remains unchanged, pathwise solutions may not. Associated with the new partial differential equation, we then derive a set of stochastic differential equations whose solutions move only forward in time. This approach has a significant computational advantage over the original formulation. Moreover, since the forward–backward approach generally requires simulating stochastic differential equations over the current to the terminal time at every step, the integration errors may accumulate and carry forward in estimating the control. This error is particularly high initially since the time of integration is the longest there. The proposed method, numerically implemented for the control of stochastically excited oscillators, is free of such errors and hence more robust. Unsurprisingly, it also exhibits lower sampling variance.