Stochastic Optimal Control using Local Sample-based Value Function Approximation

Abstract

In stochastic optimal control and partially-observable Markov decision processes, trajectory optimization methods iteratively deform a reference trajectory in a space of probability distributions such that the performance criterion associated with the problem attains an optimum. Related state-of-the-art trajectory optimization approaches are restricted to the space of Gaussian probability distributions where during optimization they perform second-order Taylor expansion of the value function at the parameters of the Gaussian, i.e. the mean and the covariance. In this paper, we propose a novel approach where trajectory optimization is performed in the space of Dirac distributions and the Taylor expansion of the value function is done at the positions of its samples. By doing so, we are able to deal with non-Gaussian distributions because Dirac distributions are often used to approximate arbitrary probability distributions. The proposed approach is demonstrated in a simulation.