Hamiltonian Monte Carlo based Path Integral for Stochastic Optimal Control

Abstract

This paper develops a path integral based model predictive control using the Hamiltonian Monte Carlo (HMC) sampling to address the stochastic optimal control (SOC) problem. The proposed control framework provides an analytically sound method for building an algorithm of optimal control based on stochastic trajectory sampling. This is achieved by using Feynman-Kac (F-K) lemma which transforms the value function of SOC problem into an expectation over all probable trajectories. The various sampling methods used in statistical analysis are bound to fail in high dimensional spaces where there is a presence of a large number of directions in which to guess. More specifically, just a singular set of directions is available that remain within the typical collection and pass the test. The HMC sampling is the Markov Chain Monte Carlo (MCMC) method that uses derivatives of density function which is being sampled to generate efficient transitions spanning the posterior. Specifically, transitions that can follow high-dimension probability mass contours and glide coherently through the typical set of the desired exploration obtained by exploiting derivatives of target distributions. As a consequence, these Hamiltonian Markov transitions provide optimal control law for the SOC problem. Finally, the model predictive path integral control using HMC sampling is efficiently implemented for a Cart-pole system.