Abstract
In this paper, we present connections between recent developments on the linearly-solvable stochastic optimal control framework with early work in control theory based on the fundamental dualities between free energy and relative entropy. We extend these connections to nonlinear stochastic systems with non-affine controls by using the generalized version of the Feynman–Kac lemma. We present alternative formulations of the linearly-solvable stochastic optimal control framework and discuss information theoretic and thermodynamic interpretations. On the algorithmic side, we present iterative stochastic optimal control algorithms and applications to nonlinear stochastic systems. We conclude with an overview of the frameworks presented and discuss limitations, differences and future directions.
Highlights
While the topic of nonlinear stochastic control has been traditionally studied within control and applied mathematics, over the past 10–15 years, there has been an increasing interest by researchers in machine learning and robotics communities to expand nonlinear stochastic optimal control in terms of theoretical generalizations and algorithms
In the first approach, which is the most traditional one, stochastic optimal control is formulated as the minimization of an objective function J(x, u) in Equation (42) subject to the controlled dynamics
The Feynman–Kac lemma is applied, and the solution of the Chapman–Kolmogorov partial differential equations (PDEs) together with the lower bound on the objective function are provided
Summary
While the topic of nonlinear stochastic control has been traditionally studied within control and applied mathematics, over the past 10–15 years, there has been an increasing interest by researchers in machine learning and robotics communities to expand nonlinear stochastic optimal control in terms of theoretical generalizations and algorithms. We expand upon our previous work on this topic [11] and present connections between the LSOC framework as presented within the machine learning and statistical physics communities with the information theoretic view of nonlinear stochastic optimal control theory using the free energy-relative entropy relationship [12,13,14,15]. The analysis leverages the generalized version of the Feynman–Kac lemma and identifies the necessary and sufficient conditions under which the aforementioned connections are valid This generalization creates future research directions towards the development of optimal control algorithms for stochastic systems nonlinear in the state and control. While typically in stochastic optimal control theory, the cost function is pre-specified, this is not the case when the stochastic optimal control framework is derived using the free energy-relative entropy relationship.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
