On the dynamical equations of conditional probability density functions, with applications to optimal stochastic control theory

Abstract

Let dx = g( x, u, t) dt + dz be a general dynamical system with control u and where z is Brownian motion ( dz dt = ξ is white Gaussian noise). Let the loss function be E ∝ t 0 T k( x, u, τ) dτ, and let dy dt = x(t) + ϵ(t) , where ϵ( t) is white Gaussian noise, be the only observable quantity. In addition, let the initial probability density function of x( t 0), P( a, t 0), be given. Two forms of (Markov process) state space representations of this system are studied. The first is the conditional probability density function of x; the second is the set of moments of the density function. A partial-differential-integral equation is derived which governs the evolution of the conditional density function, and contains a function of the observation as a driving term. This equation reduces to Kolmogorov's forward diffusion equation as the variance of the observation noise becomes infinite (hence, as the observations become valueless). Functional derivatives are used to derive a second-order partial differential equation, whose independent variables are t and P, and which is useful for a numerical approach to the solution. The set of differential equations governing the moments of P( a, t) is derived. This representation is infinite dimensional in general and several possible approximations are discussed. Previous results in stochastic control are formally extended to obtain qualitative knowledge on the optimum control. It is shown that, if the observation noise does not depend on the control, then the optimum control is bang-bang under essentially the same conditions that lead to a bang-bang solution when there is no noise in the observation. The results are obtained for a one-dimensional system but are easily extendible to the vector case. The results are also quite interesting from the point of view of some important problems in the theory of nonlinear prediction and filtering, as will be discussed in Section IV. In the general stochastic control problem where noise corrupted observations are taken, the state variables of a total system are functionals of the observations which are Markovian.