An algorithm for stochastic control through dynamic programming techniques

Abstract

An algorithm based on the concept of state and dynamic programming is derived for designing an optimum controller for a linear plant subject to noise. The controller is optimal in the sense that the behavior of the plant satisfies the expected mean quadratic performance index (EMQPI) defined in the paper. The algorithm generates the sequence of control signals which minimize the EMQPI. In addition, it gives the minimum of the EMQPI for the specified sequence of control signals. The control signal is found to consist of two components: 1) a linear combination of the system state variables, and 2) a noise-balance component which minimizes the noise-induced deviation of the actual plant output from the desired output. An example is given to illustrate the iterative procedure and the asymptotic behavior of the algorithm. The design is optimal for a class of system inputs, and is applicable to both sampling and continuous systems. The design procedure is developed to make full use of a digital computer. The basic principles of dynamic programming to the treatment of stochastic control processes are clearly illustrated in an introductory form so that it will be of interest to control engineers who may wish to familiarize themselves with dynamic programming techniques.