Universally Measurable Policies in Dynamic Programming

Abstract

Powerful dynamic programming results concerning existence and characterizations of optimal or nearly optimal policies, convergence of algorithms and characterizations of the optimal cost function have been available for some time (see Bellman [Bellman, R. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ.]), but a rigorous proof of these results has required quite restrictive hypotheses, such as countability of the state space, in order to circumvent the inherent measurability difficulties. Some of these results or weaker versions of them have been proved by Blackwell (Blackwell, D. 1965. Positive dynamic programming. Proc. Fifth Berkeley Symposium on Math. Stat. and Prob. 415–418; Blackwell, D. 1965. Discounted dynamic programming. Ann. Math. Stat. 36 226–235.), Strauch (Strauch, R. E. 1966. Negative dynamic programming. Ann. Math. Stat. 37 871–890.), Hinderer (Hinderer, K. 1970. Foundations of Nonstationary Dynamic Programming with Discrete Time Parameter. Springer, New York.), Dynkin and Juskevic (Dynkin, E. B., A. A. Juskevic. 1975. Controlled Markov Processes and Their Applications. Moscow. English translation to be published by Springer.) and others in the framework of Borel spaces and Borel measurable policies. We show that the use of universally measurable policies in the Borel space framework resolves the measurability issues so that all the basic results of dynamic programming can be obtained in the strongest possible form. In particular, ϵ-optimal policies are shown to exist, the dynamic programming algorithm is defined and conditions and bounds for its convergence to the optimal cost are given. The optimality equation is shown to hold and is used to characterize the optimal cost function and optimal policies.