Convergence of Value Functions for Finite Horizon Markov Decision Processes with Constraints

Abstract

This paper is concerned with finite horizon countable state Markov decision processes (MDPs) having an absorbing set as a constraint. Convergence of value iteration is discussed to investigate the asymptotic behavior of value functions as the time horizon tends to infinity. It turns out that the value function exhibits three different limiting behaviors according to the critical value $$\lambda _*$$ , the so-called generalized principal eigenvalue, of the associated ergodic problem. Specifically, we prove that (i) if $$\lambda _*<0$$ , then the value function converges to a solution to the corresponding stationary equation; (ii) if $$\lambda _*>0$$ , then, after a suitable normalization, it approaches a solution to the corresponding ergodic problem; (iii) if $$\lambda _*=0$$ , then it diverges to infinity with, at most, a logarithmic order. We employ this convergence result to examine qualitative properties of the optimal Markovian policy for a finite horizon MDP when the time horizon is sufficiently large.