On the Relation Between Recurrence and Ergodicity Properties in Denumerable Markov Decision Chains

Abstract

This paper studies two properties of the set of Markov chains induced by the deterministic policies in a Markov decision chain. These properties are called μ-uniform geometric ergodicity and μ-uniform geometric recurrence. μ-uniform ergodicity generalises a quasi-compactness condition. It can be interpreted as a strong version of stability, as it implies that the Markov chains generated by the deterministic stationary policies are uniformly stable. μ-uniform geometric recurrence can be shown to be equivalent to the simultaneous Doeblin condition, If μ is bounded. Both properties imply the existence of deterministic average and sensitive optimal policies. The second Key theorem in this paper shows the equivalence of μ-uniform geometric ergodicity and weak μ-uniform geometric recurrence under appropriate continuity conditions. In the literature numerous recurrence conditions have been used. The first Key theorem derives the relation between several of these conditions, which interestingly turn out to be equivalent in most cases.