Chapter II - Markov Chains

Abstract

This chapter discusses the theory of non-homogeneous Markov chains and related topics. Non-homogeneous Markov chains and systems are discussed from a mathematical point of view, with regard to asymptotic behavior, composition—direct sum and product, and decomposition. The chapter reviews “word functions” induced by Markov chains and valued Markov systems. These functions are studied with regard to characterization, equivalence, and representability by an underlying Markov chain or system. Various theorems are proven in the chapter. The difference between Markov systems and Markov chains is that in the Markov system model one studies the set of all possible products of Markov matrices taken from a finite given set of such matrices, while in the Markov chain model one investigates a specific given infinite product of Markov matrices and its possible sub-products. A homogeneous Markov chain is a particular case of both a non-homogeneous Markov chain and a Markov system—the case where only one Markov matrix and its powers is considered.