Asymptotic Properties and Associated Control Problems of Discrete-Time Singularly Perturbed Markov Chains

Abstract

This work is concerned with asymptotic properties of singularly perturbed Markov chains in discrete time with finite state spaces. We study asymptotic expansions of the probability distribution vectors and derive a mean square estimate on a sequence of occupation measures. Assuming that the state space of the underlying Markov chain can be decomposed into several groups of recurrent states and a group of transient states, by treating the states within each recurrent class as a single state, we define an aggregated process, and show that its continuous-time interpolation converges to a continuous-time Markov chain. In addition, we prove that a sequence of suitably scaled occupation measures converges to a switching diffusion process weakly. Next, control problems of large-scale nonlinear dynamic systems driven by singularly perturbed Markov chains are studied. It is demonstrated that a reduced limit system can be derived, and that by applying nearly optimal controls of the limit system to the original one, nearly optimal controls of the original system can be obtained.