Augmented truncation approximations to the solution of Poisson’s equation for Markov chains

Abstract

Poisson’s equation has a lot of applications in various areas, such as Markov decision theory, perturbation theory, central limit theorems (CLTs), etc. Usually it is hard to derive the explicit expression of the solution of Poisson’s equation for a Markov chain on an infinitely many state space. Here we will present a computational framework for the solution for both discrete-time Markov chains (DTMCs) and continuous-time Markov chains (CTMCs), by developing the technique of augmented truncation approximations. The censored Markov chain and the linear augmentation to some columns are shown to be effective truncation approximation schemes. Moreover, the convergence to the variance constant in CLTs are also considered. Finally the results obtained are applied to discrete-time single-birth processes and continuous-time single-death processes.