Numerical Iteration for Stationary Probabilities of Markov Chains

Abstract

We study numerical methods to obtain the stationary probabilities of continuous-time Markov chains whose embedded chains are periodic. The power method is applied to the balance equations of the periodic embedded Markov chains. The power method can have the convergence speed of exponential rate that is ambiguous in its application to original continuous-time Markov chains since the embedded chains are discrete-time processes. An illustrative example is presented to investigate the numerical iteration of this paper. A numerical study shows that a rapid and stable solution for stationary probabilities can be achieved regardless of periodicity and initial conditions. The limiting stationary probabilities of Markov chains are usually given by balance equations which are a system of simultaneous linear equations. These probabilities are essential in Markov modeling since the system analysis is based on many real problems. It frequently happens that the analytic so- lution is difficult to obtain; consequently, numerical solutions are considered instead. In this paper we propose an iteration method based on balance equations for the calculation of the stationary probabil- ities of a discrete time Markov chain and provide a theoretical foundation. This result can be applied to continuous time Markov chains immediately and the calculation of the stationary probabilities be- comes easy even for complex systems. Many iterative numerical methods have been developed to find the stationary probabilities of Markov chains. In O'Leary (1993) and Stewart (2000), for example, matrix algebra approaches such as LU decomposition, compact storage scheme and Grassmann-Taksar-Heyman algorithm, and itera- tion methods such as the power method, Gauss-Seidel iteration, symmetric successive overrelaxation algorithm, and preconditioned power iteration are extensively discussed. Among various iterative methods, the power method is simple to understand and easy to implement, but it may suffer from the slow speed of convergence. The problem of finding the stationary probabilities of a continuous time Markov chain can be regarded as an eigenvalue problem for the transition rate matrix; consequently, the convergence rate of the power method can be very slow if the absolute values are close between two eigenvalues which are the largest in absolute magnitude. Recently, Zhao et al. (2012) and Nes- terov and Nemirovski (2014) developed new iterative methods to solve the problem of the convergence rate. However, the power method has been used widely for real applications of the Markov modeling in that the balance equations are employed without additional transformation.