Discrete-Time Markov Chain Process

Abstract

In this chapter, we define the Discrete-Time Markov Chain Process operator, all the initial components seen in the previous chapter are applied, and the vector of final conditions as known as steady-state vector is defined and exemplified, this vector shows the final state of the process and depends on the initial state vector and the matrix of transition probabilities. The solution mechanism is shown both by the iteration of the vector-matrix product and by determining the eigenvalues and eigenvectors of the matrix of transition probabilities. In an effort to categorize the possible matrix of transition probabilities, they are illustrated as reducible form, trasient form and recurrent form. In an effort to categorize the possible matrix of transition probabilities, they are illustrated as reducible form, trasient form and recurrent form. As a direct application of the Discrete-Time Markov process, the Metropolis Algorithm is presented, as well as a regularity that can be observed in the matrix of transition of probabilities and that is described in the section Law of Large Numbers. Some full basic examples are provided to illustrate the definition and operation of this ramdon walk.