Finite State Markov Chains

Abstract

The counting processes {N(t),t≥0} of Chapters 2 and 3 have the property that N(t)changesat discrete instants of time, butis definedfor all real t ≥ 0. Such stochastic processes are generally called continuous time processes. The Markov chains to be discussed in this and the next chapter are stochastic processesdefinedonly at integer values of time, n = 0, 1,….At each integer time n ≥ 0, there is a random variable Xncalled thestateat time n, and the process is then the family of random variables {Xn,n≥0}. These processes are often called discrete time processes, but we prefer the more specific term integer time processes. An integer time process {Xn;n≥0} can also be viewed as a continuous time process {X(t);t≥0} by taking \(X(t) = X_n \) for n≤t<n+l, but since changes only occur at integer times, it is usually simpler to view the process only at integer times.