Markov Chains

Abstract

AbstractIn spite of their simplicity, Markov processes with countable state space (and discrete time) are interesting mathematical objects with which a variety of real-world phenomena can be modeled. We give an introduction to the basic concepts (Markov property, transition matrix, recurrence, transience, invariant distribution) and then study certain examples in more detail. For example, we show how to compute numerically very precisely, the expected number of returns to the origin of simple random walk on multidimensional integer lattices.The connection with discrete potential theory will be investigated later, in Chapter 19. Some readers might prefer to skip the somewhat technical construction of general Markov processes in Section 17.1.KeywordsMarkov ChainInvariant DistributionSimple Random WalkStochastic OrderStrong Markov PropertyThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.