Markov Chains on Continuous State Space

Abstract

In this chapter, we discuss Markov chains on continuous state space. We first analyze a discrete-time Markov chain on continuous state space, and then discuss a discrete-time Markov chain on a bivariate state space. Applying the censoring technique, we provide expression for the RG-factorizations, which are used to derive the stationary probability of the Markov chain. Further, we consider a continuous-time Markov chain on continuous state space. Specifically, we deal with a continuous-time level-dependent QBD process with continuous phase variable, and provide orthonormal representations for the R-, U- and G-measures, which lead to the matrix-structured computation of the stationary probability. As an application, we introduce continuousphase type (CPH) distribution and continuous-phase Markovian arrival process (CMAP), and then analyze a CMAP/CPH/1 queue. Finally, we study a piecewise deterministic Markov process, which is applied to deal with more general queues such as the GI/G/c queue.