Skip-free Markov Chains Research Articles

Fluctuation Theory of Continuous-Time, Skip-Free Downward Markov Chains with Applications to Branching Processes with Immigration

We develop a comprehensive methodology for the fluctuation theory of continuous-time, skip-free Markov chains, extending and improving the recent work of Choi and Patie for discrete-time, skip-free Markov chains. As a significant application, we use it to derive a full set of fluctuation identities regarding exiting a finite or infinite interval for Markov branching processes with immigration, thereby uncovering many new results for this classic family of continuous-time Markov chains. The theory also allows us to recover in a simple manner fluctuation identities for skip-free downward compound Poisson processes.

Skip-free Markov chains

The aim of this paper is to develop a general theory for the class of skip-free Markov chains on denumerable state space. This encompasses their potential theory via an explicit characterization of their potential kernel expressed in terms of family of fundamental excessive functions, which are defined by means of the theory of Martin boundary. We also describe their fluctuation theory generalizing the celebrated fluctuations identities that were obtained by using the Wiener-Hopf factorization for the specific skip-free random walks. We proceed by resorting to the concept of similarity to identify the class of skip-free Markov chains whose transition operator has only real and simple eigenvalues. We manage to find a set of sufficient and easy-to-check conditions on the one-step transition probability for a Markov chain to belong to this class. We also study several properties of this class including their spectral expansions given in terms of Riesz basis, derive a necessary and sufficient condition for this class to exhibit a separation cutoff, and give a tighter bound on its convergence rate to stationarity than existing results.

Sharp Moderate Maximal Inequalities for Upward Skip-Free Markov Chains

The $$L^p$$ maximal inequalities for martingales are one of the classical results in probability theory. Here we establish the sharp moderate maximal inequalities for upward skip-free Markov chains, which include the $$L^p$$ maximal inequalities as special cases. Furthermore, we apply our theory to two specific examples and obtain their moderate maximal inequalities: the first one is the M/M/1 queue and the second one is an upward skip-free Markov chain with large death jumps. These two examples have the same total birth and death rates. However, the former exhibits a phase transition phenomenon while the latter does not.

Purely Excessive Functions and Hitting Times of Continuous-Time Branching Processes

The aim of this note is to provide an original proof and derive fine properties of the excessive function that characterizes the Laplace transform of the downward first hitting time to a fixed level of a non-degenerate continuous-time branching process. It hinges on a recent result by Choi and Patie (2016) on the potential theory of skip-free Markov chains and reveals, in particular, that the fundamental excessive function that characterizes the first hitting time is a purely excessive function.

Some Sufficient Conditions for Stochastic Comparisons Between Hitting Times for Skip-free Markov Chains

We develop some sufficient conditions for the usual stochastic ordering between hitting times, of a fixed state, for two finite Markov chains with the same state-space. Our attention will be focused on the so called skip-free case and, for the proof of our results, we develop a special type of coupling. We also analyze some special cases of applications in the frame of reliability degradation and of times to occurrence of words under random sampling of letters from a finite alphabet. As will be briefly discussed such fields give rise, in a natural way, to skip-free Markov chains.

Hitting time distribution for skip-free Markov chains: A simple proof

A well-known theorem for an irreducible skip-free Markov chain on the nonnegative integers with absorbing state d, under some conditions, is that the hitting (absorbing) time of state d starting from state 0 is distributed as the sum of d independent geometric (or exponential) random variables. The purpose of this paper is to present a direct and simple proof of the theorem in the cases of both discrete and continuous time skip-free Markov chains. Our proof is to calculate directly the generation functions (or Laplace transforms) of hitting times in terms of the iteration method.

LU-Factorization Versus Wiener-Hopf Factorization for Markov Chains

Our initial motivation was to understand links between Wiener-Hopf factorizations for random walks and LU-factorizations for Markov chains as interpreted by Grassman (Eur. J. Oper. Res. 31(1):132---139, 1987). Actually, the first ones are particular cases of the second ones, up to Fourier transforms. To show this, we produce a new proof of LU-factorizations which is valid for any Markov chain with a denumerable state space equipped with a pre-order relation. Factors have nice interpretations in terms of subordinated Markov chains. In particular, the LU-factorization of the potential matrix determines the law of the global minimum of the Markov chain. For any matrix, there are two main LU-factorizations according as you decide to enter 1 in the diagonal of the first or of the second factor. When we factorize the generator of a Markov chain, one factorization is always valid while the other requires some hypothesis on the graph of the transition matrix. This dissymmetry comes from the fact that the class of sub-stochastic matrices is not stable under transposition. We generalize our work to the class of matrices with spectral radius less than one; this allows us to play with transposition and thus with time-reversal. We study some particular cases such as: skip-free Markov chains, random walks (this gives the WH-factorization), reversible Markov chains (this gives the Cholesky factorization). We use the LU-factorization to compute invariant measures. We present some pathologies: non-associativity, non-unicity; these can be cured by smooth assumptions (e.g. irreductibility).

On Hitting Times and Fastest Strong Stationary Times for Skip-Free and More General Chains

An (upward) skip-free Markov chain with the set of nonnegative integers as state space is a chain for which upward jumps may be only of unit size; there is no restriction on downward jumps. In a 1987 paper, Brown and Shao determined, for an irreducible continuous-time skip-free chain and any d, the passage time distribution from state 0 to state d. When the nonzero eigenvalues ν j of the generator on {0,…,d}, with d made absorbing, are all real, their result states that the passage time is distributed as the sum of d independent exponential random variables with rates ν j . We give another proof of their theorem. In the case of birth-and-death chains, our proof leads to an explicit representation of the passage time as a sum of independent exponential random variables. Diaconis and Miclo recently obtained the first such representation, but our construction is much simpler. We obtain similar (and new) results for a fastest strong stationary time T of an ergodic continuous-time skip-free chain with stochastically monotone time-reversal started in state 0, and we also obtain discrete-time analogs of all our results. In the paper’s final section we present extensions of our results to more general chains.

The ODE method for stability of skip-free Markov chains with applications to MCMC

Fluid limit techniques have become a central tool to analyze queueing networks over the last decade, with applications to performance analysis, simulation and optimization. In this paper, some of these techniques are extended to a general class of skip-free Markov chains. As in the case of queueing models, a fluid approximation is obtained by scaling time, space and the initial condition by a large constant. The resulting fluid limit is the solution of an ordinary differential equation (ODE) in ``most'' of the state space. Stability and finer ergodic properties for the stochastic model then follow from stability of the set of fluid limits. Moreover, similarly to the queueing context where fluid models are routinely used to design control policies, the structure of the limiting ODE in this general setting provides an understanding of the dynamics of the Markov chain. These results are illustrated through application to Markov chain Monte Carlo methods.

Limit theorems for discrete-time markov chains on the nonnegative integers conditioned on recurrence to zero

We consider a discrete–time Markov chain X on the state space with stationary one-step transition probabilities such that X is irreducible, transient, aperiodic and skip-free to the left. With denoting the modified Markov chain in which the states are aggregated into a single absorbing state, we study the conditional state probabilities of at time n, given that state 0 will be reached some time after time n. A sufficient condition for the convergence, as of these conditional probabilities to a proper distribution is determined, as well as a condition under which the limiting conditional distribution of X is the limit, as ,of the limiting conditional distribution of . For skip-free Markov chains we derive a necessary and sufficient condition for the existence of the limiting conditional distribution. As an example of a phenomenon which may be modelled by a limiting conditional distribution, we consider the backlog of a slotted ALOHA protocol

On the Regularity of Skip-Free Markov Chains

Necessary and sufficient conditions for the regularity and recurrence of right-skip-free Markov chains are established. These generalize classical results for the case of birth-and-death processes. Expressions for the general right eigenvectors of such chains are also given.

Numerical Calculation of Ruin Probabilities for Skip-Free Markov Chains

Consider a homogeneous Markov chain taking values in $\{ 0,1, \ldots ,N,N + 1\} $, where states 0 and $N + 1$ are absorbing and the probability of absorption at state 0 prior to state $N + 1$, called the ruin probability, is of interest. It is shown that if the Markov chain is skip-free, then a simple bisection search can be applied to find the ruin probabilities. This procedure is numerically stable and considerably simpler than the matrix inversion and iteration method counterparts.

Quasi-limiting distributions of Markov chains that are skip-free to the left in continuous time

A continuous-time Markov chain on the non-negative integers is called skip-free to the left (right) if the governing infinitesimal generator A = (aij ) has the property that aij = 0 for j ≦ i ‒ 2 (i ≦ j – 2). If a Markov chain is skip-free both to the left and to the right, it is called a birth-death process. Quasi-limiting distributions of birth–death processes have been studied in detail in their own right and from the standpoint of finite approximations. In this paper, we generalize, to some extent, results for birth-death processes to Markov chains that are skip-free to the left in continuous time. In particular the decay parameter of skip-free Markov chains is shown to have a similar representation to the birth-death case and a result on convergence of finite quasi-limiting distributions is obtained.

Quasi-limiting distributions of Markov chains that are skip-free to the left in continuous time

A continuous-time Markov chain on the non-negative integers is called skip-free to the left (right) if the governing infinitesimal generator A = (aij) has the property that aij = 0 for j ≦ i ‒ 2 (i ≦ j – 2). If a Markov chain is skip-free both to the left and to the right, it is called a birth-death process. Quasi-limiting distributions of birth–death processes have been studied in detail in their own right and from the standpoint of finite approximations. In this paper, we generalize, to some extent, results for birth-death processes to Markov chains that are skip-free to the left in continuous time. In particular the decay parameter of skip-free Markov chains is shown to have a similar representation to the birth-death case and a result on convergence of finite quasi-limiting distributions is obtained.

Spectral Theory for Skip-Free Markov Chains

The distribution of upward first passage times in skip-free Markov chains can be expressed solely in terms of the eigenvalues in the spectral representation, without performing a separate calculation to determine the eigenvectors. We provide insight into this result and skip-free Markov chains more generally by showing that part of the spectral theory developed for birth-and-death processes extends to skip-free chains. We show that the eigenvalues and eigenvectors of skip-free chains can be characterized in terms of recursively defined polynomials. Moreover, the Laplace transform of the upward first passage time from 0 to n is the reciprocal of the nth polynomial. This simple relationship holds because the Laplace transforms of the first passage times satisfy the same recursion as the polynomials except for a normalization.

Spectral structure of the first-passage-time densities for classes of Markov chains

Keilson [7] showed that for a birth-death process defined on non-negative integers with reflecting barrier at 0 the first-passage-time density from 0 to N (N to N + 1) has Pólya frequency of order infinity (is completely monotone). Brown and Chaganty [3] and Assaf et al. [1] studied the first-passage-time distribution for classes of discrete-time Markov chains and then produced the essentially same results as these through a uniformization. This paper addresses itself to an extension of Keilson's results to classes of Markov chains such as time-reversible Markov chains, skip-free Markov chains and birth-death processes with absorbing states. The extensions are due to the spectral representations of the infinitesimal generators governing these Markov chains. Explicit densities for those first-passage times are also given.

Spectral structure of the first-passage-time densities for classes of Markov chains

Keilson [7] showed that for a birth-death process defined on non-negative integers with reflecting barrier at 0 the first-passage-time density from 0 to N (N to N + 1) has Pólya frequency of order infinity (is completely monotone). Brown and Chaganty [3] and Assaf et al. [1] studied the first-passage-time distribution for classes of discrete-time Markov chains and then produced the essentially same results as these through a uniformization. This paper addresses itself to an extension of Keilson's results to classes of Markov chains such as time-reversible Markov chains, skip-free Markov chains and birth-death processes with absorbing states. The extensions are due to the spectral representations of the infinitesimal generators governing these Markov chains. Explicit densities for those first-passage times are also given.

An eigenvalue decomposition for first hitting times in random walks

Distributions for first hitting times in random walks (skip-free Markov chains on the non-negative integers) are studied. Their probability generating functions are expressed in terms of the eigenvalues of probability transition submatrices. The study of first hitting times leads to several classes of infinitely divisible distributions closely related to the Bondesson class of generalized convolutions of mixtures of exponential distributions which arises from the study of first hitting times in birth-death processes. Some properties of these classes are derived from their characterization by Pick functions.

Time-dependent Behaviour of Markov Processes with Boundaries

IN many applications of the theory of stochastic processes to, for example, epidemics, storage, queues, reliability and risk theory, the mathematical model that often emerges is governed by a skip free Markov chain or process which is basically very simple and is complicated only by the presence of boundaries. (A process is said to be skip free if its states are strictly ordered and passage from one particular state can only take place to its neighbouring states.) Frequently the underlying process is skip free in at least one direction. I shall describe a method for dealing with such situations and relate the discussion to a model for a finite dam in discrete time and with a discrete state space. The basic ideas have also been applied to a situation in which time and state space are continuous1,2.