General State Space Markov Chains Research Articles

Lyapunov Conditions for Differentiability of Markov Chain Expectations

We consider a family of Markov chains whose transition dynamics are affected by model parameters. Understanding the parametric dependence of (complex) performance measures of such Markov chains is often of significant interest. The derivatives and their continuity of the performance measures w.r.t. the parameters play important roles, for example, in numerical optimization of the performance measures, and quantification of the uncertainties in the performance measures when there are uncertainties in the parameters from the statistical estimation procedures. In this paper, we establish conditions that guarantee the smoothness of various types of intractable performance measures—such as the stationary and random horizon discounted performance measures—of general state space Markov chains and provide probabilistic representations for the derivatives. Funding: C.-H. Rhee is supported by the National Science Foundation [Grant CMMI-2146530].

On the limitations of single-step drift and minorization in Markov chain convergence analysis

Over the last three decades, there has been a considerable effort within the applied probability community to develop techniques for bounding the convergence rates of general state space Markov chains. Most of these results assume the existence of drift and minorization (d&m) conditions. It has often been observed that convergence rate bounds based on single-step d&m tend to be overly conservative, especially in high-dimensional situations. This article builds a framework for studying this phenomenon. It is shown that any convergence rate bound based on a set of d&m conditions cannot do better than a certain unknown optimal bound. Strategies are designed to put bounds on the optimal bound itself, and this allows one to quantify the extent to which a d&m-based convergence rate bound can be sharp. The new theory is applied to several examples, including a Gaussian autoregressive process (whose true convergence rate is known), and a Metropolis adjusted Langevin algorithm. The results strongly suggest that convergence rate bounds based on single-step d&m conditions are quite inadequate in high-dimensional settings.

Estimating the spectral gap of a trace-class Markov operator

The utility of a Markov chain Monte Carlo algorithm is, in large part, determined by the size of the spectral gap of the corresponding Markov operator. However, calculating (and even approximating) the spectral gaps of practical Monte Carlo Markov chains in statistics has proven to be an extremely difficult and often insurmountable task, especially when these chains move on continuous state spaces. In this paper, a method for accurate estimation of the spectral gap is developed for general state space Markov chains whose operators are non-negative and trace-class. The method is based on the fact that the second largest eigenvalue (and hence the spectral gap) of such operators can be bounded above and below by simple functions of the power sums of the eigenvalues. These power sums often have nice integral representations. A classical Monte Carlo method is proposed to estimate these integrals, and a simple sufficient condition for finite variance is provided. This leads to asymptotically valid confidence intervals for the second largest eigenvalue (and the spectral gap) of the Markov operator. In contrast with previously existing techniques, our method is not based on a near-stationary version of the Markov chain, which, paradoxically, cannot be obtained in a principled manner without bounds on the spectral gap. On the other hand, it can be quite expensive from a computational standpoint. The efficiency of the method is studied both theoretically and empirically.

Subgeometric rates of convergence in Wasserstein distance for Markov chains

In this paper, we provide sufficient conditions for the existence of the invariant distribution and for subgeometric rates of convergence in Wasserstein distance for general state-space Markov chains which are (possibly) not irreducible. Compared to previous work, our approach is based on a purely probabilistic coupling construction which allows to retrieve rates of convergence matching those previously reported for convergence in total variation. Our results are applied to establish the subgeometric ergodicity in Wasserstein distance of non-linear autoregressive models and of the pre-conditioned Crank-Nicolson Markov chain Monte Carlo algorithm in Hilbert space.

Spectral Theory for Weakly Reversible Markov Chains

The theory of L 2-spectral gaps for reversible Markov chains has been studied by many authors. In this paper we consider positive recurrent general state space Markov chains with stationary transition probabilities. Replacing the assumption of reversibility with a weaker assumption, we still obtain a simple necessary and sufficient condition for the spectral gap property of the associated Markov operator in terms of the isoperimetric constant. We show that this result can be applied to a large class of Markov chains, including those that are related to positive recurrent finite-range random walks on Z.

Spectral Theory for Weakly Reversible Markov Chains

The theory of L2-spectral gaps for reversible Markov chains has been studied by many authors. In this paper we consider positive recurrent general state space Markov chains with stationary transition probabilities. Replacing the assumption of reversibility with a weaker assumption, we still obtain a simple necessary and sufficient condition for the spectral gap property of the associated Markov operator in terms of the isoperimetric constant. We show that this result can be applied to a large class of Markov chains, including those that are related to positive recurrent finite-range random walks on Z.

Queueing analysis for the OFDMA downlink: Throughput regions, delay and exponential backlog bounds

While multiuser scheduling strategies have been intensively studied in the context of OFDMA downlink recently, the resulting throughput and delay performance affecting strongly end-to-end performance of wireless communication systems can still be evaluated by simulations only. In this paper we approach this problem and consider bounds for delay and queue backlog for a large class of scheduling policies. Adopting a general state space Markov chain model the concept of policy-specific throughput regions is introduced. Then, under the regime of the policy, a recursive formula for calculating all polynomial moments of the queue backlog is derived. Moreover, it is shown that even exponential decay of the tail distribution can be obtained under proper circumstances. Based on these results, upper bounds on the buffer overflow probability are derived giving insights for practical buffer dimensioning problems in UMTS LTE systems.

Evaluation of formal posterior distributions via Markov chain arguments

We consider evaluation of proper posterior distributions obtained from improper prior distributions. Our context is estimating a bounded function Φ of a parameter when the loss is quadratic. If the posterior mean of 0 is admissible for all bounded Φ, the posterior is strongly admissible. We give sufficient conditions for strong admissibility. These conditions involve the recurrence of a Markov chain associated with the estimation problem. We develop general sufficient conditions for recurrence of general state space Markov chains that are also of independent interest. Our main example concerns the p-dimensional multivariate normal distribution with mean vector 0 when the prior distribution has the form g(∥6∥ 2 ) dθ on the parameter space R p . Conditions on g for strong admissibility of the posterior are provided.

A regeneration proof of the central limit theorem for uniformly ergodic Markov chains

Let ( X n ) be a Markov chain on measurable space ( E , E ) with unique stationary distribution π . Let h : E → R be a measurable function with finite stationary mean π ( h ) ≔ ∫ E h ( x ) π ( d x ) . Ibragimov and Linnik [Ibragimov, I.A., Linnik, Y.V., 1971. Independent and Stationary Sequences of Random Variables. Wolter-Noordhoff, Groiningen] proved that if ( X n ) is geometrically ergodic, then a central limit theorem (CLT) holds for h whenever π ( | h | 2 + δ ) < ∞ , δ > 0 . Cogburn [Cogburn, R., 1972. The central limit theorem for Markov processes. In: Le Cam, L.E., Neyman, J., Scott, E.L. (Eds.), Proc. Sixth Ann. Berkley Symp. Math. Statist. and Prob., 2. pp. 485–512] proved that if a Markov chain is uniformly ergodic, with π ( h 2 ) < ∞ then a CLT holds for h . The first result was re-proved in Roberts and Rosenthal [Roberts, G.O., Rosenthal, J.S., 2004. General state space Markov chains and MCMC algorithms. Prob. Surveys 1, 20–71] using a regeneration approach; thus removing many of the technicalities of the original proof. This raised an open problem: to provide a proof of the second result using a regeneration approach. In this paper we provide a solution to this problem.

A Regeneration Proof of the Central Limit Theorem for Uniformly Ergodic Markov Chains

Central limit theorems for functionals of general state space Markov chains are of crucial importance in sensible implementation of Markov chain Monte Carlo algorithms as well as of vital theoretical interest. Different approaches to proving this type of results under diverse assumptions led to a large variety of CLT versions. However due to the recent development of the regeneration theory of Markov chains, many classical CLTs can be reproved using this intuitive probabilistic approach, avoiding technicalities of original proofs. In this paper we provide a characterization of CLTs for ergodic Markov chains via regeneration and then use the result to solve the open problem posed in [Roberts & Rosenthal 2005]. We then discuss the difference between one-step and multiple-step small set condition.

When is Eaton’s Markov chain irreducible?

Consider a parametric statistical model P(dx\0) and an improper prior distribution v(d0) that together yield a (proper) formal posterior distribution Q(d6\x). The prior is called strongly admissible if the generalized Bayes estimator of every bounded function of 0 is admissible under squared error loss. Eaton [Ann. Sta tist. 20 (1992) 1147-1179] has shown that a sufficient condition for strong admissibility of v is the local recurrence of the Markov chain whose transition function is R(0, dij) = f Q(dr)\x)P(dx\6). Applications of this result and its extensions are often greatly simplified when the Markov chain associated with R is ir reducible. However, establishing irreducibility can be difficult. In this paper, we provide a characterization of irreducibility for general state space Markov chains and use this characterization to develop an easily checked, necessary and sufficient condition for irreducibility of Eaton's Markov chain. All that is required to check this condition is a simple examination of P and v. Application of the main result is illustrated using two examples.

Analysis on adjoint non-recurrent property of nonlinear time series in random environment domain

By introducing a random interference into the typical of nonlinear time series model, this paper establishes a RENLAR model: $$X_{n+1}=T(X_n)+ e_{n+1}(Z_{n+1})$$ . The author introduces the definition of adjoint non-recurrence, and utilizing general state space Markov chain theorem, we obtain some criteria for non-recurrence and adjoint non-recurrence of nonlinear time series models in random environment domain and analyze adjoint non-recurrence of some models by using these criteria.

Measure-Valued Differentiation for Stationary Markov Chains

We study general state-space Markov chains that depend on a parameter, say, θ. Sufficient conditions are established for the stationary performance of such a Markov chain to be differentiable with respect to θ. Specifically, we study the case of unbounded performance functions and thereby extend the result on weak differentiability of stationary distributions of Markov chains to unbounded mappings. First, a closed-form formula for the derivative of the stationary performance of a general state-space Markov chain is given using an operator-theoretic approach. In a second step, we translate the derivative formula into unbiased gradient estimators. Specifically, we establish phantom-type estimators and score function estimators. We illustrate our results with examples from queueing theory.

General state space Markov chains and MCMC algorithms

This paper surveys various results about Markov chains on general (non-countable) state spaces. It begins with an introduction to Markov chain Monte Carlo (MCMC) algorithms, which provide the motivation and context for the theory which follows. Then, sufficient conditions for geometric and uniform ergodicity are presented, along with quantitative bounds on the rate of convergence to stationarity. Many of these results are proved using direct coupling constructions based on minorisation and drift conditions. Necessary and sufficient conditions for Central Limit Theorems (CLTs) are also presented, in some cases proved via the Poisson Equation or direct regeneration constructions. Finally, optimal scaling and weak convergence results for Metropolis-Hastings algorithms are discussed. None of the results presented is new, though many of the proofs are. We also describe some Open Problems.

On the Markov chain central limit theorem

The goal of this expository paper is to describe conditions which guarantee a central limit theorem for functionals of general state space Markov chains. This is done with a view towards Markov chain Monte Carlo settings and hence the focus is on the connections between drift and mixing conditions and their implications. In particular, we consider three commonly cited central limit theorems and discuss their relationship to classical results for mixing processes. Several motivating examples are given which range from toy one-dimensional settings to complicated settings encountered in Markov chain Monte Carlo.

Small sets and Markov transition densities

The theory of general state-space Markov chains can be strongly related to the case of discrete state-space by use of the notion of small sets and associated minorization conditions. The general theory shows that small sets exist for all Markov chains on state-spaces with countably generated σ-algebras, though the minorization provided by the theory concerns small sets of order n and n-step transition kernels for some unspecified n. Partly motivated by the growing importance of small sets for Markov chain Monte Carlo and Coupling from the Past, we show that in general there need be no small sets of order n=1 even if the kernel is assumed to have a density function (though of course one can take n=1 if the kernel density is continuous). However, n=2 will suffice for kernels with densities (integral kernels), and in fact small sets of order 2 abound in the technical sense that the 2-step kernel density can be expressed as a countable sum of non-negative separable summands based on small sets. This can be exploited to produce a representation using a latent discrete Markov chain; indeed one might say, inside every Markov chain with measurable transition density there is a discrete state-space Markov chain struggling to escape. We conclude by discussing complements to these results, including their relevance to Harris-recurrent Markov chains and we relate the counterexample to Turán problems for bipartite graphs.

Derandomizing Variance Estimators

One may consider a discrete-event simulation as a Markov chain evolving on a suitably rich state space. One way that regenerative cycles may be constructed for general state-space Markov chains is to generate auxiliary coin-flip random variables at each transition, with a regeneration occurring if the coin-flip results in a success. The regenerative cycles are therefore randomizedwith respect to the sequence of states visited by the Markov chain. The point estimator for a steady-state performance measure does not depend on the cycle structure of the chain, but the variance estimator (that defines the width of a confidence interval for the performance measure) does. This implies that the variance estimator is randomized with respect to the visited states. We show how to “derandomize” the variance estimator through the use of conditioning. A new variance estimator is obtained that is consistent and has lower variance than the standard estimator.

Transient analysis of deterministic and stochastic Petri nets with concurrent deterministic transitions

This paper introduces an efficient numerical algorithm for transient analysis of deterministic and stochastic Petri nets (DSPNs) and other discrete-event stochastic systems with exponential and deterministic events. The proposed approach is based on the analysis of a general state space Markov chain (GSSMC) whose state equations constitute a system of multidimensional Fredholm integral equations. Key contributions of this paper constitute the observations that the transition kernel of this system of Fredholm equations is piece-wise continuous and separable. Due to the exploitation of these properties, the GSSMC approach shows great promise for being effectively applicable for the transient analysis of large DSPNs with concurrent deterministic transitions. Moreover, for DSPNs without concurrent deterministic transitions the proposed GSSMC approach requires three orders of magnitude less computational effort than the previously known approach based on the method of supplementary variables.

On simulation output analysis for generalized semi-markov processes

The usual model for the underlying process of a discrete-event stochastic system is the generalized semi-Markov process (GSMP). A GSMP is defined in terms of a general state space Markov chain that describes the process at successive state-transition times. We provide conditions on the clock-setting distributions and state-transition probabilities of a finite state GSMP under which this underlying chain is φ-irreducible and satisfies a drift criterion for stability due to Meyn and Tweedie. If the GSMP also has a “single state” in which exactly one event is scheduled to occur, then this state is hit infinitely often with probability I and the time between successive hits has finite second moment. It follows that the standard regenerative method for analysis of simulation output can be used to obtain strongly consistent point estimates and asymptotic confidence intervals for time-average limits of the process. We also show that, under our conditions, point estimates and confidence intervals for time-average limits can be obtained using methods based on standardized time series. In particular, the method of batch means is applicable. Our results rest on a new functional central limit theorem for GSMP's together with results of Glynn and Iglehart. The standardized-time-series methods apply even when the GSMP does not have a single state or indeed any type of classical regenerative structure

Limit Theory for Performance Modeling of Future Event Set Algorithms

In a discrete-event simulation, the information related to the events scheduled to occur in the future is kept in a data structure called the future event set (FES). In this paper, we study the interaction hold model, a popular stochastic model for FES performance analysis, corresponding to the superposition of a (fixed) number of renewal processes. The general state-space Markov chain formed by the discrete-time process that keeps track, at event times, of the residual lifetimes is shown here to be recurrent in the sense of Harris, and its stationary distribution is obtained. Linked lists and indexed lists, two popular FESs, are investigated using this model. For the interaction hold model, we make rigorous certain published results as well as introduce new ones. For example, we derive the distribution of the relative position of the event to be inserted in the data structure. In the exponential case, our analytic and empirical results confirm that when events with relatively short lifetimes often get regenerated upon their occurrence, it is better to scan a list (or sublist) from its head rather than tail. In the same context and for indexed lists with sublists with constant sizes, our results suggest that subsequent sublists should be of larger sizes, i.e., the first sublist should contain the smallest number of records, the second sublist the second smallest number of records, etc.