Reversible Markov Chain Research Articles

Description and Prediction of the Development of Metabolic Syndrome: A Longitudinal Analysis Using a Markov Model Approach

BackgroundDelineating the natural history of metabolic syndrome (MetS) is prerequisite to prevention. This study aimed to build Markov models to simulate each component’s progress and to test the effect of different initial states on the development of MetS.MethodsMetS was defined with revised AHA/NHLBI criteria. Each reversible multistate Markov chain consisted of 8 states (no component, five isolated component states, 2-component state, and MetS state). Yearly transition probabilities were calculated from a five-year population-based follow up studywhich enrolled 2,247 individuals with mean aged 32.4 years at study entry.ResultsIn men, high BP or a 2-component state was most likely to initiate the progress of MetS. In women, abdominal obesity or low HDL were the most likely initiators. Metabolic components were likely to occur together. The development of MetS was an increasing monotonic function of time. MetS was estimated to develop within 15 years in 12.7% of young men with no component, and 2 components developed in 16.3%. MetS was estimated to develop in 10.6% of women with at the age of 47, and 2 components developed in 14.3%. MetS was estimated to develop in 24.6% of men and 27.6% of women with abdominal obesity, a rate higher than in individuals initiating with no component.ConclusionsThis modeling study allows estimation of the natural history of MetS. Men tended to develop this syndrome sooner than women did, i.e., before their fifth decade of life. Individuals with 1 or 2 components showed increased development of MetS.

Mixing Times are Hitting Times of Large Sets

We consider irreducible reversible discrete time Markov chains on a finite state space. Mixing times and hitting times are fundamental parameters of the chain. We relate them by showing that the mixing time of the lazy chain is equivalent to the maximum over initial states \(x\) and large sets \(A\) of the hitting time of \(A\) starting from \(x\). We also prove that the first time when averaging over two consecutive time steps is close to stationarity is equivalent to the mixing time of the lazy version of the chain.

Accelerating reversible Markov chains

Reversibility is usually applied in most popular Markov chain Monte Carlo algorithms, such as the Metropolis–Hastings algorithm and the Gibbs sampler. However, several researchers have shown that non-reversible Markov chains are better than reversible ones. In this paper, we present a method for accelerating a reversible Markov chain. For any reversible Markov chain with a cycle on the corresponding graph, we construct a non-reversible Markov chain by adding some antisymmetric perturbations to the original chain. We prove that this non-reversible Markov chain is uniformly better than the original one in the sense of having a smaller asymptotic variance. Furthermore, we propose a conjecture that no uniformly better chain exists for the acyclic case.

Subdiffusivity of random walk on the 2D invasion percolation cluster

We derive quenched subdiffusive lower bounds for the exit time τ(n) from a box of size n for the simple random walk on the planar invasion percolation cluster. The first part of the paper is devoted to proving an almost sure analogue of H. Kesten’s subdiffusivity theorem for the random walk on the incipient infinite cluster and the invasion percolation cluster using ideas of M. Aizenman, A. Burchard and A. Pisztora. The proof combines lower bounds on the intrinsic distance in these graphs and general inequalities for reversible Markov chains. In the second part of the paper, we present a sharpening of Kesten’s original argument, leading to an explicit almost sure lower bound for τ(n) in terms of percolation arm exponents. The methods give τ(n)≥n2+ϵ0+κ, where ϵ0>0 depends on the intrinsic distance and κ can be taken to be 5384 on the hexagonal lattice.

Testing spatial autocorrelation in weighted networks: the modes permutation test

In a weighted spatial network, as specified by an exchange matrix, the variances of the spatial values are inversely proportional to the size of the regions. Spatial values are no more exchangeable under independence, thus weakening the rationale for ordinary permutation and bootstrap tests of spatial autocorrelation. We propose an alternative permutation test for spatial autocorrelation, based upon exchangeable spatial modes, constructed as linear orthogonal combinations of spatial values. The coefficients obtain as eigenvectors of the standardized exchange matrix appearing in spectral clustering and generalize to the weighted case the concept of spatial filtering for connectivity matrices. Also, two proposals aimed at transforming an accessibility matrix into an exchange matrix with a priori fixed margins are presented. Two examples (inter-regional migratory flows and binary adjacency networks) illustrate the formalism, rooted in the theory of spectral decomposition for reversible Markov chains.

Bayesian nonparametric analysis of reversible Markov chains

We introduce a three-parameter random walk with reinforcement, called the $(\theta,\alpha,\beta)$ scheme, which generalizes the linearly edge reinforced random walk to uncountable spaces. The parameter $\beta$ smoothly tunes the $(\theta,\alpha,\beta)$ scheme between this edge reinforced random walk and the classical exchangeable two-parameter Hoppe urn scheme, while the parameters $\alpha$ and $\theta$ modulate how many states are typically visited. Resorting to de Finetti’s theorem for Markov chains, we use the $(\theta,\alpha,\beta)$ scheme to define a nonparametric prior for Bayesian analysis of reversible Markov chains. The prior is applied in Bayesian nonparametric inference for species sampling problems with data generated from a reversible Markov chain with an unknown transition kernel. As a real example, we analyze data from molecular dynamics simulations of protein folding.

Exact Goodness‐of‐Fit Tests for Markov Chains

Goodness-of-fit tests are useful in assessing whether a statistical model is consistent with available data. However, the usual χ² asymptotics often fail, either because of the paucity of the data or because a nonstandard test statistic is of interest. In this article, we describe exact goodness-of-fit tests for first- and higher order Markov chains, with particular attention given to time-reversible ones. The tests are obtained by conditioning on the sufficient statistics for the transition probabilities and are implemented by simple Monte Carlo sampling or by Markov chain Monte Carlo. They apply both to single and to multiple sequences and allow a free choice of test statistic. Three examples are given. The first concerns multiple sequences of dry and wet January days for the years 1948-1983 at Snoqualmie Falls, Washington State, and suggests that standard analysis may be misleading. The second one is for a four-state DNA sequence and lends support to the original conclusion that a second-order Markov chain provides an adequate fit to the data. The last one is six-state atomistic data arising in molecular conformational dynamics simulation of solvated alanine dipeptide and points to strong evidence against a first-order reversible Markov chain at 6 picosecond time steps.

LU-Factorization Versus Wiener-Hopf Factorization for Markov Chains

Our initial motivation was to understand links between Wiener-Hopf factorizations for random walks and LU-factorizations for Markov chains as interpreted by Grassman (Eur. J. Oper. Res. 31(1):132---139, 1987). Actually, the first ones are particular cases of the second ones, up to Fourier transforms. To show this, we produce a new proof of LU-factorizations which is valid for any Markov chain with a denumerable state space equipped with a pre-order relation. Factors have nice interpretations in terms of subordinated Markov chains. In particular, the LU-factorization of the potential matrix determines the law of the global minimum of the Markov chain. For any matrix, there are two main LU-factorizations according as you decide to enter 1 in the diagonal of the first or of the second factor. When we factorize the generator of a Markov chain, one factorization is always valid while the other requires some hypothesis on the graph of the transition matrix. This dissymmetry comes from the fact that the class of sub-stochastic matrices is not stable under transposition. We generalize our work to the class of matrices with spectral radius less than one; this allows us to play with transposition and thus with time-reversal. We study some particular cases such as: skip-free Markov chains, random walks (this gives the WH-factorization), reversible Markov chains (this gives the Cholesky factorization). We use the LU-factorization to compute invariant measures. We present some pathologies: non-associativity, non-unicity; these can be cured by smooth assumptions (e.g. irreductibility).

Gumbel fluctuations for cover times in the discrete torus

This work proves that the fluctuations of the cover time of simple random walk in the discrete torus of dimension at least three with large side-length are governed by the Gumbel extreme value distribution. This result was conjectured for example in Aldous and Fill (Reversible Markov chains and random walks on graphs, in preparation). We also derive some corollaries which qualitatively describe “how” covering happens. In addition, we develop a new and stronger coupling of the model of random interlacements, introduced by Sznitman (Ann Math (2) 171(3):2039–2087, 2010), and random walk in the torus. This coupling is used to prove the cover time result and is also of independent interest.

A Dirichlet principle for non reversible Markov chains and some recurrence theorems

We extend the Dirichlet principle to non-reversible Markov processes on countable state spaces. We present two variational formulas for the solution of the Poisson equation or, equivalently, for the capacity between two disjoint sets. As an application we prove a some recurrence theorems. In particular, we show the recurrence of two-dimensional cycle random walks under a second moment condition on the winding numbers.

A unified approach to the transition matrices of DNA substitution models

For a reversible finite-state continuous-time Markov chain containing similar states, the computation of the transition matrix can be expressed quite elegantly in terms of the transition matrix of an associated lumped Markov chain. This result is immensely useful for obtaining explicit transition matrices for many DNA substitution models, without diagonalizing a matrix or solving a differential equation. Furthermore, the technique works for the analogous problem in the discrete-time DNA substitution models.

Some universal estimates for reversible Markov chains

We obtain universal estimates on the convergence to equilibrium and the times of coupling for continuous time irreducible reversible finite-state Markov chains, both in the total variation and in the $L^2$ norms. The estimates in total variation norm are obtained using a novel identity relating the convergence to equilibrium of a reversible Markov chain to the increase in the entropy of its one-dimensional distributions. In addition, we propose a universal way of defining the ultrametric partition structure on the state space of such Markov chains. Finally, for chains reversible with respect to the uniform measure, we show how the global convergence to equilibrium can be controlled using the entropy accumulated by the chain.

Mixing and relaxation time for random walk on wreath product graphs

Suppose that G and H are finite, connected graphs, G regular, X is a lazy random walk on G and Z is a reversible ergodic Markov chain on H. The generalized lamplighter chain X* associated with X and Z is the random walk on the wreath product H\wr G, the graph whose vertices consist of pairs (f,x) where f=(f_v)_{v\in V(G)} is a labeling of the vertices of G by elements of H and x is a vertex in G. In each step, X* moves from a configuration (f,x) by updating x to y using the transition rule of X and then independently updating both f_x and f_y according to the transition probabilities on H; f_z for z different of x,y remains unchanged. We estimate the mixing time of X* in terms of the parameters of H and G. Further, we show that the relaxation time of X* is the same order as the maximal expected hitting time of G plus |G| times the relaxation time of the chain on H.

Reversible Markov Decision Processes with an Average-Reward Criterion

A wide variety of stochastic models have an important property known as reversibility. The analysis of reversible Markov chains is often significantly simpler than analysis of general Markov chains, particularly since there are often simple closed-form expressions for their invariant probability mass functions. In this paper we study the structure of optimal control policies for Markov decision processes with reversible dynamics. For optimal control of Markov decision processes, significant analytical and computational simplifications can arise as a result of reversibility. Here we present an analysis of a special class of reversible Markov decision processes, namely those that contain a cycle through their communication graph under any policy. These results provide encouraging evidence that similar simplifications hold for more general models, such as reversible processes without cycles and partition reversible processes. These results are demonstrated on a control-based variant of the Metropolis--Hastin...

Window annealing for pixel-labeling problems

The pixel labeling problems in computer vision are often formulated as energy minimization tasks. Algorithms such as graph cuts and belief propagation are prominent; however, they are only applicable for specific energy forms. For general optimization, Markov Chain Monte Carlo (MCMC) based simulated annealing can estimate the minima states very slowly.This paper presents a sampling paradigm for faster optimization. First, in contrast to previous MCMCs, the role of detailed balance constraint is eliminated. The reversible Markov chain jumps are essential for sampling an arbitrary posterior distribution, but they are not essential for optimization tasks. This allows a computationally simple window cluster sample. Second, the proposal states are generated from combined sets of local minima which achieve a substantial increase in speed compared to uniformly labeled cluster proposals. Third, under the coarse-to-fine strategy, the maximum window size variable is incorporated along with the temperature variable during simulated annealing. The proposed window annealing is experimentally shown to be many times faster and capable of finding lower energy compared to the previous Gibbs and Swendsen-Wang cut (SW-cut) sampler. In addition, the proposed method is compared with other deterministic algorithms like graph cut, belief propagation, and spectral method in their own specific energy forms. Window annealing displays competitive performance in all domains.

On the Functional Central Limit Theorem for Reversible Markov Chains with Nonlinear Growth of the Variance

In this paper we study the functional central limit theorem (CLT) for stationary Markov chains with a self-adjoint operator and general state space. We investigate the case when the variance of the partial sum is not asymptotically linear in n, and establish that conditional convergence in distribution of partial sums implies the functional CLT. The main tools are maximal inequalities that are further exploited to derive conditions for tightness and convergence to the Brownian motion.

Blind Despreading of Short-Code DS-CDMA Signals with Bayesian Techniques

This paper introduces an algorithm for joint spreading codes and information sequences estimation based on reversible jump Markov chain Monte Carlo (RJ-MCMC) for direct-sequence code division multiple access (DS-CDMA) signals with low signal-to-noise ratio (SNR) in non-cooperative systems, by analyzing signal model. The proposed algorithm establishes a joint posterior distribution model of signal parameters and user number, and obtains the samples of distribution to be estimated through iterative sampling. The algorithm is able to construct a reversible Markov chain sampler that jumps between parameter subspaces of different dimensionality, so that the posterior distribution of parameters to be estimated is obtained. Simulation results indicate that the proposed algorithm can be applied to low SNR with equal or unequal power and to different user number. Moreover, the estimation perfor-mance of this algorithm is a significant improvement of the existing method.

On the Functional Central Limit Theorem for Reversible Markov Chains with Nonlinear Growth of the Variance

In this paper we study the functional central limit theorem (CLT) for stationary Markov chains with a self-adjoint operator and general state space. We investigate the case when the variance of the partial sum is not asymptotically linear in n, and establish that conditional convergence in distribution of partial sums implies the functional CLT. The main tools are maximal inequalities that are further exploited to derive conditions for tightness and convergence to the Brownian motion.

Oja’s algorithm for graph clustering, Markov spectral decomposition, and risk sensitive control

Given a positive definite matrix M and an integer Nm≥1, Oja’s subspace algorithm will provide convergent estimates of the first Nm eigenvalues of M along with the corresponding eigenvectors. It is a common approach to principal component analysis. This paper introduces a normalized stochastic-approximation implementation of Oja’s subspace algorithm, as well as new applications to the spectral decomposition of a reversible Markov chain. Recall that this means that the stationary distribution satisfies the detailed balance equations (Meyn & Tweedie, 2009). Equivalently, the statistics of the process in steady state do not change when time is reversed. Stability and convergence of Oja’s algorithm are established under conditions far milder than that assumed in previous work. Applications to graph clustering, Markov spectral decomposition, and multiplicative ergodic theory are surveyed, along with numerical results.

Simplification of reversible Markov chains by removal of states with low equilibrium occupancy

We present a practical method for simplifying Markov chains on a potentially large state space when detailed balance holds. A simple and transparent technique is introduced to remove states with low equilibrium occupancy. The resulting system has fewer parameters. The resulting effective rates between the remaining nodes give dynamics identical to the original system's except on very fast timescales. This procedure amounts to using separation of timescales to neglect small capacitance nodes in a network of resistors and capacitors. We illustrate the technique by simplifying various reaction networks, including transforming an acyclic four-node network to a three-node cyclic network. For a reaction step in which a ligand binds, the law of mass action implies a forward rate proportional to ligand concentration. The effective rates in the simplified network are found to be rational functions of ligand concentration.