Reversible Markov Chain Research Articles

Fastest Mixing Reversible Markov Chain: Clique Lifted Graphs and Subgraphs

Markov chains are one of the well-known tools for modeling and analyzing stochastic systems. At the same time, they are used for constructing random walks that can achieve a given stationary distribution. This paper is concerned with determining the transition probabilities that optimize the mixing time of the reversible Markov chains towards a given equilibrium distribution. This problem is referred to as the Fastest Mixing Reversible Markov Chain (FMRMC) problem. It is shown that for a given base graph and its clique lifted graph, the FMRMC problem over the clique lifted graph is reducible to the FMRMC problem over the base graph, while the optimal mixing times on both graphs are identical. Based on this result and the solution of the semidefinite programming formulation of the FMRMC problem, the problem has been addressed over a wide variety of topologies with the same base graph. Second, the general form of the FMRMC problem is addressed on stand-alone topologies as well as subgraphs of an arbitrary graph. For subgraphs, it is shown that the optimal transition probabilities over edges of the subgraph can be determined independent of rest of the topology.

Computationally Efficient Estimation of the Spectral Gap of a Markov Chain

We consider the problem of estimating from sample paths the absolute spectral gap 1 - λ of a reversible, irreducible and aperiodic Markov chain (Xt )t€N over a finite state space Ω. We propose the UCPI (Upper Confidence Power Iteration) algorithm for this problem, a low-complexity algorithm which estimates the spectral gap in time O(n) and memory space O((lnn)2) given n samples. This is in stark contrast with most known methods which require at least memory space O((ln n)2), so that they cannot be applied to large state spaces. Furthermore, UCPI is amenable to parallel implementation.

Poincaré and logarithmic Sobolev constants for metastable Markov chains via capacitary inequalities

We investigate the metastable behavior of reversible Markov chains on possibly countable infinite state spaces. Based on a new definition of metastable Markov processes, we compute precisely the mean transition time between metastable sets. Under additional size and regularity properties of metastable sets, we establish asymptotic sharp estimates on the Poincaré and logarithmic Sobolev constant. The main ingredient in the proof is a capacitary inequality along the lines of V. Maz’ya that relates regularity properties of harmonic functions and capacities. We exemplify the usefulness of this new definition in the context of the random field Curie–Weiss model, where metastability and the additional regularity assumptions are verifiable.

Noise can speed Markov chain Monte Carlo estimation and quantum annealing.

Carefully injected noise can speed the average convergence of Markov chain Monte Carlo (MCMC) estimates and simulated annealing optimization. This includes quantum annealing and the MCMC special case of the Metropolis-Hastings algorithm. MCMC seeks the solution to a computational problem as the equilibrium probability density of a reversible Markov chain. The algorithm must cycle through a long burn-in phase until it reaches equilibrium because the Markov samples are statistically correlated. The special injected noise reduces this burn-in period in MCMC. A related theorem shows that it reduces the cooling time in simulated annealing. Simulations showed that optimal noise gave a 76% speed-up in finding the global minimum in the Schwefel optimization benchmark. The noise-boosted simulations found the global minimum in 99.8% of trials compared with only 95.4% of trials in noiseless simulated annealing. Simulations also showed that the noise boost is robust to accelerated cooling schedules and that noise decreased convergence times by more than 32% under aggressive geometric cooling. Molecular dynamics simulations showed that optimal noise gave a 42% speed-up in finding the minimum potential energy configuration of an eight-argon-atom gas system with a Lennard-Jones 12-6 potential. The annealing speed-up also extends to quantum Monte Carlo implementations of quantum annealing. Noise improved ground-state energy estimates in a 1024-spin simulated quantum annealing simulation by 25.6%. The quantum noise flips spins along a Trotter ring. The noisy MCMC algorithm brings each Markov step closer on average to equilibrium if an inequality holds between two expectations. Gaussian or Cauchy jump probabilities reduce the noise-benefit inequality to a simple quadratic inequality. Simulations show that noise-boosted simulated annealing is more likely than noiseless annealing to sample high probability regions of the search space and to accept solutions that increase the search breadth.

A Spectral Characterization for Concentration of the Cover Time

We prove that for a sequence of finite vertex-transitive graphs of increasing sizes, the cover times are asymptotically concentrated if and only if the product of the spectral gap and the expected cover time diverges. In fact, we prove this for general reversible Markov chains under the much weaker assumption (than transitivity) that the maximal hitting time of a state is of the same order as the average hitting time.

Mixing time estimation in reversible Markov chains from a single sample path

The spectral gap $\gamma_{\star}$ of a finite, ergodic and reversible Markov chain is an important parameter measuring the asymptotic rate of convergence. In applications, the transition matrix $\mathbf{{P}}$ may be unknown, yet one sample of the chain up to a fixed time $n$ may be observed. We consider here the problem of estimating $\gamma_{\star}$ from this data. Let $\boldsymbol{\pi}$ be the stationary distribution of $\mathbf{{P}}$, and $\pi_{\star}=\min_{x}\pi (x)$. We show that if $n$ is at least $\frac{1}{\gamma_{\star}\pi_{\star}}$ times a logarithmic correction, then $\gamma_{\star}$ can be estimated to within a multiplicative factor with high probability. When $\pi $ is uniform on $d$ states, this nearly matches a lower bound of $\frac{d}{\gamma_{\star}}$ steps required for precise estimation of $\gamma_{\star}$. Moreover, we provide the first procedure for computing a fully data-dependent interval, from a single finite-length trajectory of the chain, that traps the mixing time $t_{\mathrm{mix}}$ of the chain at a prescribed confidence level. The interval does not require the knowledge of any parameters of the chain. This stands in contrast to previous approaches, which either only provide point estimates, or require a reset mechanism, or additional prior knowledge. The interval is constructed around the relaxation time $t_{\mathrm{relax}}=1/\gamma_{\star}$, which is strongly related to the mixing time, and the width of the interval converges to zero roughly at a $1/\sqrt{n}$ rate, where $n$ is the length of the sample path.

Fastest mixing reversible Markov chain on friendship graph: Trade-off between transition probabilities among friends and convergence rate

A long-standing goal of social network research has been to alter the properties of network to achieve the desired outcome. In doing so, DeGroot’s consensus model has served as the popular choice for modeling the information diffusion and opinion formation in social networks. Achieving a trade-off between the cost associated with modifications made to the network and the speed of convergence to the desired state has shown to be a critical factor. This has been treated as the Fastest Mixing Markov Chain (FMMC) problem over a graph with given transition probabilities over a subset of edges. Addressing this multi-objective optimization problem over the friendship graph, this paper has provided the corresponding Pareto optimal points or the Pareto frontier. In the case of friendship graph with at least three blades, it is shown that the Pareto frontier is reduced to a global minimum point which is same as the optimal point corresponding to the minimum spanning tree of the friendship graph, i.e., the star topology. Furthermore, a lower limit for transition probabilities among friends has been provided, where values higher than this limit do not have any impact on the convergence rate.

Scaling Laws for Consensus Protocols Subject to Noise

We study the performance of discrete-time consensus protocols in the presence of additive noise. When the consensus dynamic corresponds to a reversible Markov chain, we give an exact expression for a weighted version of steady-state disagreement in terms of the stationary distribution and hitting times in an underlying graph. We then show how this result can be used to characterize the noise robustness of a class of protocols for formation control in terms of the Kemeny constant of an underlying graph.

Computationally Efficient Estimation of the Spectral Gap of a Markov Chain

We consider the problem of estimating from sample paths the absolute spectral gap 1 - λ of a reversible, irreducible and aperiodic Markov chain (X_t)_t∈N over a finite state space Ω. We propose the UCPI (Upper Confidence Power Iteration) algorithm for this problem, a low-complexity algorithm which estimates the spectral gap in time O(n) and memory space O ((łn n)^2) given n samples. This is in stark contrast with most known methods which require at least memory space O(|Ω|), so that they cannot be applied to large state spaces. We also analyze how n should scale to reach a target estimation error. Furthermore, UCPI is amenable to parallel implementation.

Quantum fast-forwarding: Markov chains and graph property testing

We introduce a new tool for quantum algorithms called quantum fast-forwarding (QFF). The tool uses quantum walks as a means to quadratically fast-forward a reversible Markov chain. More specifically, with P the Markov chain transition matrix and D = \sqrt{P\circ P^T} its discriminant matrix (D=P if P is symmetric), we construct a quantum walk algorithm that for any quantum state |v> and integer t returns a quantum state \epsilon-close to the state D^t|v>/\|D^t|v>. The algorithm uses O(|D^t|v>|^{-1}\sqrt{t\log(\epsilon\|D^t|v>})^{-1}}) expected quantum walk steps and O(\|D^t|v>|^{-1}) expected reflections around |v>. This shows that quantum walks can accelerate the transient dynamics of Markov chains, complementing the line of results that proves the acceleration of their limit behavior. We show that this tool leads to speedups on random walk algorithms in a very natural way. Specifically we consider random walk algorithms for testing the graph expansion and clusterability, and show that we can quadratically improve the dependency of the classical property testers on the random walk runtime. Moreover, our quantum algorithm exponentially improves the space complexity of the classical tester to logarithmic. As a subroutine of independent interest, we use QFF for determining whether a given pair of nodes lies in the same cluster or in separate clusters. This solves a robust version of s-t connectivity, relevant in a learning context for classifying objects among a set of examples. The different algorithms crucially rely on the quantum speedup of the transient behavior of random walks.

Reproducing kernel orthogonal polynomials on the multinomial distribution

Diaconis and Griffiths (2014) study the multivariate Krawtchouk polynomials orthogonal on the multinomial distribution. In this paper we derive the reproducing kernel orthogonal polynomials Qn(x,y;N,p) on the multinomial distribution which are sums of products of orthonormal polynomials in x and y of fixed total degree n=0,1,…,N. The Poisson kernel ∑n=0NρnQn(x,y;N,p) arises naturally from a probabilistic argument. An application to a multinomial goodness of fit test is developed, where the chi-squared test statistic is decomposed into orthogonal components which test the order of fit. A new duplication formula for the reproducing kernel polynomials in terms of the 1-dimensional Krawtchouk polynomials is derived. The duplication formula allows a Lancaster characterization of all reversible Markov chains with a multinomial stationary distribution whose eigenvectors are multivariate Krawtchouk polynomials and where eigenvalues are repeated within the same total degree. The χ2 cutoff time, and total variation cutoff time is investigated in such chains. Emphasis throughout the paper is on a probabilistic understanding of the polynomials and their applications, particularly to Markov chains.

Harmonicity and invariance on slices of the Boolean cube

In a recent work with Kindler and Wimmer we proved an invariance principle for the slice for low-influence, low-degree harmonic multilinear polynomials (a polynomial in $$x_1,\ldots ,x_n$$ is harmonic if it is annihilated by $$\sum _{i=1}^n \frac{\partial }{\partial x_i}$$ ). Here we provide an alternative proof for general low-degree harmonic multilinear polynomials, with no constraints on the influences. We show that any real-valued harmonic multilinear polynomial on the slice whose degree is $$o(\sqrt{n})$$ has approximately the same distribution under the slice and cube measures. Our proof is based on ideas and results from the representation theory of $$S_n$$ , along with a novel decomposition of random increasing paths in the cube in terms of martingales and reverse martingales. While such decompositions have been used in the past for stationary reversible Markov chains, our decomposition is applied in a non-stationary non-reversible setup. We also provide simple proofs for some known and some new properties of harmonic functions which are crucial for the proof. Finally, we provide independent simple proofs for the known facts that (1) one cannot distinguish between the slice and the cube based on functions of o(n) coordinates and (2) Boolean symmetric functions on the cube cannot be approximated under the uniform measure by functions whose sum of influences is $$o(\sqrt{n})$$ .

Understanding Our Markov Chain Significance Test: A Reply to Cho and Rubinstein-Salzedo

The article of Cho and Rubinstein-Salzedo seeks to cast doubt on our previous paper, which described a rigorous statistical test which can be applied to reversible Markov chains. In particular, Cho and Rubinstein-Salzedo seem to suggest that the test we describe might not be a reliable indicator of gerrymandering, when the test is applied to certain redistricting Markov chains. However, the examples constructed by Cho and Rubinstein-Salzedo in fact demonstrate a different point: that our test is not the same as another class of gerrymandering tests, which Cho and Rubinstein-Salzedo prefer. But we agree and emphasized this very distinction in our original paper. In this reply, we reply to the criticisms of Cho and Rubinstein-Salzedo, and discuss, more generally, the advantages of the various tests available in the context of detecting gerrymandering of political districtings.

Fastest random walk on a path

In this paper, we consider two convex optimisation problems in order to maximise the mixing rate of a Markov chain on an undirected path. In the first formulation, the holding probabilities of vertices are identical and the transition probabilities from a vertex to its neighbours are equal, whereas the second formulation is the more general reversible Markov chain with the same degree proportional stationary distribution. We derive analytical results on the solutions of the optimisation problems and compare the spectra of the associated transition probability matrices.

Multiple Ergodicity For Reversible Markov Chains

Inspired by the multiple recurrence and multiple ergodic theorems for measure preserving systems, we discuss an analogous question for measure preserving semigroups. In this note, we deal with the symmetric semigroups associated to reversible Markov chains.

Continuum versus discrete networks, graph Laplacians, and reproducing kernel Hilbert spaces

Motivated by applications to machine learning, we construct a reversible and irreducible Markov chain whose state space is a certain collection of measurable sets of a chosen l.c.h. space X. We study the resulting network (connected undirected graph), including transience, Royden and Riesz decompositions, and kernel factorization. We describe a construction for Hilbert spaces of signed measures which comes equipped with a new notion of reproducing kernels and there is a unique solution to a regularized optimization problem involving the approximation of L2 functions by functions of finite energy. The latter has applications to machine learning (for Markov random fields, for example).

Spectral clustering for non-reversible Markov chains

Spectral clustering methods are based on solving eigenvalue problems for the identification of clusters, e.g., the identification of metastable subsets of a Markov chain. Usually, real-valued eigenvectors are mandatory for this type of algorithms. The Perron Cluster Analysis (PCCA+) is a well-known spectral clustering method of Markov chains. It is applicable for reversible Markov chains, because reversibility implies a real-valued spectrum. We also extend this spectral clustering method to non-reversible Markov chains and give some illustrative examples. The main idea is to replace the eigenvalue problem by a real-valued Schur decomposition. By this extension non-reversible Markov chains can be analyzed. Furthermore, the chains do not need to have a positive stationary distribution. In addition to metastabilities, dominant cycles and sinks can also be identified. This novel method is called GenPCCA (i.e., generalized PCCA), since it includes the case of non-reversible processes. We also apply the method to real-world eye-tracking data.

Which ergodic averages have finite asymptotic variance?

We show that the class of $L^{2}$ functions for which ergodic averages of a reversible Markov chain have finite asymptotic variance is determined by the class of $L^{2}$ functions for which ergodic averages of its associated jump chain have finite asymptotic variance. This allows us to characterize completely which ergodic averages have finite asymptotic variance when the Markov chain is an independence sampler. From a practical perspective, the most important result identifies a simple sufficient condition for all ergodic averages of $L^{2}$ functions of the primary variable in a pseudo-marginal Markov chain to have finite asymptotic variance.

Performance modelling of a multirate loss system with batched Poisson arrivals under a probabilistic threshold policy

The authors consider a link accommodating batched Poisson arriving calls of different service classes. A batch of generally distributed number of calls arrive to the link at exponentially distributed time-points. Each call is treated separately from the rest, and its acceptance is decided, according to the available link bandwidth (partial batch blocking discipline). For congestion control, if the number of in-service calls of a service class exceeds a threshold (dedicated to the service class), a new call of the same service class is accepted with a probability, dependent on the system state (probabilistic threshold policy). The link is analysed as a multirate loss system, via a reversible Markov chain. The latter leads to a product form solution (PFS) for the steady-state distribution. Based on the PFS, the authors propose models for the accurate determination of time and call congestion probabilities and link utilisation. Comparison against other existing models under the complete sharing or the bandwidth reservation policy reveals the necessity and consistency of the proposed models.

A Markov theoretic description of stacking-disordered aperiodic crystals including ice and opaline silica.

This article reviews the Markov theoretic description of one-dimensional aperiodic crystals, describing the stacking-faulted crystal polytype as a special case of an aperiodic crystal. Under this description the centrosymmetric unit cell underlying a topologically centrosymmetric crystal is generalized to a reversible Markov chain underlying a reversible aperiodic crystal. It is shown that for the close-packed structure almost all stackings are irreversible when the interaction reichweite s > 4. Moreover, the article presents an analytic expression of the scattering cross section of a large class of stacking-disordered aperiodic crystals, lacking translational symmetry of their layers, including ice and opaline silica (opal CT). The observed stackings and their underlying reichweite are then related to the physics of various nucleation and growth processes of disordered ice. The article discusses how the derived expressions of scattering cross sections could significantly improve implementations of Rietveld's refinement scheme and compares this Q-space approach with the pair-distribution function analysis of stacking-disordered materials.