Irreducible Markov Chain Research Articles

Hitting time distribution for skip-free Markov chains: A simple proof

A well-known theorem for an irreducible skip-free Markov chain on the nonnegative integers with absorbing state d, under some conditions, is that the hitting (absorbing) time of state d starting from state 0 is distributed as the sum of d independent geometric (or exponential) random variables. The purpose of this paper is to present a direct and simple proof of the theorem in the cases of both discrete and continuous time skip-free Markov chains. Our proof is to calculate directly the generation functions (or Laplace transforms) of hitting times in terms of the iteration method.

Some potential-theoretic techniques in non-reversible Markov chains

Many of the well known classification theorems of irreducible Markov chains \(X\) placing them into transient or recurrent classes are proved by assuming that \(X\) is reversible. It is shown here that the ‘reversible’ condition in \(X\) can be removed if potential-theoretic methods are used.

THE DISTRIBUTION OF MIXING TIMES IN MARKOV CHAINS

The distribution of the "mixing time" or the "time to stationarity" in a discrete time irreducible Markov chain, starting in state i, can be defined as the number of trials to reach a state sampled from the stationary distribution of the Markov chain. Expressions for the probability generating function, and hence the probability distribution of the mixing time, starting in state i, are derived and special cases explored. This extends the results of the author regarding the expected time to mixing [Hunter, JJ (2006). Mixing times with applications to perturbed Markov chains. Linear Algebra and Its Applications, 417, 108–123] and the variance of the times to mixing, [Hunter, JJ (2008). Variances of first passage times in a Markov chain with applications to mixing times. Linear Algebra and Its Applications, 429, 1135–1162]. Some new results for the distribution of the recurrence and the first passage times in a general irreducible three-state Markov chain are also presented.

Dynamic sender–receiver games

Abstract We consider a dynamic version of sender–receiver games, where the sequence of states follows an irreducible Markov chain observed by the sender. Under mild assumptions, we provide a simple characterization of the limit set of equilibrium payoffs, as players become very patient. Under these assumptions, the limit set depends on the Markov chain only through its invariant measure. The (limit) equilibrium payoffs are the feasible payoffs that satisfy an individual rationality condition for the receiver, and an incentive compatibility condition for the sender.

Noise Tolerance of Expanders and Sublinear Expansion Reconstruction

We consider the problem of online sublinear expander reconstruction and its relation to random walks in “noisy expanders. Given access to an adjacency list representation of a bounded-degree graph $G$, we want to convert this graph into a bounded-degree expander $G'$ changing $G$ as little as possible. The graph $G'$ will be output by a distributed filter: this is a sublinear time procedure that, given a query vertex, outputs all its neighbors in $G'$ and can do so even in a distributed manner, ensuring consistency in all the answers. One of the main tools in our analysis is a result on the behavior of random walks in graph that are almost expanders: graphs that are formed by arbitrarily connecting a small unknown graph (the noise) to a large expander. We show that a random walk from almost any vertex in the expander part will have fast mixing properties, in the general setting of irreducible finite Markov chains. We also design sublinear time procedures to distinguish vertices of the expander part from ...

Some universal estimates for reversible Markov chains

We obtain universal estimates on the convergence to equilibrium and the times of coupling for continuous time irreducible reversible finite-state Markov chains, both in the total variation and in the $L^2$ norms. The estimates in total variation norm are obtained using a novel identity relating the convergence to equilibrium of a reversible Markov chain to the increase in the entropy of its one-dimensional distributions. In addition, we propose a universal way of defining the ultrametric partition structure on the state space of such Markov chains. Finally, for chains reversible with respect to the uniform measure, we show how the global convergence to equilibrium can be controlled using the entropy accumulated by the chain.

A New GMRES(m) Method for Markov Chains

This paper presents a class of new accelerated restarted GMRES method for calculating the stationary probability vector of an irreducible Markov chain. We focus on the mechanism of this new hybrid method by showing how to periodically combine the GMRES and vector extrapolation method into a much efficient one for improving the convergence rate in Markov chain problems. Numerical experiments are carried out to demonstrate the efficiency of our new algorithm on several typical Markov chain problems.

Efficiency in Games With Markovian Private Information

We study repeated Bayesian games with communication and observable actions in which the players' privately known payoffs evolve according to an irreducible Markov chain whose transitions are independent across players. Our main result implies that, generically, any Pareto-efficient payoff vector above a stationary minmax value can be approximated arbitrarily closely in a perfect Bayesian equilibrium as the discount factor goes to 1. As an intermediate step, we construct an approximately efficient dynamic mechanism for long finite horizons without assuming transferable utility. [PUBLICATION ABSTRACT]

Markov chain properties in terms of column sums of the transition matrix

Questions are posed regarding the influence that the column sums of the transition probabilities of a stochastic matrix (with row sums all one) have on the stationary distribution, the mean first passage times and the Kemeny constant of the associated irreducible discrete time Markov chain. Some new relationships, including some inequalities, and partial answers to the questions, are given using a special generalized matrix inverse that has not previously been considered in the literature on Markov chains.

Asymptotic analysis of the closed network with time-dependent service parameters and single-type messages

Journal of Applied Mathematics and Computational Mechanics, Prace Naukowe Instytutu Matematyki i Informatyki, Politechnika Częstochowska, Scientific Research of the Institute of Mathematics and Computer Science, Czestochowa University of Technology

Maximum entropy mixing time of circulant Markov processes

We consider both discrete-time irreducible Markov chains with circulant transition probability matrix P and continuous-time irreducible Markov processes with circulant transition rate matrix Q. In both cases we provide an expression of all the moments of the mixing time. In the discrete case, we prove that all the moments of the mixing time associated with the transition probability matrix αP+[1−α]P∗ are maximum in the interval 0≤α≤1 when α=1/2, where P∗ is the transition probability matrix of the time-reversed chain. Similarly, in the continuous case, we show that all the moments of the mixing time associated with the transition rate matrix αQ+[1−α]Q∗ are also maximum in the interval 0≤α≤1 when α=1/2, where Q∗ is the time-reversed transition rate matrix.

Analysis of the GI/Geo/<i>c</i> Queue with Working Vacations

Working vacation queue models are well applied in the modeling and analysis of the router in optical networks. The GI/Geo/c queue with working vacations is studied in this paper. Through establishing two-dimensional Markov chain and using matrix-geometric solution method, the stability condition is derived. Adopting UL-type RG-factorization of irreducible Markov chain, the stationary distribution is given. Based on these, the probability distribution of queue-length and PGF of waiting time are obtained in the end.

Persistent tracking and identification of regime‐switching systems with structural uncertainties: unmodeled dynamics, observation bias, and nonlinear model mismatch

SUMMARYThis work focuses on tracking and system identification of systems with regime‐switching parameters, which are modeled by a Markov process. It introduces a framework for persistent identification problems that encompass many typical system uncertainties, including parameter switching, stochastic observation disturbances, deterministic unmodeled dynamics, sensor observation bias, and nonlinear model mismatch. In accordance with the ‘frequency’ of the parameter switching process, we divide the problems into two classes. For fast‐switching systems, the switching parameters are stochastic processes modeled by irreducible and aperiodic Markov chains. Because accurately tracking real‐time parameters in such systems is not possible because of the uncertainty principles, the effect of parameter switching is evaluated on their average by the stationary distribution of the Markovian chain and estimated by the least squares algorithms. We derive upper and lower bounds on identification errors, which characterize how identification accuracy depends on the earlier uncertainty terms. When the system parameters switch their values infrequently in a probabilistic sense, their values can be tracked based on input/output observations. Stochastic approximation algorithms with adaptive step sizes are used for such systems. Simulation studies are carried out to demonstrate that slowly varying parameters could be tracked with reasonable accuracy.Copyright © 2012 John Wiley & Sons, Ltd.

Target containment control of multi-agent systems with random switching interconnection topologies

In this paper, the distributed containment control is considered for a second-order multi-agent system guided by multiple leaders with random switching topologies. The multi-leader control problem is investigated via a combination of convex analysis and stochastic process. The interaction topology between agents is described by a continuous-time irreducible Markov chain. A necessary and sufficient condition is obtained to make all the mobile agents almost surely asymptotically converge to the static convex leader set. Moreover, conditions on the tracking estimation are provided for the convex target set determined by moving multiple leaders.

Markov modulation of a two-sided reflected Brownian motion with application to fluid queues

In this paper, we study a reflected Markov-modulated Brownian motion with a two sided reflection in which the drift, diffusion coefficient and the two boundaries are (jointly) modulated by a finite state space irreducible continuous time Markov chain. The goal is to compute the stationary distribution of this Markov process, which in addition to the complication of having a stochastic boundary can also include jumps at state change epochs of the underlying Markov chain because of the boundary changes. We give the general theory and then specialize to the case where the underlying Markov chain has two states.

Almost sure central limit theorem for partial sums of Markov chain

The authors prove an almost sure central limit theorem for partial sums based on an irreducible and positive recurrent Markov chain using logarithmic means, which realizes the extension of the almost sure central limit theorem for partial sums from an i.i.d. sequence of random variables to a Markov chain.

Dynamic Sender-Receiver Games

We consider a dynamic version of sender-receiver games, where the sequence of states follows an irreducible Markov chain observed by the sender. Under mild assumptions, we provide a simple characterization of the limit set of equilibrium payoffs, as players become very patient. Under these assumptions, the limit set depends on the Markov chain only through its invariant measure. The (limit) equilibrium payoffs are the feasible payoffs that satisfy an individual rationality condition for the receiver, and an incentive compatibility condition for the sender.

Iterative component‐wise bounds for the steady‐state distribution of a Markov chain

SUMMARYWe prove new iterative algorithms to provide component‐wise bounds of the steady‐state distribution of an irreducible and aperiodic Markov chain. These bounds are based on simple properties of (max,+) and (min,+) sequences. The bounds are improved at each iteration. Thus, we have a clear trade‐off between tightness of the bounds (some algorithms converge to the true solution) and computation times. Copyright © 2011 John Wiley & Sons, Ltd.

Triangular and skew-symmetric splitting method for numerical solutions of Markov chains

In this paper, a theorem is presented to indicate that there exists a nonnegative constant ϵ ≥ 0 such that the matrix A = Q T + ϵ I is a positive-definite matrix, where I ∈ R n × n is an identity matrix and Q T ∈ R n × n is a matrix with positive diagonal and nonpositive off-diagonal elements. Then a class of triangular and skew-symmetric splitting (TSS) iteration method is applied to solve the positive-definite linear system A x = b for obtaining the stationary probability vector of an irreducible Markov chain. Theoretical analyses show that the TSS iteration method converges unconditionally to the unique solution of the linear system, with the upper bound of its contraction factor dependent only on the spectrum of the triangular part and independent of the eigenvectors of the matrices involved. Moreover, the inexact triangular and skew-symmetric splitting (ITSS) iteration method, which employs certain Krylov subspace methods as the inner iteration processes at each step of the outer TSS iteration method, is proposed to accelerate the convergence of the TSS iteration method. Numerical experiments are used to illustrate the effectiveness of the TSS and ITSS iteration methods.

Fast multilevel methods for Markov chains

SUMMARYThis paper describes multilevel methods for the calculation of the stationary probability vector of large, sparse, irreducible Markov chains. In particular, several recently proposed significant improvements to the multilevel aggregation method of Horton and Leutenegger are described and compared. Furthermore, we propose a very simple improvement of that method using an over‐correction mechanism. We also compare with more traditional iterative methods for Markov chains such as weighted Jacobi, two‐level aggregation/disaggregation, and preconditioned stabilized biconjugate gradient and generalized minimal residual method. Numerical experiments confirm that our improvements lead to significant speedup, and result in multilevel methods that are competitive with leading iterative solvers for Markov chains. Copyright © 2011 John Wiley & Sons, Ltd.