Irreducible Markov Chain Research Articles

On an irreducibility type condition for the ergodicity of nonconservative semigroups

We propose a simple criterion, inspired from the irreducible aperiodic Markov chains, to derive the exponential convergence of general positive semi-groups. When not checkable on the whole state space, it can be combined to the use of Lyapunov functions. It differs from the usual generalization of irreducibility and is based on the accessibility of the trajectories of the underlying dynamics. It allows to obtain new existence results of principal eigenelements, and their exponential attractiveness, for a nonlocal selection-mutation population dynamics model defined in a space-time varying environment.

Kemeny's constant for countable Markov chains

In this paper, we aim to investigate Kemeny's constant for discrete-time or continuous-time, irreducible and positive recurrent Markov chains on a countable state space. We show that Kemeny's constant is infinite for a discrete-time Markov chain on an infinitely countable state space, which confirms a conjecture in page 1031 of [3]. Moreover, it is shown that the study on Kemeny's constant for a continuous-time Markov chain with a bounded generator can be converted to that for a discrete-time Markov chain through the uniformization technique. It is different and challenging to analyze Kemeny's constant for a chain with unbounded generator. Finally, a upward skip-free process is considered and some discussions for Kemeny's constant are given.

Approximation of the exit probability of a stable Markov modulated constrained random walk

Let X be the constrained random walk on \({\mathbb {Z}}_+^2\) having increments (1, 0), \((-\,1,1)\), \((0,-\,1)\) with jump probabilities \(\lambda (M_k)\), \(\mu _1(M_k)\), and \(\mu _2(M_k)\) where M is an irreducible aperiodic finite state Markov chain. The process X represents the lengths of two tandem queues with arrival rate \(\lambda (M_k)\), and service rates \(\mu _1(M_k)\), and \(\mu _2(M_k)\); the process M represents the random environment within which the system operates. We assume that the average arrival rate with respect to the stationary measure of M is less than the average service rates, i.e., X is assumed stable. Let \(\tau _n\) be the first time when the sum of the components of X equals n for the first time. Let Y be the random walk on \({{\mathbb {Z}}} \times {{\mathbb {Z}}}_+\) having increments \((-\,1,0)\), (1, 1), \((0,-\,1)\) with probabilities \(\lambda (M_k)\), \(\mu _1(M_k)\), and \(\mu _2(M_k)\). Supposing that the queues share a joint buffer of size n, \(p_n =P_{(x_n,m)}(\tau _n < \tau _0)\) is the probability that this buffer overflows during a busy cycle of the system. To the best of our knowledge, the only methods currently available for the approximation of \(p_n\) are classical large deviations analysis giving the exponential decay rate of \(p_n\) and rare event simulation. Let \(\tau \) be the first time the components of Y are equal. For \(x \in {{\mathbb {R}}}_+^2\), \(x(1) + x(2) < 1\), \(x(1) > 0\), and \(x_n = \lfloor nx \rfloor \), we show that \(P_{(n-x_n(1),x_n(2),m)}( \tau < \infty )\) approximates \(P_{(x_n,m)}(\tau _n < \tau _0)\) with exponentially vanishing relative error as \(n\rightarrow \infty \). For the analysis we define a characteristic matrix in terms of the jump probabilities of (X, M). The 0-level set of the characteristic polynomial of this matrix defines the characteristic surface; conjugate points on this surface and the associated eigenvectors of the characteristic matrix are used to define (sub/super) harmonic functions which play a fundamental role both in our analysis and the computation/approximation of \(P_{(y,m)}(\tau < \infty )\).

Markov-Modulated Linear Regression Parameter Estimation Using a Convolution of Exponential Densities

Markov-modulated linear regression model is a special case of the Markov-additive process (𝒀, 𝑱) = {(𝒀(𝒕), 𝑱(𝒕)), 𝒕 ≥ 𝟎}, where component J is called Markov, and component Y is additive and described by a linear regression. The component J is a continuous-time homogeneous irreducible Markov chain with the known transition intensities between the states. Usually this Markov component is called the external environment or background process. Unknown regression coefficients depend on external environment state, but regressors remain constant. This research considers the case, when the Markov property is not satisfied, namely, the sojourn time in each state is not exponentially distributed. Estimation procedure for unknown model parameters is described when it’s possible to represent transition intensities as a convolution of exponential densities. An efficiency of such an approach is evaluated by a simulation.

Improvements on the distribution of maximal segmental scores in a Markovian sequence

AbstractLet $(A_i)_{i \geq 0}$ be a finite-state irreducible aperiodic Markov chain and f a lattice score function such that the average score is negative and positive scores are possible. Define $S_0\coloneqq 0$ and $S_k\coloneqq \sum_{i=1}^k f(A_i)$ the successive partial sums, $S^+$ the maximal non-negative partial sum, $Q_1$ the maximal segmental score of the first excursion above 0, and $M_n\coloneqq \max_{0\leq k\leq\ell\leq n} (S_{\ell}-S_k)$ the local score, first defined by Karlin and Altschul (1990). We establish recursive formulae for the exact distribution of $S^+$ and derive a new approximation for the tail behaviour of $Q_1$ , together with an asymptotic equivalence for the distribution of $M_n$ . Computational methods are explicitly presented in a simple application case. The new approximations are compared with those proposed by Karlin and Dembo (1992) in order to evaluate improvements, both in the simple application case and on the real data examples considered by Karlin and Altschul (1990).

Recurrence of Markov chain traces

Nous montrons que les graphes pour lesquels la marche aléatoire simple est transiente n’admettent pas de chaîne de Markov transiente aux plus proches voisins (même non réversible) visitant toutes les arêtes avec probabilité positive, tandis qu’il en existe une pour le réseau carré $\mathbb{Z}^{2}$. En particulier, le réseau $d$-dimensionnel $\mathbb{Z}^{d}$ admet une telle chaîne de Markov seulement lorsque $d=2$. Lorsque $d=2$ nous présentons un exemple de Gady Kozma, et le résultat général est obtenu en prouvant que la trace de toute chaîne de Markov sur un espace d’état dénombrable qui admet une mesure stationnaire est presque sûrement récurrente pour la marche simple. Le cas où la chaîne est réversible a été traité par Gurel-Gurevich, Lyons et le premier auteur. Nous exploitons des résultats récents de théorie du potentiel sur les chaînes de Markov non réversibles pour étendre leur résultat au cas non réversible.

Computationally Efficient Estimation of the Spectral Gap of a Markov Chain

We consider the problem of estimating from sample paths the absolute spectral gap 1 - λ of a reversible, irreducible and aperiodic Markov chain (Xt )t€N over a finite state space Ω. We propose the UCPI (Upper Confidence Power Iteration) algorithm for this problem, a low-complexity algorithm which estimates the spectral gap in time O(n) and memory space O((lnn)2) given n samples. This is in stark contrast with most known methods which require at least memory space O((ln n)2), so that they cannot be applied to large state spaces. Furthermore, UCPI is amenable to parallel implementation.

Sign-stability of Positive Markov Jump Linear Systems

This paper investigates stability of Positive Markov Jump Linear Systems (PMJLSs) in the absence of a numerical realization. It considers the situation when only signs (and not magnitudes) of the entries of the subsystem matrices and the Markov transition matrices are known. The result is an analysis of a qualitative notion of stability known as sign-stability. Although the notions of sign-stability of PMJLSs are natural extensions of standard stochastic stability concepts such as exponential almost sure, mean and mean-square stability, the sign-stability notions are proven equivalent, which is not the case for their corresponding standard concepts. Moreover, for irreducible Markov chains, the particular structure of the Markov chain is shown to have no bearing on sign-stability.

Stein’s method for stationary distributions of Markov chains and application to Ising models

We develop a new technique, based on Stein’s method, for comparing two stationary distributions of irreducible Markov chains whose update rules are close in a certain sense. We apply this technique to compare Ising models on $d$-regular expander graphs to the Curie–Weiss model (complete graph) in terms of pairwise correlations and more generally $k$th order moments. Concretely, we show that $d$-regular Ramanujan graphs approximate the $k$th order moments of the Curie–Weiss model to within average error $k/\sqrt{d}$ (averaged over size $k$ subsets), independent of graph size. The result applies even in the low-temperature regime; we also derive simpler approximation results for functionals of Ising models that hold only at high temperatures.

Finitary isomorphisms of some infinite entropy Bernoulli flows

A consequence of Ornstein theory is that the infinite entropy flows associated with Poisson processes and continuous-time irreducible Markov chains on a finite number of states are isomorphic as measure-preserving systems. We give an elementary construction of such an isomorphism that has an additional finitariness property, subject to the additional conditions that the Markov chain has a uniform holding rate and a mixing skeleton.

On the rate of convergence for the length of the longest common subsequences in hidden Markov models

AbstractLet (X, Y) = (Xn, Yn)n≥1 be the output process generated by a hidden chain Z = (Zn)n≥1, where Z is a finite-state, aperiodic, time homogeneous, and irreducible Markov chain. Let LCn be the length of the longest common subsequences of X1,..., Xn and Y1,..., Yn. Under a mixing hypothesis, a rate of convergence result is obtained for E[LCn]/n.

Dynamical properties of random walks

In this paper, we study dynamical properties such as hypercyclicity, supercyclicity, frequent hypercyclicity and chaoticity for transition operators associated to countable irreducible Markov chains. As particular cases, we consider simple random walks on [Formula: see text] and [Formula: see text].

A graph deep learning method for short‐term traffic forecasting on large road networks

AbstractShort‐term traffic flow prediction on a large‐scale road network is challenging due to the complex spatial–temporal dependencies, the directed network topology, and the high computational cost. To address the challenges, this article develops a graph deep learning framework to predict large‐scale network traffic flow with high accuracy and efficiency. Specifically, we model the dynamics of the traffic flow on a road network as an irreducible and aperiodic Markov chain on a directed graph. Based on the representation, a novel spatial–temporal graph inception residual network (STGI‐ResNet) is developed for network‐based traffic prediction. This model integrates multiple spatial–temporal graph convolution (STGC) operators, residual learning, and the inception structure. The proposed STGC operators can adaptively extract spatial–temporal features from multiple traffic periodicities while preserving the topology information of the road network. The proposed STGI‐ResNet inherits the advantages of residual learning and inception structure to improve prediction accuracy, accelerate the model training process, and reduce difficult parameter tuning efforts. The computational complexity is linearly related to the number of road links, which enables citywide short‐term traffic prediction. Experiments using a car‐hailing traffic data set at 10‐, 30‐, and 60‐min intervals for a large road network in a Chinese city shows that the proposed model outperformed various state‐of‐the‐art baselines for short‐term network traffic flow prediction.

Estimating random walk centrality in networks

Random walk centrality (equivalently, the accessibility index) for the states of a time-homogeneous irreducible Markov chain on a finite state space is considered. It is known that the accessibility index for a particular state can be written in terms of the first and second moments of the first return time to that state. Based on that observation, the problem of estimating the random walk centrality of a state is approached by taking realizations of the Markov chain, and then statistically estimating the first two moments of the corresponding first return time. In addition to the estimate of the random walk centrality, this method also yields the standard error, the bias and a confidence interval for that estimate. For the case that the directed graph of the transition matrix for the Markov chain has a cut-point, an alternate strategy for computing the random walk centrality is outlined that may be of use when the centrality values are of interest for only some of the states. In order to illustrate the effectiveness of the results, estimates of the random walk centrality arising from random walks for several directed and undirected graphs are discussed.

Computationally Efficient Estimation of the Spectral Gap of a Markov Chain

We consider the problem of estimating from sample paths the absolute spectral gap 1 - λ of a reversible, irreducible and aperiodic Markov chain (X_t)_t∈N over a finite state space Ω. We propose the UCPI (Upper Confidence Power Iteration) algorithm for this problem, a low-complexity algorithm which estimates the spectral gap in time O(n) and memory space O ((łn n)^2) given n samples. This is in stark contrast with most known methods which require at least memory space O(|Ω|), so that they cannot be applied to large state spaces. We also analyze how n should scale to reach a target estimation error. Furthermore, UCPI is amenable to parallel implementation.

Unified theory for finite Markov chains

We provide a unified framework to compute the stationary distribution of any finite irreducible Markov chain or equivalently of any irreducible random walk on a finite semigroup S. Our methods use geometric finite semigroup theory via the Karnofsky–Rhodes and the McCammond expansions of finite semigroups with specified generators; this does not involve any linear algebra. The original Tsetlin library is obtained by applying the expansions to P(n), the set of all subsets of an n element set. Our set-up generalizes previous groundbreaking work involving left-regular bands (or R-trivial bands) by Brown and Diaconis, extensions to R-trivial semigroups by Ayyer, Steinberg, Thiéry and the second author, and important recent work by Chung and Graham. The Karnofsky–Rhodes expansion of the right Cayley graph of S in terms of generators yields again a right Cayley graph. The McCammond expansion provides normal forms for elements in the expanded S. Using our previous results with Silva based on work by Berstel, Perrin, Reutenauer, we construct (infinite) semaphore codes on which we can define Markov chains. These semaphore codes can be lumped using geometric semigroup theory. Using normal forms and associated Kleene expressions, they yield formulas for the stationary distribution of the finite Markov chain of the expanded S and the original S. Analyzing the normal forms also provides an estimate on the mixing time.

Extreme Value Distributions for Two Kinds of Path Sums of Markov Chain

We consider the extreme value distributions for two kinds of path sums on condition that every path is a discrete time, homogeneous and irreducible ergodic Markov chain with a finite number of states. We not only derive the extreme value distribution for the first kind but also establish the equivalence of limit distributions of a sum and of a maximum sum for the second kind. Numerical experiments are conducted to confirm our results.

Optimal dividend policy when risk reserves follow a jump–diffusion process with a completely monotone jump density under Markov-regime switching

The paper studies optimal dividend distribution for an insurance company whose risk reserves in the absence of dividends follow a Markov-modulated jump–diffusion process with a completely monotone jump density where jump densities and parameters including discount rate are modulated by a finite-state irreducible Markov chain. The major goal is to maximize the expected cumulative discounted dividend payments until ruin time when risk reserve is less than or equal to zero for the first time. I extend the results of Jiang (2015) for a Markov-modulated jump–diffusion process from exponential jump densities to completely monotone jump densities by proving that it is also optimal to take a modulated barrier strategy at some positive regime-dependent levels and that value function as the fixed point of a contraction is explicitly characterized.

A Physical-Layer Scheduling Approach in Large-Scale Cooperative Networks

In a sensor, mesh, or ad hoc network, different transmissions take place between multiple source-destination pairs, which is referred to as data flows. In this paper, each data flow is associated with a certain duty cycle schedule to avoid collisions of multiple flows in an energy-efficient manner. Cooperative routes in the network are assumed to be linear and strip-shaped, where each hop or a level contains multiple nodes, and the distance between the nodes is kept constant. We assume all nodes in a cooperative route, which corresponds to one source-destination pair, follow the same duty cycle schedule, while schedules of different routes are orthogonal in time to avoid interference between routes. Therefore, when a new route is formed, it is important to select a duty cycle schedule that is not already in use by the nodes participating in the new route. As part of route set-up, an orthogonal frequency division multiplexing packet travels from the source to the destination, identifying the schedules in use by the nodes along the path. Our objective in this paper is to analyze the probabilities of detection for this method of detecting schedules, which is useful to gain insights into duty cycling-based medium access protocol design in the presence of multiple data flows. To this end, the successive transmissions in a linear network are modeled using an irreducible discrete time Markov chain. Also, three different approaches to detect a certain schedule are investigated, which are distinguished by the binary integration process with multiple sub-carriers corresponding to the schedule. We derive the probability transition matrix for the Markov chain based on different distributions of the received signal energy, the left eigenvector of which yields the probability of detection.

Multivariate approximation in total variation using local dependence

We establish two theorems for assessing the accuracy in total variation of multivariate discrete normal approximation to the distribution of an integer valued random vector $W$. The first is for sums of random vectors whose dependence structure is local. The second applies to random vectors~$W$ resulting from integrating the $\mathbb{Z}^d$-valued marks of a marked point process with respect to its ground process. The error bounds are of magnitude comparable to those given in Rinott and Rotar (1996), but now with respect to the stronger total variation distance. Instead of requiring the summands to be bounded, we make third moment assumptions. We demonstrate the use of the theorems in four applications: monochrome edges in vertex coloured graphs, induced triangles and $2$-stars in random geometric graphs, the times spent in different states by an irreducible and aperiodic finite Markov chain, and the maximal points in different regions of a homogeneous Poisson point process.