Aperiodic Markov Chain Research Articles

Lumpings of Markov Chains, Entropy Rate Preservation, and Higher-Order Lumpability

A lumping of a Markov chain is a coordinatewise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is strongly k-lumpable, if and only if the lumped process is a kth-order Markov chain for each starting distribution of the original Markov chain. We characterise strong k-lumpability via tightness of stationary entropic bounds. In the sparse setting, we give sufficient conditions on the lumping to both preserve the entropy rate and be strongly k-lumpable.

Lumpings of Markov Chains, Entropy Rate Preservation, and Higher-Order Lumpability

A lumping of a Markov chain is a coordinatewise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is stronglyk-lumpable, if and only if the lumped process is akth-order Markov chain for each starting distribution of the original Markov chain. We characterise strongk-lumpability via tightness of stationary entropic bounds. In the sparse setting, we give sufficient conditions on the lumping to both preserve the entropy rate and be stronglyk-lumpable.

Decentralized Learning for Multiplayer Multiarmed Bandits

We consider the problem of distributed online learning with multiple players in multiarmed bandit (MAB) models. Each player can pick among multiple arms. When a player picks an arm, it gets a reward. We consider both independent identically distributed (i.i.d.) reward model and Markovian reward model. In the i.i.d. model, each arm is modeled as an i.i.d. process with an unknown distribution with an unknown mean. In the Markovian model, each arm is modeled as a finite, irreducible, aperiodic and reversible Markov chain with an unknown probability transition matrix and stationary distribution. The arms give different rewards to different players. If two players pick the same arm, there is a collision, and neither of them get any reward. There is no dedicated control channel for coordination or communication among the players. Any other communication between the users is costly and will add to the regret. We propose an online index-based distributed learning policy called dUCB4 algorithm that trades off exploration versus exploitation in the right way, and achieves expected regret that grows at most as near- O(log2T). The motivation comes from opportunistic spectrum access by multiple secondary users in cognitive radio networks wherein they must pick among various wireless channels that look different to different users. This is the first distributed learning algorithm for multiplayer MABs with heterogeneous players (that have player-dependent rewards) to the best of our knowledge.

Limit theorems for some functionals with heavy tails of a discrete time Markov chain

Consider an irreducible, aperiodic and positive recurrent discrete time Markov chain (Xn,n ≥ 0) with invariant distribution μ. We shall investigate the long time behaviour of some functionals of the chain, in particular the additive functional \hbox{}Sn=∑i=1nf(Xi) for a possibly non square integrable function f. To this end we shall link ergodic properties of the chain to mixing properties, extending known results in the continuous time case. We will then use existing results of convergence to stable distributions, obtained in [M. Denker and A. Jakubowski, Stat. Probab. Lett. 8 (1989) 477–483; M. Tyran-Kaminska, Stochastic Process. Appl. 120 (2010) 1629–1650; D. Krizmanic, Ph.D. thesis (2010); B. Basrak, D. Krizmanic and J. Segers, Ann. Probab. 40 (2012) 2008–2033] for stationary mixing sequences. Contrary to the usual L^2L2 framework studied in [P. Cattiaux, D. Chafai and A. Guillin, ALEA, Lat. Am. J. Probab. Math. Stat. 9 (2012) 337–382], where weak forms of ergodicity are sufficient to ensure the validity of the Central Limit Theorem, we will need here strong ergodic properties: the existence of a spectral gap, hyperboundedness (or hypercontractivity). These properties are also discussed. Finally we give explicit examples.

Integrating Graph Partitioning and Matching for Trajectory Analysis in Video Surveillance

In order to track moving objects in long range against occlusion, interruption, and background clutter, this paper proposes a unified approach for global trajectory analysis. Instead of the traditional frame-by-frame tracking, our method recovers target trajectories based on a short sequence of video frames, e.g., 15 frames. We initially calculate a foreground map at each frame obtained from a state-of-the-art background model. An attribute graph is then extracted from the foreground map, where the graph vertices are image primitives represented by the composite features. With this graph representation, we pose trajectory analysis as a joint task of spatial graph partitioning and temporal graph matching. The task can be formulated by maximizing a posteriori under the Bayesian framework, in which we integrate the spatio-temporal contexts and the appearance models. The probabilistic inference is achieved by a data-driven Markov chain Monte Carlo algorithm. Given a period of observed frames, the algorithm simulates an ergodic and aperiodic Markov chain, and it visits a sequence of solution states in the joint space of spatial graph partitioning and temporal graph matching. In the experiments, our method is tested on several challenging videos from the public datasets of visual surveillance, and it outperforms the state-of-the-art methods.

An Adaptive Backoff Protocol with Markovian Contention Window Control

The backoff protocol is widely used for sharing a common channel among several stations in communication networks. The Binary Exponential Backoff (BEB) improves the system throughput but increases the capture effect, permitting a station to seize the channel for a long time. In this article, we introduce and analyze a new class of adaptive backoff protocols where a station changes its contention window after a successful transmission differently than BEB. The transitions between the contention window states are determined by a stochastic matrix which describes a finite, irreducible, aperiodic Markov chain. We derive a stationary distribution of an associated embedded Markov chain in an explicit form and then find the stationary distribution of the basic Markov chain explicitly. Preliminary simulation results show that our backoff protocol can reduce the capture effect in Ethernet and wireless networks significantly.

Hurst Index of Functions of Long-Range-Dependent Markov Chains

A positive recurrent, aperiodic Markov chain is said to be long-range dependent (LRD) when the indicator function of a particular state is LRD. This happens if and only if the return time distribution for that state has infinite variance. We investigate the question of whether other instantaneous functions of the Markov chain also inherit this property. We provide conditions under which the function has the same degree of long-range dependence as the chain itself. We illustrate our results through three examples in diverse fields: queueing networks, source compression, and finance.

Hurst Index of Functions of Long-Range-Dependent Markov Chains

A positive recurrent, aperiodic Markov chain is said to be long-range dependent (LRD) when the indicator function of a particular state is LRD. This happens if and only if the return time distribution for that state has infinite variance. We investigate the question of whether other instantaneous functions of the Markov chain also inherit this property. We provide conditions under which the function has the same degree of long-range dependence as the chain itself. We illustrate our results through three examples in diverse fields: queueing networks, source compression, and finance.

Persistent tracking and identification of regime‐switching systems with structural uncertainties: unmodeled dynamics, observation bias, and nonlinear model mismatch

SUMMARYThis work focuses on tracking and system identification of systems with regime‐switching parameters, which are modeled by a Markov process. It introduces a framework for persistent identification problems that encompass many typical system uncertainties, including parameter switching, stochastic observation disturbances, deterministic unmodeled dynamics, sensor observation bias, and nonlinear model mismatch. In accordance with the ‘frequency’ of the parameter switching process, we divide the problems into two classes. For fast‐switching systems, the switching parameters are stochastic processes modeled by irreducible and aperiodic Markov chains. Because accurately tracking real‐time parameters in such systems is not possible because of the uncertainty principles, the effect of parameter switching is evaluated on their average by the stationary distribution of the Markovian chain and estimated by the least squares algorithms. We derive upper and lower bounds on identification errors, which characterize how identification accuracy depends on the earlier uncertainty terms. When the system parameters switch their values infrequently in a probabilistic sense, their values can be tracked based on input/output observations. Stochastic approximation algorithms with adaptive step sizes are used for such systems. Simulation studies are carried out to demonstrate that slowly varying parameters could be tracked with reasonable accuracy.Copyright © 2012 John Wiley & Sons, Ltd.

Iterative component‐wise bounds for the steady‐state distribution of a Markov chain

SUMMARYWe prove new iterative algorithms to provide component‐wise bounds of the steady‐state distribution of an irreducible and aperiodic Markov chain. These bounds are based on simple properties of (max,+) and (min,+) sequences. The bounds are improved at each iteration. Thus, we have a clear trade‐off between tightness of the bounds (some algorithms converge to the true solution) and computation times. Copyright © 2011 John Wiley & Sons, Ltd.

Finite-State Markov Chains Obey Benford's Law

A sequence of real numbers (xn) is Benford if the significands, i.e. the fraction parts in the floating-point representation of (xn), are distributed logarithmically. Similarly, a discrete-time irreducible and aperiodic finite-state Markov chain with probability transition matrix P and limiting matrix P* is Benford if every component of both sequences of matrices (Pn − P*) and (Pn 1 − Pn) is Benford or eventually zero. Using recent tools that established Benford behavior both for Newton's method and for finite-dimensional linear maps, via the classical theories of uniform distribution modulo 1 and Perron-Frobenius, this paper derives a simple sufficient condition ('nonresonance') guaranteeing that P, or the Markov chain associated with it, is Benford. This result in turn is used to show that almost all Markov chains are Benford, in the sense that if the transition probabilities are chosen independently and continuously, then the resulting Markov chain is Benford with probability one. Concrete examples illustrate the various cases that arise, and the theory is complemented with several simulations and potential applications.

Coalescent theory for a Monoecious Random Mating Population with a Varying Size

Consider a monoecious diploid population with nonoverlapping generations, whose size varies with time according to an irreducible, aperiodic Markov chain with statesx1N,…,xKN, whereK≪N. It is assumed that all matings except for selfing are possible and equally probable. At time 0 a random sample ofn≪Ngenes is taken. Given two successive population sizesxjNandxiN, the numbers of gametes that individual parents contribute to offspring can be shown to be exchangeable random variables distributed asGij. Under minimal conditions on the first three moments ofGijfor alliandj, a suitable effective population sizeNeis derived. Then if time is recorded in a backward direction in units of 2Negenerations, it can be shown that coalescent theory holds.

Coalescent theory for a Monoecious Random Mating Population with a Varying Size

Consider a monoecious diploid population with nonoverlapping generations, whose size varies with time according to an irreducible, aperiodic Markov chain with statesx1N,…,xKN, whereK≪N. It is assumed that all matings except for selfing are possible and equally probable. At time 0 a random sample ofn≪Ngenes is taken. Given two successive population sizesxjNandxiN, the numbers of gametes that individual parents contribute to offspring can be shown to be exchangeable random variables distributed asGij. Under minimal conditions on the first three moments ofGijfor alliandj, a suitable effective population sizeNeis derived. Then if time is recorded in a backward direction in units of 2Negenerations, it can be shown that coalescent theory holds.

P Systems Computing the Period of Irreducible Markov Chains

<p>It is well known that any irreducible and aperiodic Markov chain has exactly one stationary distribution, and for any arbitrary initial distribution, the se- quence of distributions at time n converges to the stationary distribution, that is, the Markov chain is approaching equilibrium as n→∞.<br /> In this paper, a characterization of the aperiodicity in existential terms of some state is given. At the same time, a P system with external output is associated with any irre- ducible Markov chain. The designed system provides the aperiodicity of that Markov chain and spends a polynomial amount of resources with respect to the size of the in- put. A comparative analysis with respect to another known solution is described.</p>

Classical Capacity of Quantum Channels with General Markovian Correlated Noise

The classical capacity of a quantum channel with arbitrary Markovian correlated noise is evaluated. For the general case of a channel with long-term memory, which corresponds to a Markov chain which does not converge to equilibrium, the capacity is expressed in terms of the communicating classes of the Markov chain. For an irreducible and aperiodic Markov chain, the channel is forgetful, and one retrieves the known expression for the capacity.

System identification: Regime switching, unmodeled dynamics, and binary sensors

This paper is concerned with persistent system identification for plants that are equipped with binary sensors whose unknown parameter is a random process represented by a Markov chain. We treat two classes of problems. In the first class, the parameter is a stochastic process modeled by an irreducible and aperiodic Markov chain with transition rates sufficiently faster than adaptation rates of identification algorithms. In this case, an averaged behavior of the parameter process can be derived from the stationary measure of the Markov chain and can be estimated with empirical measures. Upper and lower error bounds are established that explicitly show impact of unmodeled dynamics. In the second class of problems, the state switches values infrequently. A moving-window maximum a posterior (MAP) algorithm is introduced for tracking the time-varying parameters. Numerical results are presented to illustrate the tracking performance of the MAP algorithm and compare it with the widely used Viterbi algorithm.

Identification of Systems With Regime Switching and Unmodeled Dynamics

This paper is concerned with persistent identification of systems that involve deterministic unmodeled dynamics and stochastic observation disturbances, and whose unknown parameters switch values (possibly large jumps) that can be represented by a Markov chain. Two classes of problems are considered. In the first class, the switching parameters are stochastic processes modeled by irreducible and aperiodic Markov chains with transition rates sufficiently faster than adaptation rates of the identification algorithms. In this case, tracking real-time parameters by output observations becomes impossible and we show that an averaged behavior of the parameter process can be derived from the stationary measure of the Markov chain and can be estimated with periodic inputs and least-squares type algorithms. Upper and lower error bounds are established that explicitly show impact of unmodeled dynamics. In contrast, the second class of problems represents systems whose state transitions occur infrequently. An adaptive algorithm with variable step sizes is introduced for tracking the time-varying parameters. Convergence and error bounds are derived. Numerical results are presented to illustrate the performance of the algorithm.

The expected hitting times for finite Markov chains

In this paper, using the theory of matrix algebra, we obtain a new expression for the expected hitting times of irreducible aperiodic Markov chains. Then, using it, we calculate the expected hitting times of random walks on several kinds of graphs. These examples show that in many cases our approach is better than the others.

On the relation between reversibility and monotonicity of fluctuation spectra for discrete time finite state Markov chains

For the irreducible aperiodic discrete time finite state Markov chain, (i) if it is reversible, we provide a necessary and sufficient condition for all fluctuation spectra to be monotonic on [ 0 , π ] and (ii) if it is irreversible, then there exist some nonmonotonic fluctuation spectra.

Long-Range Dependence of Markov Chains in Discrete Time on Countable State Space

When {Xn} is an irreducible, stationary, aperiodic Markov chain on the countable state space X = {i, j,…}, the study of long-range dependence of any square integrable functional {Yn} := {yXn} of the chain, for any real-valued function {yi: i ∈ X}, involves in an essential manner the functions Qijn = ∑r=1n(pijr − πj), where pijr = P{Xr = j | X0 = i} is the r-step transition probability for the chain and {πi: i ∈ X} = P{Xn = i} is the stationary distribution for {Xn}. The simplest functional arises when Yn is the indicator sequence for visits to some particular state i, Ini = I{Xn=i} say, in which case limsupn→∞n−1var(Y1 + ∙ ∙ ∙ + Yn) = limsupn→∞n−1 var(Ni(0, n]) = ∞ if and only if the generic return time random variable Tii for the chain to return to state i starting from i has infinite second moment (here, Ni(0, n] denotes the number of visits of Xr to state i in the time epochs {1,…,n}). This condition is equivalent to Qjin → ∞ for one (and then every) state j, or to E(Tjj2) = ∞ for one (and then every) state j, and when it holds, (Qijn / πj) / (Qkkn / πk) → 1 for n → ∞ for any triplet of states i, jk.