Reducible Markov Chains Research Articles

Metastability of exponentially perturbed Markov chains

A family of irreducible Markov chains on a finite state space is considered as an exponential perturbation of a reducible Markov chain. This is a generalization of the Freidlin-Wentzell theory, motivated by studies of stochastic Ising models, neural network and simulated annealing. It is shown that the metastability is a universal feature for this wide class of Markov chains. The metastable states are simply those recurrent states of the reducible Markov chain. Higher level attractors, related attractive basins and their pyramidal structure are analysed. The logarithmic asymptotics of the hitting time of various sets are estimated. The hitting time over its mean converges in law to the unit exponential distribution.

Hierarchical Markovian models: symmetries and reduction

Hierarchical Markovian models are a useful paradigm for the specification and quantitative analysis of models arising from complex systems. Although techniques for a very efficient analysis of large scale hierarchical Markovian models have been developed recently, the size of the Markov chain underlying a complex hierarchical model often prohibits an analysis on contemporary computer equipment. However, many realistic models contain a lot of symmetric and identical parts, allowing the construction of a reduced Markov chain yielding exact results for the complete model. Of course, to make use of symmetries in a fairly complex model, a technique is needed that generates automatically a reduced Markov chain from the specification of the model. Such an approach can be integrated in an appropriate modelling tool environment for the analysis of hierarchical models and often yields a dramatic reduction in the state space size allowing the analysis of models that are far too large to be solved by standard means.

An Introduction to Stochastic Modeling

Stochastic Modeling. Probability Review. The Major Discrete Distributions. @rtant Continuous Distributions. Some Elementary Exercises. Useful Functions, Integrals, and Sums. Conditional Probability and Conditional Expectation: The Discrete Case. The Dice Game Craps. Random Sums. Conditioning on a Continuous Random Variable. Markov Chains: Introduction: Definitions. Transition Probability Matrices of a Markov Chain. Some Markov Chain Models. First Step Analysis. Some Special Markov Chains. Functionals of Random Walks and Success Runs. Another Look at First Step Analysis. The Long Run Behavior of Markov Chains: Regular Transition Probability Matrices. Examples. The Classification of States. The Basic Limit Theorem of Markov Chains. Reducible Markov Chains. Sequential Decisions and Markov Chains. Poisson Processes: The Poisson Distribution and the Poisson Processes. The Law of Rare Events. Distributions Associated with the Poisson Process. The Uniform Distribution and Poisson Processes. Spatial Poisson Processes. Compound and Marked Poisson Processes. Continuous Time Markov Chains: Pure Birth Processes. Ptire Death Processes. Birth and Death Processes. The Limiting Behavior of Birth and Death Processes. Birth and Death Processes with Absorbing States. Finite State Continuous Time Markov Chains. Set Valued Processes. Renewal Phenomena: Definition of a Renewal Process and Related Concepts. Some Examples of Renewal Processes. The Poisson Process Viewed as a Renewal Process. The Asymptotic ]3ehavior as Renewal Process.

Computing absorption probabilities for a Markov chain

We present a new matrix construction algorithm for computing absorption probabilities for a finite, reducible Markov chain. The construction algorithm contains two steps: matrix augmentation and matrix reduction. The algorithm requires more memory and less execution time than the calculation of absorption probabilities by the LU decomposition. We apply the algorithm to a Markovian model of a production line.

Reduced Markov chains at stochastic spin models

Abstract From Glauber's stochastic spin model in discrete time, reduced Markov-chain models are constructed. The transition matrices of the reduced models utilize equilibrium correlation functions of the full N-spin system; however, the reduced models involve the time-dependent behavior of only a cluster of spins. The reduced models have as an invariant vector the exact marginal equilibrium probability for the spins in the cluster. In that sense, the reduced models have the same equilibrium as the N-spin Glauber model, but will, in general, display a different time dependence. One of the reduced models is solved exactly here for a one-dimensional lattice, a square lattice, and a simple-cubic lattice.

A Reduction Process for Perturbed Markov Chains

Given a perturbed Markov chain with generator $A( \varepsilon ) = A_0 + \varepsilon A_1 + \varepsilon ^2 A_2 + \cdots $ it is shown that to each time scale $( t,t /\varepsilon , t/\varepsilon ^2 , \cdots )$ is associated a reduced Markov chain. This process of reduction has only finitely many steps. It is given in a recursive way and each reduced chain admits as state space the recurrent classes of the reduced chain defined at the previous stage. It is shown that if $\lambda $ is an eigenvalue (different from zero) of the generator of the kth reduced chain with multiplicity p then $A( \varepsilon )$ admits p eigenvalues $\lambda ( \varepsilon )$ such that $\lambda ( \varepsilon ) = \varepsilon ^k \lambda + o( {\varepsilon ^k } )$. A basis of $\ker A_0 $ (resp. $\ker A_0^ * $) which is robust to the perturbation is constructed.

A note on the inverse problem for reducible Markov chains

Suppose a physical process is modelled by a Markov chain with transition probability on S 1 ∪ S 2, S 1 denoting the transient states and S 2 a set of absorbing states. If v denotes the output distribution on S 2, the question arises as to what input distributions (of raw materials) on S 1 produce v. In this note we give an alternative to the formulation of Ray and Margo [2] and reduce the problem to one system of linear inequalities. An application to random walk is given and the equiprobability case examined in detail.

A note on the inverse problem for reducible Markov chains

Suppose a physical process is modelled by a Markov chain with transition probability on S1 ∪ S2, S1 denoting the transient states and S2 a set of absorbing states.If v denotes the output distribution on S2, the question arises as to what input distributions (of raw materials) on S1 produce v. In this note we give an alternative to the formulation of Ray and Margo [2] and reduce the problem to one system of linear inequalities. An application to random walk is given and the equiprobability case examined in detail.

The inverse problem in reducible Markov chains

The equilibrium probability distribution over the set of absorbing states of a reducible Markov chain is specified a priori and it is required to obtain the constrained sub-space or feasible region for all possible initial probability distributions over the set of transient states. This is called the inverse problem. It is shown that a feasible region exists for the choice of equilibrium distribution. Two different cases are studied: Case I, where the number of transient states exceeds that of the absorbing states and Case II, the converse. The approach is via the use of generalised inverses and numerical examples are given.

The inverse problem in reducible Markov chains

The equilibrium probability distribution over the set of absorbing states of a reducible Markov chain is specified a priori and it is required to obtain the constrained sub-space or feasible region for all possible initial probability distributions over the set of transient states. This is called the inverse problem. It is shown that a feasible region exists for the choice of equilibrium distribution. Two different cases are studied: Case I, where the number of transient states exceeds that of the absorbing states and Case II, the converse. The approach is via the use of generalised inverses and numerical examples are given.