Markov Chain Problems Research Articles

Inexact uniformization and GMRES methods for large Markov chains

SUMMARYInexact algorithms allow certain operations (typically matrix–vector products or certain function evaluations) to be performed inexactly, either out of necessity or deliberately, in view of trading accuracy for speed. We review recent findings that show the impact of the inexact approach in the uniformization and generalized minimal residual (GMRES) methods when computing the transient, stationary, and cumulative solutions of realistic Markov chain problems of large size arising from computer systems and biochemical reactions. Copyright © 2011 John Wiley & Sons, Ltd.

First Hitting Problems for Markov Chains That Converge to a Geometric Brownian Motion

We consider a discrete-time Markov chain with state space {1,1+Δx,…,1+kΔx=N}. We compute explicitly the probability pj that the chain, starting from 1+jΔx, will hit N before 1, as well as the expected number dj of transitions needed to end the game. In the limit when Δx and the time Δt between the transitions decrease to zero appropriately, the Markov chain tends to a geometric Brownian motion. We show that pj and djΔt tend to the corresponding quantities for the geometric Brownian motion.

Fastest mixing Markov chain problem for the union of two cliques

A symmetric, random walk on a graph G can be defined by prescribing weights to the edges in such a way that for each vertex the sum of the weights of the edges incident to the vertex is at most one. The fastest mixing, Markov chain (FMMC) problem for G is to determine the weighting that yields the fastest mixing random walk. We solve the FMMC problem in the case that G is the union of two complete graphs.

A new approach to the solution of optimal stopping problem in a discrete time

An optimal stopping problem of Markov chain with infinite horizon is considered. For the case of finite number m of states, Sonin proposed an algorithm, which allows to find the value function and the stopping set in not more than steps. The algorithm is based on a modification of Markov chain on each step, related with the elimination of the states which certainly belong to the continuation set. To solve the problem with arbitrary state space and to have a possibility of generalization to the continuous time, one needs to modify the procedure. We propose a procedure which is based on a sequential modification of the pay-off function for the same chain in such a way, that the value function is the same for both problems and the modified pay-off function is greater than the initial one on some set and is equal to it on the complement. We show the efficiency of this procedure.

On the Imbedding Problem for Three-State Time Homogeneous Markov Chains with Coinciding Negative Eigenvalues

For an indecomposable 3×3 stochastic matrix (i.e., 1-step transition probability matrix) with coinciding negative eigenvalues, a new necessary and sufficient condition of the imbedding problem for time homogeneous Markov chains is shown by means of an alternate parameterization of the transition rate matrix (i.e., intensity matrix, infinitesimal generator), which avoids calculating matrix logarithm or matrix square root. In addition, an implicit description of the imbedding problem for the 3×3 stochastic matrix in Johansen (J. Lond. Math. Soc. 8:345–351, 1974) is pointed out.

Asymptotic optimality of the cross-entropy method for Markov chain problems

The correspondence between the cross-entropy method and the zero-variance approximation to simulate a rare event problem in Markov chains is shown. This leads to a sufficient condition that the cross-entropy estimator is asymptotically optimal.

Towards Improved Assessment of Functional Similarity in Large-Scale Screens: A Study on Indel Length

Although insertions and deletions are a common type of evolutionary sequence variation, their origins and their functional consequences have not been comprehensively understood. Most alignment algorithms/programs only roughly reflect the evolutionary processes that result in gaps--which typically require further evaluation. Interestingly, it is widely believed that gaps are the predominant form of sequence variation resulting in structural and functional changes. Thus it is desirable to distinguish between gaps that reflect true point mutations and alignment artifacts when it comes to assessing the functional similarity of proteins based on computational alignments. Here we introduce pair hidden Markov model-based solutions to rapidly assess the statistical significance of gaps in alignments resulting from classical Needleman-Wunsch-like alignment procedures which implement affine gap penalty scoring schemes. Surprisingly, although it has a natural formulation, the emanating Markov chain problem had no known efficient solution thus far. In this article, we present the first efficient algorithm to solve it. We demonstrate that, when comparing paralogous protein pairs (from Escherichia coli) of equal alignment identity and similarity, alignments that contain gaps of significant length are significantly less similar in terms of functionality, as measured with respect to Gene Ontology (GO) term similarity. This demonstrates for the first time, in a formally sound manner, that insertions and deletions cause more severe functional changes between proteins than substitutions. Our method can be reliably employed to quickly filter alignment outputs for protein pairs that are more likely to be functionally similar and/or divergent and establishes a sound and useful add-on for large-scale alignment studies.

Number of hidden states and memory: a joint order estimation problem for Markov chains with Markov regime

This paper deals with order identification for Markov chains with Markov regime (MCMR) in the context of finite alphabets. We define the joint order of a MCMR process in terms of the number k of states of the hidden Markov chain and the memory m of the conditional Markov chain. We study the properties of penalized maximum likelihood estimators for the unknown order (k, m) of an observed MCMR process, relying on information theoretic arguments. The novelty of our work relies in the joint estimation of two structural parameters. Furthermore, the different models in competition are not nested. In an asymptotic framework, we prove that a penalized maximum likelihood estimator is strongly consistent without prior bounds on k and m . We complement our theoretical work with a simulation study of its behaviour. We also study numerically the behaviour of the BIC criterion. A theoretical proof of its consistency seems to us presently out of reach for MCMR, as such a result does not yet exist in the simpler case where m = 0 (that is for hidden Markov models).

Hitting Time and Inverse Problems for Markov Chains

Let W n be a simple Markov chain on the integers. Suppose that X n is a simple Markov chain on the integers whose transition probabilities coincide with those of W n off a finite set. We prove that there is an M > 0 such that the Markov chain W n and the joint distributions of the first hitting time and first hitting place of X n started at the origin for the sets {-M, M} and {-(M + 1), (M + 1)} algorithmically determine the transition probabilities of X n .

Hitting Time and Inverse Problems for Markov Chains

Let Wn be a simple Markov chain on the integers. Suppose that Xn is a simple Markov chain on the integers whose transition probabilities coincide with those of Wn off a finite set. We prove that there is an M > 0 such that the Markov chain Wn and the joint distributions of the first hitting time and first hitting place of Xn started at the origin for the sets {-M, M} and {-(M + 1), (M + 1)} algorithmically determine the transition probabilities of Xn.

Decisive Markov Chains

We consider qualitative and quantitative verification problems for infinite-state Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F if it almost certainly eventually reaches either F or a state from which F can no longer be reached. While all finite Markov chains are trivially decisive (for every set F), this also holds for many classes of infinite Markov chains. Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F. In particular, this holds for probabilistic lossy channel systems (PLCS). Furthermore, all globally coarse Markov chains are decisive. This class includes probabilistic vector addition systems (PVASS) and probabilistic noisy Turing machines (PNTM). We consider both safety and liveness problems for decisive Markov chains, i.e., the probabilities that a given set of states F is eventually reached or reached infinitely often, respectively. 1. We express the qualitative problems in abstract terms for decisive Markov chains, and show an almost complete picture of its decidability for PLCS, PVASS and PNTM. 2. We also show that the path enumeration algorithm of Iyer and Narasimha terminates for decisive Markov chains and can thus be used to solve the approximate quantitative safety problem. A modified variant of this algorithm solves the approximate quantitative liveness problem. 3. Finally, we show that the exact probability of (repeatedly) reaching F cannot be effectively expressed (in a uniform way) in Tarski-algebra for either PLCS, PVASS or (P)NTM.

On Filtering of Markov Chains in Strong Noise

The filtering problem for finite-state Markov chains is revisited in the low signal-to-noise regime. We give a description of conditional measure concentration around the invariant distribution of the signal and derive asymptotic expressions for the performance indices of the minimum mean square error (MMSE) and minimum a posteriori probability (MAP) filtering estimates.

Inequalities for the spectra of symmetric doubly stochastic matrices

We prove majorization relations for symmetric doubly stochastic matrices whose nonzero off-diagonal entries lie in an edge transitive graph. These results are then used to solve the fastest mixing Markov chain problem on edge transitive graphs.

Trace Machines for Observing Continuous-Time Markov Chains

In this paper, we study several linear-time equivalences (Markovian trace equivalence, failure and ready trace equivalence) for continuous-time Markov chains that refer to the probabilities for timed execution paths. Our focus is on testing scenarios by means of push-button experiments with appropriate trace machines and a discussion of the connections between the equivalences. For Markovian trace equivalence, we provide alternative characterizations, including one that abstracts away from the time instances where actions are observed, but just reports on the average sojourn times in the states. This result is used for a reduction of the question whether two finite-state continuous-time Markov chains are Markovian trace equivalent to the probabilistic trace equivalence problem for discrete-time Markov chains (and the latter is known to be solvable in polynomial time).

Statistical Inference for Discretely Observed Markov Jump Processes

SummaryLikelihood inference for discretely observed Markov jump processes with finite state space is investigated. The existence and uniqueness of the maximum likelihood estimator of the intensity matrix are investigated. This topic is closely related to the imbedding problem for Markov chains. It is demonstrated that the maximum likelihood estimator can be found either by the EM algorithm or by a Markov chain Monte Carlo procedure. When the maximum likelihood estimator does not exist, an estimator can be obtained by using a penalized likelihood function or by the Markov chain Monte Carlo procedure with a suitable prior. The methodology and its implementation are illustrated by examples and simulation studies.

Fastest Mixing Markov Chain on a Graph

We consider a symmetric random walk on a connected graph, where each edge is labeled with the probability of transition between the two adjacent vertices. The associated Markov chain has a uniform equilibrium distribution; the rate of convergence to this distribution, i.e., the mixing rate of the Markov chain, is determined by the second largest eigenvalue modulus (SLEM) of the transition probability matrix. In this paper we address the problem of assigning probabilities to the edges of the graph in such a way as to minimize the SLEM, i.e., the problem of finding the fastest mixing Markov chain on the graph. We show that this problem can be formulated as a convex optimization problem, which can in turn be expressed as a semidefinite program (SDP). This allows us to easily compute the (globally) fastest mixing Markov chain for any graph with a modest number of edges (say, $1000$) using standard numerical methods for SDPs. Larger problems can be solved by exploiting various types of symmetry and structure in the problem, and far larger problems (say, 100,000 edges) can be solved using a subgradient method we describe. We compare the fastest mixing Markov chain to those obtained using two commonly used heuristics: the maximum-degree method, and the Metropolis--Hastings algorithm. For many of the examples considered, the fastest mixing Markov chain is substantially faster than those obtained using these heuristic methods. We derive the Lagrange dual of the fastest mixing Markov chain problem, which gives a sophisticated method for obtaining (arbitrarily good) bounds on the optimal mixing rate, as well as the optimality conditions. Finally, we describe various extensions of the method, including a solution of the problem of finding the fastest mixing reversible Markov chain, on a fixed graph, with a given equilibrium distribution.

On a quadratic matrix equation associated with an M-matrix

We study the quadratic matrix equation X 2 - EX - F = 0, where E is diagonal and F is an M-matrix. Quadratic matrix equations of this type arise in noisy Wiener-Hopf problems for Markov chains. The solution of practical interest is a particular M-matrix solution. The existence and uniqueness of M-matrix solutions and numerical methods for finding the desired M-matrix solution are discussed by transforming the equation into an equation that belongs to a special class of non-symmetric algebraic Riccati equations (AREs). We also discuss the general non-symmetric ARE and describe how we can use the Schur method to find the stabilizing or almost stabilizing solution if it exists. The desired M-matrix solution of the quadratic matrix equation (a special non-symmetric ARE by itself) turns out to be the unique stabilizing or almost stabilizing solution.

Rate of convergence for parametric estimation in a stochastic volatility model

We consider the following hidden Markov chain problem: estimate the finite-dimensional parameter θ in the equation v t=v 0+ ∫ 0 t σ(θ, v s) dW s+ drift, when we observe discrete data X i/ n at times i=0,…, n from the diffusion X t=x 0+ ∫ 0 t v s dB s+ drift . The processes ( W t ) t∈[0,1] and ( B t ) t∈[0,1] are two independent Brownian motions; asymptotics are taken as n→∞. This stochastic volatility model has been paid some attention lately, especially in financial mathematics. We prove in this note that the unusual rate n −1/4 is a lower bound for estimating θ. This rate is indeed optimal, since Gloter (CR Acad. Sci. Paris, t330, Série I, pp. 243–248), exhibited n −1/4 consistent estimators. This result shows in particular the significant difference between “high frequency data” and the ergodic framework in stochastic volatility models (compare Genon-Catalot, Jeantheau and Laredo (Bernoulli 4 (1998) 283; Bernoulli 5 (2000) 855; Bernoulli 6 (2000) 1051 and also S ørensen (Prediction-based estimating functions. Technical report, Department of Theoretical Statistics, University of Copenhagen, 1998)).

Optimal switching problem for countable Markov chains: average reward criterion

Optimal switching we consider is the following generalization of optimal stopping: (i) there are a reward function and a cost function on the state space of a Markov chain; (ii) a controller selects stopping times sequentially; (iii) at those times the controller receives rewards and pays costs in an alternating order. In this paper we treat the case of a positive recurrent countable Markov chain and the average per unit time criterion. We find an optimal strategy and the maximal average gain in terms of the solution of a variational problem with two obstacles, known also in connection with Dynkin games.

Approximation of multilevel Toeplitz matrices via multilevel trigonometric matrix spaces and applications to preconditioning

In a previous paper [34] we discussed the approximation of multilevel Toeplitz matrices generated by multivariate rectangular matrix-valued continuous functions \(\), with I=[−π,π], by means of multilevel trigonometric matrix spaces with unstructured s×t blocks and by a (multi) sequence of linear approximation operators \(\). Here we prove some theorems about strong and weak clustering around the unity of the eigenvalues/singular values of \(\) and of other preconditioned matrices based on linear approximation operators. These results represent a very uniform tool for dealing with the preconditioning problem for large dimensions, for a variety of situations (e.g., control theory, Markov chain problems), both in the Hermitian and non-Hermitian/non-square case, by devising preconditioners in any multilevel trigonometric linear space of matrices.