Original Markov Chain Research Articles

Wilson’s $6-j$ laws and stitched Markov processes

Построены новые примеры квадратичных харнессов (quadratic harnesses) с максимальным параметром $\gamma$. Конструкция основана на квадратичных функциях цепей Маркова, которые возникают в последствие временной параметризации $6-j$ законов Вильсона. Конструкция состоит из сшивания условно независимых копий таких процессов Маркова.

Lumpings of Markov Chains, Entropy Rate Preservation, and Higher-Order Lumpability

A lumping of a Markov chain is a coordinatewise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is strongly k-lumpable, if and only if the lumped process is a kth-order Markov chain for each starting distribution of the original Markov chain. We characterise strong k-lumpability via tightness of stationary entropic bounds. In the sparse setting, we give sufficient conditions on the lumping to both preserve the entropy rate and be strongly k-lumpable.

Lumpings of Markov Chains, Entropy Rate Preservation, and Higher-Order Lumpability

A lumping of a Markov chain is a coordinatewise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is stronglyk-lumpable, if and only if the lumped process is akth-order Markov chain for each starting distribution of the original Markov chain. We characterise strongk-lumpability via tightness of stationary entropic bounds. In the sparse setting, we give sufficient conditions on the lumping to both preserve the entropy rate and be stronglyk-lumpable.

Optimal Kullback–Leibler approximation of Markov chains via nuclear norm regularisation

This paper is concerned with model reduction for Markov chain models. The goal is to obtain a low-rank approximationto the original Markov chain. The Kullback–Leibler divergence rate is used to measure the similarity between two Markov chains; the nuclear norm is used to approximate the rank function. A nuclear-norm regularised optimisation problem is formulated to approximately find the optimal low-rank approximation. The proposed regularised problem is analysed and performance bounds are obtained through the convex analysis. An iterative fixed point algorithm is developed based on the proximal splitting technique to compute the optimal solutions. The effectiveness of this approach is illustrated via numerical examples.

The Entropy of Conditional Markov Trajectories

To quantify the randomness of Markov trajectories with fixed initial and final states, Ekroot and Cover proposed a closed-form expression for the entropy of trajectories of an irreducible finite state Markov chain. Numerous applications, including the study of random walks on graphs, require the computation of the entropy of Markov trajectories conditioned on a set of intermediate states. However, the expression of Ekroot and Cover does not allow for computing this quantity. In this paper, we propose a method to compute the entropy of conditional Markov trajectories through a transformation of the original Markov chain into a Markov chain that exhibits the desired conditional distribution of trajectories. Moreover, we express the entropy of Markov trajectories - a global quantity - as a linear combination of local entropies associated with the Markov chain states.

Markov Chain Simulation with Fewer Random Samples

We propose an accelerated CTMC simulation method that is exact in the sense that it produces all of the transitions involved. We call our method Trajectory Sampling Simulation as it samples from the distribution of state sequences and the distribution of time given some particular sequence. Sampling from the trajectory space rather than the transition space means that we need to generate fewer random numbers, which is an operation that is typically computationally expensive. Sampling from the time distribution involves approximating the exponential distributions that govern the sojourn times with a geometric distribution. A proper selection for the approximation parameters can ensure that the stochastic process simulated is almost identical to the simulation of the original Markov chain. Our approach does not depend on the properties of the system and it can be used as an alternative to more efficient approaches when those are not applicable.

Computing the Stationary Distribution of a Finite Markov Chain Through Stochastic Factorization

This work presents an approach for reducing the number of arithmetic operations involved in the computation of a stationary distribution for a finite Markov chain. The proposed method relies on a particular decomposition of a transition-probability matrix called stochastic factorization. The idea is simple: when a transition matrix is represented as the product of two stochastic matrices, one can swap the factors of the multiplication to obtain another transition matrix, potentially much smaller than the original. We show in the paper that the stationary distributions of both Markov chains are related through a linear transformation, which opens up the possibility of using the smaller chain to compute the stationary distribution of the original model. In order to support the application of stochastic factorization, we prove that the model derived from it retains all the properties of the original chain which are relevant to the stationary distribution computation. Specifically, we show that (i) for each recurrent class in the original Markov chain there is a corresponding class in the derived model with the same period and, given some simple assumptions about the factorization, (ii) the original chain is irreducible if and only if the derived chain is irreducible and (iii) the original chain is regular if and only if the derived chain is regular. The paper also addresses some issues associated with the application of the proposed approach in practice and briefly discusses how stochastic factorization can be used to reduce the number of operations needed to compute the fundamental matrix of an absorbing Markov chain.

Exact Statistics of a Complex Markov Chain through State Reduction: A Satellite On-board Switching Example

Satellite on-board switching offers the possibility of covering a wide area with increased total capacity. In order to show a mechanism for reducing a highly dimensional complex Markov chain, in this paper we use a terrestrial packet grouping approach to minimize the on-board switching operations. By carefully redirecting some transitions to different states, we can evaluate exact statistics of the original complex Markov chain from the analysis of a much simpler reduced Markov chain. Although we use a communication example close to our research interest, the method can have a much wider area of application.

The valuation of contingent claims using alternative numerical methods

This study compares the computational accuracy and efficiency of three numerical methods for the valuation of contingent claims written on multiple underlying assets; these are the trinomial tree, original Markov chain and Sobol–Markov chain approaches. The major findings of this study are: (i) the original Duan and Simonato (2001) Markov chain model provides more rapid convergence than the trinomial tree method, particularly in cases where the time to maturity period is less than nine months; (ii) when pricing options with longer maturity periods or with multiple underlying assets, the Sobol–Markov chain model can solve the problem of slow convergence encountered under the original Duan and Simonato (2001) Markov chain method; and (iii) since conditional density is used, as opposed to conditional probability, we can easily extend the Sobol–Markov chain model to the pricing of derivatives which are dependent on more than two underlying assets without dealing with high-dimensional integrals. We also use ‘executive stock options’ (ESOs) as an example to demonstrate that the Sobol–Markov chain method can easily be applied to the valuation of such ESOs.

A scaling analysis of a cat and mouse Markov chain

If $(C_n)$ is a Markov chain on a discrete state space ${\mathcal{S}}$, a Markov chain $(C_n,M_n)$ on the product space ${\mathcal{S}}\times{\mathcal{S}}$, the cat and mouse Markov chain, is constructed. The first coordinate of this Markov chain behaves like the original Markov chain and the second component changes only when both coordinates are equal. The asymptotic properties of this Markov chain are investigated. A representation of its invariant measure is, in particular, obtained. When the state space is infinite it is shown that this Markov chain is in fact null recurrent if the initial Markov chain $(C_n)$ is positive recurrent and reversible. In this context, the scaling properties of the location of the second component, the mouse, are investigated in various situations: simple random walks in ${\mathbb{Z}}$ and ${\mathbb{Z}}^2$ reflected a simple random walk in ${\mathbb{N}}$ and also in a continuous time setting. For several of these processes, a time scaling with rapid growth gives an interesting asymptotic behavior related to limiting results for occupation times and rare events of Markov processes.

Approximate transient analysis of large stochastic models with WinPEPSY-QNS

In this paper we present a new algorithm for the approximate transient analysis of large stochastic models. The new algorithm is based on the self-correcting analysis principle for continuous-time Markov chains (CTMC). The approach uses different time dependent aggregations of the CTMC of a stochastic model. With the method of uniformization the transient state probabilities of each aggregated CTMC for a time step are calculated. The derived probabilities are used for the construction of stronger aggregations, which are applied for the correction of the transition rates of the previous aggregations. This is done step by step, until the final time is reached. High aggregations of the original continuous-time Markov chain lead to a time and space efficient computational effort. Therefore the approximate transient analysis method based on the self-correcting aggregation can be used for models with large state spaces. For queuing networks with phase-type distributions of the service times this newly developed algorithm is implemented in WinPEPSY-QNS, a tool for performance evaluation and prediction of stochastic models based on queuing networks. It consists of a graphical editor for the construction of queuing networks and an easy-to-use evaluation component, which offers suitable analysis methods. The newly implemented algorithm is used for the analysis of several examples, and the results are compared to the results of simulation runs where exact values cannot be achieved.

Bounding the Blocking Probabilities in Multirate CDMA Networks Supporting Elastic Services

Several previous contributions have proposed calculation methods that can be used to determine the steady state (and from it the blocking probabilities) of code-division multiple-access (CDMA) systems. This present work extends the classical Kaufman-Roberts formula such that it becomes applicable in CDMA systems in which elastic services with state-dependent instantaneous bit rate and average-bit-rate-dependent residency time are supported. Our model captures the effect of soft blocking, that is, an arriving session may be blocked in virtually all system states but with a state dependent probability. The core of this method is to approximate the original irreversible Markov chain with a reversible one and to give lower and upper bounds on the so-called partially blocking macro states of the state space. We employ this extended formula to establish lower and upper bounds on the steady state and the classwise blocking probabilities.

Disjoint Decomposition of Markov Chains and Sampling Circuits in Cayley Graphs

Markov chain decomposition is a tool for analysing the convergence rate of a complicated Markov chain by studying its behaviour on smaller, more manageable pieces of the state space. Roughly speaking, if a Markov chain converges quickly to equilibrium when restricted to subsets of the state space, and if there is sufficient ergodic flow between the pieces, then the original Markov chain also must converge rapidly to equilibrium. We present a new version of the decomposition theorem where the pieces partition the state space, rather than forming a cover where pieces overlap, as was previously required. This new formulation is more natural and better suited to many applications. We apply this disjoint decomposition method to demonstrate the efficiency of simple Markov chains designed to uniformly sample circuits of a given length on certain Cayley graphs. The proofs further indicate that a Markov chain for sampling adsorbing staircase walks, a problem arising in statistical physics, is also rapidly mixing.

Using a Markov Chain to Construct a Tractable Approximation of an Intractable Probability Distribution

Abstract. Let π denote an intractable probability distribution that we would like to explore. Suppose that we have a positive recurrent, irreducible Markov chain that satisfies a minorization condition and has π as its invariant measure. We provide a method of using simulations from the Markov chain to construct a statistical estimate of π from which it is straightforward to sample. We show that this estimate is ‘strongly consistent’ in the sense that the total variation distance between the estimate and π converges to 0 almost surely as the number of simulations grows. Moreover, we use some recently developed asymptotic results to provide guidance as to how much simulation is necessary. Draws from the estimate can be used to approximate features of π or as intelligent starting values for the original Markov chain. We illustrate our methods with two examples.

Markov property for a function of a Markov chain: A linear algebra approach

In this paper, we address whether a (probabilistic) function of a finite homogeneous Markov chain still enjoys a Markov-type property. We propose a complete answer to this question using a linear algebra approach. At the core of our approach is the concept of invariance of a set under a matrix. In that sense, the framework of this paper is related to the so-called “geometric approach” in control theory for linear dynamical systems. This allows us to derive a collection of new results under generic assumptions on the original Markov chain. In particular, we obtain a new criterion for a function of a Markov chain to be a homogeneous Markov chain. We provide a deterministic polynomial-time algorithm for checking this criterion. Moreover, a non-standard notion of observability for a linear system will be used. This allows one to show that the set of all stochastic matrices for which our criterion holds, is nowhere dense in the set of stochastic matrices.

Stability estimates in the problem of average optimal switching of a Markov chain

We consider a switching model for a Markov chain xt with a transition probability p(x|B). The goal of a controller is to maximize the average gain by selecting a sequence of stopping times, in which the controller gets rewards and pays costs (depending on xt) in an alternating order. We suppose that the exact transition probability function of the original “real” chain xt is not available to the controller, and he/she is forced to rely on a given approximation \(\widetildep\) to the unknown p. The controller finds a switching policy \(\widetilde\pi\) optimal for the Markov chain with the transition probability \(\widetildep\), with a view to apply \(\widetilde\pi\) to the original Markov chain xt. Under certain restrictions on p we give an upper bound for the difference between the maximal gain attainable in switching of xt, and the gain made under the policy \(\) in the original model. The bound is expressed in terms of the total variation distance \({{\sup } \over x}{\kern 1pt} {\kern 1pt} {\kern 1pt} Var\left( {p\left( {x\left| \cdot \right.} \right),\widetildep\left( {x\left| \cdot \right.} \right)} \right)\).

Accelerating Markovian analysis of asynchronous systems using state compression

This paper presents a methodology to speed up the stationary analysis of large Markov chains that model asynchronous systems. Instead of directly working on the original Markov chain, we propose to analyze a smaller Markov chain obtained via a novel technique called state compression. Once the smaller chain is solved, the solution to the original chain is obtained via a process called expansion. The method is especially powerful when the Markov chain has a small feedback vertex set, which happens often in asynchronous systems that contain mostly bounded-delay components. Our experimental results show that the method can yield reductions of more than an order of magnitude in CPU time and facilitate the analysis of larger systems than possible using traditional techniques.

LUMPABILITY IN LRU STACK WITH MARKOVIAN PAGE REFERENCES

The dynamlic behavior of programs is studied through LRU (Least-Recently-Used) stack distribution assuming Markovian page references. We propose a new analysis method to calculate steady state probabilities for the LRU stack. We first prove that a Markov chain which describes the behavior of an LRU stack of full-length is lumpable to a Markov chain having a smaller transition matrix than that in the original Markov chain. Based on this lumpability, we propose an algorithm to calculate the invariant vector for the transition matrix of the Markov chain for a full-length LRU stack. We show numerical examples calculated with the proposed algorithm, and evaluate computational efficiency of the proposed method.

Sensitivity analysis of stationary performance measures for Markov chains

Derivative estimation is an important problem in performance analysis of discrete event dynamic systems. Derivative estimation of stationary performance measures is difficult since it generally requires the consistency of estimators. This paper proposes an algorithm for derivative estimation of stationary performance measures for Markov chains. Both discrete-time and continuous-time chains are considered. The basic idea is to simulate the original Markov chain with a modified performance measure which can be estimated by extra simulations. The computational load of the extra simulations at each step is bounded. It is shown under mild assumptions that the algorithm attains the best possible rate of convergence as the simulation time goes to infinity. An unexpected connection between the algorithm and solutions to Poisson equations is also revealed.

On the use of periodicity properties for the efficient numerical solution of certain Markov chains

AbstractMarkov chains with periodic graphs arise frequently in a wide range of modelling experiments. Application areas range from flexible manufacturing systems in which pallets are treated in a cyclic manner to computer communication networks. The primary goal of this paper is to show how advantage may be taken of this property to reduce the amount of computer memory and computation time needed to compute stationary probability vectors of periodic Markov chains. After reviewing some basic properties of Markov chains whose associated graph is periodic, we introduce a ‘reduced scheme’ in which only a subset of the probability vector need be computed. We consider the effect of the application of both direct and iterative methods to the original Markov chain (permuted to normal periodic form) as well as to the reduced system. We show how the periodicity of the Markov chain may be efficiently computed as the chain itself is being generated. Finally, some numerical experiments to illustrate the theory, are presented.