Finite State Space Markov Chain Research Articles

Random walks on the circle and Diophantine approximation.

Random walks on the circle group whose elementary steps are lattice variables with span or taken mod exhibit delicate behavior. In the rational case, we have a random walk on the finite cyclic subgroup , and the central limit theorem and the law of the iterated logarithm follow from classical results on finite state space Markov chains. In this paper, we extend these results to random walks with irrational span , and explicitly describe the transition of these Markov chains from finite to general state space as along the sequence of best rational approximations. We also consider the rate of weak convergence to the stationary distribution in the Kolmogorov metric, and in the rational case observe a phase transition from polynomial to exponential decay after steps. This seems to be a new phenomenon in the theory of random walks on compact groups. In contrast, the rate of weak convergence to the stationary distribution in the total variation metric is purelyexponential.

Virtual Markov Chains

We introduce the space of virtual Markov chains (VMCs) as a projective limit of the spaces of all finite state space Markov chains (MCs), in the same way that the space of virtual permutations is the projective limit of the spaces of all permutations of finite sets.We introduce the notions of virtual initial distribution (VID) and a virtual transition matrix (VTM), and we show that the law of any VMC is uniquely characterized by a pair of a VID and VTM which have to satisfy a certain compatibility condition.Lastly, we study various properties of compact convex sets associated to the theory of VMCs, including that the Birkhoff-von Neumann theorem fails in the virtual setting.

Direct statistical inference for finite Markov jump processes via the matrix exponential

Given noisy, partial observations of a time-homogeneous, finite-statespace Markov chain, conceptually simple, direct statistical inference is available, in theory, via its rate matrix, or infinitesimal generator, {mathsf {Q}}, since exp ({mathsf {Q}}t) is the transition matrix over time t. However, perhaps because of inadequate tools for matrix exponentiation in programming languages commonly used amongst statisticians or a belief that the necessary calculations are prohibitively expensive, statistical inference for continuous-time Markov chains with a large but finite state space is typically conducted via particle MCMC or other relatively complex inference schemes. When, as in many applications {mathsf {Q}} arises from a reaction network, it is usually sparse. We describe variations on known algorithms which allow fast, robust and accurate evaluation of the product of a non-negative vector with the exponential of a large, sparse rate matrix. Our implementation uses relatively recently developed, efficient, linear algebra tools that take advantage of such sparsity. We demonstrate the straightforward statistical application of the key algorithm on a model for the mixing of two alleles in a population and on the Susceptible-Infectious-Removed epidemic model.

Simple SIR models with Markovian control

We consider a random dynamical system, where the deterministic dynamics are driven by a finite-state space Markov chain. We provide a comprehensive introduction to the required mathematical apparatus and then turn to a special focus on the susceptible-infected-recovered epidemiological model with random steering. Through simulations we visualize the behaviour of the system and the effect of the high-frequency limit of the driving Markov chain. We formulate some questions and conjectures of a purely theoretical nature.

Speeding up Markov chains with deterministic jumps

We show that the convergence of finite state space Markov chains to stationarity can often be considerably speeded up by alternating every step of the chain with a deterministic move. Under fairly general conditions, we show that not only do such schemes exist, they are numerous.

Fitting a reversible Markov chain by maximum likelihood: Converting an awkwardly constrained optimization problem to an unconstrained one

We consider here the problem of fitting, by maximum likelihood, a discrete-time, finite-state–space Markov chain that is required to be reversible in time. The technique we use is to impose the detailed balance and other constraints by transforming the transition probabilities to a set of unconstrained ‘working parameters’, after which any general-purpose routine for unconstrained numerical optimization can be used to maximize the likelihood. The main advantages of this procedure are its simplicity and its use of standard, well-tested optimizers; very little computational expertise is needed, and all computations can easily be carried out in R on a standard machine. We provide several examples of applications, to simulated and other data, and give an example of the computation of standard errors and confidence intervals for parameters. We discuss also the variation of the problem in which it is required that the transition probabilities satisfy detailed balance with respect to a pre-specified stationary distribution, demonstrate that this case can be handled by a modification of our methods, but suggest two altogether different routes that may be preferable. A feature of our methods is the availability of excellent starting-values for the optimizations.

Sum of exit times in a series of two metastable states

We consider the problem of non degenerate in energy metastable states forming a series in the framework of reversible finite state space Markov chains. We assume that starting from the state at higher energy the system necessarily visits the second one before reaching the stable state. In this framework, we give a sharp estimate of the exit time from the metastable state at higher energy and, on the proper exponential time scale, we prove an addition rule. As an application of the theory, we study the Blume-Capel model in the zero chemical potential case.

Lipschitz partition processes

We introduce a family of Markov processes on set partitions with a bounded number of blocks, called Lipschitz partition processes. We construct these processes explicitly by a Poisson point process on the space of Lipschitz continuous maps on partitions. By this construction, the Markovian consistency property is readily satisfied; that is, the finite restrictions of any Lipschitz partition process comprise a compatible collection of finite state space Markov chains. We further characterize the class of exchangeable Lipschitz partition processes by a novel set-valued matrix operation.

Synchronization for complex networks with Markov switching via matrix measure approach

This paper devotes to almost sure synchronization and almost sure quasi-synchronization of complex networks with Markov switching. Some sufficient conditions are derived in terms of the ergodic theory of continuous time Markov chain and the matrix measure approach, which can guarantee that the dynamical networks almost surely synchronize or quasi-synchronize to a given manifold. According to the property of Markov chain and the exponential distribution of switching time sequence, we also estimate the probability distribution of the quasi-synchronization error for a two-state Markov chain and then generalize them to a finite state space Markov chain. Meanwhile, some examples with numerical simulations are given to show that the Markov chain plays an important role in synchronization of networks.

Generalized Markov Models for Real-Time Modeling of Continuous Systems

This paper presents a modeling framework based on finite-state space Markov chains (MCs) and fuzzy subsets to represent signals that vary in a continuous range. Our special attention to this extension of finite-state space MC modeling is motivated by numerous opportunities in applying MC models to represent physical variables in automotive and aerospace systems and, subsequently, using these models for fault detection, estimation, prediction, stochastic dynamic programming, and stochastic model predictive control. Our generalized MC modeling framework synergistically combines the notion of transition probabilities with information granulation based on fuzzy partitioning. As compared with the case of more familiar interval partitioning, the transition probabilities in our model are defined for transitions between fuzzy subsets rather than intervals/rectangular cells. Our framework is first introduced for scalar-valued signals and then extended to vector-valued signals. A real-time capable recursive algorithm for learning transition probabilities from measured signal data is derived. Formulas that characterize the possibility distribution of the next signal value and predict the next signal value are given. It is shown that the introduced modeling framework based on MC models defined over fuzzy partitioning inherits all properties and represents a natural extension of MC models defined over interval partitioning, while providing interpolation ability and improved prediction accuracy. In addition, we derive an alternative formulation of the Chapman-Kolmogorov equation that applies to models in possibilistic/fuzzy environment. Examples are given to illustrate the key notions and results based on modeling of the vehicle speed and road grade signals.

The SIS epidemic model with Markovian switching

Population systems are often subject to environmental noise. Motivated by Takeuchi et al. [7], we will discuss in this paper the effect of telegraph noise on the well-known SIS epidemic model. We establish the explicit solution of the stochastic SIS epidemic model, which is useful in performing computer simulations. We also establish the conditions for extinction and persistence for the stochastic SIS epidemic model and compare these with the corresponding conditions for the deterministic SIS epidemic model. We first prove these results for a two-state Markov chain and then generalise them to a finite state space Markov chain. Computer simulations based on the explicit solution and the Euler–Maruyama scheme are performed to illustrate our theory. We include a more realistic example using appropriate parameter values for the spread of Streptococcuspneumoniae in children.

Nonperiodic Inspection/Replacement Policy for Monotone Deteriorating System with Covariates

This paper discusses the condition-based non-periodic maintenance policy for a stochastic deteriorating system influenced by a dynamic environment which is described by a covariates process. The deterioration is modelled by a stochastic univariate process. The process of covariates is assumed to be a time homogeneous finite state space Markov chain. A model similar to the proportional hazards model is used to represent the influence of the covariates. In the framework of a monotone deteriorating system, we derive the optimal maintenance threshold, optimal inspection sequence to minimise the expected maintenance cost per time unit. An adequate maintenance policy in which the inspection schemes depend both on the level of degradation and on the state of covariates is studied. Comparison of the expected costs per time unit under different conditions of covariates and different maintenance policies is given by numerical results of Monte Carlo simulation.

Particle-method-based formulation of risk-sensitive filter

A novel particle implementation of risk-sensitive filters (RSF) for nonlinear, non-Gaussian state-space models is presented. Though the formulation of RSFs and its properties like robustness in the presence of parametric uncertainties are known for sometime, closed-form expressions for such filters are available only for a very limited class of models including finite state-space Markov chains and linear Gaussian models. The proposed particle filter-based implementations are based on a probabilistic re-interpretation of the RSF recursions. Accuracy of these filtering algorithms can be enhanced by choosing adequate number of random sample points called particles. These algorithms significantly extend the range of practical applications of risk-sensitive techniques and may also be used to benchmark other approximate filters, whose generic limitations are discussed. Appropriate choice of proposal density is suggested. Simulation results demonstrate the performance of the proposed algorithms.

Tail of the stationary solution of the stochastic equation [formula omitted] with Markovian coefficients

In this paper, we deal with the real stochastic difference equation Y n + 1 = a n Y n + b n , n ∈ Z , where the sequence ( a n ) is a finite state space Markov chain. By means of the renewal theory, we give a precise description of the situation where the tail of its stationary solution exhibits power law behavior.

The [formula omitted]-structures of standard and switching-regime GARCH models

This paper analyzes the probabilistic structure of Markov-switching GARCH( p , q ) models, in which the volatility process is driven by a finite state-space Markov chain. We give necessary and sufficient conditions for the existence of moments of any order. We find that the squares and higher order powers of the process have the L 2 structures of ARMA processes, and hence admit ARMA representations. These results are applicable to standard GARCH models and have statistical implications in terms of order identification and parameter estimation.

Tail of the stationary solution of the stochastic equation [formula omitted] with Markovian coefficients

In this Note, we deal with the real stochastic difference equation Y n + 1 = a n Y n + b n , n ∈ Z , where the sequence ( a n ) is a finite state space Markov chain. By means of the renewal theory, we give a precise description of the situation where the tail of its stationary solution exhibits power law behavior. To cite this article: B. de Saporta, C. R. Acad. Sci. Paris, Ser. I 340 (2005).

EXACT RESULTS FOR A FLUID MODEL WITH STATE-DEPENDENT FLOW RATES

We consider a modulated fluid system with a finite state-space Markov chain Jt as modulating process and general state-dependent net input rates. We derive differential equations for the transient and the stationary distribution of (Wt, Jt), where Wt is the content process, and the corresponding Laplace transforms with respect to time. Moreover, we study the level hitting times of Wt. Our results lead to explicit formulas in the case of two modulating states.

A note on the relation between weak derivatives and perturbation realization

Studies the relationship between two important approaches in perturbation analysis (PA)-perturbation realization (PR) and weak derivatives (WDs). Specifically, we study the relation between PR and WDs for estimating the gradient of stationary performance measures of a finite state-space Markov chain. We show that the WDs expression for the gradient of a stationary performance measure can be interpreted as the expected PR factor where the expectation is carried out with respect to a distribution that is given through the weak derivative of the transition kernel of the Markov chain. Moreover, we present unbiased gradient estimators.

A note on the relation between weak derivatives and perturbation realization

Studies the relationship between two important approaches in perturbation analysis (PA)-perturbation realization (PR) and weak derivatives (WDs). Specifically, we study the relation between PR and WDs for estimating the gradient of stationary performance measures of a finite state-space Markov chain. We show that the WDs expression for the gradient of a stationary performance measure can be interpreted as the expected PR factor where the expectation is carried out with respect to a distribution that is given through the weak derivative of the transition kernel of the Markov chain. Moreover, we present unbiased gradient estimators.

Efficiency of finite state space Monte Carlo Markov chains

The class of finite state space Markov chains, stationary with respect to a common pre-specified distribution, is considered. An easy-to-check partial ordering is defined on this class. The ordering provides a sufficient condition for the dominating Markov chain to be more efficient. Efficiency is measured by the asymptotic variance of the estimator of the integral of a specific function with respect to the stationary distribution of the chains. A class of transformations that, when applied to a transition matrix, preserves its stationary distribution and improves its efficiency is defined and studied.