Finite State Space Markov Chain Research Articles

Iterative algorithms for state estimation of jump Markov linear systems

Jump Markov linear systems (JMLSs) are linear systems whose parameters evolve with time according to a finite state Markov chain. Given a set of observations, our aim is to estimate the states of the finite state Markov chain and the continuous (in space) states of the linear system. In this paper, we present original deterministic and stochastic iterative algorithms for optimal state estimation of JMLSs. The first stochastic algorithm yields minimum mean square error (MMSE) estimates of the finite state space Markov chain and of the continuous state of the JMLS. A deterministic and a stochastic algorithm are given to obtain the marginal maximum a posteriori (MMAP) sequence estimate of the finite state Markov chain. Finally, a deterministic and a stochastic algorithm are derived to obtain the MMAP sequence estimate of the continuous state of the JMLS. Computer simulations are carried out to evaluate the performance of the proposed algorithms. The problem of deconvolution of Bernoulli-Gaussian (BG) processes and the problem of tracking a maneuvering target are addressed.

Simulated annealing for maximum a posteriori parameter estimation of hidden Markov models

Hidden Markov models are mixture models in which the populations from one observation to the next are selected according to an unobserved finite state-space Markov chain. Given a realization of the observation process, our aim is to estimate both the parameters of the Markov chain and of the mixture model in a Bayesian framework. We present an original simulated annealing algorithm which, in the same way as the EM (expectation-maximization) algorithm, relies on data augmentation, and is based on stochastic simulation of the hidden Markov chain. This algorithm is shown to converge toward the set of maximum a posteriori (MAP) parameters under suitable regularity conditions.

Perfect simulation of some point processes for the impatient user

Recently Propp and Wilson [14] have proposed an algorithm, called coupling from the past (CFTP), which allows not only an approximate but perfect (i.e. exact) simulation of the stationary distribution of certain finite state space Markov chains. Perfect sampling using CFTP has been successfully extended to the context of point processes by, amongst other authors, Häggström et al. [5]. In [5] Gibbs sampling is applied to a bivariate point process, the penetrable spheres mixture model [19]. However, in general the running time of CFTP in terms of number of transitions is not independent of the state sampled. Thus an impatient user who aborts long runs may introduce a subtle bias, the user impatience bias. Fill [3] introduced an exact sampling algorithm for finite state space Markov chains which, in contrast to CFTP, is unbiased for user impatience. Fill's algorithm is a form of rejection sampling and similarly to CFTP requires sufficient monotonicity properties of the transition kernel used. We show how Fill's version of rejection sampling can be extended to an infinite state space context to produce an exact sample of the penetrable spheres mixture process and related models. Following [5] we use Gibbs sampling and make use of the partial order of the mixture model state space. Thus we construct an algorithm which protects against bias caused by user impatience and which delivers samples not only of the mixture model but also of the attractive area-interaction and the continuum random-cluster process.

Perfect simulation of some point processes for the impatient user

Recently Propp and Wilson [14] have proposed an algorithm, called coupling from the past (CFTP), which allows not only an approximate but perfect (i.e. exact) simulation of the stationary distribution of certain finite state space Markov chains. Perfect sampling using CFTP has been successfully extended to the context of point processes by, amongst other authors, Häggström et al. [5]. In [5] Gibbs sampling is applied to a bivariate point process, the penetrable spheres mixture model [19]. However, in general the running time of CFTP in terms of number of transitions is not independent of the state sampled. Thus an impatient user who aborts long runs may introduce a subtle bias, the user impatience bias. Fill [3] introduced an exact sampling algorithm for finite state space Markov chains which, in contrast to CFTP, is unbiased for user impatience. Fill's algorithm is a form of rejection sampling and similarly to CFTP requires sufficient monotonicity properties of the transition kernel used. We show how Fill's version of rejection sampling can be extended to an infinite state space context to produce an exact sample of the penetrable spheres mixture process and related models. Following [5] we use Gibbs sampling and make use of the partial order of the mixture model state space. Thus we construct an algorithm which protects against bias caused by user impatience and which delivers samples not only of the mixture model but also of the attractive area-interaction and the continuum random-cluster process.

Asymptotic filtering for finite state Markov chains

Asymptotic formulae for the optimal filtering error for finite state space Markov chains observed in independent noise are presented. Asymptotically optimal simple filters, which do not depend on the transition rates of the chain, are also presented.

Extreme values, range and weak convergence of integrals of Markov chains

A study is made of the maximum, minimum and range on [0, t] of the integral process where S is a finite state-space Markov chain. Approximate results are derived by establishing weak convergence of a sequence of such processes to a Wiener process. For a particular family of two-state stationary Markov chains we show that the corresponding centered integral processes exhibit the Hurst phenomenon to a remarkable degree in their pre-asymptotic behaviour.

Extreme values, range and weak convergence of integrals of Markov chains

A study is made of the maximum, minimum and range on [0,t] of the integral processwhereSis a finite state-space Markov chain. Approximate results are derived by establishing weak convergence of a sequence of such processes to a Wiener process. For a particular family of two-state stationary Markov chains we show that the corresponding centered integral processes exhibit the Hurst phenomenon to a remarkable degree in their pre-asymptotic behaviour.