Finite-state Hidden Markov Models Research Articles

Exploring predictive states via Cantor embeddings and Wasserstein distance.

Predictive states for stochastic processes are a nonparametric and interpretable construct with relevance across a multitude of modeling paradigms. Recent progress on the self-supervised reconstruction of predictive states from time-series data focused on the use of reproducing kernel Hilbert spaces. Here, we examine how Wasserstein distances may be used to detect predictive equivalences in symbolic data. We compute Wasserstein distances between distributions over sequences ("predictions") using a finite-dimensional embedding of sequences based on the Cantor set for the underlying geometry. We show that exploratory data analysis using the resulting geometry via hierarchical clustering and dimension reduction provides insight into the temporal structure of processes ranging from the relatively simple (e.g., generated by finite-state hidden Markov models) to the very complex (e.g., generated by infinite-state indexed grammars).

Discovering causal structure with reproducing-kernel Hilbert space ε-machines.

We merge computational mechanics' definition of causal states (predictively equivalent histories) with reproducing-kernel Hilbert space (RKHS) representation inference. The result is a widely applicable method that infers causal structure directly from observations of a system's behaviors whether they are over discrete or continuous events or time. A structural representation-a finite- or infinite-state kernel ϵ-machine-is extracted by a reduced-dimension transform that gives an efficient representation of causal states and their topology. In this way, the system dynamics are represented by a stochastic (ordinary or partial) differential equation that acts on causal states. We introduce an algorithm to estimate the associated evolution operator. Paralleling the Fokker-Planck equation, it efficiently evolves causal-state distributions and makes predictions in the original data space via an RKHS functional mapping. We demonstrate these techniques, together with their predictive abilities, on discrete-time, discrete-value infinite Markov-order processes generated by finite-state hidden Markov models with (i) finite or (ii) uncountably infinite causal states and (iii) continuous-time, continuous-value processes generated by thermally driven chaotic flows. The method robustly estimates causal structure in the presence of varying external and measurement noise levels and for very high-dimensional data.

Learning hidden Markov models for linear Gaussian systems with applications to event-based state estimation

This work attempts to approximate a linear Gaussian system with a finite-state hidden Markov model (HMM), which is found useful in dealing with challenges in designing networked control systems An indirect approach is developed, where a state-space model (SSM) is firstly identified for a Gaussian system and the SSM is then used as an emulator for learning an HMM. In the proposed method, the training data for the HMM are obtained from the data generated by the SSM through building a quantization mapping. Parameter learning algorithms are designed to learn the parameters of the HMM, through exploiting the periodical structural characteristics of the HMM. The convergence and asymptotic properties of the proposed algorithms are analyzed. The HMM learned using the proposed algorithms is applied to event-triggered state estimation, and numerical results on model learning and state estimation demonstrate the validity of the proposed algorithms.

Spectral simplicity of apparent complexity. II. Exact complexities and complexity spectra.

The meromorphic functional calculus developed in Part I overcomes the nondiagonalizability of linear operators that arises often in the temporal evolution of complex systems and is generic to the metadynamics of predicting their behavior. Using the resulting spectral decomposition, we derive closed-form expressions for correlation functions, finite-length Shannon entropy-rate approximates, asymptotic entropy rate, excess entropy, transient information, transient and asymptotic state uncertainties, and synchronization information of stochastic processes generated by finite-state hidden Markov models. This introduces analytical tractability to investigating information processing in discrete-event stochastic processes, symbolic dynamics, and chaotic dynamical systems. Comparisons reveal mathematical similarities between complexity measures originally thought to capture distinct informational and computational properties. We also introduce a new kind of spectral analysis via coronal spectrograms and the frequency-dependent spectra of past-future mutual information. We analyze a number of examples to illustrate the methods, emphasizing processes with multivariate dependencies beyond pairwise correlation. This includes spectral decomposition calculations for one representative example in full detail.

On Finite-State Stochastic Modeling and Secure Estimation of Cyber-Physical Systems

The problem of secure state estimation and attack detection in cyber-physical systems is considered in this paper. A stochastic modeling framework is first introduced, based on which the attacked system is modeled as a finite-state hidden Markov model with switching transition probability matrices controlled by a Markov decision process. Based on this framework, a joint state and attack estimation problem is formulated and solved. Utilizing the change of probability measure approach, we show that an unnormalized joint state and attack distribution conditioned on the sensor measurement information evolves in a linear recursive form, based on which the optimal estimates can be further calculated by evaluating the normalized marginal conditional distributions. The estimation results are further applied to secure estimation of stable linear Gaussian systems, and extensions to more general systems are also discussed. The effectiveness of the results are illustrated by numerical examples and comparative simulation.

Approximating a diffusion by a finite-state hidden Markov model

For a wide class of continuous-time Markov processes evolving on an open, connected subset of Rd, the following are shown to be equivalent: (i)The process satisfies (a slightly weaker version of) the classical Donsker–Varadhan conditions;(ii)The transition semigroup of the process can be approximated by a finite-state hidden Markov model, in a strong sense in terms of an associated operator norm;(iii)The resolvent kernel of the process is ‘v-separable’, that is, it can be approximated arbitrarily well in operator norm by finite-rank kernels. Under any (hence all) of the above conditions, the Markov process is shown to have a purely discrete spectrum on a naturally associated weighted L∞ space.

Hidden Markov models with state-dependent mixtures: minimal representation, model testing and applications to clustering

Finite-state hidden Markov models (HMMs), also called Markov-dependent finite mixtures, form a popular, frequently used model class for serially dependent observations with unobserved heterogeneity. We consider HMMs in which the state-dependent distributions are themselves finite mixtures. In such models, the parametrization is not unique, since components from the state-dependent mixtures may also be represented as states in the underlying Markov chain. We determine a unique (up to label switching) representation for the HMM in which the Markov chain has a minimal number of states. Further, we propose a likelihood-ratio test for the hypothesis that the number of states in the Markov chain can be reduced without changing the distribution of the time-series model. Our method has important applications in cluster analysis and model selection. After highlighting the relevance of serial dependence for clustering, we propose two-step clustering algorithms. Starting with a BIC choice for a standard HMM (with simple state-dependent distributions), in the first step we determine the minimal representation of the HMM by testing, and in the second step we merge components in the resulting state-dependent finite mixtures by using either a local entropy or a modality-based criterion. The states in the resulting Markov chain, potentially split according to the remaining state-dependent components, are then interpreted as clusters. For model selection, we illustrate our method on a series of logarithmic returns of gold prices using normal HMMs. The AIC choice is a six-state HMM, while the BIC choice has four states. When starting with the AIC choice, successive testing results in a four-state Markov chain, with two state-dependent distributions consisting of two-component normal mixtures.

Loss of memory of hidden Markov models and Lyapunov exponents

In this paper we prove that the asymptotic rate of exponential loss of memory of a finite state hidden Markov model is bounded above by the difference of the first two Lyapunov exponents of a certain product of matrices. We also show that this bound is in fact realized, namely for almost all realizations of the observed process we can find symbols where the asymptotic exponential rate of loss of memory attains the difference of the first two Lyapunov exponents. These results are derived in particular for the observed process and for the filter; that is, for the distribution of the hidden state conditioned on the observed sequence. We also prove similar results in total variation.

Limit Theorems in Hidden Markov Models

In this paper, under mild assumptions, we derive a law of large numbers, a central limit theorem with an error estimate, an almost sure invariance principle, and a variant of the Chernoff bound in finite-state hidden Markov models. These limit theorems are of interest in certain areas of information theory and statistics. Particularly, we apply the limit theorems to derive the rate of convergence of the maximum likelihood estimator in finite-state hidden Markov models.

Genome-Wide Localization of Protein-DNA Binding and Histone Modification by a Bayesian Change-Point Method with ChIP-seq Data

Next-generation sequencing (NGS) technologies have matured considerably since their introduction and a focus has been placed on developing sophisticated analytical tools to deal with the amassing volumes of data. Chromatin immunoprecipitation sequencing (ChIP-seq), a major application of NGS, is a widely adopted technique for examining protein-DNA interactions and is commonly used to investigate epigenetic signatures of diffuse histone marks. These datasets have notoriously high variance and subtle levels of enrichment across large expanses, making them exceedingly difficult to define. Windows-based, heuristic models and finite-state hidden Markov models (HMMs) have been used with some success in analyzing ChIP-seq data but with lingering limitations. To improve the ability to detect broad regions of enrichment, we developed a stochastic Bayesian Change-Point (BCP) method, which addresses some of these unresolved issues. BCP makes use of recent advances in infinite-state HMMs by obtaining explicit formulas for posterior means of read densities. These posterior means can be used to categorize the genome into enriched and unenriched segments, as is customarily done, or examined for more detailed relationships since the underlying subpeaks are preserved rather than simplified into a binary classification. BCP performs a near exhaustive search of all possible change points between different posterior means at high-resolution to minimize the subjectivity of window sizes and is computationally efficient, due to a speed-up algorithm and the explicit formulas it employs. In the absence of a well-established “gold standard” for diffuse histone mark enrichment, we corroborated BCP's island detection accuracy and reproducibility using various forms of empirical evidence. We show that BCP is especially suited for analysis of diffuse histone ChIP-seq data but also effective in analyzing punctate transcription factor ChIP datasets, making it widely applicable for numerous experiment types.

Information symmetries in irreversible processes

We study dynamical reversibility in stationary stochastic processes from an information-theoretic perspective. Extending earlier work on the reversibility of Markov chains, we focus on finitary processes with arbitrarily long conditional correlations. In particular, we examine stationary processes represented or generated by edge-emitting, finite-state hidden Markov models. Surprisingly, we find pervasive temporal asymmetries in the statistics of such stationary processes. As a consequence, the computational resources necessary to generate a process in the forward and reverse temporal directions are generally not the same. In fact, an exhaustive survey indicates that most stationary processes are irreversible. We study the ensuing relations between model topology in different representations, the process's statistical properties, and its reversibility in detail. A process's temporal asymmetry is efficiently captured using two canonical unifilar representations of the generating model, the forward-time and reverse-time ε-machines. We analyze example irreversible processes whose ε-machine representations change size under time reversal, including one which has a finite number of recurrent causal states in one direction, but an infinite number in the opposite. From the forward-time and reverse-time ε-machines, we are able to construct a symmetrized, but nonunifilar, generator of a process--the bidirectional machine. Using the bidirectional machine, we show how to directly calculate a process's fundamental information properties, many of which are otherwise only poorly approximated via process samples. The tools we introduce and the insights we offer provide a better understanding of the many facets of reversibility and irreversibility in stochastic processes.

Implied distributions in multiple change point problems

A method for efficiently calculating exact marginal, conditional and joint distributions for change points defined by general finite state Hidden Markov Models is proposed. The distributions are not subject to any approximation or sampling error once parameters of the model have been estimated. It is shown that, in contrast to sampling methods, very little computation is needed. The method provides probabilities associated with change points within an interval, as well as at specific points.

Effects of statistical dependence on multiple testing under a hidden Markov model

The performance of multiple hypothesis testing is known to be affected by the statistical dependence among random variables involved. The mechanisms responsible for this, however, are not well understood. We study the effects of the dependence structure of a finite state hidden Markov model (HMM) on the likelihood ratios critical for optimal multiple testing on the hidden states. Various convergence results are obtained for the likelihood ratios as the observations of the HMM form an increasing long chain. Analytic expansions of the first and second order derivatives are obtained for the case of binary states, explicitly showing the effects of the parameters of the HMM on the likelihood ratios.

Estimation in hidden Markov models via efficient importance sampling

Given a sequence of observations from a discrete-time, finite-state hidden Markov model, we would like to estimate the sampling distribution of a statistic. The bootstrap method is employed to approximate the confidence regions of a multi-dimensional parameter. We propose an importance sampling formula for efficient simulation in this context. Our approach consists of constructing a locally asymptotically normal (LAN) family of probability distributions around the default resampling rule and then minimizing the asymptotic variance within the LAN family. The solution of this minimization problem characterizes the asymptotically optimal resampling scheme, which is given by a tilting formula. The implementation of the tilting formula is facilitated by solving a Poisson equation. A few numerical examples are given to demonstrate the efficiency of the proposed importance sampling scheme.

A simple proof of Kaijser’s unique ergodicity result for hidden Markov α-chains

According to a 1975 result of T. Kaijser, if some nonvanishing product of hidden Markov model (HMM) stepping matrices is subrectangular, and the underlying chain is aperiodic, the corresponding α-chain has a unique invariant limiting measure λ. Here the α-chain {αn}={(αni)} is given by αni=P(Xn=i|Yn,Yn−1,…), where {(Xn,Yn)} is a finite state HMM with unobserved Markov chain component {Xn} and observed output component {Yn}. This defines {αn} as a stochastic process taking values in the probability simplex. It is not hard to see that {αn} is itself a Markov chain. The stepping matrices M(y)=(M(y)ij) give the probability that (Xn,Yn)=(j,y), conditional on Xn−1=i. A matrix is said to be subrectangular if the locations of its nonzero entries forms a cartesian product of a set of row indices and a set of column indices. Kaijser’s result is based on an application of the Furstenberg–Kesten theory to the random matrix products M(Y1)M(Y2)⋯M(Yn). In this paper we prove a slightly stronger form of Kaijser’s theorem with a simpler argument, exploiting the theory of e chains.

Recursive estimation for hidden Markov models: a dependent case

A discrete time, finite state Hidden Markov Model is considered. This consists of a Markov chain and a finite state observation process. A recent paper considered the case when the noises in the chain and the observation process are independent; this paper discusses the more complicated dependent case. Recursive conditional normalized and unnormalized, (Zakai), estimates are obtained for: the state of the chain, the number of jumps of the chain from one state to another, the occupation time of the chain in any state, the number of transitions of the observation process and the number of joint transitions of the chain and observation process. These results allow reestimation of the parameters of the model. In particular, they provide a test for whether the noise terms in the chain and observation are independent

Hidden Markov models for fault detection in dynamic systems

Continuous monitoring of complex dynamic systems is an increasingly important issue in diverse areas such as nuclear plant safety, production line reliability, and medical health monitoring systems. Recent advances in both sensor technology and computational capabilities have made on-line permanent monitoring much more feasible than it was in the past. In this paper it is shown that a pattern recognition system combined with a finite-state hidden Markov model provides a particularly useful method for módelling temporal context in continuous monitoring. The parameters of the Markov model are derived from gross failure statistics such as the mean time between failures. The model is validated on a real-world fault diagnosis problem and it is shown that Markov modelling in this context offers significant practical benefits.