Continuous-time Hidden Markov Model Research Articles

Sampling and filtering with Markov chains

A new continuous-time Markov chain rate change formula is proven. This theorem is used to derive existence and uniqueness of novel filtering equations akin to the Duncan–Mortensen–Zakai equation and the Fujisaki–Kallianpur–Kunita equation but for Markov signals with general continuous-time Markov chain observations. The equations in this second theorem have the unique feature of being driven by both the observations and the process counting the observation transitions. A direct method of solving these filtering equations is also derived. Most results apply as special cases to the continuous-time Hidden Markov Models (CTHMM), which have become important in applications like disease progression tracking, The corresponding CTHMM results are stated as corollaries. Finally, application of our general theorems to Markov chain importance sampling, rejection sampling and branching particle filtering algorithms is also explained and these are illustrated by way of disease tracking simulations.

Estimating Hidden Markov Models (HMMs) of the cognitive process in strategic thinking using eye-tracking

Hidden Markov Models (HMMs) are used to study language, sleep, macroeconomic states, and other processes that reflect probabilistic transitions between states that can't be observed directly. This paper applies HMMs to data from location-based game theory experiments. In these location games, players choose a pixel location from an image. These players either have a common goal (choose a matching location), or competing goals, to mismatch (hide) or match (seek) in hider-seeker games. We use eye-tracking to record where players look throughout the experimental decision. Each location's numerical salience is predicted using an accurate, specialized vision science-based neural network [the Saliency Attentive Model (SAM)]. The HMM shows the pattern of transitioning from hidden states corresponding to either high or low-salience locations, combining the eye-tracking and salience data. The transitions vary based on the player's strategic goal. For example, hiders transition more often to low-salience states than seekers do. The estimated HMM is then used to do two useful things. First, a continuous-time HMM (cHMM) predicts the salience level of each player's looking over several seconds. The cHMM can then be used to predict what would happen if the same process was truncated by time pressure: This calculation makes a specific numerical prediction about how often seekers will win, and it predicts an increase in win rate but underestimates the size of the change. Second, a discrete-time HMM (dHMM) can be used to infer levels of strategic thinking from high-to-low salience eye-tracking transitions. The resulting estimates are more plausible than some maximum-likelihood models, which underestimate strategic sophistication in these games. Other applications of HMM in experimental economics are suggested.

A Novel Continuous-time Hidden Markov Model Based on a Bag of Features Extracted from MR Brain Images for Alzheimer's Stage Progression and Detection.

Mounting novel solutions for conspicuous neurodegenerative disorders that grow consistently, such as Alzheimer's disease, rely on tracking and identifying disease development, improvement, and progression. Compared to many clinical or survey-based detection methods, early Alzheimer's stage detection can be possible through computer-based MR brain images and discrete stochastic processes. In the case of Alzheimer's stage progression, the existing models illustrate that the learning problem comprises two issues: estimating posterior probabilities of the Alzheimer's stage and computing conditioned statistics of the Alzheimer's end-stage. The proposed model overcomes these issues by restructuring the estimation problem as EM-centered CT- HMM. This paper proposes a novel framework model with two phases; the first phase covers the feature extraction of magnetic resonance imaging based on many computer vision methods known as a collection of bag-of-features (BoF). In the second phase, the EM-centered learning method is used for the continuous-time hidden Markov model (CT-HMM), an efficient approach to modeling Alzheimer's disease progression with time and stages. The proposed CT-HMM is implemented with eight Alzheimer's stages (source: ADNI) to visualize and predict the stage progression of the ADNI MRI dataset. The proposed model reported the transition posterior probability as 0.765 (high to low stage progression) and 0.234 (low to high stage progression). The model's accuracy and F1 score are estimated as 97.13 and 96.51, respectively. The proposed model's accuracy and evaluation metrics reported higher results in the work on Alzheimer's stage progression and prediction.

De-biasing Particle Filtering for a Continuous Time Hidden Markov Model With a Cox Process Observation Model

De-biasing Particle Filtering for a Continuous Time Hidden Markov Model With a Cox Process Observation Model

Double-quantized-based $ H_{\infty} $ tracking control of T-S fuzzy semi-Markovian jump systems with adaptive event-triggered

<abstract><p>This paper investigates the issue of asynchronous $ H_{\infty} $ tracking control for nonlinear semi-Markovian jump systems (SMJSs) based on the T-S fuzzy model. Firstly, in order to improve the performance of network control systems (NCSs) and the efficiency of data transmission, this paper adopts a double quantization strategy which quantifies the input and output of the controllers. Secondly, for the purpose of reducing the burden of network communication, an adaptive event-triggered mechanism (AETM) is adopted. Thirdly, due to the influence of network-induce delay, the system mode information can not be transmitted to the controller synchronously, thus, a continuous-time hidden Markov model (HMM) is established to describe the asynchronous phenomenon between the system and the controller. Additionally, with the help of some improved Lyapunov-Krasovski (L-K) functions with fuzzy basis, some sufficient criteria are derived to co-guarantee the state stability and the $ H_{\infty} $ performance for the closed-loop tracking control system. Finally, a numerical example and a practical example are given to verify the effectiveness of designed mentality.</p></abstract>

Hierarchical continuous-time inhomogeneous hidden Markov model for cancer screening with extensive followup data.

Continuous-time hidden Markov models are an attractive approach for disease modeling because they are explainable and capable of handling both irregularly sampled, skewed and sparse data arising from real-world medical practice, in particular to screening data with extensive followup. Most applications in this context consider time-homogeneous models due to their relative computational simplicity. However, the time homogeneous assumption is too strong to accurately model the natural history of many diseases including cancer. Moreover, cancer risk across the population is not homogeneous either, since exposure to disease risk factors can vary considerably between individuals. This is important when analyzing longitudinal datasets and different birth cohorts. We model the heterogeneity of disease progression and regression using piece-wise constant intensity functions and model the heterogeneity of risks in the population using a latent mixture structure. Different submodels under the mixture structure employ the same types of Markov states reflecting disease progression and allowing both clinical interpretation and model parsimony. We also consider flexible observational models dealing with model over-dispersion in real data. An efficient, scalable Expectation-Maximization algorithm for inference is proposed with the theoretical guaranteed convergence property. We demonstrate our method's superior performance compared to other state-of-the-art methods using synthetic data and a real-world cervical cancer screening dataset from the Cancer Registry of Norway. Moreover, we present two model-based risk stratification methods that identify the risk levels of individuals.

Promote or inhibit? Research on the transition of consumer potential purchase intention.

A dramatic shift from offline to online has happened in consumer behavior, leading to enterprises ploughing a large number of digital advertisements to capture consumers’ attention online. To evaluate the effectiveness of different online advertising, we explore the dynamic impacts of nine different online channels on the transition of consumers’ potential purchase intention and the consumer behavior. We use a continuous-time hidden Markov model (CT-HMM) to capture the transfer path of consumers who are affected by various online channels. Our findings reveal that online advertising has a positive and statistically significant impact on the transition of consumer purchase intention, of which search advertising can significantly increase consumers’ propensity to purchase, and its effect on transferring consumers from high to low purchase intention is not very strong in comparison. However, consumers have a very low annoyance threshold to short messaging service (SMS) advertising, and they are easy to get tired of SMS advertising and transfer to low purchase intention. Most firm-initiated advertising is more likely to transfer consumers to a low purchase intention state. Advertisements which can not improve consumer purchase intention very well have fewer stimulating effects on consumers’ information collection behavior than other advertisements. Our research contributes to the literature on the effectiveness of online advertising and provide some management insights for enterprises.

Multistate capture–recapture models for irregularly sampled data

Multistate capture-recapture data comprise individual-specific sighting histories, together with information on individuals’ states related, for example, to breeding status, infection level, or geographical location. Such data are often analysed using the Arnason–Schwarz model, where transitions between states are modelled using a discrete-time Markov chain, making the model most easily applicable to regular time series. When time intervals between capture occasions are not of equal length, more complex time-dependent constructions may be required, increasing the number of parameters to estimate, decreasing interpretability, and potentially leading to reduced precision. Here we develop a multi-state model based on a state process operating in continuous time, which can be regarded as an analogue of the discrete-time Arnason–Schwarz model for irregularly sampled data. Statistical inference is carried out by regarding the capture-recapture data as realisations from a continuous-time hidden Markov model, which allows the associated efficient algorithms to be used for maximum likelihood estimation and state decoding. To illustrate the feasibility of the modelling framework, we use a long-term survey of bottlenose dolphins where capture occasions are not regularly spaced through time. Here, we are particularly interested in seasonal effects on the movement rates of the dolphins along the Scottish east coast. The results reveal seasonal movement patterns between two core areas of their range, providing information that will inform conservation management.

A distributed dynamic event-triggered mechanism to HMM-based observer design for [formula omitted] sliding mode control of Markov jump systems

This paper is devoted to investigating asynchronous state observer design for H∞ analysis of Markov jump systems with partial mode information through sliding mode control approach. In order to enhance the reliability and efficiency of communication network, a novel distributed dynamic event-triggered scheme is proposed. Firstly, due to partial information on the operation of system jumping modes, a continuous-time hidden Markov model (HMM) based asynchronous state observer relying on the triggered output measurement is designed, based on which an integral sliding surface function is proposed; Secondly, by proposing an asynchronous sliding mode control law, the finite-time reachability of estimated trajectories onto the predefined sliding surface is ensured; Thirdly, the stochastic stability with a prescribed H∞ disturbance attenuation level is analyzed for the considered closed-loop systems through linear matrix inequality; Finally, a numerical example is presented to verify the validity of the proposed control scheme.

Signal-to-noise matrix and model reduction in continuous-time hidden Markov models

Continuous-time regime-switching models are a very popular class of models for financial applications. In this work the so-called signal-to-noise matrix is introduced for hidden Markov models where the switching is driven by an unobservable Markov chain. Its relations to filtering, i.e. state estimation of the chain given the available observations, and portfolio optimization are investigated. A convergence result for the filter is derived: The filter converges to its invariant distribution if the eigenvalues of the signal-to-noise matrix converge to zero. This matrix is then also used to prove a mutual fund representation for regime-switching models and a corresponding market reduction which is consistent with filtering and portfolio optimization. Two canonical cases for the reduction are analyzed in more detail, the first based on the market regimes and the second depending on the eigenvalues. These considerations are presented both for observable and unobservable Markov chains. The results are illustrated by numerical simulations.

Progression of type 1 diabetes from latency to symptomatic disease is predicted by distinct autoimmune trajectories

Development of islet autoimmunity precedes the onset of type 1 diabetes in children, however, the presence of autoantibodies does not necessarily lead to manifest disease and the onset of clinical symptoms is hard to predict. Here we show, by longitudinal sampling of islet autoantibodies (IAb) to insulin, glutamic acid decarboxylase and islet antigen-2 that disease progression follows distinct trajectories. Of the combined Type 1 Data Intelligence cohort of 24662 participants, 2172 individuals fulfill the criteria of two or more follow-up visits and IAb positivity at least once, with 652 progressing to type 1 diabetes during the 15 years course of the study. Our Continuous-Time Hidden Markov Models, that are developed to discover and visualize latent states based on the collected data and clinical characteristics of the patients, show that the health state of participants progresses from 11 distinct latent states as per three trajectories (TR1, TR2 and TR3), with associated 5-year cumulative diabetes-free survival of 40% (95% confidence interval [CI], 35% to 47%), 62% (95% CI, 57% to 67%), and 88% (95% CI, 85% to 91%), respectively (p < 0.0001). Age, sex, and HLA-DR status further refine the progression rates within trajectories, enabling clinically useful prediction of disease onset.

Dynamic modeling of hospitalized COVID-19 patients reveals disease state-dependent risk factors.

ObjectiveThe study sought to investigate the disease state–dependent risk profiles of patient demographics and medical comorbidities associated with adverse outcomes of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections.Materials and MethodsA covariate-dependent, continuous-time hidden Markov model with 4 states (moderate, severe, discharged, and deceased) was used to model the dynamic progression of COVID-19 during the course of hospitalization. All model parameters were estimated using the electronic health records of 1362 patients from ProMedica Health System admitted between March 20, 2020 and December 29, 2020 with a positive nasopharyngeal PCR test for SARS-CoV-2. Demographic characteristics, comorbidities, vital signs, and laboratory test results were retrospectively evaluated to infer a patient’s clinical progression.ResultsThe association between patient-level covariates and risk of progression was found to be disease state dependent. Specifically, while being male, being Black or having a medical comorbidity were all associated with an increased risk of progressing from the moderate disease state to the severe disease state, these same factors were associated with a decreased risk of progressing from the severe disease state to the deceased state.DiscussionRecent studies have not included analyses of the temporal progression of COVID-19, making the current study a unique modeling-based approach to understand the dynamics of COVID-19 in hospitalized patients.ConclusionDynamic risk stratification models have the potential to improve clinical outcomes not only in COVID-19, but also in a myriad of other acute and chronic diseases that, to date, have largely been assessed only by static modeling techniques.

Adaptive monitoring scheme of stochastically failing systems under hidden degradation processes

Limited by the installation of sensors and the structure of systems, only external observation information can generate an association with the internal health status of a system. In addition, modern industrial systems are often accompanied by hidden degradation processes, which present great challenges when monitoring and alerting impending failures. To address this issue, this paper proposes an adaptive monitoring scheme for predicting faults of systems with hidden degradation processes. A multivariate, continuous-time hidden Markov model with three states (hidden healthy state, hidden unhealthy state, and observable failure state) is established to describe the degradation process of the system. An expectation maximization (EM) algorithm is presented to estimate the parameters of the stochastic model. On the basis of the hidden degradation model, an adaptive Bayesian control scheme is developed for monitoring the potential risk of the system with two switchable sampling intervals. We calculate optimal decision variables of the control scheme to achieve the minimum expected average cost by a policy iteration algorithm under a semi-Markov decision process (SMDP). The performance and characteristics of the proposed scheme are demonstrated via real oil measurement data obtained from mechanical generators. A comparison case illustrates the superiority of the prediction performance of our monitoring scheme.

Maximum approximate likelihood estimation of general continuous-time state-space models

Continuous-time state-space models (SSMs) are flexible tools for analysing irregularly sampled sequential observations that are driven by an underlying state process. Corresponding applications typically involve restrictive assumptions concerning linearity and Gaussianity to facilitate inference on the model parameters via the Kalman filter. In this contribution, we provide a general continuous-time SSM framework, allowing both the observation and the state process to be non-linear and non-Gaussian. Statistical inference is carried out by maximum approximate likelihood estimation, where multiple numerical integration within the likelihood evaluation is performed via a fine discretization of the state process. The corresponding reframing of the SSM as a continuous-time hidden Markov model, with structured state transitions, enables us to apply the associated efficient algorithms for parameter estimation and state decoding. We illustrate the modelling approach in a case study using data from a longitudinal study on delinquent behaviour of adolescents in Germany, revealing temporal persistence in the deviation of an individual's delinquency level from the population mean.

Bayesian clustering for continuous‐time hidden Markov models

We develop clustering procedures for longitudinal trajectories based on a continuous‐time hidden Markov model (CTHMM) and a generalized linear observation model. Specifically, in this article we carry out finite and infinite mixture model‐based clustering for a CTHMM and achieve inference using Markov chain Monte Carlo (MCMC). For a finite mixture model with a prior on the number of components, we implement reversible‐jump MCMC to facilitate the trans‐dimensional move between models with different numbers of clusters. For a Dirichlet process mixture model, we utilize restricted Gibbs sampling split–merge proposals to improve the performance of the MCMC algorithm. We apply our proposed algorithms to simulated data as well as a real‐data example, and the results demonstrate the desired performance of the new sampler.

A Machine-Learning Derived Huntington's Disease Progression Model: Insights for Clinical Trial Design.

Applying machine-learning algorithms to large datasets such as those available in Huntington's disease offers the opportunity to discover hidden patterns, often not discernible to clinical observation. To develop and validate a model of Huntington's disease progression using probabilistic machine learning methods. Longitudinal data encompassing 2079 assessment measures from four observational studies (PREDICT-HD, REGISTRY, TRACK-HD, and Enroll-HD) were integrated and machine-learning methods (Bayesian latent-variable analysis and continuous-time hidden Markov models) were applied to develop a probabilistic model of disease progression. The model was validated using a separate Enroll-HD dataset and compared with existing clinical reference assessments (Unified Huntington's Disease Rating Scale [UHDRS] diagnostic confidence level, total functional capacity, and total motor scores) and CAG-age product. Nine disease states were discovered based on 44 motor, cognitive, and functional measures, which correlated with reference assessments. The validation set included 3158 participants (mean age, 48.4 years) of whom 61.5% had manifest disease. Analysis of transition times showed that "early-disease" states 1 and 2, which occur before motor diagnosis, lasted ~16 years. Increasing numbers of participants had motor onset during "transition" states 3 to 5, which collectively lasted ~10 years, and the "late-disease" states 6 to 9 also lasted ~10 years. The annual probability of conversion from one of the nine identified disease states to the next ranged from 5% to 27%. The natural history of Huntington's disease can be described by nine disease states of increasing severity. The ability to derive characteristics of disease states and probabilities for progression through these states will improve trial design and participant selection. © 2021 International Parkinson and Movement Disorder Society.

Bayesian inference for continuous-time hidden Markov models with an unknown number of states

We consider the modeling of data generated by a latent continuous-time Markov jump process with a state space of finite but unknown dimensions. Typically in such models, the number of states has to be pre-specified, and Bayesian inference for a fixed number of states has not been studied until recently. In addition, although approaches to address the problem for discrete-time models have been developed, no method has been successfully implemented for the continuous-time case. We focus on reversible jump Markov chain Monte Carlo which allows the trans-dimensional move among different numbers of states in order to perform Bayesian inference for the unknown number of states. Specifically, we propose an efficient split-combine move which can facilitate the exploration of the parameter space, and demonstrate that it can be implemented effectively at scale. Subsequently, we extend this algorithm to the context of model-based clustering, allowing numbers of states and clusters both determined during the analysis. The model formulation, inference methodology, and associated algorithm are illustrated by simulation studies. Finally, we apply this method to real data from a Canadian healthcare system in Quebec.

Turn Your Online Weight Management from Zero to Hero: A Multidimensional, Continuous-Time Evaluation

Online weight-loss communities (OWCs) provide individuals with various tools to support their weight management, such as weight recorders and weight-loss journals. These tools enable individuals to focus on different aspects of their self-regulation, including weight-loss outcomes and behavioral routines. Prior research, however, has not fully incorporated individuals’ self-regulation focuses; thus, there is limited understanding of individuals’ online weight-management dynamics as well as the operating mechanisms of OWCs. This gap in the literature motivates us to develop a framework that is able to account for individuals’ multiple self-regulation focuses, termed self-regulatory dimensions in this study. We propose a multidimensional, continuous-time hidden Markov model, which can not only capture individuals’ self-regulatory dimensions jointly as a multidimensional vector, but also can incorporate a hidden layer of dynamics that depicts individuals’ cognitive states in producing weight-management behaviors. By investigating a leading noncommercial OWC in the United States, we find that individuals tend to increase their journal-recording behaviors while decreasing self-weighing behaviors after they have participated in online social activities. Given that individuals usually expend limited effort toward weight management, this result suggests that individuals may shift their focus from weight-loss outcomes (i.e., changes in weight) to weight-management behavioral routines. Therefore, neglecting either self-regulatory dimension would result in an underestimation of individuals’ engagement in conducting self-management in OWCs. Our results also provide insight into social influence on individuals’ weight-management behaviors. This study contributes to the extant literature on individuals’ engagement in online healthcare communities and the functionality of OWCs. This paper was accepted by Anandhi Bharadwaj, information systems.

A Continuous-Time Hidden Markov Model of Credit Quality Underlying Published Agencies'&amp;nbsp;Ratings

This paper presents a Continuous-Time Hidden Markov Model (CT-HMM) of debt issuers' credit quality dynamics. We observe ratings published by several agencies (e.g., Standard & Poor's, Moody's, Fitch) for one or several issuers (e.g., countries) and assume that we can find a maximum-likelihood estimate of the true rating underlying data. Using CT-HMM instead of the standard discrete HMM enables to deal with irregular credit events. Our model is specifically adapted to handle several agencies and/or several debt issuers: we can infer the hidden rating of a group without using a mean approximation. The model parameters are estimated using the Expectation-Maximization algorithm, with adapted filters and an ad-hoc numerical method with no discretization error. A set of experiments on both simulated and real sovereign credit ratings illustrates our approach.

Formation Static Output Control of Linear Multi-Agent Systems With Hidden Markov Switching Network Topologies

This paper addresses the $\mathcal {H}_{2}$ , $\mathcal {H}_\infty $ and mixed $\mathcal {H}_{2}/\mathcal {H}_\infty $ formation static output control of continuous-time linear multi-agent systems with Markovian switching network topologies. It is assumed that the operation mode of the network topology cannot be directly measured but, instead, can be estimated by an imperfect detector. To model this problem, we consider a continuous-time hidden Markov model, in which the hidden component represents the real operation mode of the network topology while the observed component represents the information emitted from the detector and available for the controller. It is also assumed that only a partial information from the state variables of the multi-agent systems is available. By using an LMI (linear matrix inequality) formulation, a distributed static output controller which switches according to the detector information is designed to guarantee the stability in the mean square sense of the closed loop system, as well as an upper-bound for an index performance. Three situations are considered for the performance criteria: the $\mathcal {H}_{2}$ norm, the $\mathcal {H}_\infty $ norm, and the mixed $\mathcal {H}_{2}/\mathcal {H}_\infty $ norms. The paper is concluded with a numerical example to illustrate the effectiveness of the theoretical results.