Continuous-time Hidden Markov Model Research Articles

Hidden Markov Modeling of the Cognitive Process in Strategic Thinking

Hidden Markov models (HMMs) are used to study language, sleep, and other processes that reflect probabilistic transitions between states that cannot be observed directly. We apply HMMs to data from experiments on visual location games. In these games, people choose a pixel from an image. They either have a common goal to match locations or have different goals in hider-seeker games. Eyetracking records where they look at a fine-grained time scale. Numerical salience of different locations is predicted, a priori, from a specialized vision science based neural network. The HMM shows the pattern of transitioning from hidden states corresponding to either high or low salience locations, using the eye-tracking and salience data. The transitions vary based on the player’s strategic goal; for example, hiders transition more often to low-salience states than seekers do. The estimated HMM is then used to do two useful things. First, a continuous-time HMM (cHMM) predicts the salience level of each player’s looking over the course of several seconds. The cHMM is then used to predict what would happen if the same process was truncated by time pressure choosing in two seconds instead of six, cHMM predicts seekers will match hiders 12% of the time; they actually match 15%. Second, dHMM is used to infer levels of strategic thinking from high-to-low transitions (a la Costa-Gomes et al. 2001 and others). The resulting estimates are more plausible than some maximum-likelihood procedures and models which appear to grossly underestimate strategic sophistication. Other applications of HMM in experimental economics are suggested.

A Continuous-Time Hidden Markov Model of Credit Quality Underlying Published Agencies&Apos; Ratings

A Continuous-Time Hidden Markov Model of Credit Quality Underlying Published Agencies&Apos; Ratings

A joint likelihood approach to the analysis of length of stay data utilising the continuous-time hidden Markov model and Coxian phase-type distribution

The Coxian phase-type distribution is a special case of phase-type distribution which represents the time to absorption of a finite Markov chain in continuous time. The distribution is able to capture subjects’ flow through a system but is unable to highlight if there are different pathways caused by an underlying latent factor. Identifying these different pathways will give healthcare providers a deeper insight and understanding of patient flow and allow them to identify and change any potential issues. This paper combines the Coxian phase-type distribution with the continuous-time hidden Markov model to highlight these paths. The theory of combining the Coxian phase-type distribution with the continuous-time hidden Markov model shall be presented along with a simulation study and an application using Italian healthcare data.

Bayesian Multiple Changepoint Detection for Stochastic Models in Continuous Time

A multiple changepoint model in continuous time is formulated as a continuous-time hidden Markov model, defined on a countable infinite state space. The new formulation of the multiple changepoint model allows the model complexities, i.e. the number of changepoints, to accrue unboundedly upon the arrivals of new data. Inference on the number of changepoints and their locations is based on a collapsed Gibbs sampler. We suggest a new version of forward-filtering backward-sampling (FFBS) algorithm in continuous time for simulating the full trajectory of the latent Markov chain, i.e. the changepoints. The FFBS algorithm is based on a randomized time-discretization for the latent Markov chain through uniformization schemes, combined with a discrete-time version of FFBS algorithm. It is shown that, desirably, both the computational cost and the memory cost of the FFBS algorithm are only quadratic to the number of changepoints. The new formulation of the multiple changepoint models allows varying scale of run lengths of changepoints to be characterized. We demonstrate the methods through simulations and a real data example for earthquakes.

Model parameter estimation and residual life span prediction of pneumatic diaphragm pump based on hidden Markov model in intelligent spraying

Pneumatic diaphragm pump is an important part in intelligent spraying. When pneumatic diaphragm pump does not work normally, the entire intelligent spraying product line will be malfunctioned. To maintain and manage pneumatic diaphragm pump effectively, the grade analysis of the health status of pneumatic diaphragm pump is generally used according to its working state. Due to the effects of condition monitoring and random faults, some observable health predictions are often inaccurate. There are very few papers dealing with the health monitoring of pneumatic diaphragm pump and their estimation of residual life span. In this article, a method with vector autoregressive model and continuous-time hidden Markov model was proposed to analyze and evaluate the life span of pneumatic diaphragm pump based on the estimation error of the health condition and the cumulative deterioration of pneumatic diaphragm pump. It is modeled through a continuous-time Markov chain with three states, which includes unobservable healthy state 0, unobservable warning state 1, and observable fault state 2. The expectation–maximization algorithm is used to estimate the model parameters of the fitted hidden Markov. Through the posterior probability of pneumatic diaphragm pump in warning state 1, the derived conditional reliability function and mean residual life span formula can be calculated to evaluate the residual life span of pneumatic diaphragm pump. The results showed that the method can effectively predict the residual life span of pneumatic diaphragm pump, illustrate the effectiveness of the model, and improve the accuracy of the health status rating.

Modeling Chronic Obstructive Pulmonary Disease Progression Using Continuous-Time Hidden Markov Models.

Understanding the progression of chronic diseases, such as chronic obstructive pulmonary disease (COPD), is important to inform early diagnosis, personalized care, and health system management. Data from clinical and administrative systems have the potential to advance this understanding, but traditional methods for modelling disease progression are not well-suited to analyzing data collected at irregular intervals, such as when a patient interacts with a healthcare system. We applied a continuous-time hidden Markov model to irregularly-spaced healthcare utilization events and patient-level characteristics in order to analyze the progression through discrete states of 76,888 patients with COPD. A 4-state model allowed classification of patients into interpretable states of disease progression and generated insights about the role of comorbidities, such as cardiovascular diseases, in accelerating severe trajectories. These results can improve the understanding of the evolution of COPD and point to new hypotheses about chronic disease management and comorbidity.

Onset of schizophrenia diagnoses in a large clinical cohort

We aimed to describe the diagnostic patterns preceding and following the onset of schizophrenia diagnoses in outpatient clinics. A large clinical sample of 26,163 patients with a diagnosis of schizophrenia in at least one outpatient visit was investigated. We applied a Continuous Time Hidden Markov Model to describe the probability of transition from other diagnoses to schizophrenia considering time proximity. Although the most frequent diagnoses before schizophrenia were anxiety and mood disorders, direct transitions to schizophrenia usually came from psychotic-spectrum disorders. The initial diagnosis of schizophrenia was not likely to change for two of every three patients if it was confirmed some months after its onset. When not confirmed, the most frequent alternative diagnoses were personality, affective or non-schizophrenia psychotic disorders. Misdiagnosis or comorbidity with affective, anxiety and personality disorders are frequent before and after the diagnosis of schizophrenia. Our findings give partial support to a dimensional view of schizophrenia and emphasize the need for longitudinal assessment.

Turn Your Online Engagement in Chronic Disease Management from Zero to Hero: A Multi-Dimensional Continuous-Time Evaluation

Individuals’ engagement in online healthcare communities (OHCs) has attracted a large body of research. However, prior research has mainly studied a single behavioral dimension of online engagement. Given that individuals’ online engagement is a complex process, overlooking certain aspects of engagement may lead to underestimating the effectiveness of OHCs. In addition, due to the progressive nature of chronic disease, chronic disease management needs to be evaluated continuously. These concerns motivate us to develop a framework that accounts for both engagement dimensions and engagement timing for studying chronic disease management in OHCs. Considering that the engagement level of disease management cannot be directly observed, in this study, we propose a multi-dimensional Continuous-Time Hidden Markov Model (CTHMM) that captures individuals’ engagement level as a latent state. We root our research in the context of weight management. Our main findings include: 1) The timing of engagement can affect an individual’s engagement level; for instance, individuals with higher participation frequency are more likely to shift to different engagement levels. 2) Participating in support-exchange activities can shift individuals’ focus from weigh-ins to journals, which are two distinct behavioral dimensions of self-monitoring. Thus, an incomplete characterization of engagement dimensions can underestimate individuals’ activeness in OHCs, which will further lead to underrating the role of OHCs in chronic disease control. 3) Different forms of social support can have statistically different effects on engagement, and these effects are mediated by individuals’ own engagement levels. Individuals need to “smartly” adopt social support tools to improve their health management.

Modeling Users’ Online Participation: Evidence from Online Weight-Loss Communities

Users’ participation in virtual communities is a complex process. On the one hand, there exist more than one dynamic sources, such as sequential dynamics and temporal dynamics. On the other hand, users may develop different participation focuses through the participation process. These aspects are underexplored in the extant literature. This research gap motivates us to develop an effective approach to calibrate users’ online behaviors. Specifically, we develop a multi-dimensional continuous-time hidden Markov model (HMM). This model is able to control both sequential and temporal dynamics of users’ online behaviors. In addition, it allows us to explore users’ multiple correlated behavioral dimensions at the same time. To gauge the effectiveness of our approach, we root our research in the context of weight management. The data is collected from a leading online weight-loss community in the US. Our main findings include 1). Timing information can bring valuable insights on users’ participation pattern in virtual communities, and it improves our model forecast performance. 2). Users may shift their participation focuses during the participation process. 3). Different social-support exchange strategies have statistically different effects on users’ weight management, and these effects are mediated by users’ community commitment. Together, these findings suggest that overlooking information imbedded in observed behaviors of users may result in inaccurate or even falsified conclusions on participation pattern. In addition, personalized health care strategy need to be implemented when individual’s condition is complicated and varying with time, such as chronic diseases like obesity and diabetes. Further, we provide several application examples to illustrate the practical values of our study, in which we combine our empirical model with machine learning techniques. These examples can be readily applied to a variety of business settings such as personalized recommendation, precision marketing and user characterization.

A Bayesian multiple changepoint model for marked poisson processes with applications to deep earthquakes

A multiple changepoint model for marked Poisson process is formulated as a continuous time hidden Markov model, which is an extension of Chib’s multiple changepoint models (J Econ 86:221–241, 1998). The inference on the locations of changepoints and other model parameters is based on a two-block Gibbs sampling scheme. We suggest a continuous time version of forward-filtering backward-sampling algorithm for simulating the full trajectories of the latent Markov chain without utilizing the uniformization method. To retrieve the optimal posterior path of the latent Markov chain, i.e. the maximum a posteriori estimation of changepoint locations, a continuous-time version of Viterbi algorithm (CT-Viterbi) is proposed. The set of changepoint locations is obtainable either from the CT-Viterbi algorithm or the posterior samples of the latent Markov chain. The number of changepoints is determined according to a modified BIC criterion tailored particularly for the multiple changepoint problems of a marked Poisson process. We then perform a simulation study to demonstrate the methods. The methods are applied to investigate the temporal variabilities of seismicity rates and the magnitude-frequency distributions of medium size deep earthquakes in New Zealand.

On the estimation of misclassification probabilities of chronic kidney disease using continuous time hidden Markov models

Introduction: The severity of chronic kidney disease (CKD) is reflected in the form of stages of CKD and can be decided on the basis of estimated glomerular filtration rate (eGFR). The computation of eFGR may have computational and measurement errors which may lead to misclassification of stages. Objectives: Estimation of transition rates, mean sojourn times, probabilities of misclassification of stages and odds ratios for misclassification probabilities. Patients and Methods: The retrospective data of 117 patients suffering from CKD during the period March 2006 to October 2016 is used. Hidden Markov model (HMM) with continuous time has been developed to present the course of progression of CKD into various stages. Results: Under the HMM, the estimated transition intensity corresponding to transition from stage 1 to stage 2 is 0.0405 and reverse transition intensities are zero. The estimated mean sojourn time corresponding to stage 1, stage 2, stage 3 and stage 4 are 15.923 years, 11.976 years, 7.936 years and 2.890 years respectively. The probability of a CKD patient with stage 1 of disease will be misclassified as a patient of stage 2 is 0.211. The odds ratios for misclassification probabilities in the presence of prognostic factors are computed. The probability of misclassification corresponding to the observed stage 2 given the true stage 1 for females is approximately 3.8 times more than that of males. Conclusion: For CKD having progression in stages, the HMM is an appropriate model to draw the course of progression of the disease.

Exponential Hidden Markov Models for <inline-formula> <tex-math notation="LaTeX">${H} _\infty$ </tex-math> </inline-formula> Control of Jumping Systems

This letter addresses the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -control problem for continuous-time Markov jump linear systems where the current mode of operation is estimated by an imperfect detector. Different from existing approaches, the joint jump process is defined in this letter by a continuous-time hidden Markov model, with the hidden process representing the dynamics of the real system while the emitted signal is represented by the information emitted from the detector. We consider static output feedback controllers which depend only on the information emitted from the detector and that stabilize the closed loop system in the mean-square sense. We develop a linear matrix inequality optimization problem which minimizes an upper bound for the H∞ norm of the closed-loop system in order to design our static output feedback controller. This letter is concluded with a numerical example.

$H_2$-Control of Continuous-Time Hidden Markov Jump Linear Systems

This technical note addresses the $H_2$ -control problem for a continuous-time Markov Jump Linear System (MJLS) with partial information on the mode of operation. The jumps of the system depend on a continuous-time hidden Markov model (CT-HMM) where the hidden process represents the dynamics of the real system, while the emitted signal represents the information available to the controller. We start by analyzing the stochastic stability control problem with the goal to design a state feedback linear controller that stochastically stabilizes the closed-loop system, relying on the information from the detector. Two dual formulations, based on LMI problems, are derived as a solution for this problem. These results are then extended to the $H_2$ guaranteed cost control problem. The technical note is concluded with a numerical example which illustrates the derived results.

Long-Term b Value Variations of Shallow Earthquakes in New Zealand: A HMM-Based Analysis

The magnitude–frequency relationship is a fundamental statistic in seismology. Customarily, the temporal variations of b values in the magnitude–frequency distribution are demonstrated via “sliding-window” approach. However, the window size is often only tuned empirically, which may cause difficulties in interpretation of b value variability. In this study, a continuous-time hidden Markov model (HMM) is applied to characterize b value variations of New Zealand shallow earthquakes over decades. HMM-based approach to the b value estimation has some appealing properties over the popular sliding-window approach. The estimation of b value is stable over a range of magnitude thresholds, which is ideal for the interpretation of b value variability. The overall b values of medium and large earthquakes across North Island and northern South Island in New Zealand vary roughly at a decade scale. It is noteworthy that periods of low b values are typically associated with the occurrences of major large earthquakes. The overall temporal variations of b values seem prevailing over many grids in space as evidenced by a comparison of spatial b values in many grids made between two periods with low or high b values, respectively. We also carry out a pre-seismic b value analysis for recent Darfield earthquake and Cook Strait swarm. it is suggested that the mainshock rupture is nucleated at the margin of or right at low b value asperities. In addition, short period of pre-seismic b value decrease is observed in both cases. The overall time-varying behavior of b values over decades is an indication of broad scale of time-varying behavior associated with subduction process, probably related to the convergence rate of the plates. The advance in the method of b value estimation will enhance our understanding of earthquake occurrence and may lead to improved risk forecasting.

Filterbased stochastic volatility in continuous-time hidden Markov models

Regime-switching models, in particular Hidden Markov Models (HMMs) where the switching is driven by an unobservable Markov chain, are widely-used in financial applications, due to their tractability and good econometric properties. In continuous time, properties of HMMs with constant and of HMMs with switching volatility can be quite different. To have a realistic model with unobservable Markov chain in continuous time and good econometric properties, a regime-switching model where the volatility depends on the filter for the underlying chain is introduced and the filtering equations are stated. Such models are motivated by agent based social learning models in economics. An approximation result for a fixed information filtration is proved and further motivation is provided by considering social learning arguments. The relation to the switching volatility model is analyzed in detail and a convergence result for the discretized model is given. Econometric properties are illustrated by numerical simulations.

PREDICTIVE MODELING OF CHOLERA OUTBREAKS IN BANGLADESH.

Despite seasonal cholera outbreaks in Bangladesh, little is known about the relationship between environmental conditions and cholera cases. We seek to develop a predictive model for cholera outbreaks in Bangladesh based on environmental predictors. To do this, we estimate the contribution of environmental variables, such as water depth and water temperature, to cholera outbreaks in the context of a disease transmission model. We implement a method which simultaneously accounts for disease dynamics and environmental variables in a Susceptible-Infected-Recovered-Susceptible (SIRS) model. The entire system is treated as a continuous-time hidden Markov model, where the hidden Markov states are the numbers of people who are susceptible, infected, or recovered at each time point, and the observed states are the numbers of cholera cases reported. We use a Bayesian framework to fit this hidden SIRS model, implementing particle Markov chain Monte Carlo methods to sample from the posterior distribution of the environmental and transmission parameters given the observed data. We test this method using both simulation and data from Mathbaria, Bangladesh. Parameter estimates are used to make short-term predictions that capture the formation and decline of epidemic peaks. We demonstrate that our model can successfully predict an increase in the number of infected individuals in the population weeks before the observed number of cholera cases increases, which could allow for early notification of an epidemic and timely allocation of resources.

Modelling the cost-effectiveness of adopting risk-stratified approaches to extended screening intervals in the national diabetic retinopathy screening programme in Scotland.

To assess the cost-effectiveness of adopting risk-stratified approaches to extended screening intervals in the national diabetic retinopathy screening programme in Scotland. A continuous-time hidden Markov model was fitted to national longitudinal screening data to derive transition probabilities between observed non-referable and referable retinopathy states. These were incorporated in a decision model simulating progression, costs and visual acuity outcomes for a synthetic cohort with a covariate distribution matching that of the Scottish diabetic screening population. The cost-effectiveness of adopting extended (2-year) screening for groups with no observed retinopathy was then assessed over a 30-year time horizon. Individuals with a current grade of no retinopathy on two consecutive screening episodes face the lowest risk of progressing to referable disease. For the cohort as a whole, the incremental cost per quality-adjusted life year gained for annual vs. biennial screening ranged from approximately £74 000 (for those with no retinopathy and a prior observed grade of mild or observable background retinopathy) to approximately £232 000 per quality-adjusted life year gained (for those with no retinopathy on two consecutive screening episodes). The corresponding incremental cost-effectiveness ratios in the subgroup with Type 1 diabetes were substantially lower; approximately £22 000 to £85 000 per quality-adjusted life year gained, respectively. Biennial screening for individuals with diabetes who have no retinopathy is likely to deliver significant savings for a very small increase in the risk of adverse visual acuity and quality of life outcomes. There is greater uncertainty regarding the long-term cost-effectiveness of adopting biennial screening in younger people with Type 1 diabetes.

A continuous-time HMM approach to modeling the magnitude-frequency distribution of earthquakes

ABSTRACTThe magnitude-frequency distribution (MFD) of earthquake is a fundamental statistic in seismology. The so-called b-value in the MFD is of particular interest in geophysics. A continuous time hidden Markov model (HMM) is proposed for characterizing the variability of b-values. The HMM-based approach to modeling the MFD has some appealing properties over the widely used sliding-window approach. Often, large variability appears in the estimation of b-value due to window size tuning, which may cause difficulties in interpretation of b-value heterogeneities. Continuous-time hidden Markov models (CT-HMMs) are widely applied in various fields. It bears some advantages over its discrete time counterpart in that it can characterize heterogeneities appearing in time series in a finer time scale, particularly for highly irregularly-spaced time series, such as earthquake occurrences. We demonstrate an expectation–maximization algorithm for the estimation of general exponential family CT-HMM. In parallel with discrete-time hidden Markov models, we develop a continuous time version of Viterbi algorithm to retrieve the overall optimal path of the latent Markov chain. The methods are applied to New Zealand deep earthquakes. Before the analysis, we first assess the completeness of catalogue events to assure the analysis is not biased by missing data. The estimation of b-value is stable over the selection of magnitude thresholds, which is ideal for the interpretation of b-value variability.

The use of statistical methodology to determine the accuracy of grading within a diabetic retinopathy screening programme.

AimsWe aimed to use longitudinal data from an established screening programme with good quality assurance and quality control procedures and a stable well‐trained workforce to determine the accuracy of grading in diabetic retinopathy screening.MethodsWe used a continuous time‐hidden Markov model with five states to estimate the probability of true progression or regression of retinopathy and the conditional probability of an observed grade given the true grade (misclassification). The true stage of retinopathy was modelled as a function of the duration of diabetes and HbA1c.ResultsThe modelling dataset consisted of 65 839 grades from 14 187 people. The median number [interquartile range (IQR)] of examinations was 5 (3, 6) and the median (IQR) interval between examinations was 1.04 (0.99, 1.17) years. In total, 14 227 grades (21.6%) were estimated as being misclassified, 10 592 (16.1%) represented over‐grading and 3635 (5.5%) represented under‐grading. There were 1935 (2.9%) misclassified referrals, 1305 were false‐positive results (2.2%) and 630 were false‐negative results (11.0%). Misclassification of background diabetic retinopathy as no detectable retinopathy was common (3.4% of all grades) but rarely preceded referable maculopathy or retinopathy.ConclusionMisclassification between lower grades of retinopathy is not uncommon but is unlikely to lead to significant delays in referring people for sight‐threatening retinopathy.

A Bayesian model of psychosis symptom trajectory in Alzheimer's disease.

Psychosis, like other neuropsychiatric symptoms of dementia, has many features that make predictive modeling of its onset difficult. For example, psychosis onset is associated with both the absolute degree of cognitive impairment and the rate of cognitive decline. Moreover, psychotic symptoms, while more likely than not to persist over time within individuals, may remit and recur. To facilitate predictive modeling of psychosis for personalized clinical decision making, including evaluating the role of risk genes in its onset, we have developed a novel Bayesian model of the dual trajectories of cognition and psychosis symptoms. Cognition was modeled as a four-parameter logistic curve with random effects for all four parameters and possible covariates for the rate and time of fall. Psychosis was modeled as a continuous-time hidden Markov model with a latent never-psychotic class and states for pre-psychotic, actively psychotic and remitted psychosis. Covariates can affect the probability of being in the never-psychotic class. Covariates and the level of cognition can affect the transition rates for the hidden Markov model. The model characteristics were confirmed using simulated data. Results from 434 AD patients show that a decline in cognition is associated with an increased rate of transition to the psychotic state. The model allows declining cognition as an input for psychosis prediction, while incorporating the full uncertainty of the interpolated cognition values. The techniques used can be used in future genetic studies of AD and are generalizable to the study of other neuropsychiatric symptoms in dementia.