Continuous-time Markov Model Research Articles

Analysis of the dynamics of transition from non-colonization to colonization and Staphylococcus aureus bacteremia in hemodialysis patients using Markov models.

Background Hemodialysis patients are frequently colonized by Staphylococcus aureus, leading to severe infections with high mortality rates. However, little is known about transition from non-colonization to colonization or bacteremia over time. The aim was to analyze the behavior of S. aureus colonization, identifying the probability of transition from non-colonized to colonized state or bacteremia, and the influence of specific covariates. Methods The study was conducted in a dialysis unit associated with a tertiary care hospital in Medellín between October 2017 and October 2019. An initial measurement was taken to evaluate S. aureus colonization, and follow-up measurements were performed 2 and 6 months later. Bacteremia evolution was monitored for 12 months. A two-state recurrent continuous-time Markov model was constructed to model transition dynamics from non-colonization to S. aureus colonization in hemodialysis patients. Subsequently, the model was applied to a third state of bacteremia. Results Of 178 patients on hemodialysis, 30.3% were colonized by S. aureus. Transition intensity from non-colonization to colonization was three times higher (0.21; CI: 0.14-0.29) than from colonization to non-colonization (0.07; CI: 0.05-0.11). The colonization risk increased in patients with previous infections (HR: 2.28; CI: 0.78-6.68), hospitalization (HR: 1.29; CI: 0.56-2.99) and antibiotics consumption (HR: 1.17; CI: 0.53-2.58). Mean non-colonized state duration was 10.9 months, while in the colonized state was 5.2 months. In the 3-state model, it was found that patients colonized were more likely to develop S. aureus infection (13.9%). Conclusion A more likely transition from non-colonization to colonization was found, which increases with factors such as previous infection. In addition, the development of bacteremia was more likely in colonized than in non-colonized patients. These results underline the importance of surveillance and proper management of S. aureus colonization to prevent serious complications, such as bacteremia, and improve prognosis in this vulnerable population.

Closed-form approximations of moments and densities of continuous–time Markov models

This paper develops power series expansions of a general class of moment functions, including transition densities and option prices, of continuous-time Markov processes, including jump–diffusions. The proposed expansions extend the ones in Kristensen and Mele (2011) to cover general Markov processes, and nest transition density and option price expansions recently developed in the literature, thereby connecting seemingly different ideas in a unified framework. We show how the general expansion can be implemented for fully general jump–diffusion models. We provide a new theory for the validity of the expansions which shows that series expansions are not guaranteed to converge as more terms are added in general once the time span of interest gets larger than some model–specific threshold. Thus, these methods should be used with caution when applied to problems with a larger time span of interest, such as long-term options or data observed at a low frequency. At the same time, the numerical studies in this paper demonstrate good performance of the proposed implementation in practice when applied to pricing options with time to maturity below three months. Thus, our expansions are particularly well suited for pricing ultra-short-term (such as “zero–day”) options.

Profiling the sleep architecture of ageing adults using a seven-state continuous-time Markov model.

Sleep is a complex biological process regulated by networks of neurons and environmental factors. As one falls asleep, neurotransmitters from sleep-wake regulating neurones work in synergy to control the switching of different sleep states throughout the night. As sleep disorders or underlying neuropathology can manifest as irregular switching, analysing these patterns is crucial in sleep medicine and neuroscience. While hypnograms represent the switching of sleep states well, current analyses of hypnograms often rely on oversimplified temporal descriptive statistics (TDS, e.g., total time spent in a sleep state), which miss the opportunity to study the sleep state switching by overlooking the complex structures of hypnograms. In this paper, we propose analysing sleep hypnograms using a seven-state continuous-time Markov model (CTMM). This proposed model leverages the CTMM to depict the time-varying sleep-state transitions, and probes three types of insomnia by distinguishing three types of wake states. Fitting the proposed model to data from 2056 ageing adults in the Multi-Ethnic Study of Atherosclerosis (MESA) Sleep study, we profiled sleep architectures in this population and identified the various associations between the sleep state transitions and demographic factors and subjective sleep questions. Ageing, sex, and race all show distinctive patterns of sleep state transitions. Furthermore, we also found that the perception of insomnia and restless sleep are significantly associated with critical transitions in the sleep architecture. By incorporating three wake states in a continuous-time Markov model, our proposed method reveals interesting insights into the relationships between objective hypnogram data and subjective sleep quality assessments.

Estimation of Transition Probabilities from a Large Cohort (> 6000) of Australians Living with Multiple Sclerosis (MS) for Changing Disability Severity Classifications, MS Phenotype, and Disease-Modifying Therapy Classifications.

Multiple sclerosis (MS) is a chronic autoimmune/neurodegenerative disease associated with progressing disability affecting mostly women. We aim to estimate transition probabilities describing MS-related disability progression from no disability to severe disability. Transition probabilities are a vital input for health economics models. In MS, this is particularly relevant for pharmaceutical agency reimbursement decisions for disease-modifying therapies (DMTs). Data were obtained from Australian participants of the MSBase registry. We used a four-state continuous-time Markov model to describe how people with MS transition between disability milestones defined by the Expanded Disability Status Scale (scale 0-10): no disability (EDSS of 0.0), mild (EDSS of 1.0-3.5), moderate (EDSS of 4.0-6.0), and severe (EDSS of 6.5-9.5). Model covariates included sex, DMT usage, MS-phenotype, and disease duration, and analysis of covariate groups were also conducted. All data were recorded by the treating neurologist. A total of N = 6369 participants (mean age 42.5 years, 75.00% female) with 38,837 person-years of follow-up and 54,570 clinical reviews were identified for the study. Annual transition probabilities included: remaining in the no, mild, moderate, and severe states (54.24%, 82.02%, 69.86%, 77.83% respectively) and transitioning from no to mild (42.31%), mild to moderate (11.38%), and moderate to severe (9.41%). Secondary-progressive MS was associated with a 150.9% increase in the hazard of disability progression versus relapsing-remitting MS. People with MS have an approximately 45% probability of transitioning from the no disability state after one year, with people with progressive MS transitioning from this health state at a much higher rate. These transition probabilities will be applied in a publicly available health economics simulation model for Australia and similar populations, intended to support reimbursement of a plethora of existing and upcoming interventions including medications to reduce progression of MS.

A Markov method for a mechanical system reliability assessment using discrete degradation data

Abstract A homogenous continuous-time Markov model (HCTMM) has the advantage of high accuracy and is usually used in reliability analysis. However, the transition intensities (rates) between any two states cannot be accurately obtained due to the discrete data obtained from a mechanical system and little expert knowledge. This paper proposes a method in which the transition intensity matrix of an HCTMM can be obtained indirectly using the actual discrete degradation data of a mechanical system. Firstly, the one-step transition probability matrix can be calculated by a hidden Markov model using the discrete degradation data. Secondly, the unit-time transition intensity matrix can be calculated from the one-step transition probability matrix. At last, an HCTMM is established to assess the reliability of a mechanical system based on the unit-time transition intensity matrix. Moreover, considering the maintenance activities cannot restore a mechanical system as a fire-new one, the random transition of a mechanical system after maintenance is expressed by a quasi-renewal process in this paper. Finally, three cases are studied to validate the proposed method.

Analysis of the dynamics of transition from non-colonization to colonization and Staphylococcus aureus bacteremia in hemodialysis patients using Markov models.

Background Hemodialysis patients are frequently colonized by Staphylococcus aureus, leading to severe infections with high mortality rates. However, little is known about transition from non-colonization to colonization or bacteremia over time. The aim was to analyze the behavior of S. aureus colonization, identifying the probability of transition from non-colonized to colonized state or bacteremia, and the influence of some covariates. Methods The study was conducted in a dialysis unit associated with a high complexity hospital in Medellín between October 2017 and October 2019. An initial measurement was taken to evaluate S. aureus colonization, and follow-up measurements were performed 2 and 6 months later. Bacteremia evolution was monitored for 12 months. A two-state recurrent continuous-time Markov model was constructed to model transition dynamics from non-colonization to S. aureus colonization in hemodialysis patients. Subsequently, the model was applied to a third state of bacteremia. Results Of 178 patients on hemodialysis, 30.3% were colonized by S. aureus. Transition intensity from non-colonization to colonization was three times higher (0.21; CI: 0.14-0.29) than from colonization to non-colonization (0.07; CI: 0.05-0.11). The colonization risk increased in patients with previous infections (HR: 2.28; CI: 0.78-6.68), hospitalization (HR: 1.29; CI: 0.56-2.99) and antibiotics consumption (HR: 1.17; CI: 0.53-2.58). Mean non-colonized state duration was 10.9 months, while in the colonized state was 5.2 months. In the 3-state model, it was found that patients colonized were more likely to develop S. aureus infection (13.9%). Conclusion A more likely transition from non-colonization to colonization was found, which increases with factors such as previous infection. In addition, the development of bacteremia was more likely in colonized than in non-colonized patients. These results underline the importance of surveillance and proper management of S. aureus colonization to prevent serious complications, such as bacteremia, and improve prognosis in this vulnerable population.

Continuous Markov Models for Analyzing the Effect of Environmental Personal Exposure on Multiple Sclerosis Progression.

Multiple sclerosis (MS) is an inflammatory autoimmune demyelinating disorder of the central nervous system, leading to progressive functional impairments. Predicting disease progression with a probabilistic and time-dependent approach might help suggest interventions for a better management of the disease. Recently, there has been increasing focus on the impact of air pollutants as environmental factors influencing disease progression. This study employs a Continuous-Time Markov Model (CMM) to explore the impact of air pollution measurements on MS progression using longitudinal data from MS patients in Italy between 2013 and 2022. Preliminary findings indicate a relationship between air pollution and MS progression, with pollutants like Particulate Matter with a diameter of 10 micrometers (PM10) or 2.5 micrometers (PM2.5), Nitrogen Dioxide (NO2), and Carbon Monoxide (CO) showing potential effects on disease activity.

The effect of weight gain and metabolic dysfunction-associated steatotic liver disease on liver fibrosis progression and regression in people with HIV.

People with HIV (PWH) have high risk of liver fibrosis. We investigated the effect of weight gain and metabolic dysfunction-associated steatotic liver disease (MASLD) on liver fibrosis dynamics. Multicenter cohort study. Fibrosis progression was defined as development of significant fibrosis [liver stiffness measurement (LSM) ≥8 kPa], or transition to cirrhosis (LSM ≥13 kPa), for those with significant fibrosis at baseline. Fibrosis regression was defined as transition to LSM less than 8 kPa, or to LSM less than 13 kPa for those with cirrhosis at baseline. MASLD was defined as hepatic steatosis (controlled attenuation parameter >248 dB/m) with at least one metabolic abnormality. A continuous-time multistate Markov model was used to describe transitions across fibrosis states. Among 1183 PWH included from three centers (25.2% with viral hepatitis coinfection), baseline prevalence of significant fibrosis and MASLD was 14.4 and 46.8%, respectively. During a median follow-up of 2.5 years (interquartile range 1.9-3.5), the incidence rate of fibrosis progression and regression was 2.8 [95% confidence interval (CI) 2.3-3.4] and 2.2 (95% CI 1.9-2.6) per 100 person-years, respectively. In Markov model, weight gain increased the odds of fibrosis progression [odds ratio (OR) 3.11, 95% CI 1.59-6.08], whereas weight gain (OR 0.30, 95% CI 0.10-0.84) and male sex (OR 0.32, 95% CI 0.14-0.75) decreased the odds of fibrosis regression. On multivariable Cox regression analysis, predictors of fibrosis progression were weight gain [adjusted hazard ratio (aHR) 3.12, 95% CI 1.41-6.90] and MASLD (aHR 2.72, 95% CI 1.05-7.02). Fibrosis transitions are driven by metabolic health variables in PWH, independently of viral hepatitis coinfection and antiretroviral class therapy.

“PEOPLE” MEET “MARKOVIANS” — INDIVIDUAL-BASED MODELING WITH HYBRID STOCHASTIC SYSTEMS

Individual-based models (IBMs) enable modelers to avoid far-reaching abstractions and strong simplifications by allowing for a state-based representation of individuals. The fact that IBMs are not represented using a standardized mathematical framework such as differential equations makes it harder to reproduce IBMs and introduces difficulties in the analysis of IBMs. We propose a model architecture based on representing individuals via Markov models. Individuals are coupled to populations — for which individuals are not explicitly represented — that are modeled by differential equations. The resulting models consisting of continuous-time finite-state Markov models coupled to systems of differential equations are examples of piecewise-deterministic Markov processes (PDMPs). We will demonstrate that PDMPs, also known as hybrid stochastic systems, allow us to design detailed state-based representations of individuals which, at the same time, can be systematically analyzed by taking advantage of the theory of PDMPs. We will illustrate design and analysis of IBMs using PDMPs via the example of a predator that intermittently feeds on a logistically growing prey by stochastically switching between a resting and a feeding state. This simple model shows a surprisingly rich dynamics which, nevertheless, can be comprehensively analyzed using the theory of PDMPs.

Bayesian learning for the robust verification of autonomous robots

Autonomous robots used in infrastructure inspection, space exploration and other critical missions operate in highly dynamic environments. As such, they must continually verify their ability to complete the tasks associated with these missions safely and effectively. Here we present a Bayesian learning framework that enables this runtime verification of autonomous robots. The framework uses prior knowledge and observations of the verified robot to learn expected ranges for the occurrence rates of regular and singular (e.g., catastrophic failure) events. Interval continuous-time Markov models defined using these ranges are then analysed to obtain expected intervals of variation for system properties such as mission duration and success probability. We apply the framework to an autonomous robotic mission for underwater infrastructure inspection and repair. The formal proofs and experiments presented in the paper show that our framework produces results that reflect the uncertainty intrinsic to many real-world systems, enabling the robust verification of their quantitative properties under parametric uncertainty.

Reliability analysis of three state linear consecutive 2-out-of-4:F system by continuous time Markov model

Reliability analysis of three state linear consecutive 2-out-of-4:F system by continuous time Markov model

Reliability analysis of three state linear consecutive 2-out-of-4:F system by continuous time Markov model

Reliability analysis of three state linear consecutive 2-out-of-4:F system by continuous time Markov model

PREDICTIVE VALUE OF NUTRITIONAL INDICES FOR SARCOPENIA TRANSITIONS IN MIDDLE-AGED AND OLDER ADULTS

Abstract Since malnutrition plays an important role in sarcopenia development, we aimed to investigate sarcopenia’s transitional dynamics and explore value of nutritional indices for predicting state transitions in middle-aged and older Asian adults. We analyzed data from 1910 participants in the WCHAT study who had complete information at baseline (2018) and at least one follow-up between 2019-2022. We defined the normal, sarcopenia, and intermediate subclinical states. Using a continuous-time multistate Markov model, we calculated transition probabilities and examined associations between nutritional indices and transitions. The probabilities of remaining stable versus progressing to the subclinical state in normal individuals were 53.4% versus 42.1% and 40.6% versus 49.0% at 2 and 4 years. In the subclinical population, their 2- and 4-year chances were 60.2% and 51.2% for keeping stable, 11.8% and 16.2% for developing sarcopenia, 28.0% and 32.6% for reverting to normal. For sarcopenic individuals, their likelihood of staying stable versus retrogressing to the subclinical state was 67.0% versus 26.3% and 48.3% versus 36.3% at 2 and 4 years. Increased albumin, BMI, calf circumference (CC), geriatric nutrition risk index (GNRI), mid-arm circumference (MAC), prealbumin, and triceps skinfold thickness (TST) was predictive for reversion to earlier states. Conversely, decreased BMI, CC, GNRI, MAC, and TST was associated with progression to sarcopenia. Vitamin D was not significantly associated with any transitions. This study reveals how sarcopenia changes over time and highlights usefulness of simple and cost-effective nutritional indices for predicting state transitions, which can help identify individuals at risk of sarcopenia and guide targeted interventions.

Sarcopenia Transitions and Influencing Factors Among Chinese Older Adults With Multistate Markov Model

Abstract Background and Objectives Little is known about the sarcopenia transition process across different stages among Chinese community-dwelling older adults. We aimed to explore dynamic transitions of sarcopenia and its influencing factors in Chinese older adults. Research Design and Methods Data were derived from the China Health and Retirement Longitudinal Study. A total of 2856 older adults with complete data in the 2011, 2013, and 2015 waves were included in our study. Participants were categorized into 3 states: no sarcopenia, possible sarcopenia, and sarcopenia according to the Asian Working Group for Sarcopenia 2019 (AWGS 2019) criteria. Continuous-time multistate Markov model was performed to estimate the 1-year transition probabilities and the associated factors of sarcopenia transitions. The association strength was expressed as hazard ratio and 95% confidence interval. Results The progression and reversion rates between no sarcopenia and sarcopenia state were 6.01 and 9.20 per 100 person-years, respectively. The 1-year progression probability to possible sarcopenia was higher compared with the likelihood of moving to the sarcopenia state (0.127 vs 0.034). The reversion probability to no sarcopenia was also higher among those with possible sarcopenia (0.281 vs 0.157). Older age, rural living, worse cognition status, higher chronic disease numbers, and lower nutrition status measured by body mass index accelerated the sarcopenia progression. Cognition status and body mass index level were related to higher chances of reverting. Discussion and Implications Possible sarcopenia might be a critical time window to promote sarcopenia reversion. Timely interventions aimed at delaying the progression and facilitating sarcopenia recovery should focus on improving cognitive function and nutrition levels.

Long-Term Preventive Failure Mitigation Strategy For Transformers Based on Markov Method

In this paper, we propose a preventive failure mitigation strategy in the power system based on Markov method. Specifically, we consider multiple units in the system, which are of different types and are managed by a single utility company. To characterize the operation, failure mitigation, and deterioration processes of the equipment, a continuous-time Markov model is formulated. By modelling the failure rate of equipment and the reinstallation rate after failures, the steady state of the proposed Markov model is analytically derived. Then to optimize the long-term net revenue of the utility company, the optimal failure mitigation rate is determined by considering the failure mitigation capacity for each equipment type as well as the overall failure mitigation capacity of the company. The performance of the proposed algorithms is demonstrated with three types of transformers in the simulation.

Transitional Dynamics of Sarcopenia and Associations of Nutritional Indices with State Transitions in Chinese aged ≥ 50.

Sarcopenia's temporal profile can be regarded as a dynamic process with distinct states, in which malnutrition plays an important role. This study aimed to address two research gaps: sarcopenia's transitional dynamics and associations of nutritional indices with state transitions in community-dwelling Chinese adults aged 50 and older. A prospective population-based cohort study. Community-based setting in western China. The analytic sample included data from 1910 participants aged ≥ 50 in the West China Health and Aging Trend study between 2018-2022. We defined three states: the initial normal state (normal muscle strength, physical performance and muscle mass), the worst sarcopenia state (low muscle mass plus low muscle strength and/or low physical performance) and the intermediate subclinical state (the other scenarios). The relevant measurement methods and cut-off points were based on the 2019 AWGS consensus. Using a continuous-time multistate Markov model, we calculated probabilities of transitions between different states over 1, 2 and 4 years; we also examined associations between nutritional indices and transitions, including body mass index (BMI), calf circumference (CC), mid-arm circumference (MAC), triceps skinfold thickness (TST), albumin (ALB), geriatric nutrition risk index (GNRI), vitamin D (VitD) and prealbumin (PA). For individuals in the normal state, their probabilities of remaining stable versus progressing to a subclinical state were 53.4% versus 42.1% at 2 years, and 40.6% versus 49.0% at 4 years. In the subclinical population, their 2- and 4-year chances were 60.2% and 51.2% for maintaining this state, 11.8% and 16.2% for developing sarcopenia, 28.0% and 32.6% for reverting to normal. For sarcopenic individuals, the likelihood of staying stable versus retrogressing to the subclinical state were 67.0% versus 26.3% at 2 years, and 48.3% versus 36.3% at 4 years. Increased BMI, CC, MAC, TST, ALB, GNRI and PA correlated with reversion from the subclinical state, among which increased TST, ALB and PA were also paralleled with reversion from sarcopenia, while decreased BMI, CC, MAC, TST and GNRI were associated with progression to sarcopenia. VitD was not significantly associated with any transitions. This study reveals how sarcopenia changes over time in a Chinese population. It also highlights the usefulness of simple and cost-effective nutritional status indices for indicating state transitions, which can help identify individuals at risk of sarcopenia and guide targeted interventions within the optimal time window.

A scalable approach for continuous time Markov models with covariates.

Existing methods for fitting continuous time Markov models (CTMM) in the presence of covariates suffer from scalability issues due to high computational cost of matrix exponentials calculated for each observation. In this article, we propose an optimization technique for CTMM which uses a stochastic gradient descent algorithm combined with differentiation of the matrix exponential using a Padé approximation. This approach makes fitting large scale data feasible. We present two methods for computing standard errors, one novel approach using the Padé expansion and the other using power series expansion of the matrix exponential. Through simulations, we find improved performance relative to existing CTMM methods, and we demonstrate the method on the large-scale multiple sclerosis NO.MS data set.

The continuous-time hidden Markov model based on discretization. Properties of estimators and applications

In this paper we consider continuous-time hidden Markov processes (CTHMM). The model considered is a two-dimensional stochastic process (X_t,Y_t), with X_t an unobserved (hidden) Markov chain defined by its generating matrix and Y_t an observed process whose distribution law depends on X_t and is called the emission function. In general, we allow the process Y_t to take values in a subset of the q-dimensional real space, for some q. The coupled process (X_t,Y_t) is a continuous-time Markov chain whose generator is constructed from the generating matrix of X and the emission distribution. We study the theoretical properties of this two-dimensional process using a formulation based on semi-Markov processes. Observations of the CTHMM are obtained by discretization considering two different scenarii. In the first case we consider that observations of the process Y are registered regularly in time, while in the second one, observations arrive at random. Maximum-likelihood estimators of the characteristics of the coupled process are obtained in both scenarii and the asymptotic properties of these estimators are shown, such as consistency and normality. To illustrate the model a real-data example and a simulation study are considered.

Probabilistic verification of diagnosability for a certain class of timed stochastic systems

This paper addresses diagnosability for a class of timed stochastic discrete event systems that are modeled with labeled continuous time Markov models. A large variety of fault patterns is considered and the paper aims to define and propose conditions for weak notions of diagnosability of fault patterns in a time stochastic framework. The proposed approach is based on a state isolation perspective. Logical and probabilistic verifiers that result by first taking the product of the system with the fault pattern, followed by the parallel composition of the resulting structure with its logical observer are designed in this perspective. The advantage of this schema is to separate explicitly the system from the pattern we are interested in, as well as from the sensoring and observational aspects. Consequently, the verifiers allow the explicit tracking of the information about the first occurrence of a fault pattern and the preservation of not only the logical information provided by the labeling function but also the timing aspects imposed by the dynamics of the system.

Reliability assessment of performance-based balanced systems with rebalancing mechanisms

Performance-based balanced systems (PBSs) are widely used as critical equipment in many fields, so their reliability assessment has become a hot topic. Taking the battery pack PBS as an example, the balance means the performance differences between components are within an acceptable range, where the common bus is used to redistribute the performance for rebalancing, and the redistribution is affected by the transmission loss and transmission capacity limit. To reflect the above rebalancing mechanisms, a reliability evaluation method is proposed for PBSs with common bus performance sharing (PBSs-CBPS) considering balance degree threshold, transmission loss, and transmission capacity limit. Firstly, a continuous-time discrete-state Markov model is built to address the transition behaviors of components. Next, considering the effects of balance degree threshold and transmission loss, the rebalancing process is formulated as a nonlinear programming problem to obtain the performance of components after rebalancing, and the system reliability model is established. Then, the universal generation function (UGF) method combined with nonlinear programming is proposed to calculate system reliability, and Paper-Cut UGF Algorithm (PCUGF) is proposed to improve computational efficiency. Finally, an analytical example and a numerical example of 6S1P lithium-ion battery packs are given to show the effectiveness of the proposed method.