Continuous-time Markov Chain Approach Research Articles

Stochastic Model of Pneumonia and Meningitis Co-infection Using Continuous Time Markov Chain Approach

Pneumonia disease is a lung infection caused by Streptococcus pneumoniae. Meningitis is an infection of the meninges and cerebrospinal fluid caused by Streptococcus pneumoniae. Both diseases may occur at the same time. A mathematical model is needed to represent the spread of pneumonia and meningitis co-infection. This study aims to build the stochastic model of pneumonia and meningitis co-infection with CTMC, determine the transition and outbreak probability, and conduct simulations to assess the effect of increasing the contact rate on pneumonia and meningitis . Based on the computer simulation undertaken, it can be concluded that if was decreased while was set to be fixed, the probability of disease outbreak decreased. If was set to be fixed while was decreased, the probability of disease outbreak decreased. However, the latter is smaller than the previous. Similarly, if was increased while was set to be fixed, the probability of disease outbreak increased. If was set to be fixed while was increased, the probability of disease outbreak increased. However, the latter is smaller than the previous. Moreover, if both and were decreased, the probability of disease outbreak was equal to zero.

A multistate modeling approach for organizational cybersecurity exploration and exploitation

This study examines the dynamic stages of exploration and exploitation efforts by organizations in their cybersecurity responses using multistate modeling. Using textual data from the annual 10-K reports of S&P 100 organizations, this study uses a combination of text analytics and Markov chain approach to quantify exploration and exploitation in organizational cybersecurity responses. The study models two and four states of exploration and exploitation based on their cybersecurity responses over time and uses a continuous-time Markov chain approach to analyze transitions between states as organizations adapt their responses over time to achieve ambidexterity. The two-state Markov model focuses on the firm-level Exploration and Exploitation states whereas the four-state model captures deeper levels of exploration and exploitation by considering Surviving, Investigating, Reinforcing, and Balancing as possible states for exploration and exploitation of cybersecurity responses. We analyze the dynamics of organizational exploration-exploitation behaviors by modeling longitudinal transition probabilities across different states. Implications for research and practice are discussed.

Modelling decay in effectiveness for evaluation of behaviour change interventions: a tutorial for public health economists

Background and purpose Recent methodological reviews of evaluations of behaviour change interventions in public health have highlighted that the decay in effectiveness over time has been mostly overlooked, potentially leading to suboptimal decision-making. While, in principle, discrete-time Markov chains—the most commonly used modelling approach—can be adapted to account for decay in effectiveness, this framework inherently lends itself to strong model simplifications. The application of formal and more appropriate modelling approaches has been supported, but limited progress has been made to date. The purpose of this paper is to encourage this shift by offering a practical guide on how to model decay in effectiveness using a continuous-time Markov chain (CTMC)-based approach.MethodsA CTMC approach is demonstrated, with a contextualized tutorial being presented to facilitate learning and uptake. A worked example based on the stylized case study in physical activity promotion is illustrated with accompanying R code.DiscussionThe proposed framework presents a relatively small incremental change from the current modelling practice. CTMC represents a technical solution which, in absence of relevant data, allows for formally testing the sensitivity of results to assumptions regarding the long-term sustainability of intervention effects and improving model transparency.Conclusions The use of CTMC should be considered in evaluations where decay in effectiveness is likely to be a key factor to consider. This would enable more robust model-based evaluations of population-level programmes to promote behaviour change and reduce the uncertainty surrounding the decision to invest in these public health interventions.

Inventory Rationing on a One-for-One Inventory Model with Backorders and Lost Sales

In this study, we are primarily motivated by the research problem of recognizing heterogeneous customer behavior towards waiting for order fulfillment under the 'threshold rationing policy' (also known as the 'critical level policy'), and aim to find its effect on system stock levels and performance measures. We assume a continuous review one-for-one ordering policy with generally distributed lead times. In the first model, we consider the case in which low-priority customer class exhibits zero patience for waiting if the demand is not satisfied immediately (a lost sale), while the demand of high-priority customer class can be backordered. This is the first study in the literature to consider this model. We provide an exact analysis for the derivation of the steady-state probability distribution and the average infinite horizon cost per unit time. We then develop an efficient procedure to minimize the average expected cost rate by deriving bounds on the optimal cost, which reduces the enumeration space considerably. In the numerical study, we show the relative savings achieved by implementing the proposed threshold rationing policy over some of the traditional industry practices. In the second model, we study the opposite case in which the high-priority customer class exhibits zero patience for waiting. The existing study in the literature uses simulation to demonstrate that the performance of the critical level policy is robust with respect to the form and variability of the lead time distribution provided the mean lead times are identical. Then it proposes using the Continuous-Time Markov Chain (CTMC) equivalent of the model as an approximation. In this study, we establish a theoretical basis for the rationale of using the CTMC approach as an approximation. We show that under certain assumptions the steady-state probabilities of the system with generally distributed lead times are identical to the steady-state probabilities of the CTMC system with the same mean. This result enables us to link the dynamics of the studied model to the CTMC model, which then allows us to provide a theoretical explanation to the empirically observed phenomenon that why the steady-state probabilities and performance measures are near-insensitive to the form and variability of the lead time distribution as long as mean lead times are identical.

Numerical Validation of the Continuous Time Markov Chain Approximation as a Service Level Estimation Method for Heterogeneous Customer Demand Classes

In this article, we study a threshold level based inventory allocation and replenishment problem for two priority-demand classes under the lot-for-lot replenishment policy. Demands of low-priority class are backordered when on-hand stock is at or below a certain threshold level, while high-priority class demands are lost in case of stock-out situations. We assume a continuous review lot-for-lot ordering policy with both deterministic and stochastic replenishment lead times. Both priority classes exhibit mutually independent, stationary, Poisson demand processes. There is no exact solution yet in the literature except for the special case of exponentially distributed lead times. In this complementary study, we aim to numerically validate the Continuous-Time Markov Chain (CTMC) approximation as a service level approximation method by thoroughly analyzing the effect of the type of lead time distribution, lead time variability, and value of system parameters on the quality of the class-specific service level estimations. As long as expected lead times are identical, we also show empirically the near-insensitivity of the effect of the type of lead time distribution and the lead time variability on the achieved service levels. The CTMC approximation works well under variety of system parameters and lead time distributions. For the cases considered in the constant lead time scenarios with service levels which we expect to see in practice (with fill rate requirements at least 60 % and 90 % for the low-priority and high priority customer classes, respectively), the average absolute errors for the CTMC approximations are calculated as 0.09 % (for the low-priority class fill rate) and 0.05 % (for the high-priority class fill rate); while the maximum absolute errors for the CTMC approximations are calculated as 0.37 % (for the low-priority fill rate) and 0.18 % (for the high-priority fill rate). The CTMC approach also provides quality approximations for the Erlang, lognormal and gamma distributed lead times.

A continuous-time Markov chain approach with the analytic likelihood in studies of behavioral changes

We develop an analytic likelihood approach for a four-state CTMC by solving the backwards Kolmogorov differential equations, reducing this bias in transition rate estimates. A simulation study is performed to assess the performance of this new method and confirms that it achieves good coverage probabilities with low bias and standard errors. Finally, we analyzed data from Project SUCCESS to estimate the study each participant’s transitions among behavioral stage, consisting of risky drinking and possible ineffective using of contraception, which comprise the primary endpoint of risk of an alcohol-exposed pregnancy.

A Stochastic Tick-Borne Disease Model: Exploring the Probability of Pathogen Persistence.

We formulate and analyse a stochastic epidemic model for the transmission dynamics of a tick-borne disease in a single population using a continuous-time Markov chain approach. The stochastic model is based on an existing deterministic metapopulation tick-borne disease model. We compare the disease dynamics of the deterministic and stochastic models in order to determine the effect of randomness in tick-borne disease dynamics. The probability of disease extinction and that of a major outbreak are computed and approximated using the multitype Galton-Watson branching process and numerical simulations, respectively. Analytical and numerical results show some significant differences in model predictions between the stochastic and deterministic models. In particular, we find that a disease outbreak is more likely if the disease is introduced by infected deer as opposed to infected ticks. These insights demonstrate the importance of host movement in the expansion of tick-borne diseases into new geographic areas.

Hybrid spectrum access with relay assisting both primary and secondary networks under imperfect spectrum sensing

This paper proposes a novel hybrid interweave-underlay spectrum access for a cognitive amplify-and-forward relay network where the relay forwards the signals of both the primary and secondary networks. In particular, the secondary network (SN) opportunistically operates in interweave spectrum access mode when the primary network (PN) is sensed to be inactive and switches to underlay spectrum access mode if the SN detects that the PN is active. A continuous-time Markov chain approach is utilized to model the state transitions of the system. This enables us to obtain the probability of each state in the Markov chain. Based on these probabilities and taking into account the impact of imperfect spectrum sensing of the SN, the probability of each operation mode of the hybrid scheme is obtained. To assess the performance of the PN and SN, we derive analytical expressions for the outage probability, outage capacity, and symbol error rate over Nakagami-m fading channels. Furthermore, we present comparisons between the performance of underlay cognitive cooperative radio networks (CCRNs) and the performance of the considered hybrid interweave-underlay CCRN in order to reveal the advantages of the proposed hybrid spectrum access scheme. Eventually, with the assistance of the secondary relay, performance improvements for the PN are illustrated by means of selected numerical results.

Computationally efficient optimization of stock pooling and allocation levels for two-demand-classes under general lead time distributions

ABSTRACTIn this article we develop a procedure for estimating service levels (fill rates) and for optimizing stock and threshold levels in a two-demand-class model managed based on a lot-for-lot replenishment policy and a static threshold allocation policy. We assume that the priority demand classes exhibit mutually independent, stationary, Poisson demand processes and non-zero order lead times that are independent and identically distributed. A key feature of the optimization routine is that it requires computation of the stationary distribution only once. There are two approaches extant in the literature for estimating the stationary distribution of the stock level process: a so-called single-cycle approach and an embedded Markov chain approach. Both approaches rely on constant lead times. We propose a third approach based on a Continuous-Time Markov Chain (CTMC) approach, solving it exactly for the case of exponentially distributed lead times. We prove that if the independence assumption of the embedded Markov chain approach is true, then the CTMC approach is exact for general lead time distributions as well. We evaluate all three approaches for a spectrum of lead time distributions and conclude that, although the independence assumption does not hold, both the CTMC and embedded Markov chain approaches perform well, dominating the single-cycle approach. The advantages of the CTMC approach are that it is several orders of magnitude less computationally complex than the embedded Markov chain approach and it can be extended in a straightforward fashion to three demand classes.

Model and performance analysis for a repair system with multiple rotable components based on a continuous-time Markov chain approach

This article presents an analytical continuous-time Markov chain approach to model a repair system with rotable components for resource allocation. As an important type of repair service system, this repair system with rotable components demonstrated that rotable components can be repaired to an as-new condition and then assembled circularly with rotable-component inventories after a piece of equipment is disassembled. However, with improved efficiency, major difficulties arise in formulating a model and analysing its performance due to the complex fork–join structure and particularly the existence of rotable-component inventories. In our study, a continuous-time Markov chain stochastic model is developed for the repair system with a single rotable component, in which the characteristic constraints to generate a state space are classified as the service capacity, quantitative conservation, and independent loading. The stationary distributions are then attained with programming, and the performance analysis is performed through numerical experiments. In particular, the relationship between the key resources and selected performance indicators is studied to provide a strategy for resource allocation. Furthermore, a multi-component system is also presented based on the proposed method.

Modeling DNA methylation dynamics with approaches from phylogenetics.

Motivation: Methylation of CpG dinucleotides is a prevalent epigenetic modification that is required for proper development in vertebrates. Genome-wide DNA methylation assays have become increasingly common, and this has enabled characterization of DNA methylation in distinct stages across differentiating cellular lineages. Changes in CpG methylation are essential to cellular differentiation; however, current methods for modeling methylation dynamics do not account for the dependency structure between precursor and dependent cell types.Results: We developed a continuous-time Markov chain approach, based on the observation that changes in methylation state over tissue differentiation can be modeled similarly to DNA nucleotide changes over evolutionary time. This model explicitly takes precursor to descendant relationships into account and enables inference of CpG methylation dynamics. To illustrate our method, we analyzed a high-resolution methylation map of the differentiation of mouse stem cells into several blood cell types. Our model can successfully infer unobserved CpG methylation states from observations at the same sites in related cell types (90% correct), and this approach more accurately reconstructs missing data than imputation based on neighboring CpGs (84% correct). Additionally, the single CpG resolution of our methylation dynamics estimates enabled us to show that DNA sequence context of CpG sites is informative about methylation dynamics across tissue differentiation. Finally, we identified genomic regions with clusters of highly dynamic CpGs and present a likely functional example. Our work establishes a framework for inference and modeling that is well suited to DNA methylation data, and our success suggests that other methods for analyzing DNA nucleotide substitutions will also translate to the modeling of epigenetic phenomena.Availability and implementation: Source code is available at www.kostkalab.net/software.Contact: tony.capra@vanderbilt.edu or kostka@pitt.edu

Stochastic analysis of epidemics on adaptive time varying networks

Many studies investigating the effect of human social connectivity structures (networks) and human behavioral adaptations on the spread of infectious diseases have assumed either a static connectivity structure or a network which adapts itself in response to the epidemic (adaptive networks). However, human social connections are inherently dynamic or time varying. Furthermore, the spread of many infectious diseases occur on a time scale comparable to the time scale of the evolving network structure. Here we aim to quantify the effect of human behavioral adaptations on the spread of asymptomatic infectious diseases on time varying networks. We perform a full stochastic analysis using a continuous time Markov chain approach for calculating the outbreak probability, mean epidemic duration, epidemic reemergence probability, etc. Additionally, we use mean-field theory for calculating epidemic thresholds. Theoretical predictions are verified using extensive simulations. Our studies have uncovered the existence of an "adaptive threshold," i.e., when the ratio of susceptibility (or infectivity) rate to recovery rate is below the threshold value, adaptive behavior can prevent the epidemic. However, if it is above the threshold, no amount of behavioral adaptations can prevent the epidemic. Our analyses suggest that the interaction patterns of the infected population play a major role in sustaining the epidemic. Our results have implications on epidemic containment policies, as awareness campaigns and human behavioral responses can be effective only if the interaction levels of the infected populace are kept in check.

Optimal Consumption and Insurance: A Continuous-time Markov Chain Approach

Personal financial decision making plays an important role in modern finance. Decision problems about consumption and insurance are in this article modelled in a continuous-time multi-state Markovian framework. The optimal solution is derived and studied. The model, the problem, and its solution are exemplified by two special cases: In one model the individual takes optimal positions against the risk of dying; in another model the individual takes optimal positions against the risk of losing income as a consequence of disability or unemployment.

Optimal Consumption and Insurance: A Continuous-time Markov Chain Approach

Personal financial decision making plays an important role in modern finance. Decision problems about consumption and insurance are in this article modelled in a continuous-time multi-state Markovian framework. The optimal solution is derived and studied. The model, the problem, and its solution are exemplified by two special cases: In one model the individual takes optimal positions against the risk of dying; in another model the individual takes optimal positions against the risk of losing income as a consequence of disability or unemployment.

Optimal Consumption and Insurance: A Continuous-Time Markov Chain Approach

Decision problems about consumption and insurance are modelled in a continuous time multistate Markovian framework. The optimal solution is derived and studied. The model, the problem, and its solution are exemplified by two special cases: In one model the individual takes optimal positions against the risk of dying; in another model the individual takes optimal positions against the risk of loosing income as consequence of disability or unemployment.