Multi-state Markov Process Research Articles

Study design for restricted mean time analysis of recurrent events and death.

The restricted mean time in favor (RMT-IF) of treatment has just been added to the analytic toolbox for composite endpoints of recurrent events and death. To help practitioners design new trials based on this method, we develop tools to calculate the sample size and power. Specifically, we formulate the outcomes as a multistate Markov process with a sequence of transient states for recurrent events and an absorbing state for death. The transition intensities, in this case the instantaneous risks of another nonfatal event or death, are assumed to be time-homogeneous but nonetheless allowed to depend on the number of past events. Using the properties of Coxian distributions, we derive the RMT-IF effect size under the alternative hypothesis as a function of the treatment-to-control intensity ratios along with the baseline intensities, the latter of which can be easily estimated from historical data. We also reduce the variance of the nonparametric RMT-IF estimator to calculable terms under a standard set-up for censoring. Simulation studies show that the resulting formulas provide accurate approximation to the sample size and power in realistic settings. For illustration, a past cardiovascular trial with recurrent-hospitalization and mortality outcomes is analyzed to generate the parameters needed to design a future trial. The procedures are incorporated into the rmt package along with the original methodology on the Comprehensive R Archive Network (CRAN).

Anytime Control Under Practical Communication Models

In this article, we investigate a novel anytime control algorithm for wireless networked control with random dropouts. The controller computes sequences of tentative future control commands using time-varying (Markovian) computational resources. The sensor–controller and controller–actuator channel states are spatial- and time-correlated, respectively, and are modeled as a multistate Markov process. To compensate the effect of packet dropouts, a dual-buffer mechanism is proposed. We develop a novel cycle-cost-based approach to obtain the stability conditions on the nonlinear plant, controller, network, and computational resources.

A study on association in time of a finite semi markov process

We derive three equivalent sufficient conditions for association in time of a finite state semi Markov process in terms of transition probabilities and crude hazard rates. This result generalizes the earlier results of Esary and Proschan (1970)for a binary Markov process and Hjort, Natvig and Funnemark (1985) for a multistate Markov process.

A Data Mining Approach for Transformer Failure Rate Modeling Based on Daily Oil Chromatographic Data

Evaluating the real-time failure rate of transformers can effectively guide the planning of maintenance and reduce their failure risk. This paper proposed a novel transformer failure rate model that considers the impact of maintenance based on daily oil chromatographic monitoring data mining. Firstly, to ensure the quality of the modeling data, an improved k-nearest neighbor (KNN) algorithm based on genetic algorithm (GA) is proposed to repair the missing monitoring data. The repaired data is then mapped to the equivalent state duration (ESD) by the M-BPNN proposed, which is used to modify the multistate Markov process of transformers so as to quantify the impact of maintenance on failure rate. Considering the changing characteristics of the dissolved gases’ content in the short period, the ESD is further merged in sequential periods to obtain the merged equivalent state duration (MESD). Finally, an analytical function of the transformer failure rate with respect to the MESD is obtained. Case studies on a typical substation demonstrate that the proposed approach has the ability to characterize the impact of maintenance and the actual failure rate, thereby improving the accuracy of the substation reliability assessment.

Doubly Enhanced Annuities (DEANs) and the Impact of Quality of Long-Term Care under a Multi-State Model of Activities of Daily Living (ADL)

The relatively small size of global voluntary immediate life annuity markets seemingly contradicts a well-known result that, under certain conditions, utility-maximizing retirees should annuitize all of their wealth upon retirement. This apparent contradiction contributes to what is called the “annuity puzzle.” The main explanations for the annuity puzzle include adverse selection, bequest motives, and retirees’ fear of health shocks that may require them to need long-term care. To explore a potential solution to this puzzle in the U.S. annuity market, we consider a cohort of retirees who are independent and identical except for a hidden immutable health parameter, , that identifies the retiree’s risk type. We assume that the “average” or “standard” retiree has θ = 1, with larger values of θ denoting less healthy individuals. Because private insurers fear that annuitants may live well beyond their life expectancy—that is, they fear adverse selection—voluntary immediate life annuities are generally priced in a manner that makes them attractive mainly to retirees who believe themselves to be healthier than average. These retirees, however, typically represent a small fraction of their cohort of retirees. To expand annuity markets, we are proposing a hybrid voluntary fixed immediate annuity product called a doubly enhanced annuity (DEAN), which we feel will be attractive to the larger retiree population; that is, retirees with . DEANs are medically underwritten to provide greater annual benefits to annuitants with shorter than average life expectancies, they provide long-term care benefits, and they satisfy a bequest motive through a death benefit. Our underlying model of retiree mortality and morbidity is based on a multi-state Markov process with states representing various levels of an insured retiree’s ability to perform activities of daily living (ADL). A major contribution of our research is to explicitly include in our multi-state Markov model a parameter, α, that reflects the level of quality of long-term care a retiree receives over her lifetime. Our main objectives are to develop a theoretical basis for pricing DEANs for the U.S. market, to determine a retiree’s expected discounted utility gained from these annuities, to determine the optimal choice of quality of care, and to explore some of the quality of life implications. A numerical example is provided.

Opportunistic Maintenance Optimization for Transmission Bays Based on Multistate Markov Process

Power equipment usually goes through multiple states during the transition from normal state to the failure state. In order to consider equipment multistate property, an opportunistic maintenance optimization model based on multistate Markov process is proposed considering economic dependence within the transmission bay. This model describe equipment state transition process using the multistate Markov process. And then, opportunistic maintenance strategy between dependent power equipment in a transmission bay is studied. The transmission bay availability is derived considering opportunistic maintenance and scheduled maintenance. Finally, system operation cost is quantified. System operation cost is minimized to optimize equipment maintenance strategies. Numerical studies demonstrate validity of the proposed model.

On Optimality of Myopic Policy in Multi-Channel Opportunistic Access

We consider the channel access problem arising in opportunistic scheduling over fading channels, cognitive radio networks, and server scheduling. The multi-channel communication system consists of $N$ channels. Each channel evolves as a time-nonhomogeneous multi-state Markov process. At each time instant, a user chooses $M$ channels to transmit information, and obtains some reward, i.e., throughput, based on the states of the chosen channels. The objective is to design an access policy, i.e., which channels should be accessed at each time instant, such that the expected accumulated discounted reward is maximised over a finite or infinite horizon. The considered problem can be cast into a restless multi-armed bandit (RMAB) problem, which is PSPACE-hard, with the optimal policy usually intractable due to the exponential computation complexity. Hence, a natural alternative is to consider the easily implementable myopic policy that only maximises the immediate reward but ignores the impact of the current strategy on the future reward. In this paper, we perform an analytical study on the performance of the myopic policy for the considered RMAB problem, and establish a set of closed-form conditions to guarantee the optimality of the myopic policy.

Optimal Observation Times for Multistate Markov Models—Applications to Pneumococcal Colonization Studies

SummaryApplications of finite state Markov transition models are numerous and the problem of estimating transition rates of such processes has been considered in many fields of science. Because these processes cannot always be followed in continuous time, the investigators often confront the question of when to measure the state of the process. The estimation of transition rates then needs to be based on a sequence of discrete time data, and the variance and estimability of the estimators greatly depend on the time spacings between consecutive observations. We study optimal time spacings for a sequence of discrete time observations to estimate the transition rates of a time homogeneous multistate Markov process. For comparative studies, optimal time spacings to estimate rate ratios are considered. Optimality criteria are formulated through the minimization of the variances of the parameter estimators of interest and are investigated assuming a stationary initial distribution. For practical purposes, we propose a simple approximation for the optimal time spacing and study the limits for its applicability. The work is motivated by studies of colonization with Streptococcus pneumoniae.

Analysis of a Transformer Time-Varying Outage Model for Operational Risk Assessment

Firstly, the operating mechanism in developing process of transformer internal latent fault and external accessory fault is analyzed. Then, dissolved gas analysis (DGA) outcome and average degree of polymerization (DP) are used as an indicator to classify transformers operating condition, the multi-state Markov process model based transformers operating condition is proposed to estimate the internal failure rate. According to the method of weather condition classification, the external accessory failure rate model is proposed by considering the influence of weather change. A time-varying transformer outage model for operational risk assessment is proposed by using the multi-state Markov process to estimate the time-varying failure probability. Finally, a numerical is presented.

Adaptive MC-CDMA Multiple Channel Estimation and Tracking Over Time-Varying Multipath Fading Channels

Multicarrier code-division multiple access (MC-CDMA) combines multicarrier transmission with direct-sequence (DS) spread spectrum techniques. In this study, several possible channel models of MC-CDMA systems are considered as a multi-state switching Markov process to match the time-varying multipath fading channel. A novel multiple channel estimator, which consists of several possible channel models and can be self-adjusted to fit different channel uncertain conditions, is proposed for MC-CDMA systems to effectively estimate channel coefficients in the frequency domain over time-varying multipath fading channels. Based on the proposed channel estimation, an enhanced minimum mean-square error (MMSE) equalizer is employed for symbol detection of MC-CDMA systems. Simulation results demonstrate that the proposed multiple channel estimator can track the channel more accurately than the conventional channel estimators. With the consideration of channel estimation error as a channel noise to improve symbol detection, the enhanced equalizer based on the proposed multiple channel estimator performs much better than those based on the conventional channel estimators in bit error rate (BER).

Using hidden multi-state Markov models with multi-parameter volcanic data to provide empirical evidence for alert level decision-support

For the purposes of eruption forecasting and hazard mitigation, a volcanic crisis may be represented as a staged progression of states of unrest, each with its own timescale and likelihood of transition to other states (or to climactic eruption). If the state conditions can be interpreted physically, e.g., in terms of advancing materials failure, this knowledge could be used directly to inform decisions on alert level setting. A multi-state Markov process provides one simple model for defining states and for estimating rates of switching between states. However, for eruptive processes, such states are not directly observable and must be inferred from latent markers, such as seismic activity, gas output, deformation rates, etc., some of which may be contradictory. Interpretations of uncertain data will be liable to error, so a model is needed which can simultaneously estimate both elements: the transition likelihood of a hidden process and the probabilities of state misclassification. We describe the concept and underlying principles of continuous-time hidden Markov models and demonstrate them in a decision-support context with a preliminary working implementation using MULTIMO data. Where multi-parameter streams of raw, processed or conditioned data of different kinds are available, these can be input in near real-time to appropriate hidden multi-state Markov models, the outputs of each providing their own objective analyses of eruptive state in probabilistic terms. These separate, multiple indicators of state can then be input into a Bayesian Belief Network framework for weighing and combining them as different strands of evidence, together with other observations, data, interpretations and expert judgment.

Cost-effectiveness analysis of breast cancer adjuvant treatment: FEC 50 versus FEC 100 (FASG05 study)

Background: The aim of the study was to assess the incremental cost-effectiveness ratio (ICER) of the FEC 100 compared with the FEC 50 in the FASG05 trial.Materials and methods: A cost-effectiveness analysis was performed using a multi-state Markov process model. Relevant clinical data introduced into the model were obtained from 10-year follow-up of the clinical trial FASG05. Survival curves for each health state were assessed by survival parametric model. The model allowed assessments from the start of adjuvant chemotherapy until death. The costs of adjuvant treatment and follow-up were estimated. The costs of recurrence were evaluated from the medical records of 146 patients. A prospective survey was performed on a cohort of 87 patients to quantify the resources external to the hospital (including cost of transportation). The inpatient costs were evaluated using the French diagnosis-related groups. The ambulatory costs were assessed using the French nomenclature. Costs were expressed in 2002 Euro (€), according to the French societal perspective. The ICER assessed the cost of one additional life year saved. A discount rate of 5% per year was used for cost, and alternatively 0%, 3% and 5% for effectiveness. We validated the results with a probabilistic sensitivity analysis incorporating parametric and non-parametric bootstraps, and with the acceptability curves.Results: The mean total discounting cost of adjuvant treatments was 11465€ for FEC 50 and 13815€ for FEC 100; the mean total discounting cost of recurrences was 14636€ and 13503€, respectively. According to the discount rate of effectiveness, the life expectancy was 16.5, 11.4 and 9.3 years for FEC 50 and 18.4, 12.5 and 10.2 years for FEC 100. The ICER (cost per life year saved) were 642€, 1084€ and 1460€, respectively. The probability according to which FEC 50 is strictly dominated by FEC 100 was 0.15.Conclusion: The clinical benefit of FEC 100 generates a negligible cost increase when compared with FEC 50.

Stochastic Nonlinear Dynamics: How Many Ion Channels are in a Single Neuron?

We apply our stochastic cell model to simulate the noisy shape of single neuronal action potentials on the basis of the nonlinear oscillatory Brusselator model. By comparison with the experimental action potentials as measured by Laurent and co-workers in the olfactory system of the locust, we conclude that at least ∼14 000 ion channels in a local neuron (LN) and ∼70 000 ion channels in a project neueron (PN) are responsible for the stochastic appearance of the measured action potentials. Instead of treating the stochastic gating of ionic channels as a multistate Markov process, we use a computer simulation that performs a direct count of all events of each individual reaction step on the basis of their Poisson distributions. Our stochastic analysis is independent of the particular architecture of the neuronal network.

Multi-state models and outcome prediction in bone marrow transplantation.

Multi-state models have proved versatile and useful in the statistical analysis of the complicated course of events after bone marrow transplantation. Working from data from the International Bone Marrow Transplant Registry, we show that summary probability calculations may be useful to explore hypothetical scenarios where some transition intensities are set by the researcher. A multi-state Markov process model is specified with six states: the initial state 0; acute; chronic and both acute and chronic graft-versus-host disease A, C and AC; relapse R and death in remission D. Transition rates between the states are estimated using Nelson-Aalen estimators and Cox regression models and combined to transition probability estimators using Aalen-Johansen product integration. Besides the estimated transition probabilities to D and R we explore hypothetical probabilities obtained by artificially changing certain transition intensities, with the general purposes of getting summary views of the development for actual patients 'in this world' and of exploring the intrinsic information from real patients about consequences of various changed conditions.

Cost-effectiveness of interferon alfa in chronic myelogenous leukemia.

To evaluate the cost-effectiveness of interferon alfa (IFN alpha) treatment of patients with chronic myelogenous leukemia relative to conventional chemotherapy. A decision-analysis model that involved a multistate Markov process was designed to estimate the expected cost and quality-adjusted life expectancies for two cohorts of patients to be administered conventional chemotherapy or IFN alpha. Two IFN alpha strategies were modeled: prolonged treatment for patients who achieved a hematologic response (scenario A) or only for patients who achieved a cytogenetic remission in a 2-year period (scenario B). Data on response and transition probabilities between health states were obtained from the literature by a MEDLINE search and pooled with a meta-analytic method. Costs were based on local charges. Expected survival was adjusted for quality of life on the basis of an expert panel judgment. Baseline analysis showed IFN alpha treatment to increase the quality-adjusted life expectancy by 15.5 and 12.5 months relative to conventional chemotherapy, in scenarios A and B, respectively. Marginal cost-effectiveness was $89,500 and $63,500 per quality-adjusted life-year (QALY) gained. Sensitivity analysis confirmed IFN alpha as the most effective approach. Cost-effectiveness results were sensitive to the cost of IFN alpha therapy and to the assumptions about the rate of cytogenetic remission. Reducing the drug dose, as suggested by a recent report, would decrease the marginal cost-effectiveness to less than $20,000. IFN alpha is substantially superior to conventional chemotherapy in terms of quality-adjusted survival, but, at the current doses, marginal cost-effectiveness ranges from $50,000 to $100,000 per QALY gained under most of our assumptions.

P88 A multi-state markov process model for quality of life and survival in large heart failure clinical trials

P88 A multi-state markov process model for quality of life and survival in large heart failure clinical trials

Association in time of a finite semi-Markov process

We derive three equivalent sufficient conditions for association in time of a finite state semi-Markov process in terms of transition probabilities and crude hazard rates. This result generalizes the earlier results of Esary and Proschan (1970), for a binary Markov process and Hjort et al. (1985), for a multistate Markov process.

ORDMKV: a computer program fitting proportional odds model for multi-state Markov process

ORDMKV is a computer program designed to fit a multi-state discrete-time Markov model for k-stages disease processes having an ordinal structure. The model consists of k transient states representing the increasing severity of the disease process, and the final state can be optionally chosen to be an absorbing state in cases such as death. The ordinal structure of the stages of the disease is modelled by using ordinal response models. Each row of the one-step transition probability matrix is modelled using a proportional odds model based on the cumulative transition probabilities. By using these ordinal response models, the number of parameters used to model the disease process can be reduced significantly not only with respect to a general discrete-time model, but also compared with a parsimonuos continuous-time model. A restricted model can be fitted by assuming that the effect of the covariables in the cumulative probability has common regression coefficients in all stages of the disease process. This assumption, if it holds, reduces the number of regression coefficients associated with each covariate to only one. The regression coefficients of this model are estimated via the method of maximum likelihood, using a quasi-Newton optimization algorithm. When the last state is considered as an absorbing state, it is possible to compute survival curves from the transient states of the process. The program was written in standard FORTRAN 77 and is illustrated using a four-state model to determine factors influencing diabetic retinopathy in young subjects with insulin-dependent diabetes mellitus.

Reliability Modeling & Evaluation for Networks under Multiple & Fluctuating Operational Conditions

A simple multistate Markov process models a component and a system of any configuration under several operational conditions. Based on it, a simple tie-set approach calculates availability and frequencies-of-system-failure during different conditions; a cut-set approach calculates probabilities and frequencies of system-failure during various conditions. There are two interpretations of operational conditions: 1) environment such as weather or stress, and 2) operating conditions such as occur in several sequential or fluctuating phases. As the system is assumed at any moment under a condition with a certain probability or weight, some relative reliability measures can be further obtained simply to identify which conditions are disastrous.

Optimal periodic maintenance policy for machines subject to deterioration and random breakdown: D. N. Khandelwal, Jaydev Sharma and L. M. Ray. IEEE Trans. Reliab., R-28, (4) 328 (October 1979)

We derive three equivalent sufficient conditions for association in time of a finite state semi-Markov process in terms of transition probabilities and crude hazard rates. This result generalizes the earlier results of Esary and Proschan (1970), for a binary Markov process and Hjort et al. (1985), for a multistate Markov process.