Homogeneous Continuous-time Markov Chain Research Articles

Interpretation of the Evolution of the Homogeneous Markov System (or, equivalently, of the Embedded Markov Chain) as the Deformation of a Viscoelastic Medium. The 3-D Case

Every attainable structure of a continuous time homogeneous Markov chain (HMC) with n states, or of a closed Markov system with an embedded HMC with n states, or more generally of a Markov system driven by an HMC, is considered as a point-particle of ℜ n . Then, the motion of the attainable structure corresponds to the motion of the respective point-particle in ℜ n . Under the assumption that “the motion of every particle at every time point is due to the interaction with its surroundings”, ℜ n (and equivalently the set of the accosiated attainable structures of the homogeneous Markov system (HMS), or alternatively of the underlying embedded HMC) becomes a continuum. Thus, the evolution of the set of the attainable structures corresponds to the motion of the continuum. In this paper it is shown that the evolution of a three-dimensional HMS (n = 3) or simply of an HMC, can be interpreted through the evolution of a two-dimensional isotropic viscoelastic medium.

On a reward rate estimation for the finite irreducible continuous-time Markov chain

A continuous-time homogeneous irreducible Markov chain {X(t)}, t ϵ [0; ∞), taking values on N = {1,..., k}, k <∞, is considered. Matrix λ = (λij) of the intensity of transition λij from state i to state j is known. A unit of the sojourn time in state i gives reward βi so the total reward during time t is \(Y(t) = \mathop \smallint \limits_0^t {\beta _{X(s)}}ds\). The reward rates {βi} are not known and it is necessary to estimate them. For that purpose the following statistical data on r observations are at our disposal: (1) t, observation time; (2) i, initial state X(0); (3) j, final state X(t); and (4) y, acquired reward Y(t). Two methods are used for the estimation: the weighted least-squares method and the saddle-point method for the Laplace transformation of the reward. Simulation study illustrates the suggested approaches.

Optimization of hospital ward resources with patient relocation using Markov chain modeling

Overcrowding of hospital wards is a well-known and often revisited problem in the literature, yet it appears in many different variations. In this study, we present a mathematical model to solve the problem of ensuring sufficient beds to hospital wards by re-distributing beds that are already available to the hospital. Patient flow is modeled using a homogeneous continuous-time Markov chain and optimization is conducted using a local search heuristic. Our model accounts for patient relocation, which has not been done analytically in literature with similar scope. The study objective is to ensure that patient occupancy is reflected by our Markov chain model, and that a local optimum can be derived within a reasonable runtime.Using a Danish hospital as our case study, the Markov chain model is statistically found to reflect occupancy of hospital beds by patients as a function of how hospital beds are distributed. Furthermore, our heuristic is found to efficiently derive the optimal solution. Applying our model to the hospital case, we found that relocation of daily arrivals can be reduced by 11.7% by re-distributing beds that are already available to the hospital.

The Limit Behaviour of Imprecise Continuous-Time Markov Chains

We study the limit behaviour of a nonlinear differential equation whose solution is a superadditive generalisation of a stochastic matrix, prove convergence, and provide necessary and sufficient conditions for ergodicity. In the linear case, the solution of our differential equation is equal to the matrix exponential of an intensity matrix and can then be interpreted as the transition operator of a homogeneous continuous-time Markov chain. Similarly, in the generalised nonlinear case that we consider, the solution can be interpreted as the lower transition operator of a specific set of non-homogeneous continuous-time Markov chains, called an imprecise continuous-time Markov chain. In this context, our convergence result shows that for a fixed initial state, an imprecise continuous-time Markov chain always converges to a limiting distribution, and our ergodicity result provides a necessary and sufficient condition for this limiting distribution to be independent of the initial state.

Malaria Incidence Rates from Time Series of 2-Wave Panel Surveys.

Methodology to estimate malaria incidence rates from a commonly occurring form of interval-censored longitudinal parasitological data—specifically, 2-wave panel data—was first proposed 40 years ago based on the theory of continuous-time homogeneous Markov Chains. Assumptions of the methodology were suitable for settings with high malaria transmission in the absence of control measures, but are violated in areas experiencing fast decline or that have achieved very low transmission. No further developments that can accommodate such violations have been put forth since then. We extend previous work and propose a new methodology to estimate malaria incidence rates from 2-wave panel data, utilizing the class of 2-component mixtures of continuous-time Markov chains, representing two sub-populations with distinct behavior/attitude towards malaria prevention and treatment. Model identification, or even partial identification, requires context-specific a priori constraints on parameters. The method can be applied to scenarios of any transmission intensity. We provide an application utilizing data from Dar es Salaam, an area that experienced steady decline in malaria over almost five years after a larviciding intervention. We conducted sensitivity analysis to account for possible sampling variation in input data and model assumptions/parameters, and we considered differences in estimates due to submicroscopic infections. Results showed that, assuming defensible a priori constraints on model parameters, most of the uncertainty in the estimated incidence rates was due to sampling variation, not to partial identifiability of the mixture model for the case at hand. Differences between microscopy- and PCR-based rates depend on the transmission intensity. Leveraging on a method to estimate incidence rates from 2-wave panel data under any transmission intensity, and from the increasing availability of such data, there is an opportunity to foster further methodological developments, particularly focused on partial identifiability and the diversity of a priori parameter constraints associated with different human-ecosystem interfaces. As a consequence there can be more nuanced planning and evaluation of malaria control programs than heretofore.

A Multi-State Physics Modeling approach for the reliability assessment of Nuclear Power Plants piping systems

A Multi-State Physics Modeling (MSPM) approach is here proposed for degradation modeling and failure probability quantification of Nuclear Power Plants (NPPs) piping systems. This approach integrates multi-state modeling to describe the degradation process by transitions among discrete states (e.g., no damage, micro-crack, flaw, rupture, etc.), with physics modeling by (physic) equations to describe the continuous degradation process within the states. We propose a Monte Carlo (MC) simulation method for the evaluation of the time-dependent transition rates between the states of the MSPM. Accountancy is given for the uncertainty in the parameters and external factors influencing the degradation process. The proposed modeling approach is applied to a benchmark problem of a piping system of a Pressurized Water Reactor (PWR) undergoing thermal fatigue. The results are compared with those obtained by a continuous-time homogeneous Markov Chain Model.

Credit Risk Valuation with Rating Transitions and Partial Information

This work intend to shed some light on a new use of Phase-type distributions in credit risk, taking into account different flows of information without huge numerical calculations. We consider credit migration models with partial information and study the influence of a deficit of information on prices of credit linked securities. The transitions through the various credit classes are modeled via a homogeneous continuous time Markov chain but they are not directly observable by investors in the secondary market. We first consider the case of one bond issuer and study three settings of partial information. In a first model, information about ratings arrives at predetermined dates with delay periods. In a second model, information arrives randomly according to an exogenous Poisson process, whereas in a third model, information arrives randomly according to an endogenous rule (transitions are observed only when they lead the Markov chain to a class with a lower credit rating). We infer in the three settings bonds and options prices, and we provide an explicit description of the dynamics of bond prices under real and pricing measures. We also consider the case of two bond issuers and we analyze the cross effects of deficit of information and contagion on bonds prices and correlation of default times. We conclude the paper by numerical illustrations.

Optimal Control of a Partially Observable Failing System with Costly Multivariate Observations

We model a partially observable deteriorating system subject to random failure. The state process follows an unobservable continuous time homogeneous Markov chain. At equidistant sampling times vector-valued observations having multivariate normal distribution with state-dependent mean and covariance matrix are obtained at a positive cost. At each sampling epoch a decision is made either to run the system until the next sampling epoch or to carry out full preventive maintenance, which is assumed to be less costly than corrective maintenance carried out upon system failure. The objective is to determine the optimal control policy that minimizes the long-run expected average cost per unit time. We show that the optimal preventive maintenance region is a convex subset of Euclidean space. We also analyze the practical three-state version of this problem in detail and show that in this case the optimal policy is a control limit policy. An efficient computational algorithm is developed for the three-state problem, illustrated by a numerical example.

Parameter estimation for partially observable systems subject to random failure

In this paper, we present a parameter estimation procedure for a condition‐based maintenance model under partial observations. Systems can be in a healthy or unhealthy operational state, or in a failure state. System deterioration is driven by a continuous time homogeneous Markov chain and the system state is unobservable, except the failure state. Vector information that is stochastically related to the system state is obtained through condition monitoring at equidistant sampling times. Two types of data histories are available — data histories that end with observable failure, and censored data histories that end when the system has been suspended from operation but has not failed. The state and observation processes are modeled in the hidden Markov framework and the model parameters are estimated using the expectation–maximization algorithm. We show that both the pseudolikelihood function and the parameter updates in each iteration of the expectation–maximization algorithm have explicit formulas. A numerical example is developed using real multivariate spectrometric oil data coming from the failing transmission units of 240‐ton heavy hauler trucks used in the Athabasca oil sands of Alberta, Canada. Copyright © 2012 John Wiley &amp; Sons, Ltd.

On the Numerical Analysis of Inhomogeneous Continuous-Time Markov Chains

Inhomogeneous continuous-time Markov chains play an important role in different application areas. In contrast to homogeneous continuous-time Markov chains, where a large number of numerical analysis techniques are available and have been compared, few results about the performance of numerical techniques in the inhomogeneous case are known. This paper presents a new variant of the uniformization technique, the most efficient approach for homogeneous Markov chains. The new uniformization technique allows for the stable computation of strict bounds for the transient distribution of inhomogeneous continuous-time Markov chains, which is not possible with other numerical techniques that provide only an approximation of the distribution and asymptotic bounds. Furthermore, another variant of uniformization is presented that computes an approximation of the transient distribution and is shown to outperform standard differential equation solvers if transition rates change slowly.

Optimal server resource reservation policies for priority classes of users under cyclic non-homogeneous markov modeling

Abstract Resource availability optimization is studied on a server–client system where different users are partitioned into priority classes. The aim is to provide higher resource availability according to the priority of each class. For this purpose, resource reservation is modeled by a homogeneous continuous time Markov chain (CTMC), but also by a cyclic non-homogeneous Markov chain (CNHMC) as there is a cyclic behavior of the users’ requests for resources. The contribution of the work presented consists in the formulation of a multiobjective optimization problem for both the above cases that aims to determine the optimal resource reservation policy providing higher levels of resource availability for all classes. The optimization problem is solved either with known methods or with a proposed kind of heuristic algorithm. Finally, explicit generalized approximate inverse preconditioning methods are adopted for solving efficiently sparse linear systems that are derived, in order to compute resource availability.

Similar States in Continuous-Time Markov Chains

In a homogeneous continuous-time Markov chain on a finite state space, two states that jump to every other state with the same rate are called similar. By partitioning states into similarity classes, the algebraic derivation of the transition matrix can be simplified, using hidden holding times and lumped Markov chains. When the rate matrix is reversible, the transition matrix is explicitly related in an intuitive way to that of the lumped chain. The theory provides a unified derivation for a whole range of useful DNA base substitution models, and a number of amino acid substitution models.

Similar States in Continuous-Time Markov Chains

In a homogeneous continuous-time Markov chain on a finite state space, two states that jump to every other state with the same rate are called similar. By partitioning states into similarity classes, the algebraic derivation of the transition matrix can be simplified, using hidden holding times and lumped Markov chains. When the rate matrix is reversible, the transition matrix is explicitly related in an intuitive way to that of the lumped chain. The theory provides a unified derivation for a whole range of useful DNA base substitution models, and a number of amino acid substitution models.

A note on a production control model for a facility with limited storage capacity in a random environment

Abstract This note presents an extension of our earlier paper on a production–inventory system with limited storage capacity in a multi-state random environment [i.e., E. Mohebbi, A production–inventory model with randomly changing environmental conditions, Eur. J Res. 174 (2006) 539–552]. More specifically, it assumes that the production rate as well as the occurrence rate and batch sizes of a compound Poisson demand stream are influenced by the state of a continuous-time homogeneous Markov chain representing the random environment. We present an adaptation of the system-point method of level crossings which yields the limiting distribution of inventory level (stock-on-hand) for a two-parameter production control policy when stockouts are lost.

An inventory model with non-resuming randomly interruptible lead time

We assume that an unreliable supplier in a single-item stochastic inventory system alternates randomly between two possible states (i.e., available and unavailable), following a two-state continuous-time homogeneous Markov chain. For a compound Poisson stream of demands and Erlang lead times, our model considers the scenario where the processing of the outstanding order (if any) is interrupted at every supplier's transition epoch from the available to the unavailable state, and is restarted from the outset upon the supplier's regaining its available state. We derive the stationary distribution of the on-hand inventory under a continuous-review policy and provide some numerical results.

Allelomimetic synchronization in Merino sheep

Changes between the inactive (resting/ruminating) and active (grazing, walking) states in groups of Merino sheep were studied in the field for different group sizes (two, four, six or eight) of either male or female animals over 6-h periods. The amount of synchrony within groups was high (60–80%) and is attributed to the mutual adjustment of behaviour by group members. To quantify this process, changes in the number of active individuals were fitted by a time homogeneous continuous time Markov chain model. We found that the probability of an individual becoming active increased with the number of active conspecifics in the group and decreased with the number of inactive conspecifics. The reverse effect was found for the probability of becoming inactive. A model of this individual decision-making process is fitted to the data and predictions of the model are shown to account for the synchrony observed within the group. Group synchronization is thus presented as a self-organized dynamic system, where collective oscillations between activity and inactivity arise stochastically from the coupling between individual Markov processes.

Past states of continuous-time Markov models for ecological communities

Discrete-time Markov chains are often used to model communities of sessile organisms. The community is described by a set of discrete states, which may represent species or groups of species. Transitions between states are modelled using a stochastic matrix. A recent study showed how the time-reversal of such a Markov chain can be used to estimate the distribution of time since the last occurrence of some state of interest (such as empty space) at a point, given the current state of the point. However, if the underlying process operates in continuous time but is observed at regular intervals, this distribution describes the time since the last possible observation of the state of interest, rather than the time since its last occurrence. We show how to obtain the distribution of time since the last occurrence of a state of interest for a continuous-time homogeneous Markov chain. The expected time since the last occurrence of an initial state can be interpreted as a measure of the successional rank of a state. We show how to distinguish between different ways in which a state can have high successional rank. We apply our results to a marine subtidal community.

When supplier’s availability affects the replenishment lead time—An extension of the supply-interruption problem

This paper presents an exact treatment of a continuous-review inventory system with compound Poisson demand, Erlang-distributed lead times and random supply interruptions. In contrast with the existing models in the literature, we take into account the supplier’s availability in characterizing the lead time of a replenishment order. Assuming that the supplier’s availability can be described by a continuous-time homogeneous Markov chain with two states ( on and off) and that stockouts are lost, we derive the stationary distribution of the inventory level (stock-on-hand) under an ( s, Q)-type control policy. This probability distribution is then used to formulate an exact expression for the long-run average cost per unit time of operating the inventory system. Some numerical results are also provided.

A production-inventory model with randomly changing environmental conditions

We consider a limited-capacity production-inventory system with linear production rate and compound Poisson demand in which both production and demand processes are subject to independently and randomly changing environmental conditions. Assuming that these conditions are represented by two continuous-time homogeneous Markov chains and shortages are lost, we derive the limiting distribution of the inventory level and present some numerical results in terms of the ensuing performance measures.

Performability analysis of clustered systems with rejuvenation under varying workload

This paper develops time-based rejuvenation policies to improve the performability measures of a cluster system. Three rejuvenation policies, namely standard rejuvenation, delayed rejuvenation and mixed rejuvenation, are designed to improve the cluster’s performability under varying workload. Analytic models are built to evaluate these three policies. Since deterministic transitions are used in this paper and analytical models based on homogeneous continuous-time Markov chains (CTMC) do not allow non-exponential distributions, we utilize deterministic and stochastic Petri nets (DSPN), in which the underlying stochastic process is a Markov regenerative process (MRGP), to capture both exponential and deterministic distributions. System performability measures under these three rejuvenation policies are derived based on the DSPN models. We show that the mixed rejuvenation policy achieves the maximum performability among the three policies, which results in 12% improvement on the system throughput in the example shown in this paper. The delayed rejuvenation is better than the standard rejuvenation with respect to the optimal job blocking probability and system throughput. For longer rejuvenation-triggering intervals, the standard rejuvenation yields a better result than delayed rejuvenation, while for shorter rejuvenation-triggering intervals the delayed rejuvenation policy outperforms standard rejuvenation policy.