Discrete Markov Process Research Articles

On asymptotic description of a probabilistic change-point model for a two-state discrete Markov process

For jump variation of parameters of a two-state discrete Markov process, we obtain an asymptotic representation of the likelihood ratio. Its use in the joint change-point detection and estimation problem is illustrated by an example.

A rigorous framework for multiscale simulation of stochastic cellular networks

Noise and stochasticity are fundamental to biology and derive from the very nature of biochemical reactions where thermal motion of molecules translates into randomness in the sequence and timing of reactions. This randomness leads to cell-cell variability even in clonal populations. Stochastic biochemical networks are modeled as continuous time discrete state Markov processes whose probability density functions evolve according to a chemical master equation (CME). The CME is not solvable but for the simplest cases, and one has to resort to kinetic Monte Carlo techniques to simulate the stochastic trajectories of the biochemical network under study. A commonly used such algorithm is the stochastic simulation algorithm (SSA). Because it tracks every biochemical reaction that occurs in a given system, the SSA presents computational difficulties especially when there is a vast disparity in the timescales of the reactions or in the number of molecules involved in these reactions. This is common in cellular networks, and many approximation algorithms have evolved to alleviate the computational burdens of the SSA. Here, we present a rigorously derived modified CME framework based on the partition of a biochemically reacting system into restricted and unrestricted reactions. Although this modified CME decomposition is as analytically difficult as the original CME, it can be naturally used to generate a hierarchy of approximations at different levels of accuracy. Most importantly, some previously derived algorithms are demonstrated to be limiting cases of our formulation. We apply our methods to biologically relevant test systems to demonstrate their accuracy and efficiency.

An efficient solution to a retrial queue for the performability evaluation of DHCP

The Dynamic Host Configuration Protocol (DHCP) is designed to provide an automatic mechanism for the allocation, configuration and management of IP addresses and TCP/IP protocol stack parameters of computers and devices in IP networks. The important feature of DHCP is a “dynamic allocation” mechanism, which assigns an IP address to a client for a limited period of time (called a lease time). Therefore, a previously allocated IP address can automatically be assigned to another host by a DHCP server upon the expiration of the lease time if a host does not renew the use of an allocated IP address. This paper proposes a retrial queueing model to approximate the performability of the DHCP dynamic allocation mechanism. An efficient computational algorithm is developed to calculate the steady state probabilities of a continuous time discrete state Markov process. A comparison of our approximate model with a detailed simulation model of the DHCP dynamic allocation mechanism shows an excellent agreement between the analytical and the simulation results. The impact of the lease time parameter on the performability of the DHCP dynamic allocation mechanism is also illustrated through a numerical study.

Optimal search for a moving target with the option to wait

AbstractWe investigate the problem in which an agent has to find an object that moves between two locations according to a discrete Markov process (Pollock, Operat Res 18 (1970) 883–903). At every period, the agent has three options: searching left, searching right, and waiting. We assume that waiting is costless whereas searching is costly. Moreover, when the agent searches the location that contains the object, he finds it with probability 1 (i.e. there is no overlooking). Waiting can be useful because it could induce a more favorable probability distribution over the two locations next period. We find an essentially unique (nearly) optimal strategy, and prove that it is characterized by two thresholds (as conjectured by Weber, J Appl Probab 23 (1986) 708–717). We show, moreover, that it can never be optimal to search the location with the lower probability of containing the object. The latter result is far from obvious and is in clear contrast with the example in Ross (1983) for the model without waiting. © 2009 Wiley Periodicals, Inc. Naval Research Logistics 2009

A stochastic model for the evolution of metabolic networks with neighbor dependence

Most current research in network evolution focuses on networks that follow a Duplication Attachment model where the network is only allowed to grow. The evolution of metabolic networks, however, is characterized by gain as well as loss of reactions. It would be desirable to have a biologically relevant model of network evolution that could be used to calculate the likelihood of homologous metabolic networks. We describe metabolic network evolution as a discrete space continuous time Markov process and introduce a neighbor-dependent model for the evolution of metabolic networks where the rates with which reactions are added or removed depend on the fraction of neighboring reactions present in the network. We also present a Gibbs sampler for estimating the parameters of evolution without exploring the whole search space by iteratively sampling from the conditional distributions of the paths and parameters. A Metropolis-Hastings algorithm for sampling paths between two networks and calculating the likelihood of evolution is also presented. The sampler is used to estimate the parameters of evolution of metabolic networks in the genus Pseudomonas. An implementation of the Gibbs sampler in Java is available at http://www.stats.ox.ac.uk/ approximately mithani/networkGibbs/. Supplementary data are available at the Bioinformatics online.

Evolutionary dynamics on graphs: Efficient method for weak selection

Investigating the evolutionary dynamics of game theoretical interactions in populations where individuals are arranged on a graph can be challenging in terms of computation time. Here, we propose an efficient method to study any type of game on arbitrary graph structures for weak selection. In this limit, evolutionary game dynamics represents a first-order correction to neutral evolution. Spatial correlations can be empirically determined under neutral evolution and provide the basis for formulating the game dynamics as a discrete Markov process by incorporating a detailed description of the microscopic dynamics based on the neutral correlations. This framework is then applied to one of the most intriguing questions in evolutionary biology: the evolution of cooperation. We demonstrate that the degree heterogeneity of a graph impedes cooperation and that the success of tit for tat depends not only on the number of rounds but also on the degree of the graph. Moreover, considering the mutation-selection equilibrium shows that the symmetry of the stationary distribution of states under weak selection is skewed in favor of defectors for larger selection strengths. In particular, degree heterogeneity--a prominent feature of scale-free networks--generally results in a more pronounced increase in the critical benefit-to-cost ratio required for evolution to favor cooperation as compared to regular graphs. This conclusion is corroborated by an analysis of the effects of population structures on the fixation probabilities of strategies in general 2 x 2 games for different types of graphs. Computer simulations confirm the predictive power of our method and illustrate the improved accuracy as compared to previous studies.

The Optimal Stopping Time for Detecting Changes in Discrete Time Markov Processes

In this paper we prove that Page's stopping time (CUSUM test) is optimal for detecting the changes in Markov processes that have invariant distribution.

Listening to the noise: random fluctuations reveal gene network parameters

The cellular environment is abuzz with noise originating from the inherent random motion of reacting molecules in the living cell. In this noisy environment, clonal cell populations show cell-to-cell variability that can manifest significant phenotypic differences. Noise-induced stochastic fluctuations in cellular constituents can be measured and their statistics quantified. We show that these random fluctuations carry within them valuable information about the underlying genetic network. Far from being a nuisance, the ever-present cellular noise acts as a rich source of excitation that, when processed through a gene network, carries its distinctive fingerprint that encodes a wealth of information about that network. We show that in some cases the analysis of these random fluctuations enables the full identification of network parameters, including those that may otherwise be difficult to measure. This establishes a potentially powerful approach for the identification of gene networks and offers a new window into the workings of these networks.

Using iterations for finding the optimal virtual channel in telecommunication networks

We model telecommunication systems using two-state discrete Markov processes. Using the methods of dynamic programming we find the optimal virtual channel with minimal probability of a busy state. A simple example is also considered for the purpose of illustration.

Analysis of censored discrete longitudinal data: Estimation of mean response

The study of longitudinal data is usually concerned with one or several response variables measured, possibly along with some covariates, at different points in time. In real-life situations this is often complicated by missing observations due to what we usually refer to as 'censoring'. In this paper we consider missingness of a monotone kind; subjects that dropout, i.e. are censored, fail to participate in the study at any of the subsequent observation times. Our scientific objective is to make inference about the mean response in a hypothetical population without any dropouts. There are several methods and approaches that address this problem, and we will present two existing methods (the linear-increments method and the inverse-probability-weighting method), as well as propose a new method, based on a discrete Markov process. We examine the performance of the corresponding estimators and compare these with respect to bias and variability. To demonstrate the effectiveness of the approaches in estimating the mean of a response variable, we analyse simulated data of different multistate models with a Markovian structure. Analyses of substantive data from (1) a study of symptoms experienced after a traumatic brain injury, and (2) a study of cognitive function among the elderly, are used as illustrations of the methods presented.

Ruin problems in a discrete Markov risk model

In this paper, we extend the compound binomial risk model to a Markov dependent model in which the claim occurrence and the claim amount are both regulated by a discrete time Markov process. The explicit expression for the “discounted” joint probability function of the surplus before ruin and the deficit at ruin is derived when the initial surplus u = 0 , and a recursive formula to calculate such “discounted” joint probability function when the initial surplus u > 0 is also obtained.

The Half-Cycle Correction Explained: Two Alternative Pedagogical Approaches

Students of Markov decision models are often taught to add a half-cycle's worth of incremental utility to the cumulative total for each health state. The reason for this "half-cycle correction'' is often illustrated by a graph of the proportion of the hypothetical Markov cohort remaining in a given state. The ideal graph is shown as a smooth, declining, curve that represents the transition of patients randomly throughout each cycle. On the same graph, the effect of the accounting of state membership at the end of each cycle in discrete, computer-based approximations of the ideal Markov process is shown. Students are able to clearly see that the cumulative incremental utility in the discrete case underestimates the desired quantity. Likewise, they find the concept of shifting the ideal curve to the right by one-half cycle to reduce the latter discrepancy to be intuitive. However, students often find the approximate equivalence of shifting the ideal state membership curve and adding a half-cycle's worth of incremental utility to the total for the state at the beginning of a discrete Markov process to be a difficult cognitive leap. This article describes 2 pedagogical devices, algebraic and intuitive/visual approaches, that may assist the instructor of Markov theory to convey the latter concept. Elements of adult learning theory are discussed, which may help the instructor to choose which approach to employ. Implementation of the half-cycle correction in commonly used decision-analytic software is also discussed.

Modeling Real-Time Balancing Power Market Prices Using Combined SARIMA and Markov Processes

This paper describes modeling of real-time balancing power market prices by using combined seasonal auto regressive integrated moving average (SARIMA) and discrete Markov processes. The combination of such processes allows generation of price series with periods where no demand for balancing power exists. The purpose of the model is simulation of prices to construct scenario trees representing possible realization of the stochastic prices. Such scenario trees can be used in planning models based on stochastic optimization to generate bid sequences to the balancing market. The spread of the prices in the tree and the shape of the scenarios are of central importance. Model parameter estimation methods reflecting the demands on scenario trees have therefore been used. The proposed model is also applied to data from the Nordic power market. The conclusion of this paper is that the developed model is appropriate for modeling real-time balancing power prices.

Stokes efficiency of molecular motors with inertia

A molecular motor utilizes chemical free energy to generate a unidirectional motion in a viscous media. The stochastic motion of a motor is governed by a Langevin equation coupled to the chemical occupancy state. The change of chemical occupancy state is governed by a discrete Markov process. The Stokes efficiency was introduced to measure how “efficiently” the motor uses chemical free energy to drive through the surrounding fluid. For the overdamping case where the effect of inertia is ignored, it was proved that the Stokes efficiency is bounded by 100% [H. Wang, G. Oster, The Stokes efficiency for molecular motors and its applications, Europhysics Letters 57 (2002) 134–140]. Here we present a proof for the general case.

A Markov chain Monte Carlo Expectation Maximization Algorithm for Statistical Analysis of DNA Sequence Evolution with Neighbor-Dependent Substitution Rates

The evolution of DNA sequences can be described by discrete state continuous time Markov processes on a phylogenetic tree. We consider neighbor-dependent evolutionary models where the instantaneous rate of substitution at a site depends on the states of the neighboring sites. Neighbor-dependent substitution models are analytically intractable and must be analyzed using either approximate or simulation-based methods. We describe statistical inference of neighbor-dependent models using a Markov chain Monte Carlo expectation maximization (MCMC-EM) algorithm. In the MCMC-EM algorithm, the high-dimensional integrals required in the EM algorithm are estimated using MCMC sampling. The MCMC sampler requires simulation of sample paths from a continuous time Markov process, conditional on the beginning and ending states and the paths of the neighboring sites. An exact path sampling algorithm is developed for this purpose.

Dynamic and structural features of intifada violence: a Markov process approach

This paper analyzes the daily incidence of violence during the Second Intifada. We compare several alternative statistical models with different dynamic and structural stability characteristics while keeping modelling complexity to a minimum by only maintaining the assumption that the process under consideration is at most a second order discrete Markov process. For the pooled data, the best model is one with asymmetric dynamics, where one Israeli and two Palestinian lags determine the conditional probability of violence. However, when we allow for structural change, the evidence strongly favors the hypothesis of structural instability across political regime sub-periods, within which dynamics are generally weak.

Stochastic model of adsorption of particles on macromolecules at an arbitrary filling

The theory of adsorption of particles on macromolecules is considered. The theory is based on the notion that the adsorption is a discrete Markovian process. In the context of this theory the average number and the variance of the number of ligands adsorbed on macromolecules are calculated exactly. The results are compared with the frequently used approximation where the number of adsorbed particles is smaller than that in the solution. It is shown that for small values of the equilibrium constant the exact and approximate values coincide practically.

A continuous time version and a generalization of a Markov-recapture model for trapping experiments

Wileyto et al. [E.P. Wileyto, W.J. Ewens, M.A. Mullen, Markov-recapture population estimates: a tool for improving interpretation of trapping experiments, Ecology 75 (1994) 1109] propose a four-state discrete time Markov process, which describes the structure of a marking–capture experiment as a method of population estimation. They propose this method primarily for estimation of closed insect populations. Their method provides a mark–recapture estimate from a single trap observation by allowing subjects to mark themselves. The estimate of the unknown population size is based on the assumption of a closed population and a simple Markov model in which the rates of marking, capture, and recapture are assumed to be equal. Using the one step transition probability matrix of their model, we illustrate how to go from an embedded discrete time Markov process to a continuous time Markov process assuming exponentially distributed holding times. We also compute the transition probabilities after time t for the continuous time case and compare the limiting behavior of the continuous and discrete time processes. Finally, we generalize their model by relaxing the assumption of equal per capita rates for marking, capture, and recapture. Other questions about how their results change when using a continuous time Markov process are examined.

Stokes Efficiency of Molecular Motor-Cargo Systems

A molecular motor utilizes chemical free energy to generate a unidirectional motion through the viscous fluid. In many experimental settings and biological settings, a molecular motor is elastically linked to a cargo. The stochastic motion of a molecular motor-cargo system is governed by a set of Langevin equations, each corresponding to an individual chemical occupancy state. The change of chemical occupancy state is modeled by a continuous time discrete space Markov process. The probability density of a motor-cargo system is governed by a two-dimensional Fokker-Planck equation. The operation of a molecular motor is dominated by high viscous friction and large thermal fluctuations from surrounding fluid. The instantaneous velocity of a molecular motor is highly stochastic: the past velocity is quickly damped by the viscous friction and the new velocity is quickly excited by bombardments of surrounding fluid molecules. Thus, the theory for macroscopic motors should not be applied directly to molecular motors without close examination. In particular, a molecular motor behaves differently working against a viscous drag than working against a conservative force. The Stokes efficiency was introduced to measure how efficiently a motor uses chemical free energy to drive against viscous drag. For a motor without cargo, it was proved that the Stokes efficiency is bounded by 100% [H. Wang and G. Oster, (2002)]. Here, we present a proof for the general motor-cargo system.

Accelerated Stochastic Simulation of Large Chemical Systems

For efficient simulation of chemical systems with large number ofreactions, we report a fast and exact algorithm for direct simulation ofchemical discrete Markov processes. The approach adopts the scheme oforganizing the reactions into hierarchical groups. By generating arandom number, the selection of the next reaction that actually occursis accomplished by a few successive selections in the hierarchicalgroups. The algorithm which is suited for simulating systems with largenumber of reactions is much faster than the direct method or theoptimized direct method. For a demonstration of its efficiency, theaccelerated algorithm is applied to simulate the reaction-diffusionBrusselator model on a discretized space.