Continuous-time Markov Chain Research Articles

Analysis of a Multi-Server Queue with Group Service and Service Time Dependent on the Size of a Group as a Model of a Delivery System

In this paper, we consider a multi-server queue with a finite buffer. Request arrivals are defined by the Markov arrival process. Service is provided to groups of requests. The minimal and maximal group sizes are fixed. The service time of a group has a phase-type distribution with an irreducible representation depending on the size of the group. The requests are impatient. The patience time for an arbitrary request has an exponential distribution. After this time expires, the request is lost if all servers are busy or, if some server is idle, with a certain probability, all requests staying in the buffer start their service even if their number is below the required minimum. The behavior of the system is described by a multi-dimensional continuous-time Markov chain that does not belong to the class of level-independent quasi-birth-and-death processes. The algorithm for the computation of the stationary distribution of this chain is presented, and expressions for the computation of the queuing system’s performance characteristics are derived. The description of a delivery system operation in terms of the analyzed queuing model is given, and the problem of the optimization of its operation is numerically solved. Multi-server queues with a phase-type distribution for the group service time that are dependent on the size of the group, the account of request impatience, and the correlated arrival process have not previously been analyzed in the existing literature. However, they represent a precise model of many real-world objects, including delivery systems.

Stochastic chemical kinetics of cell fate decision systems: From single cells to populations and back.

Stochastic chemical kinetics is a widely used formalism for studying stochasticity of chemical reactions inside single cells. Experimental studies of reaction networks are generally performed with cells that are part of a growing population, yet the population context is rarely taken into account when models are developed. Models that neglect the population context lose their validity whenever the studied system influences traits of cells that can be selected in the population, a property that naturally arises in the complex interplay between single-cell and population dynamics of cell fate decision systems. Here, we represent such systems as absorbing continuous-time Markov chains. We show that conditioning on non-absorption allows one to derive a modified master equation that tracks the time evolution of the expected population composition within a growing population. This allows us to derive consistent population dynamics models from a specification of the single-cell process. We use this approach to classify cell fate decision systems into two types that lead to different characteristic phases in emerging population dynamics. Subsequently, we deploy the gained insights to experimentally study a recurrent problem in biology: how to link plasmid copy number fluctuations and plasmid loss events inside single cells to growth of cell populations in dynamically changing environments.

A rapidly mixing Markov chain from any gapped quantum many-body system

We consider the computational task of sampling a bit string x from a distribution π(x)=|⟨x|ψ⟩|2, where ψ is the unique ground state of a local Hamiltonian H. Our main result describes a direct link between the inverse spectral gap of H and the mixing time of an associated continuous-time Markov Chain with steady state π. The Markov Chain can be implemented efficiently whenever ratios of ground state amplitudes ⟨y|ψ⟩/⟨x|ψ⟩ are efficiently computable, the spectral gap of H is at least inverse polynomial in the system size, and the starting state of the chain satisfies a mild technical condition that can be efficiently checked. This extends a previously known relationship between sign-problem free Hamiltonians and Markov chains. The tool which enables this generalization is the so-called fixed-node Hamiltonian construction, previously used in Quantum Monte Carlo simulations to address the fermionic sign problem. We implement the proposed sampling algorithm numerically and use it to sample from the ground state of Haldane-Shastry Hamiltonian with up to 56 qubits. We observe empirically that our Markov chain based on the fixed-node Hamiltonian mixes more rapidly than the standard Metropolis-Hastings Markov chain.

Reliability analysis and redundancy optimization of k-out-of-n systems with random variable k using continuous time Markov chain and Monte Carlo simulation

This article discusses the problems associated with the redundancy of structures k-out-of-n as a method of increasing system availability. A parallel k-out-of-n system was considered, with k repairable and homogeneous components. The other (n – k) objects are in hot redundancy mode. Failures and repairs are independent processes, and each repaired object is treated as full-fledged in terms of operation. The system operates under dynamically changing conditions, so the minimum number of operable components essential for its proper operation is a random variable with a specific distribution. Component damages and repairs are independent stochastic processes. A probabilistic and simulation approach based on Continuous Time Markov Chain and Monte Carlo simulation is proposed. This study aims to optimize the number of components in the k-out-of-n structure, which will ensure the system operation with a certain availability and performance. The proposed methods were applied to real transport systems, for which models were developed with parameters estimated on the basis of empirical data. The convergence of the results obtained, using two methods, testifies to the correctness of the methodology used and the reliability of the models developed. The risk to system performance associated with changing input parameters was assessed using model sensitivity analysis. As indicated, the model is not susceptible to changes in the damage and repair intensity. It has also been shown that optimization of the k-out-of-n structure brings a significant reduction in the costs incurred for system development and operation.

Marginal and joint distribution of inter-departure times for a PH/PH/c queue

In this paper, we introduce the Laplace–Stieltjes transform (LST) of the inter-departure time distribution in a PH/PH/c queue, and the two-dimensional joint LST of two consecutive inter-departure times to construct their correlation structure. We exploit the properties of phase-type (PH) random variables, as well as the steady-state distribution of the underlying continuous-time Markov chain in a PH/PH/c queue to construct these LSTs. We demonstrate our approach through numerical examples, while validating the results. Later, we analyze the correlation between two consecutive inter-departure times for various PH/PH/c queues. We observe that, if the fundamental elements of the queue have high (low) variability, then the correlation is positive (negative).

The linear framework II: using graph theory to analyse the transient regime of Markov processes.

The linear framework uses finite, directed graphs with labelled edges to model biomolecular systems. Graph vertices represent chemical species or molecular states, edges represent reactions or transitions and edge labels represent rates that also describe how the system is interacting with its environment. The present paper is a sequel to a recent review of the framework that focussed on how graph-theoretic methods give insight into steady states as rational algebraic functions of the edge labels. Here, we focus on the transient regime for systems that correspond to continuous-time Markov processes. In this case, the graph specifies the infinitesimal generator of the process. We show how the moments of the first-passage time distribution, and related quantities, such as splitting probabilities and conditional first-passage times, can also be expressed as rational algebraic functions of the labels. This capability is timely, as new experimental methods are finally giving access to the transient dynamic regime and revealing the computations and information processing that occur before a steady state is reached. We illustrate the concepts, methods and formulas through examples and show how the results may be used to illuminate previous findings in the literature.

Algorithmic Minimization of Uncertain Continuous-Time Markov Chains

The assumption of perfect knowledge of rate parameters in continuous-time Markov chains (CTMCs) is undermined when confronted with reality, where they may be uncertain due to lack of information or because of measurement noise. Here we consider uncertain CTMCs (UCTMCs), where rates are assumed to vary non-deterministically with time from bounded continuous intervals. An uncertain CTMC can be therefore seen as a specific type of Markov decision process for which the analysis is computationally difficult. To tackle this, we develop a theory of minimization which generalizes the notion of lumpability for CTMCs. Our first result is a quantitative and logical characterization of minimization. Specifically, we show that the reduced UCTMC model has a macro-state for each block of a partition of the state space, which preserves value functions and logical formulae whenever rewards are equal within each block. The second result is an efficient minimization algorithm for UCTMCs by means of partition refinement. As application, we show that reductions in a number of CTMCbenchmark models are robust with respect to uncertainties in original rates.

Optimal friction matrix for underdamped Langevin sampling

We propose a procedure for optimising the friction matrix of underdamped Langevin dynamics when used for continuous time Markov Chain Monte Carlo. Starting from a central limit theorem for the ergodic average, we present a new expression of the gradient of the asymptotic variance with respect to friction matrix. In addition, we present an approximation method that uses simulations of the associated first variation/tangent process. Our algorithm is applied to a variety of numerical examples such as toy problems with tractable asymptotic variance, diffusion bridge sampling and Bayesian inference problems for high dimensional logistic regression.

Antidisturbance Control for Hidden Markovian Jump Systems: Asynchronous Disturbance Observer Approach

This paper investigates the anti-disturbance control issue for a class of Markovian jump systems (MJSs) with matched/mismatched disturbances by using disturbance observer approach. A key feature here is that the external disturbance is considered to be generated by a Markovian jump exogenous dynamics, in which the disturbance mode may not be synchronized with the system mode. Meanwhile, it is supposed that both system and disturbance modes can just be estimated via a hidden Markov mode detector. To overcome the possible obstacle in handling dual hidden Markov modes, a mapping technique for continuous-time Markov chain is provided firstly. Then, a novel asynchronous disturbance observer is developed to estimate the external disturbance by utilizing the detected mode. Based upon the estimated disturbance, asynchronous anti-disturbance controllers are investigated for matched and mismatched disturbance cases, respectively. By employing a decoupling technique with the idea of completing the square, sufficient conditions are established for guaranteeing the stability in probability of the resultant closed-loop systems. Finally, two numerical examples are provided to verify the effectiveness of the proposed asynchronous anti-disturbance control approaches under inaccessible system and disturbance modes.

Dimensioning the pending interest table in content-centric networks

In chunk-based interest-driven content-centric networks, each interest packet forwarded upstream by a node on a face implies the return of at most one data chunk to the node from that face shortly after. As a consequence, the congestion of the downstream transmission buffer in the data path of the node is highly correlated with the occupancy of the pending interest table (PIT). Therefore a systematic study and analysis of the PIT occupancy are of paramount importance to understanding congestion in CCN. In particular, in this paper, we propose an analytical model to estimate the PIT occupancy distribution via a continuous-time Markov chain (CTMC) model that considers the effects of interest blocking, interest timeout and retries. To validate our model and assumptions, we invoke simulation with realistic traffic streams and show how the filtering effects of caching and interest aggregation make the Markov assumption reasonable in the nodes that are the most susceptible to experiencing congestion. To solve the model numerically, we use two alternative approximations, state space truncation and state aggregation, and then give some numerical results to demonstrate the accuracy of our approximations.

Failure time prediction for vehicle dynamics under performance degradation of dampers and track evolution

During long-term service, railway vehicles will experience damper performance degradation and track evolution, both have significant effects on the vehicle dynamics performance. This study aims to develop a mathematical model for assessing the failure time of the vehicle system dynamics when considering two random factors. As for the methodology, Wiener processes are used to propose a damper degradation model. Moreover, a continuous-time Markov chain is adopted to describe the state transaction of the track irregularities. In addition, the Monte Carlo method is employed to sample the dynamic responses of the vehicle using stochastic parameters. Finally, the Bayesian theory is utilized to establish the reliability model of the Weibull mixtures under the Dirichlet process prior. The results are obtained through a numerical example: the dampers’ individual differences directly affect effect on the rate of degradation of the vehicle dynamics performance. Furthermore, different indexes positively correlate with each other, mixture models are composed of a small number of components, and Weibull mixtures closely approximate the distribution of the failure time. Compared to other dampers, the anti-yaw dampers have a more significant impact on the vehicle system dynamics. Other results show that each type of damper can adopt a uniform replacement period whereas anti-yaw dampers should be more effectively monitored.

Gene Transfer-Based Phylogenetics: Analytical Expressions and Additivity via Birth-Death Theory.

The genomic era has opened up vast opportunities in molecular systematics, one of which is deciphering the evolutionary history in fine detail. Under this mass of data, analyzing the point mutations of standard markers is often too crude and slow for fine-scale phylogenetics. Nevertheless, genome dynamics (GD) events provide alternative, often richer information. The synteny index (SI) between a pair of genomes combines gene order and gene content information, allowing the comparison of genomes of unequal gene content, together with order considerations of their common genes. Recently, genome dynamics has been modeled as a continuous-time Markov process, and gene distance in the genome as a birth-death-immigration process. Nevertheless, due to complexities arising in this setting, no precise and provably consistent estimators could be derived, resulting in heuristic solutions. Here, we extend this modeling approach by using techniques from birth-death theory to derive explicit expressions of the system's probabilistic dynamics in the form of rational functions of the model parameters. This, in turn, allows us to infer analytically accurate distances between organisms based on their SI. Subsequently, we establish additivity of this estimated evolutionary distance (a desirable property yielding phylogenetic consistency). Applying the new measure in simulation studies shows that it provides accurate results in realistic settings and even under model extensions such as gene gain/loss or over a tree structure. In the real-data realm, we applied the new formulation to unique data structure that we constructed-the ordered orthology DB-based on a new version of the EggNOG database, to construct a tree with more than 4.5K taxa. To the best of our knowledge, this is the largest gene-order-based tree constructed and it overcomes shortcomings found in previous approaches. Constructing a GD-based tree allows to confirm and contrast findings based on other phylogenetic approaches, as we show.

Performance modeling and analysis for randomly walking mobile users with Markov chains

We treat user equipments (UEs) and mobile edge clouds (MECs) as M/G/1 queueing systems, which are the most suitable, powerful, and manageable models. We propose a computation offloading strategy which can satisfy all UEs served by an MEC and develop an efficient method to find such a strategy. We use discrete-time Markov chains, continuous-time Markov chains, and semi-Markov processes to characterize the mobility of UEs, and calculate the joint probability distribution of the locations of UEs at any time. We extend our Markov chains to incorporate mobility cost into consideration, and are able to obtain the average response time of a UE with location change penalty. We can algorithmically predict the overall average response time of tasks generated on a UE and also demonstrate numerical data and examples. We consider the power constrained MEC speed setting problem and develop an algorithm to solve the problem for two power consumption models.

Performability analysis of multi-service call admission control schemes in LTE networks

Two important Quality-of-Service (QoS) measures for cellular networks are the fractions of New Calls (NCs) and Handoff Calls (HCs). These types of calls are usually blocked or dropped for two reasons: radio channels are occupied (performance reason) or failure of radio channels (availability reason). Call Admission Control (CAC) is a technique that accepts or rejects user requests in a network. This paper proposes two new CAC schemes for Long Term Evolution (LTE) networks supporting two classes of services (Real-Time (RT) and Non-Real-Time (NRT)). The proposed CAC schemes are composites, meaning that they consider performance changes associated with the failure and recovery of radio channels. We use the Probabilistic Model Checking (PMC) to perform a comparative analysis between the performability measures (combination of performance and availability measures) of the proposed composite CAC schemes and an existing pure performance model. First, we model the composite CAC schemes with multidimensional Continuous Time Markov Chain (CTMC). Then, we specify QoS requirements through the CTMC using the Continuous-time Stochastic Logic (CSL). Finally, we quantify the performability measures of the composite CAC schemes by checking CSL steady-state, transient, and path formulas using the PRISM model checker.

Statistical inference for mixed jump processes by Markov switching model with application to identify seismicity levels

Describing the non-stationarity of mixed stochastic jump processes presents a formidable challenge. This study employs a mixed compound Poisson process by Markov switching model (CPMSM) to address the issue of overdispersed data of random jumps. The Poisson rate is governed by a continuous-time Markov chain in this model. By assuming independence of the compound Poisson process and jump sizes, we present a generalized algorithm to estimate the parameters of CPMSM and try to predict the future behavior of this stochastic system. To investigate the potential application of this model and the accuracy of our algorithm, we provide a numerical simulation example in the context of the auto insurance payout scenario. The results suggest that CPMSM is an effective method of describing jump behavior controlled by random counts with mode switching via a Markov chain. Furthermore, we demonstrate a specific application of this model in identifying seismicity levels of Greek earthquake data and identify four hidden states. Finally, we compare its fitting effectiveness with the ETAS model using two public earthquake datasets in Japan and Iran.

A fast staffing algorithm for multistage call centers with impatient customers and time-dependent overflow

Ensuring customer satisfaction is one of the main objectives of a call center. We focus on the question of how many agents are necessary and how they should be allocated to maintain a service level threshold and reduce the expected waiting time of the customers. In this paper, we consider a multistage call center that consists of a front and a back office, impatient customers, and an overflow mechanism. Based on the performance evaluation of such a system using a continuous-time Markov chain, a configuration of agents is determined using a binary search algorithm. We focus on structural insights, e.g., convexity conditions, to obtain a quick solution for the staffing problem. Since monotonicity does not always hold, the approach is heuristic. The numerical results show the performance of the algorithm. The influence of the fraction requiring second-level service in the back office and the impatience rate for the minimum number of agents is shown.

Numerical Computation of Distributions in Finite-State Inhomogeneous Continuous Time Markov Chains, Based on Ergodicity Bounds and Piecewise Constant Approximation

In this paper it is shown, that if a possibly inhomogeneous Markov chain with continuous time and finite state space is weakly ergodic and all the entries of its intensity matrix are locally integrable, then, using available results from the perturbation theory, its time-dependent probability characteristics can be approximately obtained from another Markov chain, having piecewise constant intensities and the same state space. The approximation error (the taxicab distance between the state probability distributions) is provided. It is shown how the Cauchy operator and the state probability distribution for an arbitrary initial condition can be calculated. The findings are illustrated with the numerical examples.

Aerial computing: Enhancing mobile cloud computing with unmanned aerial vehicles as data bridges—A Markov chain based dependability quantification

Aerial Computing, utilizing unmanned aerial vehicles (UAVs), has emerged as a promising solution to enhance mobile cloud computing (MCC) infrastructure for the Internet of Things (IoT). The continuous generation of vast amounts of data by IoT devices requires efficient processing and monitoring for timely decision-making. However, wireless connections between IoT devices and remote servers can be unreliable, resulting in data loss. UAVs, with their increasing processing power and autonomy, can act as bridges between IoT devices and remote servers such as edge or cloud computing. In that context, this paper proposes a continuous time Markov chain (CTMC) models for an aerial computing system to evaluate system dependability metrics including availability and reliability. Sensitivity analysis is conducted to provide extended CTMC models with improved system availability. The proposed advanced model reduces downtime by 62 h compared to the baseline model, showcasing the potential of UAVs in enhancing the availability and reliability of MCC infrastructures. The use of UAVs and MCC in aerial computing is believed to be a win–win solution for cost-effective and energy-saving communication and computation services in various environments.

The asymptotic tails of limit distributions of continuous-time Markov chains

AbstractThis paper investigates tail asymptotics of stationary distributions and quasi-stationary distributions (QSDs) of continuous-time Markov chains on subsets of the non-negative integers. Based on the so-called flux-balance equation, we establish identities for stationary measures and QSDs, which we use to derive tail asymptotics. In particular, for continuous-time Markov chains with asymptotic power law transition rates, tail asymptotics for stationary distributions and QSDs are classified into three types using three easily computable parameters: (i) super-exponential distributions, (ii) exponential-tailed distributions, and (iii) sub-exponential distributions. Our approach to establish tail asymptotics of stationary distributions is different from the classical semimartingale approach, and we do not impose ergodicity or moment bound conditions. In particular, the results also hold for explosive Markov chains, for which multiple stationary distributions may exist. Furthermore, our results on tail asymptotics of QSDs seem new. We apply our results to biochemical reaction networks, a general single-cell stochastic gene expression model, an extended class of branching processes, and stochastic population processes with bursty reproduction, none of which are birth–death processes. Our approach, together with the identities, easily extends to discrete-time Markov chains.

Applications of the Linear Quadratic Regulator Optimal Control With Completely and Partially Observed Markovian Switching

Abstract We study optimal control problems where the dynamic system evolves according to a linear stochastic differential equation with (a) multiplicative and additive white noise, (b) a mixture of white noise, and (c) white noise and colored noise. The drift rate and the diffusion matrix of the linear dynamic system depend on a continuous-time Markov chain with finite state space that is (I) partially observed and (II) completely observed. Using the results of Wonham filter theory, we reduce the partially observed problems to one with complete observation. We solve the control optimal problems explicitly with the help of dynamic programing technique, and three applications are presented to illustrate our theoretical results.