Continuous-time Markov Chain Research Articles

A fast staffing algorithm for multistage call centers with impatient customers and time-dependent overflow

Ensuring customer satisfaction is one of the main objectives of a call center. We focus on the question of how many agents are necessary and how they should be allocated to maintain a service level threshold and reduce the expected waiting time of the customers. In this paper, we consider a multistage call center that consists of a front and a back office, impatient customers, and an overflow mechanism. Based on the performance evaluation of such a system using a continuous-time Markov chain, a configuration of agents is determined using a binary search algorithm. We focus on structural insights, e.g., convexity conditions, to obtain a quick solution for the staffing problem. Since monotonicity does not always hold, the approach is heuristic. The numerical results show the performance of the algorithm. The influence of the fraction requiring second-level service in the back office and the impatience rate for the minimum number of agents is shown.

Numerical Computation of Distributions in Finite-State Inhomogeneous Continuous Time Markov Chains, Based on Ergodicity Bounds and Piecewise Constant Approximation

In this paper it is shown, that if a possibly inhomogeneous Markov chain with continuous time and finite state space is weakly ergodic and all the entries of its intensity matrix are locally integrable, then, using available results from the perturbation theory, its time-dependent probability characteristics can be approximately obtained from another Markov chain, having piecewise constant intensities and the same state space. The approximation error (the taxicab distance between the state probability distributions) is provided. It is shown how the Cauchy operator and the state probability distribution for an arbitrary initial condition can be calculated. The findings are illustrated with the numerical examples.

Aerial computing: Enhancing mobile cloud computing with unmanned aerial vehicles as data bridges—A Markov chain based dependability quantification

Aerial Computing, utilizing unmanned aerial vehicles (UAVs), has emerged as a promising solution to enhance mobile cloud computing (MCC) infrastructure for the Internet of Things (IoT). The continuous generation of vast amounts of data by IoT devices requires efficient processing and monitoring for timely decision-making. However, wireless connections between IoT devices and remote servers can be unreliable, resulting in data loss. UAVs, with their increasing processing power and autonomy, can act as bridges between IoT devices and remote servers such as edge or cloud computing. In that context, this paper proposes a continuous time Markov chain (CTMC) models for an aerial computing system to evaluate system dependability metrics including availability and reliability. Sensitivity analysis is conducted to provide extended CTMC models with improved system availability. The proposed advanced model reduces downtime by 62 h compared to the baseline model, showcasing the potential of UAVs in enhancing the availability and reliability of MCC infrastructures. The use of UAVs and MCC in aerial computing is believed to be a win–win solution for cost-effective and energy-saving communication and computation services in various environments.

The asymptotic tails of limit distributions of continuous-time Markov chains

AbstractThis paper investigates tail asymptotics of stationary distributions and quasi-stationary distributions (QSDs) of continuous-time Markov chains on subsets of the non-negative integers. Based on the so-called flux-balance equation, we establish identities for stationary measures and QSDs, which we use to derive tail asymptotics. In particular, for continuous-time Markov chains with asymptotic power law transition rates, tail asymptotics for stationary distributions and QSDs are classified into three types using three easily computable parameters: (i) super-exponential distributions, (ii) exponential-tailed distributions, and (iii) sub-exponential distributions. Our approach to establish tail asymptotics of stationary distributions is different from the classical semimartingale approach, and we do not impose ergodicity or moment bound conditions. In particular, the results also hold for explosive Markov chains, for which multiple stationary distributions may exist. Furthermore, our results on tail asymptotics of QSDs seem new. We apply our results to biochemical reaction networks, a general single-cell stochastic gene expression model, an extended class of branching processes, and stochastic population processes with bursty reproduction, none of which are birth–death processes. Our approach, together with the identities, easily extends to discrete-time Markov chains.

Applications of the Linear Quadratic Regulator Optimal Control With Completely and Partially Observed Markovian Switching

Abstract We study optimal control problems where the dynamic system evolves according to a linear stochastic differential equation with (a) multiplicative and additive white noise, (b) a mixture of white noise, and (c) white noise and colored noise. The drift rate and the diffusion matrix of the linear dynamic system depend on a continuous-time Markov chain with finite state space that is (I) partially observed and (II) completely observed. Using the results of Wonham filter theory, we reduce the partially observed problems to one with complete observation. We solve the control optimal problems explicitly with the help of dynamic programing technique, and three applications are presented to illustrate our theoretical results.

Pricing European Vulnerable Options with Jumps and Stochastic Default Obstacles Barrier under Regime Switching

In this paper, we propose an enhanced model for pricing vulnerable options. Specifically, our model assumes that parameters such as interest rates, jump intensity, and asset value volatility are governed by an observable continuous-time finite-state Markov chain. We take into account European vulnerable options that are exposed to both default risk and rare shocks from underlying and counterparty assets. We also consider stochastic default barriers driven by a regime-switching model and geometric Brownian motion, thus improving upon the assumption of fixed default barriers. The risky assets follow a related jump-diffusion process, whereas the default barriers are influenced by a geometric Brownian motion correlated with the risky assets. Within the framework of our model, we derive an explicit pricing formula for European vulnerable options. Furthermore, we conduct numerical simulations to examine the effects of default barriers and other related parameters on option prices. Our findings indicate that stochastic default barriers increase credit risk, resulting in a decrease in option prices. By considering the aforementioned factors, our research contributes to a better understanding of pricing vulnerable options in the context of counterparty credit risk in over-the-counter trading.

On the study of slow–fast dynamics, when the fast process has multiple invariant measures

Motivated by applications to mathematical biology, we study the averaging problem for slow–fast systems, in the case in which the fast dynamics is a stochastic process with multiple invariant measures . We consider both the case in which the fast process is decoupled from the slow process and the case in which the two components are fully coupled. We work in the setting in which the slow process evolves according to an ordinary differential equation (ODE) and the fast process is a continuous time Markov process with finite state space and show that, in this setting, the limiting (averaged) dynamics can be described as a random ODE (i.e. an ODE with random coefficients).

Optimal stopping and impulse control in the presence of an anticipated regime switch

We consider a class of stochastic optimal stopping and impulse control problems where the agent solving the problem anticipates that a regime switch will happen at a random time in the future. We assume that there are only two regimes, the regime switching time is exponentially distributed, the underlying stochastic process is a linear, regular, time-homogeneous diffusion in both regimes and the payoff may be regime-dependent. This is in contrast with most existing literature on the topic, where regime switching is modulated by a continuous-time Markov chain and the underlying process and payoff belong to the same parametric family in all regimes. We state a set of easily verifiable sufficient conditions under which the solutions to these problems are given by one-sided threshold strategies. We prove uniqueness of the thresholds and characterize them as solutions to certain algebraic equations. We also study how anticipation affects optimal policies i.e. we present various comparison results for problems with and without regime switching. It may happen that the anticipative value functions and optimal policies coincide with the usual ones even if the regime switching structure is non-trivial. We illustrate our results with practical examples.

Analytical model for performability evaluation of Practical Byzantine Fault-Tolerant systems

Designing systems tolerant to faults is crucial to assure continuity of service for mission critical applications. However, their implementation may be costly and challenging. In this study, analytical models are presented for performance evaluation of systems equipped with Practical Byzantine Fault-Tolerant consensus protocols. Byzantine Fault Tolerance is particularly compelling, since it can provide a robust consensus mechanism to implement decentralized platforms, like Decentralized Ledger Technology and, notably, blockchains. The performability model is based on continuous-time Markov chains, in which the processes involved follow the exponential distribution. The numerical results presented report an inverse non-linear relation between number of nodes and performability. Performance decreases also as the ratio between break-down rate and repair rate increases.

Criteria on recurrence and uniqueness of time-continuous Markov chains with uniformly upward or downward finite ranges

Criteria on recurrence and uniqueness of time-continuous Markov chains with uniformly upward or downward finite ranges

Investigation of continuous time Markov chain approximate transient analysis via subgraph generation in the case of a car sharing system model

We present an investigation of the generation of partially absorbing continuous time Markov chains (CTMCs) for the purpose of approximating transient analysis of the underlying Markov chain. The Markov chain model of a car-sharing system, considering realistic parameter values such as fleet size and number of zones, is too large to be analyzed in its entirety. To obtain an approximate estimation of the changes in system characteristics starting from a given initial state, it is sufficient to generate and analyze a small fraction of the complete CTMC. We address the problem of identifying which states should be included for analysis by employing subgraph generation algorithms. The numerical results validate some of our insights about the problem and also provide suggestions for further research.

Analysis of a Pre-Emptive Two-Priority Queuing System with Impatient Customers and Heterogeneous Servers

This paper presents a queuing system model that incorporates multiple priorities, multiple abandonments, and heterogeneous servers. Waiting for service easily leads to impatient behaviors. The impact of two kinds of impatient behaviors, balking and reneging, on queueing system performance is examined. The problem is formulated as continuous-time Markov chains. It also introduces a special state called the non-sojourn state to record the number of customers who abandon the system. The state transition rate matrix is transformed into a block tridiagonal matrix by appropriately setting the state numbers. A novel indicator called interstate transition frequency is proposed, which aids in distinguishing state transitions during the system evaluation process. Based on the interstate transition frequency, a set of indicators is derived to offer additional analytical perspectives for the queuing system. Finally, the proposed model is applied to an automobile repair shop to validate its effectiveness in practical scenarios.

Metastability from the large deviations point of view: A [formula omitted]-expansion of the level two large deviations rate functional of non-reversible finite-state Markov chains

Consider a sequence of continuous-time Markov chains (Xt(n):t≥0) evolving on a fixed finite state space V. Let In be the level two large deviations rate functional for Xt(n), as t→∞. Under a hypothesis on the jump rates, we prove that In can be written as In=I(0)+∑1≤p≤q(1/θn(p))I(p) for some rate functionals I(p). The weights θn(p) correspond to the time-scales at which the sequence of Markov chains Xt(n) exhibit a metastable behavior, and the zero level sets of the rate functionals I(p) identify the metastable states.

Analysis of Green Closed-Loop Supply Chain Efficiency under Generalized Stochastic Petri Nets

In this paper, we aim to explore the operational performance of a green closed-loop supply chain under random events. A green closed-loop supply chain model based on generalized stochastic Petri nets (GSPN) is built using the Petri nets theory. According to the isomorphic relationship between GSPN and continuous-time Markov chains, the relevant Markov model is converted from GSPN, and the steady-state probability of the model is then calculated. Finally, the model is analyzed from the aspects of time performance and operation efficiency of each link. Compared to previous studies, this paper finds that: when the whole green closed-loop supply chain system reaches a dynamic equilibrium state, the product has a steady-state probability at all stages, and thus the overall operational performance of the system can be obtained; compared with the recycling of waste products, the green product takes a longer time in the production and distribution stages; since marketing, packaging processing, market feedback, and market demand formulation account for a high level of utilization throughout the life cycle of green products, decision makers need to focus on the supervision and management of these links. Managers of green closed-loop supply chain systems need to adjust their decision-making strategies in a timely manner according to the performance level of the system in the steady state to realize the efficient operation of the system. This paper not only provides theoretical support for the improvement of the operational efficiency of green closed-loop supply chain system, but also provides new ideas for the research of green closed-loop supply chain operation mode.

Joint Distribution of Ages of Information in Networks

We study a general setting of status updating systems in which a set of source nodes provide status updates about some physical process(es) to a set of monitors. The freshness of information available at each monitor is quantified in terms of the Age of Information (AoI), and the vector of AoI processes at the monitors (or equivalently the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">age vector</i> ) models the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">continuous</i> state of the system. While the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">marginal</i> distributional properties of each AoI process have been studied for a variety of settings using the stochastic hybrid system (SHS) approach, we lack a counterpart of this approach to systematically study their <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint</i> distributional properties. Developing such a framework is the main contribution of this paper. In particular, we model the discrete state of the system as a finite-state continuous-time Markov chain, and describe the coupled evolution of the continuous and discrete states of the system by a piecewise linear SHS with linear reset maps. Using the notion of tensors, we first derive first-order linear differential equations for the temporal evolution of both the joint moments and the joint moment generating function (MGF) for an arbitrary set of age processes. We then characterize the conditions under which the derived differential equations are asymptotically stable. The generality of our framework is demonstrated by recovering several existing results as special cases. Finally, we apply our framework to derive closed-form expressions of the stationary joint MGF in a multi-source updating system under non-preemptive and source-agnostic/ source-aware preemptive in service queueing disciplines.

Blocking Rate Analysis in Heterogeneous Multicell Cognitive-Radio 5G Networks

Cognitive radio (CR) technology is being recognized as one of the main enabling technologies for supporting massive high-speed connectivity in multicell fifth-generation (5G) networks by allowing dynamic spectrum access policies according to predetermined etiquette. Under this paradigm, the available spectrum is opportunistically allocated to the different cells. A key metric that is heavily employed in designing efficient CR-based multicell spectrum-allocation mechanisms is the blocking rate. An efficient spectrum-allocation mechanism aims to adaptively allocate channels to the different cells such that the overall blocking rate is minimized. In this article, we use the theory of continuous-time Markov chains to derive a closed-form expression for the blocking rate in multicell CR-based 5G networks. In our derivations, we build two detailed mathematical models describing the frequency-spectrum usage of both primary and secondary users. We combine the two models in a hierarchical fashion. As opposed to related models in the literature which suffer high complexity, the proposed hierarchical structure yields simplicity and tractability. Furthermore, while existing research works overlook spectrum heterogeneity, the proposed model addresses this important factor. Simulation results demonstrate high accuracy in evaluating the blocking rate across the different cells in the network.

A periodic review inventory model facing different disruption profiles

We study a periodic review inventory system with alternations of periods with supply availability and unavailability. The system is controlled by a periodic review base stock policy (S,T). While periodic review inventory models have been studied assuming supply disruption, they mainly invoke the limiting disruption probabilities overlooking that these probabilities may correspond to different disruption profiles (ranging from rare-long disruptions to frequent-short ones). In this context, we use a continuous-time Markov chain (CTMC), in order to formulate a periodic review inventory model that can distinguish the different disruption profiles. Through theoretical results, we determine the policy variables (order-up-to level and review interval) that minimize the long-run average cost. We show that when the disruption profile is considered, the optimal order-up-to level, although Newsvendor-styled form, is not always linear in the review interval. We present analytical properties demonstrating the impact of the different disruption profiles (under the same limiting disruption probability) and the impact of the limiting probability on the optimal policy. Furthermore, we propose an approximation to obtain a closed-form (near-)optimal solution for the review interval. Finally, numerical comparisons highlight the impact on system's cost performance of deploying approaches that either use the above mentioned closed-form solution for the review interval or the limiting disruption probabilities instead of the time dependent ones in the cost formulation.

Stochastic Reaction Networks Within Interacting Compartments.

Stochastic reaction networks, which are usually modeled as continuous-time Markov chains on [Formula: see text], and simulated via a version of the "Gillespie algorithm," have proven to be a useful tool for the understanding of processes, chemical and otherwise, in homogeneous environments. There are multiple avenues for generalizing away from the assumption that the environment is homogeneous, with the proper modeling choice dependent upon the context of the problem being considered. One such generalization was recently introduced in Duso and Zechner (Proc Nat Acad Sci 117(37):22674-22683 , Duso and Zechner (2020)), where the proposed model includes a varying number of interacting compartments, or cells, each of which contains an evolving copy of the stochastic reaction system. The novelty of the model is that these compartments also interact via the merging of two compartments (including their contents), the splitting of one compartment into two, and the appearance and destruction of compartments. In this paper we begin a systematic exploration of the mathematical properties of this model. We (i) obtain basic/foundational results pertaining to explosivity, transience, recurrence, and positive recurrence of the model, (ii) explore a number of examples demonstrating some possible non-intuitive behaviors of the model, and (iii) identify the limiting distribution of the model in a special case that generalizes three formulas from an example in Duso and Zechner (Proc Nat Acad Sci 117(37):22674-22683 , Duso and Zechner (2020)).

The entropy production of stationary diffusions

The entropy production rate is a central quantity in non-equilibrium statistical physics, scoring how far a stochastic process is from being time-reversible. In this paper, we compute the entropy production of diffusion processes at non-equilibrium steady-state under the condition that the time-reversal of the diffusion remains a diffusion. We start by characterising the entropy production of both discrete and continuous-time Markov processes. We investigate the time-reversal of time-homogeneous stationary diffusions and recall the most general conditions for the reversibility of the diffusion property, which includes hypoelliptic and degenerate diffusions, and locally Lipschitz vector fields. We decompose the drift into its time-reversible and irreversible parts, or equivalently, the generator into symmetric and antisymmetric operators. We show the equivalence with a decomposition of the backward Kolmogorov equation considered in hypocoercivity theory, and a decomposition of the Fokker-Planck equation in GENERIC form. The main result shows that when the time-irreversible part of the drift is in the range of the volatility matrix (almost everywhere) the forward and time-reversed path space measures of the process are mutually equivalent, and evaluates the entropy production. When this does not hold, the measures are mutually singular and the entropy production is infinite. We verify these results using exact numerical simulations of linear diffusions. We illustrate the discrepancy between the entropy production of non-linear diffusions and their numerical simulations in several examples and illustrate how the entropy production can be used for accurate numerical simulation. Finally, we discuss the relationship between time-irreversibility and sampling efficiency, and how we can modify the definition of entropy production to score how far a process is from being generalised reversible.

Power utility maximization problem under rough stochastic local volatility models with continuous-time Markov chain approximation

This paper studies the portfolio optimization problem based on rough stochastic local volatility models. Due to the challenge that this model is neither a semimartingale nor a Markov process, we first introduce the semimartingale approximate optimization problem and present the convergence analysis. We use the classic stochastic control method and continuous-time Markov chain approximation approach to solve the approximate optimization problem respectively. Then the optimal investment strategy and the HJB equation satisfied by the value function are obtained respectively. Furthermore, for the classic stochastic control method, we demonstrate the existence of solutions for the associated partial differential equation via the nonlinear Feynman–Kac formula. For continuous-time Markov chain approximation method, we prove the convergence of the value function. Finally, we provide a numerical analysis to illustrate the above results by using the power utility function. The numerical analysis shows that continuous-time Markov chain approximation method is effective in solving the portfolio optimization problem under rough stochastic local volatility models.