Jump Markov Processes Research Articles

Controllable Markov Jump Processes. II. Monitoring and Optimization of TCP Connections

Controllable Markov Jump Processes. II. Monitoring and Optimization of TCP Connections

Large deviations theory for Markov jump models of chemical reaction networks

We prove a sample path Large Deviation Principle (LDP) for a class of jump processes whose rates are not uniformly Lipschitz continuous in phase space. Building on it, we further establish the corresponding Wentzell–Freidlin (W-F) (infinite time horizon) asymptotic theory. These results apply to jump Markov processes that model the dynamics of chemical reaction networks under mass action kinetics, on a microscopic scale. We provide natural sufficient topological conditions for the applicability of our LDP and W-F results. This then justifies the computation of nonequilibrium potential and exponential transition time estimates between different attractors in the large volume limit, for systems that are beyond the reach of standard chemical reaction network theory.

Impact of Mode Decision Delay on Estimation Error in Continuous-Time Controlled System

For highly maneuvering target interception in terminal guidance, the maximal admissible mode decision delay (MAMDD) and the least required mode sojourn time (LRMST) are calculated in continuous-time controlled system. The lateral acceleration command of the evader is modeled as a jump-Markov process. Suppose that the mode switches are observable and the evader’s motion mode is correctly identified by a separate mode decision-maker with a fixed time delay, then the error model of the state estimation is derived. By limiting the state estimation error to a certain range, MAMDD and LRMST are calculated. The derivation is validated by a typical instance of tactical ballistic missile interception. The results in this paper provide a useful tool for guiding the design of the mode decision-maker and evaluating the performance of the logic-based integrated estimation and guidance system.

Multinomial approximation to the Kolmogorov Forward Equation for jump (population) processes

We develop a simulation method for Markov Jump processes with finite time steps based in a quasilinear approximation of the process and in multinomial random deviates. The second-order approximatio...

Conditional Joint Probability Distributions of First Exit Times to Overlapping Absorbing Sets of the Mixture of Markov Jump Processes

New results on conditional joint probability distributions of first exit times are presented for a continuous-time stochastic process defined as the mixture of Markov jump processes moving at different speeds on the same finite state space, while the mixture occurs at a random time. Such mixture was first proposed by Frydman \cite{Frydman2005} and Frydman and Schuermann \cite{Frydman2008} as a generalization of the mover-stayer model of Blumen et at. \cite{Blumen}, and was recently extended by Surya \cite{Surya2018}, in which explicit distributional identities of the process are given, in particular in the presence of an absorbing state. We revisit \cite{Surya2018} for a finite mixture with overlapping absorbing sets. The contribution of this paper is two fold. First, we generalize distributional properties of the mixture discussed in \cite{Frydman2008} and \cite{Surya2018}. Secondly, we give distributional identities of the first exit times explicitly in terms of intensity matrices of the underlying Markov processes and the Bayesian updates of switching probability and of the probability distribution of states, despite the fact that the mixture itself is non-Markov. They form non-stationary functions of time and have the ability to capture heterogeneity and path dependence when conditioning on the available information (either full or partial) of the process. In particular, initial profile of the distributions forms of a generalized mixture of multivariate phase-type distributions of Assaf et al. \cite{Assaf1984}. When the underlying processes move at the same speed, in which case the mixture becomes a simple Markov process, these features are removed, and the initial distributions reduce to \cite{Assaf1984}. Some explicit and numerical examples are discussed to illustrate the main results.

Construction of sampling trajectories of coupled diffusion\\ processes and its simulation algorithm

Coupled diffusion process is a system in which a diffusion process is coupled with a Markov jumping process. To simulate such a hybrid process, we consider a product space formed by a Lebesgue space on $[0,1]^{\infty}$ and a Wiener space on $\mathcal{C}(\mathbb R_+,\mathbb R^r)$, and intuitively construct a coupled diffusion process on this probability space. We prove that the constructed process is a Markov process, and present the probability distribution of the residual lifetime as well as the joint probability distribution of the next reaction index and the residual lifetime. We further consider the infinitesimal generator and the martingale problem of the process, and prove the solution to the martingale problem is unique in distribution. Based on the sampling process we constructed, we propose an algorithm to simulate the coupled diffusion process.

A Uniformization Approach for the Dynamic Control of Queueing Systems with Abandonments

We consider queueing systems with general abandonment. Abandonment times are approximated by a particular Cox distribution with all phase exponential rates being the same. We prove that this distribution arbitrarily closely approximates any nonnegative distribution. By explicitly modeling the waiting time of the first customer in line, we obtain a natural bounded jump Markov process allowing for uniformization. This approach is useful to solve, via dynamic programming, various optimization problems where the objectives and/or constraints involve the distributions of the performance measures, not only their expected values. It is also useful for the performance analysis of queueing systems with general abandonment times.

Fluctuation theorem: A critical review.

Fluctuation theorem for entropy production is revisited in the framework of stochastic processes. The applicability of the fluctuation theorem to physico-chemical systems and the resulting stochastic thermodynamics were analyzed. Some unexpected limitations are highlighted in the context of jump Markov processes. We have shown that these limitations handicap the ability of the resulting stochastic thermodynamics to correctly describe the state of non-equilibrium systems in terms of the thermodynamic properties of individual processes therein. Finally, we considered the case of diffusion processes and proved that the fluctuation theorem for entropy production becomes irrelevant at the stationary state in the case of one variable systems.

Asymptotic expansions for Laplace transforms of Markov processes

Let μϵ be the probability measures on D[0,T] of suitable Markov processes {ξtϵ}0≤t≤T (possibly with small jumps) depending on a small parameter ϵ>0, where D[0,T] denotes the space of all functions on [0,T] which are right continuous with left limits. In this paper we investigate asymptotic expansions for the Laplace transforms ∫D[0,T]exp⁡{ϵ−1F(x)}μϵ(dx) as ϵ→0 for smooth functionals F on D[0,T]. This study not only recovers several well-known results, but more importantly provides new expansions for jump Markov processes. Besides several standard tools such as exponential change of measures and Taylor's expansions, the novelty of the proof is to implement the expectation asymptotic expansions on normal deviations which were recently derived in [13].

Hausdorff Dimension of the Range and the Graph of Stable-Like Processes

We determine the Hausdorff dimension for the range of a class of pure jump Markov processes in $\mathbb{R}^d$, which turns out to be random and depends on the trajectories of these processes. The key argument is carried out through the SDE representation of these processes. The method developed here also allows to compute the Hausdorff dimension for the graph.

Kolmogorov’s equations for jump Markov processes with unbounded jump rates

As is well-known, transition probabilities of jump Markov processes satisfy Kolmogorov’s backward and forward equations. In the seminal 1940 paper, William Feller investigated solutions of Kolmogorov’s equations for jump Markov processes. Recently the authors solved the problem studied by Feller and showed that the minimal solution of Kolmogorov’s backward and forward equations is the transition probability of the corresponding jump Markov process if the transition rate at each state is bounded. This paper presents more general results. For Kolmogorov’s backward equation, the sufficient condition for the described property of the minimal solution is that the transition rate at each state is locally integrable, and for Kolmogorov’s forward equation the corresponding sufficient condition is that the transition rate at each state is locally bounded.

Decentralized quantized control for NCSs under periodic protocol

This paper is concerned with the problem of designing decentralized quantized controller for networked control systems (NCSs) that are subject to time-varying transmission intervals, packet dropout, communication constraints and limited bandwidth. The convex overapproximation method proposed in Donkers and Heemels (2011) is used to handle with time-varying transmission intervals. Packet dropout process in this paper is modeled as both independent identically distributed process and homogeneous Markov jumping process. The uniform quantizer is adopted here to deal with limited bandwidth. In quantization strategy, which switches between “zooming out” and “zooming in” modes, a variable called proportionality coefficient of saturation value is used to handle with the complex coupling relationship between system states and quantized variables. Periodic protocol is used here to deal with communication constraints, under which only one node can access the network and send its information per transmission. Within this setup, we derive stability results in terms of linear matrix inequalities (LMIs). An illustrative example is given to show the usefulness of the proposed framework for the stability analysis of some classes of NCSs.

Game Theoretic Decentralized Feedback Controls in Markov Jump Processes

This paper studies a decentralized routing problem over a network, using the paradigm of mean-field games with large number of players. Building on a state-space extension technique, we turn the problem into an optimal control one for each single player. The main contribution is an explicit expression of the optimal decentralized control which guarantees the convergence both to local and to global equilibrium points. Furthermore, we study the stability of the system also in the presence of a delay which we model using an hysteresis operator. As a result of the hysteresis, we prove existence of multiple equilibrium points and analyze convergence conditions. The stability of the system is illustrated via numerical studies.

A Large Deviations Analysis of Certain Qualitative Properties of Parallel Tempering and Infinite Swapping Algorithms

Parallel tempering, or replica exchange, is a popular method for simulating complex systems. The idea is to run parallel simulations at different temperatures, and at a given swap rate exchange configurations between the parallel simulations. From the perspective of large deviations it is optimal to let the swap rate tend to infinity and it is possible to construct a corresponding simulation scheme, known as infinite swapping. In this paper we propose a novel use of large deviations for empirical measures for a more detailed analysis of the infinite swapping limit in the setting of continuous time jump Markov processes. Using the large deviations rate function and associated stochastic control problems we consider a diagnostic based on temperature assignments, which can be easily computed during a simulation. We show that the convergence of this diagnostic to its a priori known limit is a necessary condition for the convergence of infinite swapping. The rate function is also used to investigate the impact of asymmetries in the underlying potential landscape, and where in the state space poor sampling is most likely to occur.

Conditional Phase-Type Distribution Under Doubly Stochastic Jump Markov Processes with Observed Covariates

Empirical evidences on corporate bond found e.g. in Frydman (2005) and Frydman and Schuermann (2008) suggest the facts that the attribution of constant intensity in the credit rating dynamics, represented by finite-state jump Markov processes, is not borne out by actual data. On the other hand, it is known facts that continuous-time finite-state jump Markov processes can be represented by means of Poisson process and embedded discrete-time Markov chains, see, e.g., Ch. 7 of Pardoux (2008), Sec. 5.10 of Resnick (2002) and and Jakubowski and Nieweglowski (2010, 2008). Motivated by the above facts, we consider representation of jump Markov processes in terms of doubly stochastic Poisson process (see e.g. Cox (1955), Kingman (1964), Serfozo (1972), and Bremaud (1981)) whose intensity is driven by observed explanatory covariates. However, we impose weaker conditions than that of specified in Kingman (1964) and Jakubowski and Nieweglowski (2010, 2008) in the construction of such jump Markov process in which the conditional rate of jump arrival in the conditional Poisson process follows the landmarking approach. This approach has been recently introduced in event history analysis by van Houwelingen (2007) and van Houwelingen and Putter (2012). By this approach, we do not require the whole trajectory (dynamics) of the observed covariates. Analogous to Cox (1955) and Jakubowski and Nieweglowski (2010), we call such representation as doubly stochastic jump Markov processes. We derive lifetime distributions until its absorption of doubly stochastic finite state absorbing jump Markov process, and propose generalization of the phase-type distribution introduced in Neuts (1981, 1975). Also, we derive conditional forward intensity of future occurrences of the jump Markov process. The new distribution and intensity are given in closed form and have ability to capture explanatory covariates and heterogeneity. The results can be seen as an alternative structural approach to the reduced-form model of credit default discussed in Duffie et al. (2009, 2007), and Duan et al. (2012). Some numerical examples are discussed to motivate the main results.

Consensus and Polarization of Binary Opinions in Structurally Balanced Networks

In this paper, we propose a new model for binary opinion dynamics in a (fully connected) structurally balanced network. In a structurally balanced network, agents are classified into two clusters and two agents in the same cluster (resp. different clusters) are connected with a positive (resp. negative) edge. Initially, every agent is assigned with one of the two opinions randomly. In every time slot, three agents are randomly selected to have their opinions updated. If the three agents belong to the same cluster, the majority rule (MR) is used to update their opinions. On the other hand, if the three agents belong to two different clusters, with probability $p$ , a consensus is reached by the MR, and with probability $1-p$ , a polarization (in line with the signs of the three edges) is reached. The probability $p$ , called the rationality probability, plays a significant role for measuring how rational the agents in a network behave when they encounter different opinions. By applying a fluid limit theorem for jump Markov processes, we derive a system of differential equations for the density functions of opinions for large networks. We show that the equilibrium points corresponding to consensus and polarization are the only stable equilibrium points. All other equilibrium points are all unstable. As such, as time goes on, the network eventually reaches a consensus or a polarization, depending on the rationality probability and the initial state of the network.

Stability and structural properties of gene regulation networks with coregulation rules

Coregulation of the expression of groups of genes has been extensively demonstrated empirically in bacterial and eukaryotic systems. Such coregulation can arise through the use of shared regulatory motifs, which allow the coordinated expression of modules (and module groups) of functionally related genes across the genome. Coregulation can also arise through the physical association of multi-gene complexes through chromosomal looping, which are then transcribed together. We present a general formalism for modeling coregulation rules in the framework of Random Boolean Networks (RBN), and develop specific models for transcription factor networks with modular structure (including module groups, and multi-input modules (MIM) with autoregulation) and multi-gene complexes (including hierarchical differentiation between multi-gene complex members). We develop a mean-field approach to analyse the dynamical stability of large networks incorporating coregulation, and show that autoregulated MIM and hierarchical gene-complex models can achieve greater stability than networks without coregulation whose rules have matching activation frequency. We provide further analysis of the stability of small networks of both kinds through simulations. We also characterize several general properties of the transients and attractors in the hierarchical coregulation model, and show using simulations that the steady-state distribution factorizes hierarchically as a Bayesian network in a Markov Jump Process analogue of the RBN model.

Sign properties of Metzler matrices with applications

Several results about sign properties of Metzler matrices are obtained. It is first established that checking the sign-stability of a Metzler sign-matrix can be either characterized in terms of the Hurwitz stability of the unit sign-matrix in the corresponding qualitative class, or in terms the negativity of the diagonal elements of the Metzler sign-matrix and the acyclicity of the associated directed graph. Similar results are obtained for the case of Metzler block-matrices and Metzler mixed-matrices, the latter being a class of Metzler matrices containing both sign- and real-type entries. The problem of assessing the sign-stability of the convex hull of a finite and summable family of Metzler matrices is also solved, and a necessary and sufficient condition for the existence of common Lyapunov functions for all the matrices in the convex hull is obtained. The concept of sign-stability is then generalized to the concept of Ker+(B)-sign-stability, a problem that arises in the analysis of certain jump Markov processes. A sufficient condition for the Ker+(B)-sign-stability of Metzler sign-matrices is obtained and formulated using inverses of sign-matrices and the concept of L+-matrices. Several applications of the results are discussed in the last section.

Constrained BSDEs representation of the value function in optimal control of pure jump Markov processes

We consider a classical finite horizon optimal control problem for continuous-time pure jump Markov processes described by means of a rate transition measure depending on a control parameter and controlled by a feedback law. For this class of problems the value function can often be described as the unique solution to the corresponding Hamilton–Jacobi-Bellman equation. We prove a probabilistic representation for the value function, known as nonlinear Feynman–Kac formula. It relates the value function with a backward stochastic differential equation (BSDE) driven by a random measure and with a sign constraint on its martingale part. We also prove existence and uniqueness results for this class of constrained BSDEs. The connection of the control problem with the constrained BSDE uses a control randomization method recently developed by several authors. This approach also allows to prove that the value function of the original non-dominated control problem coincides with the value function of an auxiliary dominated control problem, expressed in terms of equivalent changes of probability measures.

Representing stochastic damage evolution in disordered media as a jump Markov process using the fiber bundle model

The damage evolution in quasi-brittle materials is inherently stochastic due to the presence of strong disorder in the form of heterogeneities, voids, and microcracks. The final macroscopic failure is foreshadowed by accumulation of a significant amount of distributed damage that results in precursory events observed as avalanches in experiments and simulations. Simulations on spring lattice models of disordered media have been widely used to understand the collective nature of the quasi-brittle material failure process. In this study, we use the jump Markov process to model stochastic damage evolution, which is informed by the avalanche size distributions for a given material. The jump Markov process is defined based on the probability distributions of the jump sizes, the wait-time between consecutive jumps, and the failure strength. The fiber bundle model is used as an example to obtain the required inputs and test the viability of the proposed approach. The stochasticity and size-dependence of the damage evolution process is inherently captured through the inputs provided for the jump Markov process. The avalanche and strength distributions are used to describe the effect of microscopic information present in the form of disorder, on the macroscopic damage evolution behavior.