Jump Markov Processes Research Articles

Moderate deviation principles for weakly interacting particle systems

Moderate deviation principles for empirical measure processes associated with weakly interacting Markov processes are established. Two families of models are considered: the first corresponds to a system of interacting diffusions whereas the second describes a collection of pure jump Markov processes with a countable state space. For both cases the moderate deviation principle is formulated in terms of a large deviation principle (LDP), with an appropriate speed function, for suitably centered and normalized empirical measure processes. For the first family of models the LDP is established in the path space of an appropriate Schwartz distribution space whereas for the second family the LDP is proved in the space of \(l_2\) (the Hilbert space of square summable sequences)-valued paths. Proofs rely on certain variational representations for exponential functionals of Brownian motions and Poisson random measures.

Unbiased Bayesian inference for population Markov jump processes via random truncations

We consider continuous time Markovian processes where populations of individual agents interact stochastically according to kinetic rules. Despite the increasing prominence of such models in fields ranging from biology to smart cities, Bayesian inference for such systems remains challenging, as these are continuous time, discrete state systems with potentially infinite state-space. Here we propose a novel efficient algorithm for joint state/parameter posterior sampling in population Markov Jump processes. We introduce a class of pseudo-marginal sampling algorithms based on a random truncation method which enables a principled treatment of infinite state spaces. Extensive evaluation on a number of benchmark models shows that this approach achieves considerable savings compared to state of the art methods, retaining accuracy and fast convergence. We also present results on a synthetic biology data set showing the potential for practical usefulness of our work.

Large Deviations for Finite State Markov Jump Processes with Mean-Field Interaction Via the Comparison Principle for an Associated Hamilton–Jacobi Equation

We prove the large deviation principle for the trajectory of a broad class of mean field interacting Markov jump processes via a general analytic approach based on viscosity solutions. Examples include generalized Ehrenfest models as well as Curie-Weiss spin flip dynamics with singular jump rates. The main step in the proof of the large deviation principle, which is of independent interest, is the proof of the comparison principle for an associated collection of Hamilton-Jacobi equations. Additionally, we show that the large deviation principle provides a general method to identify a Lyapunov function for the associated McKean-Vlasov equation.

An ensemble Kalman filtering algorithm for state estimation of jump Markov systems

This paper presents a target model whose dynamics is modelled as a jump Markov process. The problem of state estimation of jump Markov system using ensemble Kalman filter (EnKF) is dealt here. The EnKF is designed for higher order nonlinear state estimation. In target tracking using jump process, accuracy of the state estimate is defined as a function of ensemble size. The jump Markov system minimises state error at the state estimation and assumes that all probability distributions involved are to be Gaussian. This paper proposes state estimation of jump Markov systems using EnKF, at each step the ensemble of state estimates are calculated. From the result, calculate the error statistics from ensemble forecasts which are dynamically propagated through nonlinear system.

Jump Markov models and transition state theory: the quasi-stationary distribution approach.

We are interested in the connection between a metastable continuous state space Markov process (satisfying e.g. the Langevin or overdamped Langevin equation) and a jump Markov process in a discrete state space. More precisely, we use the notion of quasi-stationary distribution within a metastable state for the continuous state space Markov process to parametrize the exit event from the state. This approach is useful to analyze and justify methods which use the jump Markov process underlying a metastable dynamics as a support to efficiently sample the state-to-state dynamics (accelerated dynamics techniques). Moreover, it is possible by this approach to quantify the error on the exit event when the parametrization of the jump Markov model is based on the Eyring-Kramers formula. This therefore provides a mathematical framework to justify the use of transition state theory and the Eyring-Kramers formula to build kinetic Monte Carlo or Markov state models.

Viscosity Solution of Mean-Variance Portfolio Selection of a Jump Markov Process with No-Shorting Constraints

We consider the so-called mean-variance portfolio selection problem in continuous time under the constraint that the short-selling of stocks is prohibited where all the market coefficients are random processes. In this situation the Hamilton-Jacobi-Bellman (HJB) equation of the value function of the auxiliary problem becomes a coupled system of backward stochastic partial differential equation. In fact, the value functionVoften does not have the smoothness properties needed to interpret it as a solution to the dynamic programming partial differential equation in the usual (classical) sense; however, in such casesVcan be interpreted as a viscosity solution. Here we show the unicity of the viscosity solution and we see that the optimal and the value functions are piecewise linear functions based on some Riccati differential equations. In particular we solve the open problem posed by Li and Zhou and Zhou and Yin.

Numerical methods for the quadratic hedging problem in Markov models with jumps

We develop algorithms for the numerical computation of the quadratic hedging strategy in incomplete markets modeled by a pure jump Markov process. Using the Hamilton-Jacobi-Bellman approach, the value function of the quadratic hedging problem can be related to a triangular system of parabolic partial integrodifferential equations (PIDEs), which can be shown to possess unique smooth solutions in our setting. The first equation is nonlinear, but does not depend on the payoff of the option to hedge (the pure investment problem), while the other two equations are linear. We propose convergent finite-difference schemes for the numerical solution of these PIDEs and illustrate our results with an application to electricity markets, where time-inhomogeneous pure jump Markov processes naturally occur.

Moderate deviation principles for weakly interacting particle systems

Moderate deviation principles for empirical measure processes associated with weakly interacting Markov processes are established. Two families of models are considered: the first corresponds to a system of interacting diffusions whereas the second describes a collection of pure jump Markov processes with a countable state space. For both cases the moderate deviation principle is formulated in terms of a large deviation principle (LDP), with an appropriate speed function, for suitably centered and normalized empirical measure processes. For the first family of models the LDP is established in the path space of an appropriate Schwartz distribution space whereas for the second family the LDP is proved in the space of $l_2$ (the Hilbert space of square summable sequences)-valued paths. Proofs rely on certain variational representations for exponential functionals of Brownian motions and Poisson random measures.

Markov state modeling and dynamical coarse-graining via discrete relaxation path sampling.

A method is derived to coarse-grain the dynamics of complex molecular systems to a Markov jump process (MJP) describing how the system jumps between cells that fully partition its state space. The main inputs are relaxation times for each pair of cells, which are shown to be robust with respect to positioning of the cell boundaries. These relaxation times can be calculated via molecular dynamics simulations performed in each cell separately and are used in an efficient estimator for the rate matrix of the MJP. The method is illustrated through applications to Sinai billiards and a cluster of Lennard-Jones discs.

The Stationary Distribution of a Markov Jump Process Glued Together from Two State Spaces at Two Vertices

We compute the stationary distribution of a continuous-time Markov chain that is constructed by gluing together two finite, irreducible Markov chains by identifying a pair of states of one chain with a pair of states of the other and keeping all transition rates from either chain. The result expresses the stationary distribution of the glued chain in terms of quantities of the two original chains. Special emphasis is given to the cases when the stationary distribution of the glued chain is a multiple of the equilibria of the original chains, and when not, for which bounds are derived.

On the large deviation rate function for the empirical measures of reversible jump Markov processes

The large deviations principle for the empirical measure for both continuous and discrete time Markov processes is well known. Various expressions are available for the rate function, but these expressions are usually as the solution to a variational problem, and in this sense not explicit. An interesting class of continuous time, reversible processes was identified in the original work of Donsker and Varadhan for which an explicit expression is possible. While this class includes many (reversible) processes of interest, it excludes the case of continuous time pure jump processes, such as a reversible finite state Markov chain. In this paper, we study the large deviations principle for the empirical measure of pure jump Markov processes and provide an explicit formula of the rate function under reversibility.

Stochastic Nonlinear Equations Describing the Mesoscopic Voltage-Gated Ion Channels

We propose a stochastic nonlinear system to model the gating activity coupled with the membrane potential for a typical neuron. It distinguishes two different levels: a macroscopic one, for the membrane potential, and a mesoscopic one, for the gating process through the movement of its voltage sensors. Such a nonlinear system can be handled to form a Hodgkin-Huxley-like model, which links those two levels unlike the original deterministic Hodgkin-Huxley model which is positioned at a macroscopic scale only. Also, we show that an interacting particle system can be used to approximate our model, which is an approximation technique similar to the jump Markov processes, used to approximate the original Hodgkin-Huxley model.

Path-integral methods for analyzing the effects of fluctuations in stochastic hybrid neural networks.

We consider applications of path-integral methods to the analysis of a stochastic hybrid model representing a network of synaptically coupled spiking neuronal populations. The state of each local population is described in terms of two stochastic variables, a continuous synaptic variable and a discrete activity variable. The synaptic variables evolve according to piecewise-deterministic dynamics describing, at the population level, synapses driven by spiking activity. The dynamical equations for the synaptic currents are only valid between jumps in spiking activity, and the latter are described by a jump Markov process whose transition rates depend on the synaptic variables. We assume a separation of time scales between fast spiking dynamics with time constant tau_{a} and slower synaptic dynamics with time constant τ. This naturally introduces a small positive parameter epsilon=tau _{a}/tau, which can be used to develop various asymptotic expansions of the corresponding path-integral representation of the stochastic dynamics. First, we derive a variational principle for maximum-likelihood paths of escape from a metastable state (large deviations in the small noise limit epsilonrightarrow0). We then show how the path integral provides an efficient method for obtaining a diffusion approximation of the hybrid system for small ϵ. The resulting Langevin equation can be used to analyze the effects of fluctuations within the basin of attraction of a metastable state, that is, ignoring the effects of large deviations. We illustrate this by using the Langevin approximation to analyze the effects of intrinsic noise on pattern formation in a spatially structured hybrid network. In particular, we show how noise enlarges the parameter regime over which patterns occur, in an analogous fashion to PDEs. Finally, we carry out a 1/epsilon-loop expansion of the path integral, and use this to derive corrections to voltage-based mean-field equations, analogous to the modified activity-based equations generated from a neural master equation.

Learning Hybrid Models with Guarded Transitions

Innovative methods have been developed for diagnosis, activity monitoring, and state estimation that achieve high accuracy through the use of stochastic models involving hybrid discrete and continuous behaviors. A key bottleneck is the automated acquisition of these hybrid models, and recent methods have focused predominantly on Jump Markov processes and piecewise autoregressive models. In this paper, we present a novel algorithm capable of performing unsupervised learning of guarded Probabilistic Hybrid Automata (PHA) models, which extends prior work by allowing stochastic discrete mode transitions in a hybrid system to have a functional dependence on its continuous state. Our experiments indicate that guarded PHA models can yield significant performance improvements when used by hybrid state estimators, particularly when diagnosing the true discrete mode of the system, without any noticeable impact on their real-time performance.

Global Heat Kernel Estimates for Symmetric Markov Processes Dominated by Stable-Like Processes in Exterior C 1,η Open Sets

In this paper, we establish sharp two-sided heat kernel estimates for a large class of symmetric Markov processes in exterior C 1,� open sets for all t > 0. The processes are symmetric pure jump Markov processes with jumping kernel intensity

Bayesian inference for Markov jump processes with informative observations.

In this paper we consider the problem of parameter inference for Markov jump process (MJP) representations of stochastic kinetic models. Since transition probabilities are intractable for most processes of interest yet forward simulation is straightforward, Bayesian inference typically proceeds through computationally intensive methods such as (particle) MCMC. Such methods ostensibly require the ability to simulate trajectories from the conditioned jump process. When observations are highly informative, use of the forward simulator is likely to be inefficient and may even preclude an exact (simulation based) analysis. We therefore propose three methods for improving the efficiency of simulating conditioned jump processes. A conditioned hazard is derived based on an approximation to the jump process, and used to generate end-point conditioned trajectories for use inside an importance sampling algorithm. We also adapt a recently proposed sequential Monte Carlo scheme to our problem. Essentially, trajectories are reweighted at a set of intermediate time points, with more weight assigned to trajectories that are consistent with the next observation. We consider two implementations of this approach, based on two continuous approximations of the MJP. We compare these constructs for a simple tractable jump process before using them to perform inference for a Lotka-Volterra system. The best performing construct is used to infer the parameters governing a simple model of motility regulation in Bacillus subtilis.

Bootstrapping least-squares estimates in biochemical reaction networks

The paper proposes new computational methods of computing confidence bounds for the least-squares estimates (LSEs) of rate constants in mass action biochemical reaction network and stochastic epidemic models. Such LSEs are obtained by fitting the set of deterministic ordinary differential equations (ODEs), corresponding to the large-volume limit of a reaction network, to network's partially observed trajectory treated as a continuous-time, pure jump Markov process. In the large-volume limit the LSEs are asymptotically Gaussian, but their limiting covariance structure is complicated since it is described by a set of nonlinear ODEs which are often ill-conditioned and numerically unstable. The current paper considers two bootstrap Monte-Carlo procedures, based on the diffusion and linear noise approximations for pure jump processes, which allow one to avoid solving the limiting covariance ODEs. The results are illustrated with both in-silico and real data examples from the LINE 1 gene retrotranscription model and compared with those obtained using other methods.

Sign-error adaptive filtering algorithms involving Markovian parameters

Motivated by reduction of computational complexity,this work develops sign-error adaptive filtering algorithms for estimatingrandomly time-varyingsystemparameters.Different from the existing work on sign-error algorithms,the parameters are time-varying and their dynamics are modeled by a discrete-timeMarkov chain.Another distinctive feature of the algorithms is themulti-time-scale framework for characterizing parameter variations and algorithm updating speeds.This is realized by considering the stepsize of the estimation algorithms anda scaling parameter that defines the transition rate of the Markovjump process. Depending on the relative time scales of these two processes, suitably scaled sequencesof the estimatesare shown to converge to either anordinary differential equation, or a set of ordinary differential equations modulated by random switching, or a stochastic differential equation,or stochastic differential equations with random switching. Using weak convergence methods, convergence and rates of convergence of the algorithmsare obtained for all these cases. Simulation results are provided for demonstration.

Bayesian inference for hybrid discrete-continuous stochastic kinetic models

We consider the problem of efficiently performing simulation and inference for stochastic kinetic models. Whilst it is possible to work directly with the resulting Markov jump process (MJP), computational cost can be prohibitive for networks of realistic size and complexity. In this paper, we consider an inference scheme based on a novel hybrid simulator that classifies reactions as either ‘fast’ or ‘slow’ with fast reactions evolving as a continuous Markov process whilst the remaining slow reaction occurrences are modelled through a MJP with time-dependent hazards. A linear noise approximation (LNA) of fast reaction dynamics is employed and slow reaction events are captured by exploiting the ability to solve the stochastic differential equation driving the LNA. This simulation procedure is used as a proposal mechanism inside a particle MCMC scheme, thus allowing Bayesian inference for the model parameters. We apply the scheme to a simple application and compare the output with an existing hybrid approach and also a scheme for performing inference for the underlying discrete stochastic model.

Automated State-Dependent Importance Sampling for Markov Jump Processes via Sampling from the Zero-Variance Distribution

Many complex systems can be modeled via Markov jump processes. Applications include chemical reactions, population dynamics, and telecommunication networks. Rare-event estimation for such models can be difficult and is often computationally expensive, because typically many (or very long) paths of the Markov jump process need to be simulated in order to observe the rare event. We present a state-dependent importance sampling approach to this problem that is adaptive and uses Markov chain Monte Carlo to sample from the zero-variance importance sampling distribution. The method is applicable to a wide range of Markov jump processes and achieves high accuracy, while requiring only a small sample to obtain the importance parameters. We demonstrate its efficiency through benchmark examples in queueing theory and stochastic chemical kinetics.