Ergodic Markov Process Research Articles

Ergodic aspects of some Ornstein–Uhlenbeck type processes related to Lévy processes

This work concerns the Ornstein–Uhlenbeck type process associated to a positive self-similar Markov process (X(t))t≥0 which drifts to ∞, namely U(t)≔e−tX(et−1). We point out that U is always a (topologically) recurrent ergodic Markov process. We identify its invariant measure in terms of the law of the exponential functional Iˆ≔∫0∞exp(ξˆs)ds, where ξˆ is the dual of the real-valued Lévy process ξ related to X by the Lamperti transformation. This invariant measure is infinite (i.e. U is null-recurrent) if and only if ξ1∉L1(P). In that case, we determine the family of Lévy processes ξ for which U fulfills the conclusions of the Darling–Kac theorem. Our approach relies crucially on a remarkable connection due to Patie (Patie, 2008) with another generalized Ornstein–Uhlenbeck process that can be associated to the Lévy process ξ, and properties of time-substitutions based on additive functionals.

Stochastic dynamics and stable equilibrium of evolutionary optional public goods game in finite populations

Continuous noise caused by mutation is widely present in evolutionary systems. Considering the noise effects and under the optional participation mechanism, a stochastic model for evolutionary public goods game in a finite size population is established. The evolutionary process of strategies in the population is described as a multidimensional ergodic and continuous time Markov process. The stochastic stable state of the system is analyzed by the limit distribution of the stochastic process. By numerical experiments, the influences of the fixed income coefficient for non-participants and the investment income coefficient of the public goods on the stochastic stable equilibrium of the system are analyzed. Through the numerical calculation results, we found that the optional participation mechanism can change the evolutionary dynamics and the equilibrium of the public goods game, and there is a range of parameters which can effectively promote the evolution of cooperation. Further, we obtain the accurate quantitative relationship between the parameters and the probabilities for the system to choose different stable equilibriums, which can be used to realize the control of cooperation.

Inferring the parameters of a Markov process from snapshots of the steady state

We seek to infer the parameters of an ergodic Markov process from samples taken independently from the steady state. Our focus is on non-equilibrium processes, where the steady state is not described by the Boltzmann measure, but is generally unknown and hard to compute, which prevents the application of established equilibrium inference methods. We propose a quantity we call propagator likelihood, which takes on the role of the likelihood in equilibrium processes. This propagator likelihood is based on fictitious transitions between those configurations of the system which occur in the samples. The propagator likelihood can be derived by minimising the relative entropy between the empirical distribution and a distribution generated by propagating the empirical distribution forward in time. Maximising the propagator likelihood leads to an efficient reconstruction of the parameters of the underlying model in different systems, both with discrete configurations and with continuous configurations. We apply the method to non-equilibrium models from statistical physics and theoretical biology, including the asymmetric simple exclusion process (ASEP), the kinetic Ising model, and replicator dynamics.

Numerical Simulation of Polynomial-Speed Convergence Phenomenon

We provide a hybrid method that captures the polynomial speed of convergence and polynomial speed of mixing for Markov processes. The hybrid method that we introduce is based on the coupling technique and renewal theory. We propose to replace some estimates in classical results about the ergodicity of Markov processes by numerical simulations when the corresponding analytical proof is difficult. After that, all remaining conclusions can be derived from rigorous analysis. Then we apply our results to two 1D microscopic heat conduction models. The mixing rate of these two models are expected to be polynomial but very difficult to prove. In both examples, our numerical results match the expected polynomial mixing rate well.

An Information-Spectrum Approach to the Capacity Region of the Interference Channel

In this paper, a general formula for the capacity region of a general interference channel with two pairs of users is derived, which reveals that the capacity region is the union of a family of rectangles. In the region, each rectangle is determined by a pair of spectral inf-mutual information rates. The presented formula provides us with useful insights into the interference channels in spite of the difficulty of computing it. Specially, when the inputs are discrete, ergodic Markov processes and the channel is stationary memoryless, the formula can be evaluated by the BCJR (Bahl-Cocke-Jelinek-Raviv) algorithm. Also the formula suggests that considering the structure of the interference processes contributes to obtaining tighter inner bounds than the simplest one (obtained by treating the interference as noise). This is verified numerically by calculating the mutual information rates for Gaussian interference channels with embedded convolutional codes. Moreover, we present a coding scheme to approach the theoretical achievable rate pairs. Numerical results show that the decoding gains can be achieved by considering the structure of the interference.

A Note on the Birkhoff Ergodic Theorem

The classical Birkhoff ergodic theorem states that for an ergodic Markov process the limiting behaviour of the time average of a function (having finite p-th moment, \(p\ge 1\), with respect to the invariant measure) along the trajectories of the process, starting from the invariant measure, is a.s. and in the p-th mean constant and equals to the space average of the function with respect to the invariant measure. The crucial assumption here is that the process starts from the invariant measure, which is not always the case. In this paper, under the assumptions that the underlying process is a Markov process on Polish space, that it admits an invariant probability measure and that its marginal distributions converge to the invariant measure in the \(L^{1}\)-Wasserstein metric, we show that the assertion of the Birkhoff ergodic theorem holds in the p-th mean, \(p\ge 1\), for any bounded Lipschitz function and any initial distribution of the process.

Reducibility by Moments of Linear Impulse Markov Dynamical Systems with Almost Constant Coefficients

Abstract This paper deals with linear impulse dynamical systems on ℝd whose parameters depend on an ergodic piece-wise constant Markov process with values from some phase space 𝕐 and on a small parameter ɛ. Trajectories of Markov process x(t,y)∈ ℝd satisfy a system of linear differential equations with close to constant coefficients on its continuity intervals, while its phase coordinate changes discontinuously when Markov process switching occur. Jump sizes depend linearly on the phase coordinate and are proportional to the small parameter ɛ. We propose a method and an algorithm for choosing the base 𝔹(t,y) of the space ℝd that provides approximation of average phase trajectories E{x(t,y)} by a solution of a system of linear differential equations with constant coefficients.

Finding Dominant Structures of Nonreversible Markov Processes

Finding metastable sets as dominant structures of Markov processes has been shown to be especially useful in modeling interesting slow dynamics of various real world complex processes. Furthermore, coarse graining of such processes based on their dominant structures leads to better understanding and dimension reduction of observed systems. However, in many cases, e.g., for nonreversible Markov processes, dominant structures are often not formed by metastable sets but by important cycles or a mixture of both. This paper aims at understanding and identifying these different types of dominant structures for reversible as well as nonreversible ergodic Markov processes. Our algorithmic approach generalizes spectral-based methods for reversible processes by using Schur decomposition techniques which can also tackle nonreversible cases. We illustrate the mathematical construction of our new approach by numerical experiments.

A new look at Markov processes of G/M/1-type

We present a new method for deriving the stationary distribution of an ergodic Markov process of G/M/1-type in continuous-time, by deriving and making use of a new representation for each element of the rate matrices contained in these distributions. This method can also be modified to derive the Laplace transform of each transition function associated with Markov processes of G/M/1-type.

On the final size of epidemics in random environment

This paper extends the final size result of the classical SIR epidemic model in constant and periodic environments to random environment. Conditionally on the basic reproduction number R0 recently defined for random environment and the initial infected population fraction, we prove a final size result of an epidemic governed by the SIR model with time-depending parameters. The parameters are driven by an ergodic inhomogeneous time-periodic Markov process with finite state space. We also analyze the classical SEIR epidemic model in random environment.

Irreversible Langevin samplers and variance reduction: a large deviations approach

In order to sample from a given target distribution (often of Gibbs type), the Monte Carlo Markov chain method consists of constructing an ergodic Markov process whose invariant measure is the target distribution. By sampling the Markov process one can then compute, approximately, expectations of observables with respect to the target distribution. Often the Markov processes used in practice are time-reversible (i.e. they satisfy detailed balance), but our main goal here is to assess and quantify how the addition of a non-reversible part to the process can be used to improve the sampling properties. We focus on the diffusion setting (overdamped Langevin equations) where the drift consists of a gradient vector field as well as another drift which breaks the reversibility of the process but is chosen to preserve the Gibbs measure. In this paper we use the large deviation rate function for the empirical measure as a tool to analyze the speed of convergence to the invariant measure. We show that the addition of an irreversible drift leads to a larger rate function and it strictly improves the speed of convergence of ergodic average for (generic smooth) observables. We also deduce from this result that the asymptotic variance decreases under the addition of the irreversible drift and we give an explicit characterization of the observables whose variance is not reduced reduced, in terms of a nonlinear Poisson equation. Our theoretical results are illustrated and supplemented by numerical simulations.

Limits of relative entropies associated with weakly interacting particle systems

The limits of scaled relative entropies between probability distributions associated with N-particle weakly interacting Markov processes are considered. The convergence of such scaled relative entropies is established in various settings. The analysis is motivated by the role relative entropy plays as a Lyapunov function for the (linear) Kolmogorov forward equation associated with an ergodic Markov process, and Lyapunov function properties of these scaling limits with respect to nonlinear finite-state Markov processes are studied in the companion paper [6].

Limits of relative entropies associated with weakly interacting particle systems

The limits of scaled relative entropies between probability distributions associated with N-particle weakly interacting Markov processes are considered. The convergence of such scaled relative entropies is established in various settings. The analysis is motivated by the role relative entropy plays as a Lyapunov function for the (linear) Kolmogorov forward equation associated with an ergodic Markov process, and Lyapunov function properties of these scaling limits with respect to nonlinear finite-state Markov processes are studied in the companion paper [6].

On quantitative convergence to quasi-stationarity

The quantitative long time behavior of absorbing, finite, irreducible Markov processes is considered. Via Doob transforms, it is shown that only the knowledge of the ratio of the values of the underlying first Dirichlet eigenvector is necessary to come back to the well-investigated situation of the convergence to equilibrium of ergodic finite Markov processes. This leads to explicit estimates on the convergence to quasi-stationarity, in particular via functional inequalities. When the process is reversible, the optimal exponential rate consisting of the spectral gap between the two first Dirichlet eigenvalues is recovered. Several simple examples are provided to illustrate the bounds obtained.

Stationary and Transition Probabilities in Slow Mixing, Long Memory Markov Processes

We observe a length-n sample generated by an unknown, stationary ergodic Markov process (model) over a finite alphabet A. Given any string w of symbols from A we want estimates of the conditional probability distribution of symbols following w, as well as the stationary probability of w. Two distinct problems that complicate estimation in this setting are: 1) long memory and 2) slow mixing, which could happen even with only one bit of memory. Any consistent estimator in this setting can only converge pointwise over the class of all ergodic Markov models. Namely, given any estimator and any sample size n, the underlying model could be such that the estimator performs poorly on a sample of size n with high probability. But can we look at a length-n sample and identify if an estimate is likely to be accurate? Since the memory is unknown a-priori, a natural approach is to estimate a potentially coarser model with memory k <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> = O(log n). As n grows, pointwise consistent estimates that hold eventually almost surely (e.a.s.) are known so long as the scaling of k <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> is not superlogarithmic in n. Here, rather than e.a.s. convergence results, we want the best answers possible with a length-n sample. Combining results in universal compression with Aldous' coupling arguments, we obtain sufficient conditions on the lengthn sample (even for slow mixing models) to identify when naive: 1) estimates of the conditional probabilities and 2) estimates related to the stationary probabilities are accurate, and also bound the deviations of the naive estimates from true values.

Geometric ergodicity for classes of homogeneous Markov chains

The paper deals with non asymptotic computable bounds for the geometric convergence rate of homogeneous ergodic Markov processes. Some sufficient conditions are stated for simultaneous geometric ergodicity of Markov chain classes. This property is applied to nonparametric estimation in ergodic diffusion processes.

Scheduling control for Markov-modulated single-server multiclass queueing systems in heavy traffic

This paper studies a scheduling control problem for a single-server multiclass queueing network in heavy traffic, operating in a changing environment. The changing environment is modeled as a finite-state Markov process that modulates the arrival and service rates in the system. Various cases are considered: fast changing environment, fixed environment, and slowly changing environment. In all cases, the arrival rates are environment dependent, whereas the service rates are environment dependent when the environment Markov process is changing fast, and are assumed to be constant in the other two cases. In each of the cases, using weak convergence analysis, in particular functional limit theorems for Poisson processes and ergodic Markov processes, it is shown that an appropriate "averaged" version of the classical $$c\mu $$ c μ -policy (the priority policy that favors classes with higher values of the product of holding cost $$c$$ c and service rate $$\mu $$ μ ) is asymptotically optimal for an infinite horizon discounted cost criterion.

Synchronization control for a class of tele-operation system with stochastically time-varying communication delays

Synchronization control methods for a class of tele-operation system with stochastically time-varying communication delays are surveyed, aiming at the existing communication delays, packet loss, and so on, during the ground-based tele-operation of unmanned aerial vehicles. A control model for this class of master-slave tele-operation system with communication time delays is established. Then, the time delays within the remote control communication network are depicted by a time-varying-delay function, of which the modal parameters are transferred based on an ergodic Markov process. After that, the synchronization control problem of the system is solved by stability analysis of the mean square within the framework of the stochastic systems. Finally, the semi-physical simulation platforms of an unmanned micro-quadrotor are carried out to verify the methods presented in this paper; and the elementary simulation results show that the loci of the master-slave sides are all bounded through designing the controllers by the feedback linearization separately in the methods.

Sampled-Parameter Feedback Control of Discrete-time Linear Stochastic Parameter-Varying Systems

Feedback stabilization of a discrete-time linear stochastic parameter-varying system is explored. The parameter of the system is modeled as a discrete-time stationary, ergodic, and aperiodic Markov process on a Euclidean space. We develop a stabilizing control framework for the case where the system parameter is observed (sampled) periodically. We obtain sufficient conditions under which almost sure asymptotic stabilization of the closed-loop stochastic parameter-varying system is guaranteed by our proposed control law, which depends only on the sampled version of the system parameter.

Large deviations for excursions of non-homogeneous Markov processes

In this paper, the large deviations at the trajectory level for ergodic Markov processes are studied. These processes take values in the non-negative quadrant of the two-dimensional lattice and are concentrated on step-wise functions. The rates of jumps towards the axes (downward jumps) depend on the position of the process - the higher the position, the greater the rate. The rates of jumps going in the same direction as the axes (upward jumps) are constants. Therefore the processes are ergodic. The large deviations are studied under equal scalings of both space and time. The scaled versions of the processes converge to 0. The main result is that the probabilities of excursions far from 0 tend to 0 exponentially fast with an exponent proportional to the square of the scaling parameter. The proportionality coefficient is an integral of a linear combination of path components. A rate function of the large deviation principle is calculated for continuous functions only.