Stationary Markov Process Research Articles

Langevin dynamics for the probability of finite state Markov processes

Langevin dynamics for the probability of finite state Markov processes

Information bounds for Gaussian copula parameter in stationary semiparametric Markov models

Let {Vt}t=1n be any univariate stationary first-order semiparametric Markov process generated from an unknown invariant marginal distribution and a bivariate Gaussian copula with unknown correlation coefficient α0∈(−1,1). We prove that 1−α02 is the semiparametric efficient variance bound for estimating the correlation parameter α0 in any Gaussian copula generated first-order stationary Markov models. Surprisingly, this variance bound is strictly larger than 1−α022 (when α0≠0), which is the semiparametric efficient variance bound derived by Klaassen and Wellner (1997) for estimating the correlation parameter using any i.i.d. data {(Xi,Yi)}i=1n generated from a bivariate Gaussian copula with two unknown marginal distributions.

MGF-based SNC for stationary independent Markovian processes with localized application of martingales

MGF-based SNC for stationary independent Markovian processes with localized application of martingales

New graphical models for sequential data and the improved state estimations by data-conditioned driving noises

A prevalent problem in statistical signal processing, applied statistics, and time series analysis arises from the attempt to identify the hidden state of Markov process based on a set of available noisy observations. In the context of sequential data, filtering refers to the probability distribution of the underlying Markovian system given the measurements made at or before the time of the estimated state. In addition to the filtering, the smoothing distribution is obtained from incorporating measurements made after the time of the estimated state into the filtered solution. This work proposes a number of new filters and smoothers that, in contrast to the traditional schemes, systematically make use of the process noises to give rise to enhanced performances in addressing the state estimation problem. In doing so, our approaches for the resolution are characterized by the application of the graphical models; the graph-based framework not only provides a unified perspective on the existing filters and smoothers but leads us to design new algorithms in a consistent and comprehensible manner. Moreover, the graph models facilitate the implementation of the suggested algorithms through message passing on the graph.

Stable laws for random dynamical systems

Abstract In this paper, we consider random dynamical systems formed by concatenating maps acting on the unit interval $[0,1]$ in an independent and identically distributed (i.i.d.) fashion. Considered as a stationary Markov process, the random dynamical system possesses a unique stationary measure $\nu $ . We consider a class of non-square-integrable observables $\phi $ , mostly of form $\phi (x)=d(x,x_0)^{-{1}/{\alpha }}$ , where $x_0$ is a non-recurrent point (in particular a non-periodic point) satisfying some other genericity conditions and, more generally, regularly varying observables with index $\alpha \in (0,2)$ . The two types of maps we concatenate are a class of piecewise $C^2$ expanding maps and a class of intermittent maps possessing an indifferent fixed point at the origin. Under conditions on the dynamics and $\alpha $ , we establish Poisson limit laws, convergence of scaled Birkhoff sums to a stable limit law, and functional stable limit laws in both the annealed and quenched case. The scaling constants for the limit laws for almost every quenched realization are the same as those of the annealed case and determined by $\nu $ . This is in contrast to the scalings in quenched central limit theorems where the centering constants depend in a critical way upon the realization and are not the same for almost every realization.

Dynamics caused by the mean-reverting Ornstein–Uhlenbeck process in a stochastic predator–prey model with stage structure

There are many species that go through different life stages in nature, they have different ecological characteristics in different stages, besides, stochastic variation exists in the natural environment. Based on this biological phenomenon, we develop and investigate dynamics in a predator–prey system with stage structure and the mean-reverting Ornstein–Uhlenbeck processes. We prove that this system satisfies the existence and uniqueness of globally positive equilibrium. The sufficient conditions for the existence of a stationary Markov process in stochastic system are attained. We deal with explicit expression and the existence of density function of the system which make a great contribution to biological intrinsic essence for the system. In addition, the sufficient criteria for extinction of the predator populations is derived. Finally, numerical simulations are furnished to illustrate the analysis results.

Thermodynamic Formalism for Continuous-Time Quantum Markov Semigroups: the Detailed Balance Condition, Entropy, Pressure and Equilibrium Quantum Processes

Let [Formula: see text] denote the set of [Formula: see text] by [Formula: see text] complex matrices. Consider continuous time quantum semigroups [Formula: see text], [Formula: see text], where [Formula: see text] is the infinitesimal generator. If we assume that [Formula: see text], we will call [Formula: see text], [Formula: see text] a quantum Markov semigroup. Given a stationary density matrix [Formula: see text], for the quantum Markov semigroup [Formula: see text], [Formula: see text], we can define a continuous time stationary quantum Markov process, denoted by [Formula: see text], [Formula: see text] Given an a priori Laplacian operator [Formula: see text], we will present a natural concept of entropy for a class of density matrices on [Formula: see text]. Given a Hermitian operator [Formula: see text] (which plays the role of a Hamiltonian), we will study a version of the variational principle of pressure for [Formula: see text]. A density matrix [Formula: see text] maximizing pressure will be called an equilibrium density matrix. From [Formula: see text] we will derive a new infinitesimal generator [Formula: see text]. Finally, the continuous time quantum Markov process defined by the semigroup [Formula: see text], [Formula: see text], and an initial stationary density matrix, will be called the continuous time equilibrium quantum Markov process for the Hamiltonian [Formula: see text]. It corresponds to the quantum thermodynamical equilibrium for the action of the Hamiltonian [Formula: see text].

Improved dynamic event-triggered [formula omitted] control for networked fuzzy singularly perturbed systems with hidden Markov fading channels

In this paper, the dynamic event-triggered H∞ control problem is explored for networked fuzzy slow sampling singularly perturbed systems (S3PSs) subject to hidden Markov fading channels. To alleviate the communication burden more effectively, an improved dynamic event-triggered scheme (DETS) is constructed for networked fuzzy S3PSs, under which two auxiliary internal dynamic variables are considered into event-triggered condition. The fading channel phenomena with randomly time-varying amplitudes are characterized by the finite state Markov process. Meanwhile, considering the actual network mode may be unmeasurable, a hidden Markov mode detector is exploited to estimate the actual network mode, by which the control signal can be adjusted appropriately. By constructing the singular perturbation parameter (SPP)-dependent Lyapunov functional, sufficient conditions are obtained to guarantee the mean square stability of networked fuzzy S3PSs. Besides, the numerical ill-conditioned problem of networked fuzzy S3PSs can be avoided. Finally, a tunnel diode circuit example is proposed to illustrate the effectiveness of the main results.

Asymptotic behaviors of aggregated Markov processes

AbstractFinite state Markov processes and their aggregated Markov processes have been extensively studied, especially in ion channel modeling and reliability modeling. In reliability field, the asymptotic behaviors of repairable systems modeled by both processes have been paid much attention to. For a Markov process, it is well-known that limiting measures such as availability and transition probability do not depend on the initial state of the process. However, for an aggregated Markov process, it is difficult to directly know whether this conclusion holds true or not from the limiting measure formulas expressed by the Laplace transforms. In this paper, four limiting measures expressed by Laplace transforms are proved to be independent of the initial state through Tauber’s theorem. The proof is presented under the assumption that the rank of transition rate matrix is one less than the dimension of state space for the Markov process, which includes the case that all states communicate with each other. Some numerical examples and discussions based on these are presented to illustrate the results directly and to show future related research topics. Finally, the conclusion of the paper is given.

Reliability Assessment of Distribution Power Transformers Considering Load Transfer Capability

A number of reliability models based on stationary Markov processes were developed at Hydro One to evaluate the reliability of a group of similar distribution power transformers. Those models accounted for a number of factors that affected the reliability of the group. The factors included, in addition to the basic Markov model parameters, the redundancy and mobile unit transformer capability at each individual station. There was one factor namely the load transfer capability between the stations that was not considered in the reliability assessment. The objective of load transfer is to further reduce the impact of transformer outages on customers and therefore improve the overall reliability of the system. The policy of load transfer between distribution stations can be simple or complicated and that can have a significant impact on the overall system reliability. This paper describes a probabilistic method based on stationary Markov models for evaluating the impacts of load transfers on the reliability of distribution stations. An example of a real distribution system is used to illustrate the proposed method of assessment.

An HIV stochastic model with cell-to-cell infection, B-cell immune response and distributed delay

In this study, a delayed HIV stochastic model with virus-to-cell infection, cell-to-cell transmission and B-cell immune response is proposed. We first transform the stochastic differential equation with distributed delay into a high-dimensional degenerate stochastic differential equation, and then theoretically analyze the dynamic behaviour of the degenerate model. The unique global solution of the model is given by rigorous analysis. By formulating suitable Lyapunov functions, the existence of the stationary Markov process is obtained if the stochastic B-cell-activated reproduction number is greater than one. We also use the law of large numbers theorem and the spectral radius analysis method to deduce that the virus can be cleared if the stochastic B-cell-inactivated reproduction number is less than one. Through uncertainty and sensitivity analysis, we obtain key parameters that determine the value of the stochastic B-cell-activated reproduction number. Numerically, we examine that low level noise can maintain the number of the virus and B-cell populations at a certain range, while high level noise is helpful for the elimination of the virus. Furthermore, the effect of the cell-to-cell infection on model behaviour, and the influence of the key parameters on the size of the stochastic B-cell-activated reproduction number are also investigated.

Correlation matrices of Gaussian Markov random fields over cycle graphs

Gaussian Markov Random Fields over graphs have been widely used in many fields of application. Here, we address the matrix construction problem that arises in the study of Gaussian Markov Random Fields with uniform correlation, i.e., those in which all correlations between adjacent nodes in the graph are equal. We provide a characterization of the correlation matrix of a Gaussian Markov Random Field with uniform correlation over a cycle graph, which is circulant and has a sparse inverse matrix, and study the relationship with the stationary Gaussian Markov Process on the circle. Two methods for computing the correlation matrix are also provided. Ultimately, asymptotic results for cycle graphs of large order point out the relation between Gaussian Markov Random Fields with uniform correlation over cycle and path graphs.

Training Beam Sequence Design for mmWave Tracking Systems With and Without Environmental Knowledge

In this paper, we consider a millimeter wave multiple-input single-output tracking system, where the time-varying angle of departure (AoD) is assumed to change following a discrete state Markov process. Depending on whether the associated AoD transition function is available or not, we propose two different training beam sequence design approaches. Specifically, in the case when the AoD transition function is available, we leverage the maximum a posteriori criterion to estimate the updated AoD in each beam tracking period. Since it is infeasible to derive an explicit expression for the resultant estimation error rate, we turn to its upper bound, which possesses a closed-form expression and is therefore used as the objective function to optimize the training beam sequence. Considering the complicated objective function and the unit modulus constraints imposed by the analog phase shifters, we resort to a particle swarm algorithm to solve the formulated optimization problem. In the case when the AoD transition function is unavailable, we turn to the maximum likelihood criterion for AoD estimation. To cope with the unknown AoD transition function, we reformulate the beam tracking problem as a partially observable Markov decision process problem and develop an actor-critic reinforcement learning framework to obtain an efficient training beam sequence design. Numerical results demonstrate superiorities of the proposed training beam sequence design approaches for both two cases.

Markovianity and the Thompson Group F

We show that representations of the Thompson group F in the automorphisms of a noncommutative probability space yield a large class of bilateral stationary noncommutative Markov processes. As a partial converse, bilateral stationary Markov processes in tensor dilation form yield representations of F. As an application, and building on a result of Kümmerer, we canonically associate a representation of F to a bilateral stationary Markov process in classical probability.

Asymptotic analysis of a nonlinear stochastic eco-epidemiological system with feedback control

This paper proposes a new stochastic eco-epidemiological model with nonlinear incidence rate and feedback controls. First, we prove that the stochastic system has a unique global positive solution. Second, by constructing a series of appropriate stochastic Lyapunov functions, the asymptotic behaviors around the equilibria of deterministic model are obtained, and we demonstrate that the stochastic system exists a stationary Markov process. Third, the conditions for persistence in the mean and extinction of the stochastic system are established. Finally, we carry out some numerical simulations with respect to different stochastic parameters to verify our analytical results. The obtained results indicate that the stochastic perturbations and feedback controls have crucial effects on the survivability of system.

Stationary Markovian arrival processes: Results and open problems

We consider two classes of irreducible Markovian arrival processes specified by the matrices \(C\) and \(D\): the Markov-modulated Poisson process (MMPP) and the Markov-switched Poisson process (MSPP). The former exhibits a diagonal matrix D while the latter exhibits a diagonal matrix C. For these two classes we consider the following four statements: (I) the counting process is overdispersed; (II) the hazard rate of the event-stationary interarrival time is nonincreasing; (III) the squared coefficient of variation of the event-stationary process is greater than or equal to one; (IV) there is a stochastic order showing that the time-stationary interarrival time dominates the event-stationary interarrival time. For general MSPPs and order two MMPPs, we show that (I)–(IV) hold. Then for general MMPPs, it is easy to establish (I), while (II) is shown to be false by a counter-example. For general simple point processes, (III) follows from (IV). For MMPPs, we conjecture that (IV) and thus (III) hold. We also carry out some numerical experiments that fail to disprove this conjecture. Importantly, modelling folklore has often treated MMPPs as “bursty”, and implicitly assumed that (III) holds. However, to the best of our knowledge, proving this is still an open problem. doi: https://doi.org/10.1017/S1446181122000013

Langevin dynamic for the 2D Yang–Mills measure

We define a natural state space and Markov process associated to the stochastic Yang–Mills heat flow in two dimensions.To accomplish this we first introduce a space of distributional connections for which holonomies along sufficiently regular curves (Wilson loop observables) and the action of an associated group of gauge transformations are both well-defined and satisfy good continuity properties. The desired state space is obtained as the corresponding space of orbits under this group action and is shown to be a Polish space when equipped with a natural Hausdorff metric.To construct the Markov process we show that the stochastic Yang–Mills heat flow takes values in our space of connections and use the “DeTurck trick” of introducing a time dependent gauge transformation to show invariance, in law, of the solution under gauge transformations.Our main tool for solving for the Yang–Mills heat flow is the theory of regularity structures and along the way we also develop a “basis-free” framework for applying the theory of regularity structures in the context of vector-valued noise – this provides a conceptual framework for interpreting several previous constructions and we expect this framework to be of independent interest.

Hidden Markov modeling for maximum probability neuron reconstruction

Recent advances in brain clearing and imaging have made it possible to image entire mammalian brains at sub-micron resolution. These images offer the potential to assemble brain-wide atlases of neuron morphology, but manual neuron reconstruction remains a bottleneck. Several automatic reconstruction algorithms exist, but most focus on single neuron images. In this paper, we present a probabilistic reconstruction method, ViterBrain, which combines a hidden Markov state process that encodes neuron geometry with a random field appearance model of neuron fluorescence. ViterBrain utilizes dynamic programming to compute the global maximizer of what we call the most probable neuron path. We applied our algorithm to imperfect image segmentations, and showed that it can follow axons in the presence of noise or nearby neurons. We also provide an interactive framework where users can trace neurons by fixing start and endpoints. ViterBrain is available in our open-source Python package brainlit.

Deep Q-Learning: Theoretical Insights From an Asymptotic Analysis

Deep Q-Learning is an important reinforcement learning algorithm, which involves training a deep neural network, called Deep Q-Network (DQN), to approximate the well-known Q-function. Although wildly successful under laboratory conditions, serious gaps between theory and practice as well as a lack of formal guarantees prevent its use in the real world. Adopting a dynamical systems perspective, we provide a theoretical analysis of a popular version of Deep Q-Learning under realistic and verifiable assumptions. More specifically, we prove an important result on the convergence of the algorithm, characterizing the asymptotic behavior of the learning process. Our result sheds light on hitherto unexplained properties of the algorithm and helps understand empirical observations, such as performance inconsistencies even after training. Unlike previous theories, our analysis accommodates state Markov processes with multiple stationary distributions. In spite of the focus on Deep Q-Learning, we believe that our theory may be applied to understand other deep learning algorithms

Comparative Analysis of Pitching Prediction Algorithms

Two algorithms are described in the paper; one of them is the Kalman filter, which is based on the use of a pitching mathematical model, and the second uses a neural network in which the model is considered unknown. The results of the algorithms sensitivity analysis to the parameters of the model and its influence on the potential accuracy of prediction are presented. A stationary narrow-band second-order Markov process is used as a model of the ship pitching, which was used to form the input signal of the algorithms. Also, the results of the algorithms simulation in predicting real data are presented.