State Markov Chain Research Articles

Phase-type models for competing risks, with emphasis on identifiability issues

We first review some main results for phase-type distributions, including a discussion of Coxian distributions and their canonical representations. We then consider the extension of phase-type modeling to cover competing risks. This extension involves the consideration of finite state Markov chains with more than one absorbing state, letting each absorbing state correspond to a particular risk. The non-uniqueness of Markov chain representations of phase-type distributions is well known. In the paper we study corresponding issues for the competing risks case with the aim of obtaining identifiable parameterizations. Statistical inference for the Coxian competing risks model is briefly discussed and some real data are analyzed for illustration.

Empirical process theory for locally stationary processes

We provide a framework for empirical process theory of locally stationary processes using the functional dependence measure. Our results extend known results for stationary Markov chains and mixing sequences by another common possibility to measure dependence and allow for additional time dependence. Our main result is a functional central limit theorem for locally stationary processes. Moreover, maximal inequalities for expectations of sums are developed. We show the applicability of our theory in some examples, for instance, we provide uniform convergence rates for nonparametric regression with locally stationary noise.

Learn Quasi-Stationary Distributions of Finite State Markov Chain.

We propose a reinforcement learning (RL) approach to compute the expression of quasi-stationary distribution. Based on the fixed-point formulation of quasi-stationary distribution, we minimize the KL-divergence of two Markovian path distributions induced by candidate distribution and true target distribution. To solve this challenging minimization problem by gradient descent, we apply a reinforcement learning technique by introducing the reward and value functions. We derive the corresponding policy gradient theorem and design an actor-critic algorithm to learn the optimal solution and the value function. The numerical examples of finite state Markov chain are tested to demonstrate the new method.

A Markov Chain Approach for Myopic Multi-Hop Relaying: Outage and Diversity Analysis

In this paper, a cooperative protocol is investigated for a multi-hop network consisting of relays with buffers of finite size, which may operate in different communication modes. The protocol is based on the myopic decode-and-forward strategy, where each node of the network cooperates with a limited number of neighboring nodes for the transmission of the signals. Each relay stores in its buffer the messages that were successfully decoded, in order to forward them through the appropriate channel links, based on its supported communication modes. A complete theoretical framework is investigated that models the evolution of the buffers and the transitions at the operations of each relay as a state Markov chain (MC). We analyze the performance of the proposed protocol in terms of outage probability and derive an expression for the achieved diversity-multiplexing tradeoff, by using the state transition matrix and the related steady state of the MC. Our results show that the proposed protocol outperforms the conventional multi-hop relaying scheme and the system's outage probability as well as the achieved diversity order depend on the degree of cooperation among neighboring nodes and the communication model that is considered for every relay of the network.

Approximating quasi-stationary distributions with interacting reinforced random walks

We propose two numerical schemes for approximating quasi-stationary distributions (QSD) of finite state Markov chains with absorbing states. Both schemes are described in terms of certain interacting chains in which the interaction is given in terms of the total time occupation measure of all particles in the system and has the impact of reinforcing transitions, in an appropriate fashion, to states where the collection of particles has spent more time. The schemes can be viewed as combining the key features of the two basic simulation-based methods for approximating QSD originating from the works of Fleming and Viot (1979) and Aldous, Flannery and Palacios (1998), respectively. The key difference between the two schemes studied here is that in the first method one starts witha(n) particles at time 0 and number of particles stays constant over time whereas in the second method we start with one particle and at most one particle is added at each time instant in such a manner that there area(n) particles at timen. We prove almost sure convergence to the unique QSD and establish Central Limit Theorems for the two schemes under the key assumption thata(n) =o(n). Whena(n) ~n, the fluctuation behavior is expected to be non-standard. Some exploratory numerical results are presented to illustrate the performance of the two approximation schemes.

Sequential stratified regeneration: MCMC for large state spaces with an application to subgraph count estimation

This work considers the general task of estimating the sum of a bounded function over the edges of a graph, given neighborhood query access and where access to the entire network is prohibitively expensive. To estimate this sum, prior work proposes Markov chain Monte Carlo (MCMC) methods that use random walks started at some seed vertex and whose equilibrium distribution is the uniform distribution over all edges, eliminating the need to iterate over all edges. Unfortunately, these existing estimators are not scalable to massive real-world graphs. In this paper, we introduce Ripple, an MCMC-based estimator that achieves unprecedented scalability by stratifying the Markov chain state space into ordered strata with a new technique that we denote sequential stratified regenerations. We show that the Ripple estimator is consistent, highly parallelizable, and scales well. We empirically evaluate our method by applying Ripple to the task of estimating connected, induced subgraph counts given some input graph. Therein, we demonstrate that Ripple is accurate and can estimate counts of up to 12-node subgraphs, which is a task at a scale that has been considered unreachable, not only by prior MCMC-based methods but also by other sampling approaches. For instance, in this target application, we present results in which the Markov chain state space is as large as \(10^{43}\), for which Ripple computes estimates in less than 4 h, on average.

A novel method for estimating the failure possibility by combining the adaptive Kriging model with the Markov chain simulation

Under the condition of fuzzy inputs, the structure safety level is generally measured by the failure possibility. Differing from the level cut optimization method and the fuzzy simulation method, a novel method for estimating the failure possibility is proposed in this paper by combining the adaptive Kriging surrogate model with the Markov chain simulation. Based on the principle of fuzzy simulation, the fuzzy design point is defined as the failure point with the maximal joint membership function in the fuzzy simulation sample pool, and the problem of estimating the failure possibility can be transformed into that of searching the fuzzy design point. Then the Markov chain simulation which can quickly simulate the samples in the interest region is employed to search the fuzzy design point by a designed transfer criteria. To efficiently recognize the candidate states of the Markov chain, the Kriging surrogate model is adaptively trained in the candidate sample pool generated by the Markov chain states. By replacing the actual performance function with the convergent Kriging model, the efficiency of Markov chain simulation can be improved greatly. The presented examples are used to prove the rationality and high efficiency of the proposed method.

ON-OFF Privacy in the Presence of Correlation

We formulate and study the problem of ON-OFF privacy. ON-OFF privacy algorithms enable a user to continuously switch his privacy between ON and OFF. An obvious example is the incognito mode in internet browsers. But beyond internet browsing, ON-OFF privacy can be a desired feature in most online applications. The challenge is that the statistical correlation over time of a user’s online behavior can lead to leakage of information. We consider the setting in which a user is interested in retrieving the latest message generated by one of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> sources. The user’s privacy status can change between ON and OFF over time. When privacy is ON the user wants to hide his request. Moreover, since the user’s requests depend on personal attributes such as age, gender, and political views, they are typically correlated over time. As a consequence, the user cannot simply ignore privacy when privacy is OFF. We model the correlation between user’s requests by an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> state Markov chain. The goal is to design query schemes with optimal download rate, that preserve privacy in an ON-OFF privacy setting. In this paper, we present inner and outer bounds on the achievable download rate for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> sources. We also devise an efficient algorithm to construct an ON-OFF privacy scheme achieving the inner bound and prove its optimality in the case <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N=2$ </tex-math></inline-formula> sources. For <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N &gt; 2$ </tex-math></inline-formula> , finding tighter outer bounds and efficient constructions of ON-OFF privacy schemes that would achieve them remains an open question.

Backward stochastic differential equations with Markov chains and associated PDEs

In this paper we study the regularity of the solutions for backward stochastic differential equations (BSDEs) with finite state Markov chains and establish its link with associated partial differential equations (PDEs) in classical sense. Moreover, we study the existence and uniqueness of solutions for such BSDEs under Lipschitz conditions on f in the space Lρ2(Rd;Rk)⊗Lρ2(Rd;Rd×k)⊗Lρ2(Rd×I;Rk). In this way, we establish a new connection between Lρ2(Rd;Rk)⊗Lρ2(Rd;Rd×k)⊗Lρ2(Rd×I;Rk) valued solutions of BSDEs and the solutions of PDEs in a Sobolev space.

Markov (Set) chains application to predict mortality rates using extended Milevsky–Promislov generalized mortality models

The mortality rates () measure the frequency of deaths in a fixed: population and time interval. The ability to model and forecast allows determining, among others, fundamental characteristics of life expectancy tables, e.g. used to determine the amount of premium in life insurance, adequate to the risk of death. The article proposes a new method of modelling and forecasting , using the class of stochastic Milevsky–Promislov switch models with excitations. The excitations are modelled by second, fourth and sixth order polynomials of outputs from the non-Gaussian Linear Scalar Filter (nGLSF) model and taking into account the Markov (Set) chain. The Markov (Set) chain state space is defined based on even orders of the nGLSF polynomial. The model order determines the theoretical values of the death rates. The obtained results usually provide a more precise forecast of the mortality rates than the commonly used Lee–Carter model.

Markov chain based on neutrosophic numbers in decision making

Markov chain is a stochastic model for estimating the equilibrium of any system. It is a unique mathematical model in which the future behavior of the system depends only on the present. Often biased possibilities can be used over biased probabilities, for handling uncertain information to define Markov chain using fuzzy environment. Indeterminacy is different from randomness due to its construction type where the items involved in the space are true and false in the same time. In this context as an extension of conventional and fuzzy probabilities neutrosophic probability (NP) was introduced. These neutrosophic probabilities can be captured as neutrosophic numbers. In this paper, Markov chain based on neutrosophic numbers is introduced and a new approach to the ergoticity for the traffic states in the neutrosophic Markov chain based on neutrosophic numbers is verified. The proposed approach is applied to decision-making in the prediction of traffic volume.

A Locally Stationary Markov Chain Model for Labor Dynamics

Labor market surveys usually partition individuals into three states: employed, unemployed, and out of the labor force. In particular, the Argentine “ Encuesta Permanente de Hogares (EPH)” follows a rotating scheme so that each selected household is interviewed four times within two years. Each time, the current labor state of individuals is recorded, together with extensive demographic information. We model those labor paths as consecutive observations from independent Markov chains, were transition matrixes are related to covariates through a multivariate logistic link. Because the EPH is severely affected by attrition, a significant fraction of the surveyed paths contain just one single point. Instead of discarding those observations, we opt to base estimation on the full data by (i) assuming the Markov chains are stationary and (ii) incorporating the chronological time of the first interview as an additional covariate for each individual. This novel treatment represents a convenient approximation, which we illustrate with data from Argentina in the period 1995-2002 via maximum likelihood estimation. Several interesting labor market indexes, which are functionally related to the transition matrixes, are also presented in the last portion of the paper and illustrated with real data.

Optimal Investment and Proportional Reinsurance in a Regime-Switching Market Model under Forward Preferences

In this paper, we study the optimal investment and reinsurance problem of an insurance company whose investment preferences are described via a forward dynamic exponential utility in a regime-switching market model. Financial and actuarial frameworks are dependent since stock prices and insurance claims vary according to a common factor given by a continuous time finite state Markov chain. We construct the value function and we prove that it is a forward dynamic utility. Then, we characterize the optimal investment strategy and the optimal proportional level of reinsurance. We also perform numerical experiments and provide sensitivity analyses with respect to some model parameters.

Lyapunov Exponents for Quantum Channels: An Entropy Formula and Generic Properties

We denote by Mk the set of k by k matrices with complex entries. We consider quantum channels φL of the form: given a measurable function L: Mk → Mk and a measure µ on Mk we define the linear operator φL : Mk → Mk , by the law ρ → φL (ρ) = ∫ Mk L(v)ρL(v)† dµ(v). In a previous work, the authors show that for a fixed measure µ the Φ-Erg property is generic on the function L (also irreducibility). Here we will show that the purification property is also generic on L for a fixed µ. Given L and µ there are two related stochastic processes: one takes values on the projective space P (ℂ k ) and the other on matrices in Mk . The Φ-Erg property and the purification condition are the nice hypothesis for the discrete time evolution given by the natural transition probability. In this way it will follow that generically on L, if ∫ |L(v)|2 log |L(v)| dµ(v) < ∞, the Lyapunov exponents ∞ > γ 1 ≥ γ 2 ≥ … ≥ γk ≥ −∞ are well defined. In a previous work, the concepts of entropy of a channel and Gibbs channel were presented; and also an example (associated to a stationary Markov chain) in which this definition of entropy (for a quantum channel) matches the Kolmogorov-Shanon definition of entropy. We estimate here the larger Lyapunov exponent for the mentioned example and we show that it is equal to −1/2 h, where h is the entropy of the associated Markov invariant probability.

Study of magnetization relaxation in molecular spin clusters using an innovative kinetic Monte Carlo method

Modeling blocking temperature in molecular magnets has been a long standing problem in the field of molecular magnetism. We investigate this problem using a kinetic Monte Carlo (kMC) approach on an assembly of 100,000 short molecular magnetic chains (SMMCs), each of six identical spins with nearest neighbour anisotropic ferromagnetic exchange interactions. Each spin is also anisotropic with an uniaxial anisotropy. The site spin on these SMMCs take values $1$, $3/2$ or $2$. Using eigenstates of these SMMCs as the states of Markov chain, we carry out a kMC simulation starting with an initial state in which all SMMCs are completely spin polarized and assembled on a one-dimensional lattice so as to experience ferromagnetic spin-dipolar interaction with each other. From these simulations we obtain the relaxation time $\tau_r$ as a function of temperature and the associated blocking temperature. We study this for different exchange anisotropy, on-site anisotropy and strength of dipolar interactions. The magnetization relaxation times show non-Arrhenius behaviour for weak on-site interactions. The energy barrier to magnetization relaxation increases with increase in on-site anisotropy, exchange anisotropy and strength of spin dipolar interactions; more strongly on the last parameter. In all cases the barrier saturates at large on-site anisotropy. The barrier also increases with site spin. The large barrier observed in rare-earth single ion magnets can be attributed to large dipolar interactions due to short intermolecular distances, owing to their small size and large spin of the rare earth ion in the molecule.

Graph transformation and shortest paths algorithms for finite Markov chains.

The graph transformation (GT) algorithm robustly computes the mean first-passage time to an absorbing state in a finite Markov chain. Here we present a concise overview of the iterative and block formulations of the GT procedure and generalize the GT formalism to the case of any path property that is a sum of contributions from individual transitions. In particular, we examine the path action, which directly relates to the path probability, and analyze the first-passage path ensemble for a model Markov chain that is metastable and therefore numerically challenging. We compare the mean first-passage path action, obtained using GT, with the full path action probability distribution simulated efficiently using kinetic path sampling, and with values for the highest-probability paths determined by the recursive enumeration algorithm (REA). In Markov chains representing realistic dynamical processes, the probability distributions of first-passage path properties are typically fat-tailed and therefore difficult to converge by sampling, which motivates the use of exact and numerically stable approaches to compute the expectation. We find that the kinetic relevance of the set of highest-probability paths depends strongly on the metastability of the Markov chain, and so the properties of the dominant first-passage paths may be unrepresentative of the global dynamics. Use of a global measure for edge costs in the REA, based on net productive fluxes, allows the total reactive flux to be decomposed into a finite set of contributions from simple flux paths. By considering transition flux paths, a detailed quantitative analysis of the relative importance of competing dynamical processes is possible even in the metastable regime.

Dynamical systems under random perturbations with fast switching and slow diffusion: Hyperbolic equilibria and stable limit cycles

This work is devoted to the study of long-term qualitative behavior of randomly perturbed dynamical systems. The focus is on certain stochastic differential equations (SDE) with Markovian switching, when the switching is fast varying and the diffusion (white noise) is slowly changing. Consider the system d X ε , δ ( t ) = f ( X ε , δ ( t ) , α ε ( t ) ) d t + δ σ ( X ε , δ ( t ) , α ε ( t ) ) d W ( t ) , X ε , δ ( 0 ) = x , α ε ( 0 ) = i , where α ε ( t ) is a finite state Markov chain with irreducible generator Q = ( q ι ℓ ) . The relative changing rates of the switching and the diffusion are highlighted by two small parameters ε and δ . Associated with the above stochastic differential equation, there is an averaged ordinary differential equation (ODE) d X ‾ ( t ) = f ‾ ( X ‾ ( t ) ) d t , X ‾ ( 0 ) = x , where f ‾ ( ⋅ ) = ∑ ι = 1 m 0 f ( ⋅ , ι ) ν ι and ( ν 1 , … , ν m 0 ) is the unique stationary distribution of the Markov chain with generator Q . Suppose that for each pair ( ε , δ ) , the process has an invariant probability measure μ ε , δ , and that the averaged ODE has a limit cycle in which there is an averaged occupation measure μ 0 for the averaged equation. It is proved in this paper that under weak conditions, if f ‾ has finitely many stable or hyperbolic fixed points, then μ ε , δ converges weakly to μ 0 as ε → 0 and δ → 0 . Our results generalize to the setting where the switching process α ε is state-dependent in that P { α ε ( t + Δ ) = ℓ | α ε = ι , X ε , δ ( s ) , α ε ( s ) , s ≤ t } = q ι ℓ ( X ε , δ ( t ) ) Δ + o ( Δ ) , ι ≠ ℓ as long as the generator Q ( ⋅ ) = ( q ι ℓ ( ⋅ ) ) is locally bounded, locally Lipschitz, and irreducible for all x ∈ R d . Finally, we provide two examples in two and three dimensions to showcase our results.

Finite-memory elephant random walk and the central limit theorem for additive functionals

The Central Limit Theorem (CLT) for additive functionals of Markov chains is a well-known result with a long history. In this paper, we present applications to two finite-memory versions of the Elephant Random Walk, solving a problem from Gut and Stadtmüeller (2018). We also present a derivation of the CLT for additive functionals of finite state Markov chains, which is based on positive recurrence, the CLT for IID sequences and some elementary linear algebra, and which focuses on characterization of the variance.

Robust parametric inference for finite Markov chains

We consider the problem of statistical inference in a parametric finite Markov chain model and develop a robust estimator of the parameters defining the transition probabilities via minimization of a suitable (empirical) version of the popular density power divergence. Based on a long sequence of observations from a first-order stationary Markov chain, we have defined the minimum density power divergence estimator (MDPDE) of the underlying parameter and rigorously derived its asymptotic and robustness properties under appropriate conditions. Performance of the MDPDEs is illustrated theoretically as well as empirically for some common examples of finite Markov chain models. Its applications in robust testing of statistical hypotheses are also discussed along with (parametric) comparison of two Markov chain sequences. Several directions for extending the MDPDE and related inference are also briefly discussed for multiple sequences of Markov chains, higher order Markov chains and non-stationary Markov chains with time-dependent transition probabilities. Finally, our proposal is applied to analyze corporate credit rating migration data of three international markets.

Predicting Future CD4 Cell Counts of HIV/AIDS Patients by Non Stationary Markov Chain: A Case Study of Anambra State

A total of 1094 HIV patients were involved in a cohort study (from January-December 2010) with follow-up in their CD4 cell transition counts and grouped according to their immunological states into five(5) states developed by Guiseppe Di Biase et al (2007). The five states (5) considered were: State one (CD4 &gt; 500 cells/mm3 ), State two (350 &lt; CD4 500 cells /mm3 ) State three(200 &lt; CD4 350 cells/mm3 ), State four(CD4 200 cells/mm3 ), State five(Death). These states de ne the seriousness of the sickness based on the epidemiological states of the patients CD4 cell counts. We use the non-stationary Markov chain model for the prediction. The estimation of the non-stationary probabilities were done using the exponential smoothing technique. The result of the prediction showed a gradual decrease of the CD4 cells as we move from Jan-Dec. Furthermore, the result showed that the patients in the study cannot survive death from the month Dec. 2011, if they are not subjected to therapy, using highly active antiretrovirals (HAART). The results also showed that the model can be used for the testing of the drug e efficacy administered to patients within a given period.