Finite Markov Chain Research Articles

Graph transformation and shortest paths algorithms for finite Markov chains.

The graph transformation (GT) algorithm robustly computes the mean first-passage time to an absorbing state in a finite Markov chain. Here we present a concise overview of the iterative and block formulations of the GT procedure and generalize the GT formalism to the case of any path property that is a sum of contributions from individual transitions. In particular, we examine the path action, which directly relates to the path probability, and analyze the first-passage path ensemble for a model Markov chain that is metastable and therefore numerically challenging. We compare the mean first-passage path action, obtained using GT, with the full path action probability distribution simulated efficiently using kinetic path sampling, and with values for the highest-probability paths determined by the recursive enumeration algorithm (REA). In Markov chains representing realistic dynamical processes, the probability distributions of first-passage path properties are typically fat-tailed and therefore difficult to converge by sampling, which motivates the use of exact and numerically stable approaches to compute the expectation. We find that the kinetic relevance of the set of highest-probability paths depends strongly on the metastability of the Markov chain, and so the properties of the dominant first-passage paths may be unrepresentative of the global dynamics. Use of a global measure for edge costs in the REA, based on net productive fluxes, allows the total reactive flux to be decomposed into a finite set of contributions from simple flux paths. By considering transition flux paths, a detailed quantitative analysis of the relative importance of competing dynamical processes is possible even in the metastable regime.

Analysis of Single-Server Multi-Class Queue with Unreliable Service, Batch Correlated Arrivals, Customers Impatience, and Dynamical Change of Priorities

A single-server non-pre-emptive priority queueing system of a finite capacity with many types of customers is analyzed. Inter-arrival times can be correlated and batch arrivals are allowed. Possible unreliability of the server, implying the loss of a customer or the necessity of its service from the early beginning or some phase of the service, is taken into account. Initial priorities provided to various types of customers at the arrival moment can be varied (increased or decreased) after the random amount of time during the customer stay in the buffer. Such a type of queues arises in the modeling operation of various emergency care systems, information, and perishable goods delivering systems, etc. The stationary behavior of the system is described by the finite state multi-dimensional continuous-time Markov chain with the upper-Hessenberg block structure of the generator. The stationary distribution of the system states and some important characteristics of the system are calculated. The presented numerical examples illustrate opportunities to quantitatively evaluate the impact of the buffer capacity and customers’ mean arrival rate on the most important characteristics of the system. The possibility of solving optimization problems is briefly shown.

Dynamical systems under random perturbations with fast switching and slow diffusion: Hyperbolic equilibria and stable limit cycles

This work is devoted to the study of long-term qualitative behavior of randomly perturbed dynamical systems. The focus is on certain stochastic differential equations (SDE) with Markovian switching, when the switching is fast varying and the diffusion (white noise) is slowly changing. Consider the system d X ε , δ ( t ) = f ( X ε , δ ( t ) , α ε ( t ) ) d t + δ σ ( X ε , δ ( t ) , α ε ( t ) ) d W ( t ) , X ε , δ ( 0 ) = x , α ε ( 0 ) = i , where α ε ( t ) is a finite state Markov chain with irreducible generator Q = ( q ι ℓ ) . The relative changing rates of the switching and the diffusion are highlighted by two small parameters ε and δ . Associated with the above stochastic differential equation, there is an averaged ordinary differential equation (ODE) d X ‾ ( t ) = f ‾ ( X ‾ ( t ) ) d t , X ‾ ( 0 ) = x , where f ‾ ( ⋅ ) = ∑ ι = 1 m 0 f ( ⋅ , ι ) ν ι and ( ν 1 , … , ν m 0 ) is the unique stationary distribution of the Markov chain with generator Q . Suppose that for each pair ( ε , δ ) , the process has an invariant probability measure μ ε , δ , and that the averaged ODE has a limit cycle in which there is an averaged occupation measure μ 0 for the averaged equation. It is proved in this paper that under weak conditions, if f ‾ has finitely many stable or hyperbolic fixed points, then μ ε , δ converges weakly to μ 0 as ε → 0 and δ → 0 . Our results generalize to the setting where the switching process α ε is state-dependent in that P { α ε ( t + Δ ) = ℓ | α ε = ι , X ε , δ ( s ) , α ε ( s ) , s ≤ t } = q ι ℓ ( X ε , δ ( t ) ) Δ + o ( Δ ) , ι ≠ ℓ as long as the generator Q ( ⋅ ) = ( q ι ℓ ( ⋅ ) ) is locally bounded, locally Lipschitz, and irreducible for all x ∈ R d . Finally, we provide two examples in two and three dimensions to showcase our results.

Reliability modeling for multi-state systems with a protective device considering multiple triggering mechanism

Failures of safety-critical systems may result in irretrievable economic losses and significant safety hazards, thus enhancing the reliability of safety-critical system is crucial. As applied widely in engineering fields, protective devices are commonly equipped for the systems operating in shock environment to reduce external damage, which has not been taken into consideration in existing literatures. This paper investigates the reliability of multi-state systems with competing failure patterns supported by a protective device. According to the system failure modes, state-based and shock number-based triggering mechanism of the protective device are developed. That is, the protective device is triggered once the system state or cumulative number of shocks exceeds corresponding critical thresholds respectively. After being triggered, the protective device can reduce the probability of damaging shocks for the system. The protective device fails when the number of consecutive valid shocks reaches a threshold. Based on the constructed model, a finite Markov chain imbedding approach is employed to derive reliability indices including distribution functions of system lifetime and residual lifetime, together with expected operating time of the protective device. Moreover, two age-based replacement policies together with a condition-based replacement policy are developed to accommodate different maintenance scenarios and corresponding optimal solutions are acquired. Numerical illustrations based on the application of cooling systems in engines are presented to validate the results.

Variable Length Memory Chains: Characterization of stationary probability measures

Variable Length Memory Chains (VLMC), which are generalizations of finite order Markov chains, are an essential tool to modelize random sequences in many domains, as well as an interesting object in contemporary probability theory. The question of existence of stationary probability measures leads us to introduce a key combinatorial structure for words produced by a VLMC: the Longest Internal Suffix. This notion allows us to state a necessary and sufficient condition for a VLMC to admit a unique invariant probability measure. This condition turns out to get a much simpler form for a subclass of VLMC: the stable VLMC. This natural subclass, unlike the general case, enjoys a renewal property. Namely, a stable VLMC induces a semi-Markov chain on an at most countable state space. Unfortunately, this discrete time renewal process does not contain the whole information of the VLMC, preventing the study of a stable VLMC to be reduced to the study of its induced semi-Markov chain. For a subclass of stable VLMC, the convergence in distribution of a VLMC towards its stationary probability measure is established. Finally, finite state space semi-Markov chains turn out to be very special stable VLMC, shedding some new light on their limit distributions.

Finite-memory elephant random walk and the central limit theorem for additive functionals

The Central Limit Theorem (CLT) for additive functionals of Markov chains is a well-known result with a long history. In this paper, we present applications to two finite-memory versions of the Elephant Random Walk, solving a problem from Gut and Stadtmüeller (2018). We also present a derivation of the CLT for additive functionals of finite state Markov chains, which is based on positive recurrence, the CLT for IID sequences and some elementary linear algebra, and which focuses on characterization of the variance.

Robust parametric inference for finite Markov chains

We consider the problem of statistical inference in a parametric finite Markov chain model and develop a robust estimator of the parameters defining the transition probabilities via minimization of a suitable (empirical) version of the popular density power divergence. Based on a long sequence of observations from a first-order stationary Markov chain, we have defined the minimum density power divergence estimator (MDPDE) of the underlying parameter and rigorously derived its asymptotic and robustness properties under appropriate conditions. Performance of the MDPDEs is illustrated theoretically as well as empirically for some common examples of finite Markov chain models. Its applications in robust testing of statistical hypotheses are also discussed along with (parametric) comparison of two Markov chain sequences. Several directions for extending the MDPDE and related inference are also briefly discussed for multiple sequences of Markov chains, higher order Markov chains and non-stationary Markov chains with time-dependent transition probabilities. Finally, our proposal is applied to analyze corporate credit rating migration data of three international markets.

The entropy method under curvature-dimension conditions in the spirit of Bakry-Émery in the discrete setting of Markov chains

We consider continuous-time (not necessarily finite) Markov chains on discrete spaces and identify a curvature-dimension inequality, the condition CDϒ(κ,∞), which serves as a natural analogue of the classical Bakry-Émery condition CD(κ,∞) in several respects. In particular, it is tailor-made to the classical approach of proofing the modified logarithmic Sobolev inequality via computing and estimating the second time derivative of the entropy along the heat flow generated by the generator of the Markov chain. We prove that curvature bounds in the sense of CDϒ are preserved under tensorization, discuss links to other notions of discrete curvature and consider a variety of examples including complete graphs, the hypercube and birth-death processes. We further consider power type entropies and determine, in the same spirit, a natural CD condition which leads to Beckner inequalities. The CDϒ condition is also shown to be compatible with the diffusive setting, in the sense that corresponding hybrid processes enjoy a tensorization property.

Gambler’s ruin estimates on finite inner uniform domains

Gambler’s ruin estimates can be viewed as harmonic measure estimates for finite Markov chains which are absorbed (or killed) at boundary points. We relate such estimates to properties of the underlying chain and its Doob transform. Precisely, we show that gambler’s ruin estimates reduce to a good understanding of the Perron–Frobenius eigenfunction and eigenvalue whenever the underlying chain and its Doob transform are Harnack Markov chains. Finite inner-uniform domains (say, in the square grid Zn) provide a large class of examples where these ideas apply and lead to detailed estimates. In general, understanding the behavior of the Perron–Frobenius eigenfunction remains a challenge.

MODELS AND ALGORITHMS FOR THE MANAGEMENT OF FIRE AND RESCUE UNITS IN EXTINGUISHING FIRES AT METALLURGICAL ENTERPRISES WHEN UNLOADING RAW MATERIALS FROM ROLLING STOCK

The approbation of the application of models and algorithms for supporting the control of forces and means of a fire and rescue garrison based on optimal decision-making models (stochastic or probabilistic) within the framework of the existing conditions for achieving the main task of fire departments, based on the theory of finite Markov chains, has been carried out. Study of the consequences of fires at metallurgical enterprises, namely, fires that occur on conveyor belts, including when unloading raw materials from a transport rolling stock, as well as the process of optimizing the management of fire departments when extinguishing these fires. Researchers have considered a model for managing fire and rescue units using a method that includes a process of discrete sequential conditions for making managerial decisions (Markov chains). To achieve the set goals, a practical experiment was carried out on burning samples of conveyor belts in a muffle furnace and a calorimetric installation, the results obtained were assessed, and a model for managing fire and rescue units was described. The subject of the research is support for the management of fire and rescue units in extinguishing a fire at metallurgical enterprises when unloading raw materials from rolling stock onto conveyor belts of the enterprise. Results. The destructiveness and scale of the possible consequences of a fire at metalworking enterprises have been experimentally confirmed, and the need to apply preventive measures and extinguish fires at facilities of this type has been determined. The results obtained can be used to control fire and rescue units when extinguishing fires at metallurgical enterprises when unloading raw materials from rolling trains onto conveyor belts of the enterprise. This work is intended for persons who make managerial decisions and who manage forces and means when extinguishing fires at metallurgical enterprises.

Reliability and optimal triggering policy for multi-state systems subject to shocks and supported by a protective device

Failures of safety-critical systems could result in irretrievable economic loss and significant environmental damage. To mitigate the failure risk of safety-critical systems, a protective device is commonly designed to alleviate the damage caused by external shocks. In this paper, a multi-state system subject to shocks and supported by a protective device is considered. The protective device can be triggered to defense valid shocks when the system state is worse than a predetermined threshold. During the operating process of the protective device, the system is assumed to be failure-free due to the resistant effect of the protective device. After resisting a finite number of valid shocks for the system, the protective device loses the protective ability. A finite Markov chain imbedding method is employed to derive the system reliability characteristics. Based on the established reliability model, the state based triggering policy of the protective device is designed and an optimization model is constructed to determine the optimal triggering threshold. Finally, numerical examples are given to illustrate the proposed model and obtained results.

Reduced rank multinomial logistic regression in Markov chains with application to cognitive data.

Finite Markov chains are useful tools for studying transitions among health states; these chains can be complex consisting of a mix of transient and absorbing states. The transition probabilities, which are often affected by covariates, can be difficult to estimate due to the presence of many covariates and/or a subset of transitions that are rarely observed. The purpose of this article is to show how to estimate the effect of a subset of covariates of interest after adjusting for the presence of multiple other covariates by applying multidimensional dimension reduction to the latter. The case in which transitions within each row of the one-step transition probability matrix are estimated by multinomial logistic regression is discussed in detail. Dimension reduction for the adjustment covariates involves estimating the effect of the covariates by a product of matrices iteratively; at each iteration one matrix in the product is fixed while the second is estimated using either standard software or nonlinear estimation, depending on which of the matrices in the product is fixed. The algorithm is illustrated by an application where the effect of at least one Apolipoprotein-E (APOE) gene allele on transition probability is estimated in a Markov Chain that includes adjustment for eight covariates and focuses on transitions from normal cognition to several forms of mild cognitive impairment, with possible absorption into dementia. Data were drawn from annual cognitive assessments of 649 participants enrolled in the BRAiNS cohort at the University of Kentucky's Alzheimer's Disease Research Center.

Reliability Modeling for Sparsely Connected Homogeneous Multistate Consecutive-k-Out-of-n: GSystems

Compared to binary-state system models, multistate (MS) system models are more practical and useful when describing complex systems in applications, such as communication systems, oil pipeline systems, and grid or cloud computing infrastructures. This paper models a generalized sparsely connected MS consecutive- ${k}$ -out-of- ${n}$ : ${G}$ system. The proposed model is regarded as a natural extension of the consecutive- ${k}$ -out-of- ${n}$ system. Moreover, the proposed system model could cover three special MS submodels, i.e., a decreasing MS: ${G}$ system model, an increasing MS: ${G}$ system model, and a nonmonotonic MS: ${G}$ system model. Three corresponding real-world examples are given to illustrate the applications of the model. Meanwhile, by utilizing the well-known finite Markov chain imbedding method, the explicit system reliability and state distribution for the three kinds of submodels are presented, respectively. Finally, the developed indexes and analysis method are demonstrated by numerical examples.

Analysis of a Batch Arrival, Batch Service Queuing-Inventory System with Processing of Inventory While on Vacation

A single-server queuing-inventory system in which arrivals are governed by a batch Markovian arrival process and successive arrival batch sizes form a finite first-order Markov chain is considered in this paper. Service is provided in batches according to a batch Markovian service process, with consecutive service batch sizes forming a finite first-order Markov chain. A service starts for the next batch on completion of the current service, provided that inventory is available at that epoch; otherwise, there will be a delay in starting the next service. When the service of a batch is completed, the inventory decreases by 1 unit, irrespective of batch size. A control policy in which the server goes on vacation when a service process is frozen until a quorum can initiate the next batch service is proposed to ensure idle-time utilization. During the vacation, the server produces inventory (items) for future services until it hits a specified level L or until the number of customers in the system reaches a maximum service batch size N, with whichever occurring first. In the former case, a server stays idle once the processed inventory level reaches L until the number of customers reaches (or even exceeds because of batch arrival) a maximum service batch size N. The time required for processing one unit of inventory follows a phase-type distribution. In this paper, the steady-state probability vector of this infinite system is computed. The distributions of inventory processing time in a vacation cycle, idle time in a vacation cycle, and vacation cycle length are found. The effect of correlation in successive inter-arrival times and service times on performance measures for such a queuing system is illustrated with a numerical example. An optimization problem is considered. The proposed system is then compared with a queuing-inventory system without the Markov-dependent assumption on successive arrivals as well as service batch sizes using numerical examples.

Reliability for consecutive-[formula omitted]-out-of-[formula omitted]: F systems with shared components between adjacent subsystems

Consecutive-k systems have well known for outstanding developments and extensive applications. In this paper, we propose consecutive-k-out-of-n: F systems with shared components between adjacent subsystems which extend the consecutive-k-out-of-n: F linear and circular models and the consecutive-kr-out-of-nr: F linear zigzag structure and circular polygon structure. They are referred to as an extension of linear zigzag structure and an extension of circular polygon structure respectively. The idea of computation for reliability is to cut the system into m subsystems and to get the reliability of every subsystem, then to obtain the system reliability in terms of reliabilities of all subsystems, the probability of occurrence of each event and the system structure. Based on this view, we obtain the reliability for the above two structures by combining the sum of disjoint products with finite Markov chain imbedding approach (FMCIA). A numerical example demonstrates the efficiency of the proposed method for our new models.

Optimum component reassignment for balanced systems with multi-state components operating in a shock environment

For systems with functionally exchangeable components, those components may work in different health states after operating for a period due to distinct working loads or external environment. To explore the full use of all components and extend the overall system lifetime, the components could be reassigned to different positions at a proper time. Thus, a component reassignment problem (CRP) needs to be solved to determine the optimum reassignment time and plan. Nowadays, the reliability analysis for balanced systems has attracted many scholars’ attention in engineering fields such as new energy storage, aeronautics, and astronautics. Because of the symmetrical feature of a balanced system, it is meaningful to solve the CRP problem. In this paper, we investigated the CRP for a balanced system with multi-state components working in a shock environment. In such a balanced system, the components are multi-state and system balance is defined based on component performance levels. Firstly, some reliability indices are derived by a two-step finite Markov chain imbedding approach. Secondly, to determine the optimum component reassignment time and strategy, an optimization model is constructed to maximize the system reliability at a certain time. Finally, an application of tire rotation is presented to demonstrate the proposed model.

Prognostic and Classification of Dynamic Degradation in a Mechanical System Using Variance Gamma Process

Recently, maintaining a complex mechanical system at the appropriate times is considered a significant task for reliability engineers and researchers. Moreover, the development of advanced mechanical systems and the dynamics of the operating environments raises the complexity of a system’s degradation behaviour. In this aspect, an efficient maintenance policy is of great importance, and a better modelling of the operating system’s degradation is essential. In this study, the non-monotonic degradation of a centrifugal pump system operating in the dynamic environment is considered and modelled using variance gamma stochastic process. The covariates are introduced to present the dynamic environmental effects and are modelled using a finite state Markov chain. The degradation of the system in the presence of covariates is modelled and prognostic results are analysed. Two machine learning algorithms k-nearest-neighbour (KNN) and neural network (NN) are applied to identify the various characteristics of degradation and the environmental conditions. A predefined degradation threshold is assigned and used to propose a prognostic result for each classification state. It was observed that this methodology shows promising prognostic results.

A Proximal/Gradient Approach for Computing the Nash Equilibrium in Controllable Markov Games

This paper proposes a new algorithm for computing the Nash equilibrium based on an iterative approach of both the proximal and the gradient method for homogeneous, finite, ergodic and controllable Markov chains. We conceptualize the problem as a poly-linear programming problem. Then, we regularize the poly-linear functional employing a regularization approach over the Lagrange functional for ensuring the method to converge to some of the Nash equilibria of the game. This paper presents two main contributions: The first theoretical result is the proposed iterative approach, which employs both the proximal and the gradient method for computing the Nash equilibria in Markov games. The method transforms the game theory problem in a system of equations, in which each equation itself is an independent optimization problem for which the necessary condition of a minimum is computed employing a nonlinear programming solver. The iterated approach provides a quick rate of convergence to the Nash equilibrium point. The second computational contribution focuses on the analysis of the convergence of the proposed method and computes the rate of convergence of the step-size parameter. These results are interesting within the context of computational and algorithmic game theory. A numerical example illustrates the proposed approach.

Computation of survival signatures for multi-state consecutive-k systems

In this paper, the survival signature is introduced in a matrix form for a multi-state coherent system with multi-state components and then some related properties are discussed. Using the finite Markov chain imbedding approach and the relationship between survival signatures and state distributions, a computational method for survival signatures is proposed for multi-state consecutive type systems. Some illustrative examples are also presented to demonstrate how survival signatures are calculated by this method and as to how they are applied.

Propagation of chaos for a general balls into bins dynamics

Consider $N$ balls initially placed in $L$ bins. At each time step take a ball from each non-empty bin and \emph{randomly} reassign the balls into the bins.We call this finite Markov chain \emph{General Repeated Balls into Bins} process. It is a discrete time interacting particles system with parallel updates. Assuming a \emph{quantitative} chaotic condition on the reassignment rule we prove a \emph{quantitative} propagation of chaos for this model. We furthermore study some equilibrium properties of the limiting nonlinear process.