Finite Markov Chain Research Articles

Simulation of Tennis Behaviour Using Finite Markov Chains

Markov chains are a well-known tool in mathematical modelling. In performance analysis, finite Markov chains and the connected transition probabilities can be used to evaluate game behavior. Herein finite Markov chains are especially suited in the modelling of net games like tennis, due to their structure as an alternating sequence of discrete strokes. Finite Markov chain modelling thereby can be used as a descriptive tool, as well as to gain insight in the relationship between sports behavior and outcomes. The transition matrix can be used to display the frequency of game actions in relation to the respective game structure. Likewise, the model calculations from the theory of Markov chains enable determining the relevance of actions therein, concerning overall performance. To permit the usage of finite Markov chain modelling, adherence to the Markov property, often described as the property of “memorylessness”, must be assured. Present study implemented finite Markov chain modelling for elite level tennis at the Australian and French Open. The goal of the present study was to verify the usage of Markov chain modelling in tennis, using a newly designed transition matrix. Furthermore, the aim was to gain insight in the game structure of tennis. Results showed that the new model adhered to the Markov property in an extent, which allowed the usage of selected model predictions for the analysis. Further the analysis revealed insights in the current game structure of tennis, as well as connected influence of the factors court surface and gender of players.

Approximating quasi-stationary distributions with interacting reinforced random walks

We propose two numerical schemes for approximating quasi-stationary distributions (QSD) of finite state Markov chains with absorbing states. Both schemes are described in terms of certain interacting chains in which the interaction is given in terms of the total time occupation measure of all particles in the system and has the impact of reinforcing transitions, in an appropriate fashion, to states where the collection of particles has spent more time. The schemes can be viewed as combining the key features of the two basic simulation-based methods for approximating QSD originating from the works of Fleming and Viot (1979) and Aldous, Flannery and Palacios (1998), respectively. The key difference between the two schemes studied here is that in the first method one starts witha(n) particles at time 0 and number of particles stays constant over time whereas in the second method we start with one particle and at most one particle is added at each time instant in such a manner that there area(n) particles at timen. We prove almost sure convergence to the unique QSD and establish Central Limit Theorems for the two schemes under the key assumption thata(n) =o(n). Whena(n) ~n, the fluctuation behavior is expected to be non-standard. Some exploratory numerical results are presented to illustrate the performance of the two approximation schemes.

Covering a compact space by fixed-radius or growing random balls

Simple random coverage models, well studied in Euclidean space, can also be defined on a general compact metric space. By analogy with the geometric models, and with the discrete coupon collector's problem and with cover times for finite Markov chains, one expects a weak concentration bound for the distribution of the cover time to hold under minimal assumptions. We give two such results, one for random fixed-radius balls and the other for sequentially arriving randomly-centered and deterministically growing balls. Each is in fact a simple application of a different more general bound, the former concerning coverage by i.i.d. random sets with arbitrary distribution, and the latter concerning hitting times for Markov chains with a strong monotonicity property. The growth model seems generally more tractable, and we record some basic results and open problems for that model.

Decisiveness of stochastic systems and its application to hybrid models

In 2007, Abdulla et al. introduced the concept of decisiveness, an interesting tool for lifting good properties of finite Markov chains to denumerable ones. Later, this concept was extended to more general stochastic transition systems (STSs), allowing the design of various verification algorithms for large classes of (infinite) STSs. We further improve the understanding and utility of decisiveness in two ways.First, we provide a general criterion for proving the decisiveness of general STSs. This criterion, which is very natural but whose proof is rather technical, (strictly) generalizes all known criteria from the literature.Second, we focus on stochastic hybrid systems (SHSs), a stochastic extension of hybrid systems. We establish the decisiveness of a large class of SHSs and, under a few classical hypotheses from mathematical logic, we show how to decide reachability problems in this class, even though they are undecidable for general SHSs. This provides a decidable stochastic extension of o-minimal hybrid systems.

Exit problems for Kou's process in a Markovian environment

In this paper, we consider a path-wise sum of a Brownian motion plus a compound Poisson process with exponentially distributed positive and negative jumps with parameters that depend on some finite Markov chain. Using known fluctuation identities we investigate one-sided and two-sided exit problems generalizing some results for Kou's processes to the setting of regime switching models without exploiting the fluid embedding technique. The generating function for the hitting time of the state-dependent levels is analyzed. For the case of two states, the numerical examples are given.

A Probabilistic Higher-order Fixpoint Logic

We introduce PHFL, a probabilistic extension of higher-order fixpoint logic, which can also be regarded as a higher-order extension of probabilistic temporal logics such as PCTL and the $\mu^p$-calculus. We show that PHFL is strictly more expressive than the $\mu^p$-calculus, and that the PHFL model-checking problem for finite Markov chains is undecidable even for the $\mu$-only, order-1 fragment of PHFL. Furthermore the full PHFL is far more expressive: we give a translation from Lubarsky's $\mu$-arithmetic to PHFL, which implies that PHFL model checking is $\Pi^1_1$-hard and $\Sigma^1_1$-hard. As a positive result, we characterize a decidable fragment of the PHFL model-checking problems using a novel type system.

Invariance of State Estimate for a Finite Markov Chain with Discrete Time

Invariance of State Estimate for a Finite Markov Chain with Discrete Time

Reliability modelling for linear and circular [formula omitted]-out-of-[formula omitted]: F systems with shared components

Reliability analysis for k-out-of-n systems has received much attention in the field of reliability due to its practical importance. In this work, models for linear and circular k-out-of-n: F systems with shared components, consisting of m subsystems, are discussed. The finite Markov chain imbedding approach cannot be used directly to derive reliability of the considered systems because of the common area existing between adjacent subsystems. For this reason, we first derive the probability that the number of failed components is a given value in the m common areas. Then, all disjoint cases, each of which is regarded as a series system consisting of m subsystems with different initial state probability distributions, are obtained, with which the nith step transition rate matrix is calculated by using the Markov chain technique, for i=1,2,…,m. Finally, the reliability of the system is obtained by summing the reliabilities of all cases along with the finite Markov chain imbedding approach. Some numerical examples are presented to illustrate the efficiency of the proposed reliability evaluation method.

Nearly reducible finite Markov chains: Theory and algorithms.

Finite Markov chains, memoryless random walks on complex networks, appear commonly as models for stochastic dynamics in condensed matter physics, biophysics, ecology, epidemiology, economics, and elsewhere. Here, we review exact numerical methods for the analysis of arbitrary discrete- and continuous-time Markovian networks. We focus on numerically stable methods that are required to treat nearly reducible Markov chains, which exhibit a separation of characteristic timescales and are therefore ill-conditioned. In this metastable regime, dense linear algebra methods are afflicted by propagation of error in the finite precision arithmetic, and the kinetic Monte Carlo algorithm to simulate paths is unfeasibly inefficient. Furthermore, iterative eigendecomposition methods fail to converge without the use of nontrivial and system-specific preconditioning techniques. An alternative approach is provided by state reduction procedures, which do not require additional a priori knowledge of the Markov chain. Macroscopic dynamical quantities, such as moments of the first passage time distribution for a transition to an absorbing state, and microscopic properties, such as the stationary, committor, and visitation probabilities for nodes, can be computed robustly using state reduction algorithms. The related kinetic path sampling algorithm allows for efficient sampling of trajectories on a nearly reducible Markov chain. Thus, all of the information required to determine the kinetically relevant transition mechanisms, and to identify the states that have a dominant effect on the global dynamics, can be computed reliably even for computationally challenging models. Rare events are a ubiquitous feature of realistic dynamical systems, and so the methods described herein are valuable in many practical applications.

Mixed shock model for multi-state weighted k-out-of-n: F systems with degraded resistance against shocks

In previous studies of shock models, the probability of shocks causing damage to the system during operation is always constant. However, it is more practical for the system to become more vulnerable to shocks under a severe condition and the probabilities of subsequent shocks causing damage to the system increase. Furthermore, few studies have focused on the shock environment of weighted k-out-of-n: F systems. To fill these research gaps, this paper proposes a novel mixed shock model for a multi-state weighted k-out-of-n: F system with a consideration of its resistance against shocks. The components in the system operate in a two-stage process and different weights are taken for their different states. The system has multiple states determined by the total system weight/performance. System failure results from the competing failure criteria of insufficient system weight/performance and from reaching the critical number of failed components. A combination of finite Markov chain imbedding approach, universal generating function technique and phase-type distribution is adopted to analyze the probabilistic indices of the components and entire system. Numerical illustrations based on a power-generating plant are presented to demonstrate the applicability of new shock model and the effectiveness of proposed method.

Backward stochastic differential equations with Markov chains and associated PDEs

In this paper we study the regularity of the solutions for backward stochastic differential equations (BSDEs) with finite state Markov chains and establish its link with associated partial differential equations (PDEs) in classical sense. Moreover, we study the existence and uniqueness of solutions for such BSDEs under Lipschitz conditions on f in the space Lρ2(Rd;Rk)⊗Lρ2(Rd;Rd×k)⊗Lρ2(Rd×I;Rk). In this way, we establish a new connection between Lρ2(Rd;Rk)⊗Lρ2(Rd;Rd×k)⊗Lρ2(Rd×I;Rk) valued solutions of BSDEs and the solutions of PDEs in a Sobolev space.

Lattice approximations of the first-order mean field type differential games

The theory of first-order mean field type differential games examines the systems of infinitely many identical agents interacting via some external media under assumption that each agent is controlled by two players. We study the approximations of the value function of the first-order mean field type differential game using solutions of model finite-dimensional differential games. The model game appears as a mean field type continuous-time Markov game, i.e., the game theoretical problem with the infinitely many agents and dynamics of each agent determined by a controlled finite state nonlinear Markov chain. Given a supersolution (resp. subsolution) of the Hamilton–Jacobi equation for the model game, we construct a suboptimal strategy of the first (resp. second) player and evaluate the approximation accuracy using the modulus of continuity of the reward function and the distance between the original and model games. This gives the approximations of the value function of the mean field type differential game by values of the finite-dimensional differential games. Furthermore, we present the way to build a finite-dimensional differential game that approximates the original game with a given accuracy.

Virtual Markov Chains

We introduce the space of virtual Markov chains (VMCs) as a projective limit of the spaces of all finite state space Markov chains (MCs), in the same way that the space of virtual permutations is the projective limit of the spaces of all permutations of finite sets.We introduce the notions of virtual initial distribution (VID) and a virtual transition matrix (VTM), and we show that the law of any VMC is uniquely characterized by a pair of a VID and VTM which have to satisfy a certain compatibility condition.Lastly, we study various properties of compact convex sets associated to the theory of VMCs, including that the Birkhoff-von Neumann theorem fails in the virtual setting.

Reliability assessment of multi-state performance sharing systems with transmission loss and random shocks

In this article, a performance sharing system with transmission loss and a shock operation environment is studied. Such systems are widely found in power distribution systems, distributed computing systems, data transmission systems, communication systems, and so on. The system consists of n components and each of them works to satisfy its demand and shares its performance surplus with others through a common bus. When the system operates, it may suffer a variety of stresses from its operating environment, which can be regarded as random external shocks, and the transmission loss is also wildly seen in engineering systems. Therefore, the random shocks and transmission loss are considered in this article. The performance level of a component is affected by three types of random external shocks – invalid shocks, valid shocks and extreme shocks. The system fails if at least one component cannot satisfy its demand. A finite Markov chain imbedding approach and phase-type distributions are used to estimate the performance level for each component and the universal generating function technique is applied to analyze system reliability. Analysis of a power distribution system is given to show the application of the model under study and the effectiveness of the proposed method.

Reliability evaluation of consecutive k-out-of-m: F balanced systems with a symmetry line

Based on some real backgrounds, a new balanced system structure, a consecutive k-out-of- m: F system with a symmetry line, is proposed in this paper. Considering different state numbers of a subsector, the new balanced system is analyzed under two situations respectively: the subsector with binary-state and the subsector with multi-state, while the multi-state balanced systems have not been studied in the previous research. Besides, two models are developed in terms of assumptions for the two situations, respectively. For this system, several methods, such as the finite Markov chain imbedding approach, the order statistics technique and the phase-type distributions, are used on the models. In addition to system reliability formulas, the means and variances of the system lifetimes under two models for different situations are given. Finally, numerical examples are presented to illustrate the results obtained in this paper.

Run-length distribution of multivariate control charts for covariance matrix with runs rules using finite Markov chain imbedding

Run-length distribution of multivariate control charts for covariance matrix with runs rules using finite Markov chain imbedding

Bounds on the Rate of Convergence for MtX/MtX/1 Queueing Models

We apply the method of differential inequalities for the computation of upper bounds for the rate of convergence to the limiting regime for one specific class of (in)homogeneous continuous-time Markov chains. Such an approach seems very general; the corresponding description and bounds were considered earlier for finite Markov chains with analytical in time intensity functions. Now we generalize this method to locally integrable intensity functions. Special attention is paid to the situation of a countable Markov chain. To obtain these estimates, we investigate the corresponding forward system of Kolmogorov differential equations as a differential equation in the space of sequences l1.

Optimal Investment and Proportional Reinsurance in a Regime-Switching Market Model under Forward Preferences

In this paper, we study the optimal investment and reinsurance problem of an insurance company whose investment preferences are described via a forward dynamic exponential utility in a regime-switching market model. Financial and actuarial frameworks are dependent since stock prices and insurance claims vary according to a common factor given by a continuous time finite state Markov chain. We construct the value function and we prove that it is a forward dynamic utility. Then, we characterize the optimal investment strategy and the optimal proportional level of reinsurance. We also perform numerical experiments and provide sensitivity analyses with respect to some model parameters.

Numerical analysis of first-passage processes in finite Markov chains exhibiting metastability.

We describe state-reduction algorithms for the analysis of first-passage processes in discrete- and continuous-time finite Markov chains. We present a formulation of the graph transformation algorithm that allows for the evaluation of exact mean first-passage times, stationary probabilities, and committor probabilities for all nonabsorbing nodes of a Markov chain in a single computation. Calculation of the committor probabilities within the state-reduction formalism is readily generalizable to the first hitting problem for any number of alternative target states. We then show that a state-reduction algorithm can be formulated to compute the expected number of times that each node is visited along a first-passage path. Hence, all properties required to analyze the first-passage path ensemble (FPPE) at both a microscopic and macroscopic level of detail, including the mean and variance of the first-passage time distribution, can be computed using state-reduction methods. In particular, we derive expressions for the probability that a node is visited along a direct transition path, which proceeds without returning to the initial state, considering both the nonequilibrium and equilibrium (steady-state) FPPEs. The reactive visitation probability provides a rigorous metric to quantify the dynamical importance of a node for the productive transition between two endpoint states and thus allows the local states that facilitate the dominant transition mechanisms to be readily identified. The state-reduction procedures remain numerically stable even for Markov chains exhibiting metastability, which can be severely ill-conditioned. The rare event regime is frequently encountered in realistic models of dynamical processes, and our methodology therefore provides valuable tools for the analysis of Markov chains in practical applications. We illustrate our approach with numerical results for a kinetic network representing a structural transition in an atomic cluster.

Computing Genomic Signatures Using de Bruijn Chains

Genomic DNA sequences have both deterministic and random aspects and exhibit features at numerous scales, from codons to regions of conserved or divergent gene order. Genomic signatures work by capturing one or more such features efficiently into a compact mathematical structure. We examine the unique manner in which oligonucleotides constitute a genome, within a graph-theoretic setting. A de Bruijn chain (DBC) is a kind of de Bruijn graph that includes a finite Markov chain. By representing a DNA sequence as a walk over a DBC and retaining specific information at nodes and edges, we obtain the de Bruijn chain genomic signature θdbc, based on graph structure and the stationary distribution of the DBC. We demonstrate that the θdbc signature is information-rich, efficient, sufficiently representative of the sequence from which it is derived, and superior to existing genomic signatures such as the dinucleotide odds ratio and word frequency based signatures. We develop a mathematical framework to elucidate the power of the θdbc signature to distinguish between sequences hypothesized to be generated by DBCs of distinct parameters. We study the effect of order of the θdbc signature, genome size, and variation within a genome on accuracy. We illustrate its superior performance over existing genomic signatures in predicting the origin of short DNA sequences.