Infinite-state Markov Chains Research Articles

Some Old and New Problems in Thermodynamic Formalism of Infinite State Markov Chains

Some Old and New Problems in Thermodynamic Formalism of Infinite State Markov Chains

Upper and Lower Bounds for the Synchronizer Performance in Systems with Probabilistic Message Loss

In this paper, we revisit the performance of the α-synchronizer in distributed systems with probabilistic message loss as introduced in Függer et al. [Perf. Eval. 93(2015)]. In sharp contrast to the infinite-state Markov chain resp. the exponential-size finite-state upper bound presented in the original paper, we introduce a polynomial-size finite-state Markov chain for a new synchronizer variant alpha ^{prime }, which provides a new upper bound on the performance of the α-synchronizer. Both analytic and simulation results show that our new upper bound is strictly better than the existing one. Moreover, we show that a modified version of the alpha ^{prime }-synchronizer provides a lower bound on the performance of the α-synchronizer. By means of elaborate simulation results, we show that our new lower bound is also strictly better than the lower bound presented in the original paper.

Exponential Stability and Robust H∞ Control for Discrete-Time Time-Delay Infinite Markov Jump Systems

In this paper, exponential stability and robust H∞ control problem are investigated for a class of discrete-time time-delay stochastic systems with infinite Markov jump and multiplicative noises. The jumping parameters are modeled as an infinite-state Markov chain. By using a novel Lyapunov-Krasovskii functional, a new sufficient condition in terms of matrix inequalities is derived to guarantee the mean square exponential stability of the equilibrium point. Then some sufficient conditions for the existence of feedback controller are presented to guarantee that the resulting closed-loop system has mean square exponential stability for the zero exogenous disturbance and satisfies a prescribed H∞ performance level. Numerical simulations are exploited to validate the applicability of developed theoretical results.

H∞ Control for Nonlinear Infinite Markov Jump Systems

In this paper, we discuss the infinite horizon H∞ control problem for a class of nonlinear stochastic systems with state, control, and disturbance dependent noise. The jumping parameters are modelled as an infinite-state Markov chain. Based on the solvability of a set of coupled Hamilton-Jacobi inequalities (HJIs), the exponential mean square H∞ controller for the considered nonlinear stochastic systems is obtained. A numerical example is given to show the effectiveness of the proposed design method.

Probability distributions for Markov chain based quantum walks

We analyze the probability distributions of the quantum walks induced from Markov chains by Szegedy (2004). The first part of this paper is devoted to the quantum walks induced from finite state Markov chains. It is shown that the probability distribution on the states of the underlying Markov chain is always convergent in the Cesaro sense. In particular, we deduce that the limiting distribution is uniform if the transition matrix is symmetric. In the case of a non-symmetric Markov chain, we exemplify that the limiting distribution of the quantum walk is not necessarily identical with the stationary distribution of the underlying irreducible Markov chain. The Szegedy scheme can be extended to infinite state Markov chains (random walks). In the second part, we formulate the quantum walk induced from a lazy random walk on the line. We then obtain the weak limit of the quantum walk. It is noted that the current quantum walk appears to spread faster than its counterpart-quantum walk on the line driven by the Grover coin discussed in literature. The paper closes with an outlook on possible future directions.

Mixed Vehicle Flow At Signalized Intersection: Markov Chain Analysis

Abstract We assume that a Poisson flow of vehicles arrives at isolated signalized intersection, and each vehicle, independently of others, represents a random number X of passenger car units (PCU’s). We analyze numerically the stationary distribution of the queue process {Zn}, where Zn is the number of PCU’s in a queue at the beginning of the n-th red phase, n → ∞. We approximate the number Yn of PCU’s arriving during one red-green cycle by a two-parameter Negative Binomial Distribution (NBD). The well-known fact is that {Zn} follow an infinite-state Markov chain. We approximate its stationary distribution using a finite-state Markov chain. We show numerically that there is a strong dependence of the mean queue length E[Zn] in equilibrium on the input distribution of Yn and, in particular, on the ”over dispersion” parameter γ= Var[Yn ]/E[Yn ]. For Poisson input, γ = 1. γ > 1 indicates presence of heavy-tailed input. In reality it means that a relatively large ”portion” of PCU’s, considerably exceeding the average, may arrive with high probability during one red-green cycle. Empirical formulas are presented for an accurate estimation of mean queue length as a function of load and g of the input flow. Using the Markov chain technique, we analyze the mean ”virtual” delay time for a car which always arrives at the beginning of the red phase.

A Multistep Extending Truncation Method towards Model Construction of Infinite-State Markov Chains

The model checking of Infinite-State Continuous Time Markov Chains will inevitably encounter the state explosion problem when constructing the CTMCs model; our method is to get a truncated model of the infinite one; to get a sufficient truncated model to meet the model checking of Continuous Stochastic Logic based system properties, we propose a multistep extending advanced truncation method towards model construction of CTMCs and implement it in the INFAMY model checker; the experiment results show that our method is effective.

Opportunistic Cooperation in Cognitive Femtocell Networks

We investigate opportunistic cooperation between secondary (femtocell) users and primary (macrocell) users in cognitive femtocell networks. We consider two models for such cooperation. In the first model, called the Cooperative Relay Model, a secondary user cannot transmit its own data concurrently with a primary user. However, it can employ cooperative relaying of primary user data in order to improve the latter's effective transmission rate. In the second model, called the Interference Model, a secondary user is allowed to transmit its data concurrently with a primary user. However, the secondary user can "cooperate" by deferring its transmissions when the primary user is busy. In both models, the secondary users must make intelligent cooperation decisions as they seek to maximize their own throughput subject to average power constraints. The decision options are different during idle and busy periods of the primary user, and the decisions in turn influence the durations of these periods according to a controllable infinite state Markov chain. Such problems can be formulated as constrained Markov decision problems, and conventional solution techniques require either extensive knowledge of the system dynamics or learning based approaches that suffer from large convergence times. However, using a generalized Lyapunov optimization technique, we design a novel greedy and online control algorithm that overcomes these challenges. Remarkably, this algorithm does not require any knowledge of the network arrival rates and is provably optimal.

Model checking probabilistic systems against pushdown specifications

Model checking is a fully automatic verification technique traditionally used to verify finite-state systems against regular specifications. Although regular specifications have been proven to be feasible in practice, many desirable specifications are non-regular. For instance, requirements which involve counting cannot be formalized by regular specifications but using pushdown specifications, i.e., context-free properties represented by pushdown automata. Research on model-checking techniques for pushdown specifications is, however, rare and limited to the verification of non-probabilistic systems.In this paper, we address the probabilistic model-checking problem for systems modeled by discrete-time Markov chains and specifications that are provided by deterministic pushdown automata over infinite words. We first consider finite-state Markov chains and show that the quantitative and qualitative model-checking problem is solvable via a product construction and techniques that are known for the verification of probabilistic pushdown automata. Then, we consider recursive systems modeled by probabilistic pushdown automata with an infinite-state Markov chain semantics. We first show that imposing appropriate compatibility (visibility) restrictions on the synchronizations between the pushdown automaton for the system and the specification, decidability of the probabilistic model-checking problem can be established. Finally we prove that slightly departing from this compatibility assumption leads to the undecidability of the probabilistic model-checking problem, even for qualitative properties specified by deterministic context-free specifications.

Censoring Technique Applied to a Map/G/1 Queue with Set-up Time and Multiple Vacations

Infinite state Markov chains with block-structured transition matrix find extensive applications in many areas, including telecommunications, and queueing systems, for example. In this paper, we use the censoring technique to study a MAP/G/1 queue with set-up time and multiple vacations, whose transition matrix can be transformed to a block-Toeplitz or block-repeating structured. Hence, we are able to relate the boundary conditions of the system to a Markov chain of M/G/1 type. This leads to a solution of the boundary equations, which is crucial for solving the system of differential equations. We also provide expressions for the distribution of stationary queue length, virtual waiting time and the busy period, respectively.

Stochastic and substochastic solutions for infinite-state Markov chains with applications to matrix-analytic methods

This paper deals with censoring of infinite-state banded Markov chains. Censoring involves reducing the time spent in states outside a certain set of states to 0 without affecting the number of visits within this set. We show that, if all states are transient, there is, besides the standard censored Markov chain, a nonstandard censored Markov chain which is stochastic. Both the stochastic and the substochastic solutions are found by censoring a sequence of finite transition matrices. If all matrices in the sequence are stochastic, the stochastic solution arises in the limit, whereas the substochastic solution arises if the matrices in the sequence are substochastic. We also show that, if the Markov chain is recurrent, the only solution is the stochastic solution. Censoring is particularly fruitful when applied to quasi-birth-and-death (QBD) processes. It turns out that key matrices in such processes are not unique, a fact that has been observed by several authors. We note that the stochastic solution is important for the analysis of finite queues.

Stochastic and substochastic solutions for infinite-state Markov chains with applications to matrix-analytic methods

This paper deals with censoring of infinite-state banded Markov chains. Censoring involves reducing the time spent in states outside a certain set of states to 0 without affecting the number of visits within this set. We show that, if all states are transient, there is, besides the standard censored Markov chain, a nonstandard censored Markov chain which is stochastic. Both the stochastic and the substochastic solutions are found by censoring a sequence of finite transition matrices. If all matrices in the sequence are stochastic, the stochastic solution arises in the limit, whereas the substochastic solution arises if the matrices in the sequence are substochastic. We also show that, if the Markov chain is recurrent, the only solution is the stochastic solution. Censoring is particularly fruitful when applied to quasi-birth-and-death (QBD) processes. It turns out that key matrices in such processes are not unique, a fact that has been observed by several authors. We note that the stochastic solution is important for the analysis of finite queues.

CSL model checking algorithms for QBDs

We present an in-depth treatment of model checking algorithms for a class of infinite-state continuous-time Markov chains known as quasi-birth death processes. The model class is described in detail, as well as the logic CSL to express properties of interest. Using a new property-independency concept, we provide model checking algorithms for all the CSL operators. Special emphasis is given to the time-bounded until operator for which we present a new and efficient computational procedure named uniformization with representatives. By the use of an application-driven dynamic stopping criterion, the algorithm stops whenever the property to be checked can be certified (or falsified). A comprehensive case study of a connection management system shows the versatility of our new algorithms.

A note on the attractor-property of infinite-state Markov chains

In the past 5 years, a series of verification algorithms has been proposed for infinite Markov chains that have a finite attractor, i.e., a set that will be visited infinitely often almost surely s...

Block-structured stochastic process algebra and its applications to queueing systems

A new and general stochastic process algebra with block structure, called PEPABS, is introduced, which is a generalisation of PEPAph∞. For PEPABS, the activity durations may be allowed to be generally distributed, and the corresponding transition probability matrix has a block-partitioned structure. Specifically, PEPABS is suitable for describing and analysing performance of general systems with block-structured transition probability matrices, such as Markov chains of GI/G/1 type. The steady-state probabilities of the PEPABS model can be calculated by means of censoring technique and the RG-factorisation, which have been successfully applied to study infinite-state Markov chain or Markov renewal process. Some practical examples show that the formal method is convenient to model and analyse non-Markovian systems, and can efficiently tackle an infinite-state problem under an algorithmic framework.

A note on the attractor-property of infinite-state Markov chains

In the past 5 years, a series of verification algorithms has been proposed for infinite Markov chains that have a finite attractor, i.e., a set that will be visited infinitely often almost surely starting from any state. In this paper, we establish a sufficient criterion for the existence of an attractor. We show that if the states of a Markov chain can be given levels (positive integers) such that the expected next level for states at some level n > 0 is less than n − Δ for some positive Δ, then the states at level 0 constitute an attractor for the chain. As an application, we obtain a direct proof that some probabilistic channel systems combining message losses with duplication and insertion errors have a finite attractor.

Extension of the PAC framework to finite and countable Markov chains

We consider a model of learning in which the successive observations follow a certain Markov chain. The observations are labeled according to a membership to some unknown target set. For a Markov chain with finitely many states we show that, if the target set belongs to a family of sets with a finite Vapnik-Chervonenkis (1995) dimension, then probably approximately correct (PAC) learning of this set is possible with polynomially large samples. Specifically for observations following a random walk with a state space /spl Xscr/ and uniform stationary distribution, the sample size required is no more than /spl Omega/(t/sub 0//1-/spl lambda//sub 2/log(t/sub 0/|/spl chi/|1//spl delta/)), where /spl delta/ is the confidence level, /spl lambda//sub 2/ is the second largest eigenvalue of the transition matrix, and t/sub 0/ is the sample size sufficient for learning from independent and identically distributed (i.i.d.) observations. We then obtain similar results for Markov chains with countably many states using Lyapunov function technique and results on mixing properties of infinite state Markov chains.

Censoring, Factorizations, and Spectral Analysis for Transition Matrices with Block-Repeating Entries

In this paper, we use the Markov chain censoring technique to study infinite state Markov chains whose transition matrices possess block-repeating entries. We demonstrate that a number of important probabilistic measures are invariant under censoring. Informally speaking, these measures involve first passage times or expected numbers of visits to certain levels where other levels are taboo;they are closely related to the so-called fundamental matrix of the Markov chain which is also studied here. Factorization theorems for the characteristic equation of the blocks of the transition matrix are obtained. Necessary and sufficient conditions are derived for such a Markov chain to be positive recurrent, null recurrent, or transient based either on spectral analysis, or on a property of the fundamental matrix. Explicit expressions are obtained for key probabilistic measures, including the stationary probability vector and the fundamental matrix, which could be potentially used to develop various recursivealgorithms for computing these measures.

Performance of Multiclass Markovian Queueing Networks Via Piecewise Linear Lyapunov Functions

We study the distribution of steady-state queue lengths in multiclass queueing networks under a stable policy. We propose a general methodology based on Lyapunov functions for the performance analysis of infinite state Markov chains and apply it specifically to Markovian multiclass queueing networks. We establish a deeper connection between stability and performance of such networks by showing that if there exist linear or piecewise linear Lyapunov functions that show stability, then these Lyapunov functions can be used to establish geometric-type lower and upper bounds on the tail probabilities, and thus bounds on the expectation of the queue lengths. As an example of our results, for a reentrant line queueing network with two processing stations operating under a work-conserving policy, we show that $\mathrm{E}[L] = O(\frac{1}{1 - \rho^*)^2})$, where $L$ is the total number of customers in the system, and $\rho^*$ is the maximal actual or virtual traffic intensity in the network. In a Markovian setting, this extends a recent result by Dai and Vande Vate, which states that a reentrant line queueing network with two stations is globally stable if $\rho^*<1$. We also present several results on the performance of multiclass queueing networks operating under general Markovian and,in particular,priority policies. The results in this paper are the first that establish explicit geometric-type upper and lower bounds on tail probabilities of queue lengths for networks of such generality. Previous results provide numerical bounds and only on the expectation, not the distribution, of queue lengths.

APPROXIMATIONS FOR MARKOV CHAINS WITH UPPER HESSENBERG TRANSITION MATRICES

Abstract We present an approximation for the stationary distribution T of a countably infinite-state Markov chain with transition probability matrix P = (pq) of upper Hessenberg form. Our approximation makes use of an associated upper Hessenberg matrix which is spatially homogeneous one P ( ~ ) except for a finite number of rows obtained by letting p q = pj-i+l, i > N + 1, for some distribution p = {pj } with mean p 1. We prove that there exists an optimal p, say p*(N) with which our method provides exact probabilities up to the level N . However, in general to find this optimal p*(N) is practically impossible unless one has the exact distribution v . Therefore, we propose a number of approximations to p* (N) and prove that a better approximation than that given by finite truncation methods can be obtained in the sense of smaller li-distance between exact distribution of its approximation. Numerical experiments are implemented for the M/h4/ l retrial queue.