Infinite Markov Chains Research Articles

On finite long run costs and rewards in infinite Markov chains

Abstract Conditions for the finiteness of long run costs and rewards associated with infinite recurrent Markov chains that may be discrete or continuous in time are considered. Without resorting to results from the theory of Markov processes on general state spaces we provide instructive proofs in the course of which we derive auxiliary results that are of interest in themselves. Potential applications of the finiteness conditions are outlined in order to elucidate their high practical relevance.

REPRESENTING MARKOV CHAINS WITH TRANSITION DIAGRAMS

Stochastic processes have many useful applications and are taught in several university programmes. Students often encounter difficulties in learning s tochastic processes and Markov chains, in particula r. In this article we describe a teaching strategy that u ses transition diagrams to represent a Markov chain and to re-define properties of its states in simple terms of directed graphs. This strategy utilises the stud ents’ intuition and makes the learning of complex concept s about Markov chains faster and easier. The method is illustrated by worked examples. The described strat egy helps students to master properties of finite M arkov chains, so they have a solid basis for the study of infinite Markov chains and other stochastic proces ses.

Opportunistic Cooperation in Cognitive Femtocell Networks

We investigate opportunistic cooperation between secondary (femtocell) users and primary (macrocell) users in cognitive femtocell networks. We consider two models for such cooperation. In the first model, called the Cooperative Relay Model, a secondary user cannot transmit its own data concurrently with a primary user. However, it can employ cooperative relaying of primary user data in order to improve the latter's effective transmission rate. In the second model, called the Interference Model, a secondary user is allowed to transmit its data concurrently with a primary user. However, the secondary user can "cooperate" by deferring its transmissions when the primary user is busy. In both models, the secondary users must make intelligent cooperation decisions as they seek to maximize their own throughput subject to average power constraints. The decision options are different during idle and busy periods of the primary user, and the decisions in turn influence the durations of these periods according to a controllable infinite state Markov chain. Such problems can be formulated as constrained Markov decision problems, and conventional solution techniques require either extensive knowledge of the system dynamics or learning based approaches that suffer from large convergence times. However, using a generalized Lyapunov optimization technique, we design a novel greedy and online control algorithm that overcomes these challenges. Remarkably, this algorithm does not require any knowledge of the network arrival rates and is provably optimal.

Censoring Technique Applied to a Map/G/1 Queue with Set-up Time and Multiple Vacations

Infinite state Markov chains with block-structured transition matrix find extensive applications in many areas, including telecommunications, and queueing systems, for example. In this paper, we use the censoring technique to study a MAP/G/1 queue with set-up time and multiple vacations, whose transition matrix can be transformed to a block-Toeplitz or block-repeating structured. Hence, we are able to relate the boundary conditions of the system to a Markov chain of M/G/1 type. This leads to a solution of the boundary equations, which is crucial for solving the system of differential equations. We also provide expressions for the distribution of stationary queue length, virtual waiting time and the busy period, respectively.

On the mixing property and the ergodic principle for nonhomogeneous Markov chains

We study different types of limit behavior of infinite dimension discrete time nonhomogeneous Markov chains. We show that the geometric structure of the set of those Markov chains which have asymptotically stationary density depends on the considered topologies. We generalize and correct some results from Ganikhodjaev et al. (2006) [3].

Large Closed Queueing Networks in Semi-Markov Environment and Their Application

The paper studies closed queueing networks containing a server station and k client stations. The server station is an infinite server queueing system, and client stations are single-server queueing systems with autonomous service, i.e. every client station serves customers (units) only at random instants generated by a strictly stationary and ergodic sequence of random variables. The total number of units in the network is N. The expected times between departures in client stations are (Nμj)−1. After a service completion in the server station, a unit is transmitted to the jth client station with probability pj (j=1,2,…,k), and being processed in the jth client station, the unit returns to the server station. The network is assumed to be in a semi-Markov environment. A semi-Markov environment is defined by a finite or countable infinite Markov chain and by sequences of independent and identically distributed random variables. Then the routing probabilities pj (j=1,2,…,k) and transmission rates (which are expressed via parameters of the network) depend on a Markov state of the environment. The paper studies the queue-length processes in client stations of this network and is aimed to the analysis of performance measures associated with this network. The questions risen in this paper have immediate relation to quality control of complex telecommunication networks, and the obtained results are expected to lead to the solutions to many practical problems of this area of research.

Decisive Markov Chains

We consider qualitative and quantitative verification problems for infinite-state Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F if it almost certainly eventually reaches either F or a state from which F can no longer be reached. While all finite Markov chains are trivially decisive (for every set F), this also holds for many classes of infinite Markov chains. Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F. In particular, this holds for probabilistic lossy channel systems (PLCS). Furthermore, all globally coarse Markov chains are decisive. This class includes probabilistic vector addition systems (PVASS) and probabilistic noisy Turing machines (PNTM). We consider both safety and liveness problems for decisive Markov chains, i.e., the probabilities that a given set of states F is eventually reached or reached infinitely often, respectively. 1. We express the qualitative problems in abstract terms for decisive Markov chains, and show an almost complete picture of its decidability for PLCS, PVASS and PNTM. 2. We also show that the path enumeration algorithm of Iyer and Narasimha terminates for decisive Markov chains and can thus be used to solve the approximate quantitative safety problem. A modified variant of this algorithm solves the approximate quantitative liveness problem. 3. Finally, we show that the exact probability of (repeatedly) reaching F cannot be effectively expressed (in a uniform way) in Tarski-algebra for either PLCS, PVASS or (P)NTM.

Algorithmic Analysis of the Sparre Andersen Model in Discrete Time

In this paper, we show that the delayed Sparre Andersen insurance risk model in discrete time can be analyzed as a doubly infinite Markov chain. We then describe how matrix analytic methods can be used to establish a computational procedure for calculating the probability distributions associated with fundamental ruin-related quantities of interest, such as the time of ruin, the surplus immediately prior to ruin, and the deficit at ruin. Special cases of the model, namely the ordinary and stationary Sparre Andersen models, are considered in several numerical examples.

Algorithmic Analysis of the Sparre Andersen Model in Discrete Time

In this paper, we show that the delayed Sparre Andersen insurance risk model in discrete time can be analyzed as a doubly infinite Markov chain. We then describe how matrix analytic methods can be used to establish a computational procedure for calculating the probability distributions associated with fundamental ruin-related quantities of interest, such as the time of ruin, the surplus immediately prior to ruin, and the deficit at ruin. Special cases of the model, namely the ordinary and stationary Sparre Andersen models, are considered in several numerical examples.

Bounding stationary results of Tandem networks with MAP input and PH service time distributions

In this paper, we propose a new approach to compute bounds on stationary measures of queueing systems with an input process described by a Markovian Arrival Process (MAP) and a sequence of stations with Phase Type (PH) service time distributions. Such queueing systems cannot be solved exactly since they have an infinite state space in several natural dimensions. Based on earlier work on the computation of bounds for specific classes of infinite Markov chains, the paper presents a new approach specifically tailored to the analysis of the mentioned class of queueing networks. By increasing the size of the state space of the aggregated Markov chain to be solved for bound computation, bounds can be made arbitrarily tight, but practical limits come up due to the computational complexity. However, we show by means of several examples that tight bounds can be derived with low effort for a large set of queueing systems in the mentioned class.

A note on the attractor-property of infinite-state Markov chains

In the past 5 years, a series of verification algorithms has been proposed for infinite Markov chains that have a finite attractor, i.e., a set that will be visited infinitely often almost surely s...

Multi-Server Queueing Systems with Multiple Priority Classes

We present the first near-exact analysis of an M/PH/k queue with m > 2 preemptive-resume priority classes. Our analysis introduces a new technique, which we refer to as Recursive Dimensionality Reduction (RDR). The key idea in RDR is that the m-dimensionally infinite Markov chain, representing the m class state space, is recursively reduced to a 1-dimensionally infinite Markov chain, that is easily and quickly solved. RDR involves no truncation and results in only small inaccuracy when compared with simulation, for a wide range of loads and variability in the job size distribution. Our analytic methods are then used to derive insights on how multi-server systems with prioritization compare with their single server counterparts with respect to response time. Multi-server systems are also compared with single server systems with respect to the effect of different prioritization schemes--smart prioritization (giving priority to the smaller jobs) versus stupid prioritization (giving priority to the larger jobs). We also study the effect of approximating m class performance by collapsing the m classes into just two classes.

A note on the attractor-property of infinite-state Markov chains

In the past 5 years, a series of verification algorithms has been proposed for infinite Markov chains that have a finite attractor, i.e., a set that will be visited infinitely often almost surely starting from any state. In this paper, we establish a sufficient criterion for the existence of an attractor. We show that if the states of a Markov chain can be given levels (positive integers) such that the expected next level for states at some level n > 0 is less than n − Δ for some positive Δ, then the states at level 0 constitute an attractor for the chain. As an application, we obtain a direct proof that some probabilistic channel systems combining message losses with duplication and insertion errors have a finite attractor.

Markov chain-based method for generating long-range dependence

This paper describes a model for generating time series which exhibit the statistical phenomenon known as long-range dependence (LRD). A Markov modulated process based on an infinite Markov chain is described. The work described is motivated by applications in telecommunications where LRD is a known property of time series measured on the Internet. The process can generate a time series exhibiting LRD with known parameters and is particularly suitable for modeling Internet traffic because the time series is in terms of ones and zeros, which can be interpreted as data packets and interpacket gaps. The method is extremely simple, both computationally and analytically, and could prove more tractable than other methods described in the literature.

ON THE SPH-DISTRIBUTION CLASS

Following up Neuts’ idea, the SPH-distribution class associated with bounded Q matrices for infinite Markov chains is defined. The main result in this paper is to characterize the SPH class through the derivatives of the distribution functions. Based on the characterization theorem, closure properties, the expansion, uniform approximation, and the matrix representations of the SPH class are also discussed by the derivatives of the distribution functions at origin.

A recursive analysis technique for multi-dimensionally infinite Markov chains

Performance analysis of multiserver systems with multiple classes of jobs often has a common source of difficulty: the state space needed to capture the system behavior grows infinitely in multiple dimensions. For example, consider two processors, each serving its own M/M/1 queue, where one of the processors (the "donor") can help the other processor (the "beneficiary") with its jobs, during times when the donor processor is idle [5, 16] or when some threshold conditions are met [14, 15]. Since the behavior of beneficiary jobs depends on the number of donor jobs in system, performance analysis of beneficiary jobs involves a two dimensionally infinite (2D-infinite) state space, where one dimension corresponds to the number of beneficiary jobs and the other dimension corresponds to the number of donor jobs. Another example is an M/M/2 queue with two priority classes, where high priority jobs have preemptive priority over low priority jobs (see for example [1, 3, 4, 8, 10, 11, 12, 17] and references therein). Since the behavior of low priority jobs depends on the number of high priority jobs in system, performance analysis of low priority jobs involves 2D-infinite state space, where each dimension corresponds to the number of each class of jobs in system. As we will see, when there are <i>m</i> priority classes, performance analysis of the lowest priority classes involves <i>m</i> dimensionally infinite state space.

Extension of the PAC framework to finite and countable Markov chains

We consider a model of learning in which the successive observations follow a certain Markov chain. The observations are labeled according to a membership to some unknown target set. For a Markov chain with finitely many states we show that, if the target set belongs to a family of sets with a finite Vapnik-Chervonenkis (1995) dimension, then probably approximately correct (PAC) learning of this set is possible with polynomially large samples. Specifically for observations following a random walk with a state space /spl Xscr/ and uniform stationary distribution, the sample size required is no more than /spl Omega/(t/sub 0//1-/spl lambda//sub 2/log(t/sub 0/|/spl chi/|1//spl delta/)), where /spl delta/ is the confidence level, /spl lambda//sub 2/ is the second largest eigenvalue of the transition matrix, and t/sub 0/ is the sample size sufficient for learning from independent and identically distributed (i.i.d.) observations. We then obtain similar results for Markov chains with countably many states using Lyapunov function technique and results on mixing properties of infinite state Markov chains.

Censoring, Factorizations, and Spectral Analysis for Transition Matrices with Block-Repeating Entries

In this paper, we use the Markov chain censoring technique to study infinite state Markov chains whose transition matrices possess block-repeating entries. We demonstrate that a number of important probabilistic measures are invariant under censoring. Informally speaking, these measures involve first passage times or expected numbers of visits to certain levels where other levels are taboo;they are closely related to the so-called fundamental matrix of the Markov chain which is also studied here. Factorization theorems for the characteristic equation of the blocks of the transition matrix are obtained. Necessary and sufficient conditions are derived for such a Markov chain to be positive recurrent, null recurrent, or transient based either on spectral analysis, or on a property of the fundamental matrix. Explicit expressions are obtained for key probabilistic measures, including the stationary probability vector and the fundamental matrix, which could be potentially used to develop various recursivealgorithms for computing these measures.

GI / G /1 type processes

A new general solution method is derived for the general GI / G /1 type processes --- for the steady-state distribution of infinite block-structured Markov chains with repetitive structure. While matrix inversion is needed in each iterational step of other general (and of more special) matrix analytical procedures, the method presented here uses matrix addition and matrix multiplication only. In exchange, the computational complexity and the memory requirement is increasing in each iterational step of the proposed method. This paper, however, lays priority on the theoretical aspect of the general solution.

Performance of Multiclass Markovian Queueing Networks Via Piecewise Linear Lyapunov Functions

We study the distribution of steady-state queue lengths in multiclass queueing networks under a stable policy. We propose a general methodology based on Lyapunov functions for the performance analysis of infinite state Markov chains and apply it specifically to Markovian multiclass queueing networks. We establish a deeper connection between stability and performance of such networks by showing that if there exist linear or piecewise linear Lyapunov functions that show stability, then these Lyapunov functions can be used to establish geometric-type lower and upper bounds on the tail probabilities, and thus bounds on the expectation of the queue lengths. As an example of our results, for a reentrant line queueing network with two processing stations operating under a work-conserving policy, we show that $\mathrm{E}[L] = O(\frac{1}{1 - \rho^*)^2})$, where $L$ is the total number of customers in the system, and $\rho^*$ is the maximal actual or virtual traffic intensity in the network. In a Markovian setting, this extends a recent result by Dai and Vande Vate, which states that a reentrant line queueing network with two stations is globally stable if $\rho^*<1$. We also present several results on the performance of multiclass queueing networks operating under general Markovian and,in particular,priority policies. The results in this paper are the first that establish explicit geometric-type upper and lower bounds on tail probabilities of queue lengths for networks of such generality. Previous results provide numerical bounds and only on the expectation, not the distribution, of queue lengths.