Countable State Research Articles

Stay-in-a-set games

There exists a Nash equilibrium (e-Nash equilibrium) for every n-person stochastic game with a finite (countable) state space and finite action sets for the players if the payoff to each player i is one when the process of states remains in a given set of states Gi and is zero otherwise.

N-Person stochastic games with upper semi-continuous payoffs

Every n-person stochastic game with a countable state space, finite action sets for the players and bounded, upper semi-continuous payoffs has an e-equilibrium for every e>0.

Markovian couplings staying in arbitrary subsets of the state space

Let (X t ) and (Y t ) be continuous-time Markov chains with countable state spaces E and F and let K be an arbitrary subset of E x F. We give necessary and sufficient conditions on the transition rates of (X t ) and (Y t ) for the existence of a coupling which stays in K. We also show that when such a coupling exists, it can be chosen to be Markovian and give a way to construct it. In the case E=F and K ⊆ E x E, we see how the problem of construction of the coupling can be simplified. We give some examples of use and application of our results, including a new concept of lumpability in Markov chains.

Markovian couplings staying in arbitrary subsets of the state space

Let (Xt) and (Yt) be continuous-time Markov chains with countable state spaces E and F and let K be an arbitrary subset of E x F. We give necessary and sufficient conditions on the transition rates of (Xt) and (Yt) for the existence of a coupling which stays in K. We also show that when such a coupling exists, it can be chosen to be Markovian and give a way to construct it. In the case E=F and K ⊆ E x E, we see how the problem of construction of the coupling can be simplified. We give some examples of use and application of our results, including a new concept of lumpability in Markov chains.

Average optimal switching of a Markov chain with a Borel state space

We extend results on average per unit time optimality criterion in a switching model from a countable state space to a Borel state space. In the model we consider, a controller selects an increasing sequence of stopping times with respect to a Markov chain, and gets rewards and pays costs at them in an alternating order. The rewards and costs depend on the state of the chain. We find the optimal average gain and construct an optimal strategy. The basic tool is a variational problem with two obstacles that appears also in Dynkin games.

Deviation Matrix, Laurent Series and Blackwell Optimality in Countable State Markov Decision Processes

This paper presents a recurrent condition on Markov decision processes with a countable state space and bounded rewards. The condition is sufficient for the existence of a Blackwell optimal stationary policy, having the Laurent series expansion with continuous coefficients. It is so relaxed that the Markov chain corresponding to a stationary policy may have countably many periodic recurrent classes. Our method finds the deviation matrix in an explicit form.

$\varepsilon$-Equilibria for Stochastic Games with Uncountable State Space and Unbounded Costs

We study a class of noncooperative stochastic games with unbounded cost functions and an uncountable state space. It is assumed that the transition law is absolutely continuous with respect to some probability measure on the state space. Undiscounted stochastic games with expected average costs are considered first. It is shown under a uniform geometric ergodicity assumption that there exists a stationary $\varepsilon$-equilibrium for each $\varepsilon > 0 $. The proof is based on recent results on uniform bounds for convergence rates of Markov chains [S. P. Meyn and R. L. Tweedie, Ann. Appl. Probab., 4 (1994), pp. 981--1011] and on an approximation method similar to that used in [A. S. Nowak, J. Optim. Theory Appl., 45 (1985), pp. 591--602], where an $\varepsilon$-equilibrium in stationary policies was shown to exist for the bounded discounted costs. The stochastic game is approximated by one with a countable state space for which a stationary Nash equilibrium exists (see [E. Altman, A. Hordijk, and F. M. Spieksma, Math. Oper. Res., 22 (1997), pp. 588--618]); this equilibrium determines an $\epsilon$-equilibrium for the original game. Finally, new results for the existence of stationary $\varepsilon$-equilibrium for discounted stochastic games are given.

Compact factors of countable state Markov shifts

We study continuous shift commuting maps from transitive countable state Markov shifts into compact subshifts. The closure of the image is a coded system. On the other hand, any coded system is the surjective image of some transitive Markov shift, which may be chosen locally compact by construction. These two results yield a formal analogy to “the transitive sofic systems are the subshift factors of transitive shifts of finite type”. Then we consider factor maps which have bounded coding length in some graph presentation (label maps). Now the image has to be synchronized, but not every synchronized system can be obtained in this way. We show various restrictions for a surjective label map to exist.

The reliability of an object with semi-Markov failure rate

In this paper the properties of the reliability function of an object with the failure rate modeled by a semi-Markov process defined on an at most countable state space are presented. Applying the introduced system of renewal equations, the object reliability functions for the alternating and the Poisson failure rate process are obtained. The Laplace–Stieltjes transform of the object reliability function and its mean time to failure of are obtained.

Moderate deviations for Markov chains with atom

We obtain in this paper moderate deviations for functional empirical processes of general state space valued Markov chains with atom under weak conditions: a tail condition on the first time of return to the atom, and usual conditions on the class of functions. Our proofs rely on the regeneration method and sharp conditions issued of moderate deviations of independent random variables. We prove our result in the nonseparable case for additive and unbounded functionals of Markov chains, extending the work of de Acosta and Chen (J. Theoret. Probab. (1998) 75–110) and Wu (Ann. Probab. (1995) 420–445). One may regard it as the analog for the Markov chains of the beautiful characterization of moderate deviations for i.i.d. case of Ledoux 1992. Some applications to Markov chains with a countable state space are considered.

Passage-Time Moments for Positively Recurrent Markov Chains

Fractional moments of the passage-times are considered for positively recurrent Markov chains with countable state spaces. A criterion of the finiteness of the fractional moments is obtained in terms of the convergence rate of the transition probability to the stationary distribution. As an application it is proved that the passage time of a direct product process of Markov chains has the same order of the fractional moments as that of the single Markov chain.

Stopped decision processes in conjunction with general utility

A combined model of Markov decision process and the stopping problem, called Stopped Decision Processes, with a general utility is considered in the paper. This model is important for the various applications including Bandit problems and piecewise deterministic Markov processes, etc. Our aim is to formulate the general utility-treatment of processes with a countable state space and give a functional characterization from points of seeking an optimal pair, that is, a policy and a stopping time. And the further results concerning an optimality equation of our model are given. If we restrict the problem as the choice of a policy for a given fixed stopping time, these results are consistent with our previous paper (1998). Especially, in the case of the exponential utility functions, the optimal pair can be derived concretely using the idea of the onestep look ahead policy. Also, a simple numerical example is given to illustrate the results.

Decay rates and cutoff for convergence and hitting times of Markov chains with countably infinite state space

For a positive recurrent continuous-time Markov chain on a countable state space, we compare the access time to equilibrium to the hitting time of a particular state. For monotone processes, the exponential rates are ranked. When the process starts far from equilibrium, a cutoff phenomenon occurs at the same instant, in the sense that both the access time to equilibrium and the hitting time of a fixed state are equivalent to the expectation of the latter. In the case of Markov chains on trees, that expectation can be computed explicitly. The results are illustrated on the M/M/∞ queue.

Decay rates and cutoff for convergence and hitting times of Markov chains with countably infinite state space

For a positive recurrent continuous-time Markov chain on a countable state space, we compare the access time to equilibrium to the hitting time of a particular state. For monotone processes, the exponential rates are ranked. When the process starts far from equilibrium, a cutoff phenomenon occurs at the same instant, in the sense that both the access time to equilibrium and the hitting time of a fixed state are equivalent to the expectation of the latter. In the case of Markov chains on trees, that expectation can be computed explicitly. The results are illustrated on the M/M/∞ queue.

Correlation decay for Markov maps on a countable state space

We estimate the decay of correlations for some Markov maps on a countable state space. A necessary and sufficient condition is given for the transfer operator to be quasi-compact on the space of locally Lipschitz functions. In the non-quasi-compact case, the decay of correlations depends on the contribution to the transfer operator of the complementary of finitely many cylinders. Estimates are given for some non-uniformly expanding maps and for maps with bounded jumps.

The CLT for Markov chains with a countable state space embedded in the space lp

We find necessary and sufficient conditions for the CLT for Markov chains with a countable state space embedded in the space l p for p⩾1. This result is an extension of the uniform CLT over the family of indicator functions in Levental (Stochastic Processes Appl. 34 (1990) 245–253), where the result is equivalent to our case p=1. A similar extension for the uniform CLT over a family of possibly unbounded functions in Tsai (Taiwan. J. Math. 1(4) (1997) 481–498) is also obtained.

One-dimensional branching random walks in a Markovian random environment

We study a one-dimensional supercritical branching random walk in a non-i.i.d. random environment, which considers both the branching mechanism and the step transition. This random environment is constructed using a recurrent Markov chain on a finite or countable state space. Criteria of (strong) recurrence and transience are presented for this model.

One-dimensional branching random walks in a Markovian random environment

We study a one-dimensional supercritical branching random walk in a non-i.i.d. random environment, which considers both the branching mechanism and the step transition. This random environment is constructed using a recurrent Markov chain on a finite or countable state space. Criteria of (strong) recurrence and transience are presented for this model.

Nonzero-sum non-stationary discounted Markov game model

The goal of this paper is provide a theory of K-person non-stationary Markov games with unbounded rewards, for a countable state space and action spaces. We investigate both the finite and infinite horizon problems. We define the concept of strong Nash equilibrium and present conditions for both problems for which strong Nash or Nash equilibrium strategies exist for all players within the Markov strategies, and show that the rewards in equilibrium satisfy the optimality equations.

Multiplicative ergodicity and large deviations for an irreducible Markov chain

The paper examines multiplicative ergodic theorems and the related multiplicative Poisson equation for an irreducible Markov chain on a countable state space. The partial products are considered for a real-valued function on the state space. If the function of interest satisfies a monotone condition, or is dominated by such a function, then 1. The mean normalized products converge geometrically quickly to a finite limiting value. 2. The multiplicative Poisson equation admits a solution. 3. Large deviation bounds are obtainable for the empirical measures.