Denumerable State Space Research Articles

Entropy and Large Deviations for Discrete-Time Markov Chains

Let E be a denumerable state space and X be a homogeneous Markov chain on E with kernel P. Then the chain X verifies a weak Sanov's theorem, i.e., a weak large deviation principle holds for the law of the pair empirical measure. This LDP holds for any discrete state space Markov chain, not necessarily ergodic or irreducible. It is also known that a strong LDP cannot hold in the present framework. The result is obtained by a new method, which allows us to extend the LDP from a finite state space setting to a denumerable one, somehow like the projective limit approach. The analysis presented here offers some by-products, among which there are a contraction principle for the weak LDP, leading directly to a weak Sanov's theorem for the one-dimensional empirical measure. A refined analysis of the ubiquitous entropy function H proves to be useful in other frameworks, e.g., continuous time or stochastic networks, and allows us to improve the sharpness of asymptotics.

Adaptive control of average Markov decision chains under the Lyapunov stability condition

This note concerns discrete-time Markov decision processes with denumerable state space. A control policy is graded by the long-run expected average reward criterion, and the main feature of the model is that the reward function and the transition law depend on an unknown parameter. Besides standard continuity-compactness restrictions, it is supposed that the controller can use the observed history to generate a consistent estimation scheme, and that the system's transition-reward structure satisfies an adaptive version of the Lyapunov function condition. Within this context, a special implementation of the non stationary value iteration method is studied, and it is shown that this technique produces convergent approximations to the solution of the optimality equation, result that is used to build an optimal adaptive policy.

STRONG AVERAGE OPTIMALITY FOR CONTROLLED NONHOMOGENEOUS MARKOV CHAINS1*

In this paper, the problem of three kinds of average criteria (totally so-called strong average criterion) for controlled nonhomogeneous Markov chains with possibly unbounded rewards and denumerable state space is studied. We proposed a new set of conditions under which the existence of both a solution to the optimality equations and the strong average ∈-optimal Markov policies is derived by martingale theory. Furthermore, an algorithm for computing strong average ∈(>0)-optimal Markov policies is presented.

A note on optimality conditions for continuous-time Markov decision processes with average cost criterion

This note deals with continuous-time Markov decision processes with a denumerable state space and the average cost criterion. The transition rates are allowed to be unbounded, and the action set is a Borel space. We give a new set of conditions under which the existence of optimal stationary policies is ensured by using the optimality inequality. Our results are illustrated with a controlled queueing model. Moreover, we use an example to show that our conditions do not imply the existence of a solution to the optimality equations in the previous literature.

Large Deviation Principle for Markov Chains in Continuous Time

Let Yt be a homogeneous nonexplosive Markov process with generator R defined on a denumerable state space E (not necessarily ergodic). We introduce the empirical generator Gt of Yt and prove the Ruelle–Lanford property, which implies the weak LDP. In a fairly broad setting, we show how to perform almost all classical operations (e.g., contraction) on the weak LDP under suitable assumptions, whence Sanov's theorem follows.

A new strong optimality criterion for nonstationary Markov decision processes

This paper deals with a new optimality criterion consisting of the usual three average criteria and the canonical triplet (totally so-called strong average-canonical optimality criterion) and introduces the concept of a strong average-canonical policy for nonstationary Markov decision processes, which is an extension of the canonical policies of Herna´ndez-Lerma and Lasserre [16] (pages: 77) for the stationary Markov controlled processes. For the case of possibly non-uniformly bounded rewards and denumerable state space, we first construct, under some conditions, a solution to the optimality equations (OEs), and then prove that the Markov policies obtained from the OEs are not only optimal for the three average criteria but also optimal for all finite horizon criteria with a sequence of additional functions as their terminal rewards (i.e. strong average-canonical optimal). Also, some properties of optimal policies and optimal average value convergence are discussed. Moreover, the error bound in average reward between a rolling horizon policy and a strong average-canonical optimal policy is provided, and then a rolling horizon algorithm for computing strong average e(>0)-optimal Markov policies is given.

A Note on the Existence of Optimal Policies in Total Reward Dynamic Programs with Compact Action Sets

This work deals with Markov decision processes (MDPs) with expected total rewards, discrete state spaces, and compact action sets. Within this framework, a question on the existence of optimal stationary policies, formulated by Puterman (1994, p. 326), is considered. The paper concerns the possibility of obtaining an affirmative answer when additional assumptions are imposed on the decision model. Three conditions ensuring the existence of average optimal stationary policies in finite-state MDPs are analyzed, and it is shown that only the so-called structural continuity condition is a natural sufficient assumption under which the existence of total-reward optimal stationary policies can be guaranteed. In particular, this existence result holds for unichain MDPs with finite state space, but an example is provided to show that this general conclusion does not have an extension to the denumerable state space case.

Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces

This note concerns Markov decision processes on a discrete state space. It is supposed that the reward function is nonnegative, and that the decision maker has a nonnull constant risk-sensitivity, which leads to grade random rewards via the expectation of an exponential utility function. The perfomance index is the risk-sensitive expected-total reward criterion, and the existence of approximately optimal stationary policies, in the absolute and relative senses, is studied. The main results, derived under mild conditions, extend classical theorems in risk-neutral positive dynamic programming and can be summarized as follows: Assuming that the optimal value function is finite, it is proved that (i) e-optimal stationary policies exist when the state and action spaces are both finite, and (ii) this conclusion is extended to the denumerable state space case whenever (a) the decision maker is risk-averse, and (b) the optimal value function is bounded. This latter result is a (weak) risk-sensitive version of a classical theorem formulated by Ornstein (1969).

Existence of Risk-Sensitive Optimal Stationary Policies for Controlled Markov Processes

In this paper we are concerned with the existence of optimal stationary policies for infinite-horizon risk-sensitive Markov control processes with denumerable state space, unbounded cost function, and long-run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk-sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy.

Sensitive equilibria for ergodic stochastic games with countable state spaces

We consider stochastic games with countable state spaces and unbounded immediate payoff functions. Our assumptions on the transition structure of the game are based on a recent work by Meyn and Tweedie [19] on computable bounds for geometric convergence rates of Markov chains. The main results in this paper concern the existence of sensitive optimal strategies in some classes of zero-sum stochastic games. By sensitive optimality we mean overtaking or 1-optimality. We also provide a new Nash equilibrium theorem for a class of ergodic nonzero-sum stochastic games with denumerable state spaces.

Nearly optimal stationary policies in negative dynamic programmin g

This work concerns controlled Markov chains with denumerable state space and discrete time parameter. The reward function is assumed to be≤0 and the performance of a control policy is measured by the expected total-reward criterion. Within this context, sufficient conditions are given so that the existence of a stationary policy which is e-optimal at every state is guaranteed.

Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions

We study controlled Markov chains with denumerable state space and bounded costs per stage. A (long-run) risk-sensitive average cost criterion, associated to an exponential utility function with a constant risk sensitivity coefficient, is used as a performance measure. The main assumption on the probabilistic structure of the model is that the transition law satisfies a simultaneous Doeblin condition. Working within this framework, the main results obtained can be summarized as follows: If the constant risk-sensitivity coefficient is small enough, then an associated optimality equation has a bounded solution with a constant value for the optimal risk-sensitive average cost; in addition, under further standard continuity-compactness assumptions, optimal stationary policies are obtained. However, it is also shown that the above conclusions fail to hold, in general, for large enough values of the risk-sensitivity coefficient. Our results therefore disprove previous claims on this topic. Also of importance is the fact that our developments are very much self-contained and employ only basic probabilistic and analysis principles.

A pause control approach to the value iteration scheme in average Markov decision processes

This work concerns average Markov decision chains with denumerable state space. Assuming that the Lyapunov function condition holds, it is shown that the value iteration scheme yields convergent approximations to the solution of the average cost optimality equation. This result is obtained using a particular implementation of the value iteration procedure involving an artificial control action under which the system remains static.

Blackwell Optimality in Borelian Continuous-in-Action Markov Decision Processes

We prove the existence of stationary Blackwell optimal policies in Markov decision processes with a Borel state space, compact action sets, and continuous-in-action and bounded transition densities and rewards, satisfying a simultaneous Doeblin-type condition. The proof is based on a compactification of the randomized stationary policy space in a weak-strong topology, on the continuity of Laurent coefficients of the discounted rewards in this topology, and on a lexicographical policy improvement. Until now similar results were obtained for the models with a denumerable state space or with a Borel state space and finite action sets.

Value iteration in a class of average controlled Markov chains with unbounded costs: necessary and sufficient conditions for pointwise convergence

This work concerns controlled Markov chains with denumerable state space, (possibly) unbounded cost function, and an expected average cost criterion. Under a Lyapunov function condition, together with mild continuity-compactness assumptions, a simple necessary and sufficient criterion is given so that the relative value functions and differential costs produced by the value iteration scheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above.

Value iteration in a class of average controlled Markov chains with unbounded costs: necessary and sufficient conditions for pointwise convergence

This work concerns controlled Markov chains with denumerable state space, (possibly)unboundedcost function, and an expectedaveragecost criterion. Under aLyapunov function condition, together with mild continuity-compactness assumptions, a simplenecessary and sufficientcriterion is given so that the relative value functions and differential costs produced by thevalue iterationscheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above.

Risk sensitive control of Markov processes in countable state space

In this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved by proving, under suitable conditions, that there exists a bounded solution to the dynamic programming equation. The dynamic programming equation is transformed into an Isaacs equation for a stochastic game, and the vanishing discount method is used to study its solution. In addition, we prove that the existence conditions are also necessary.

Value Iteration in a Class of Communicating Markov Decision Chains with the Average Cost Criterion

Markov decision processes with denumerable state space and discrete time parameter are considered. The performance index of a control policy is the (lim sup expected) average cost criterion, and the the main structural restrictions on the model are the following: (i) under the action of any stationary policy, the state space is a communicating class; (ii) the cost function has an almost monotone---or penalized---structure [V. S. Borkar, SIAM J. Control Optim.}, 21 (1983), pp. 652--666; 22 (1984), pp. 965--978]; and (iii) some stationary policy induces an ergodic chain with finite average cost. In this context it is shown that the value iteration scheme can be used to construct convergent approximations of a solution to the optimality equation, as well as a sequence of stationary policies whose limit points are optimal.

Denumerable controlled Markov chains with strong average optimality criterion: Bounded & unbounded costs

Studies discrete-time controlled stochastic systems, modeled by controlled Markov chains (CMC) with denumerable state space and compact actions space, and with an infinite planning horizon. One optimality criterion often used is the long-run expected average cost (AC). The authors' main interest in this paper is to study the relation between the notions of AC optimality and strong average cost (SAC) optimality. The latter criterion is introduced to assess the performance of a policy over long but finite horizons, as well as in the long-run average sense. The authors show that for bounded one-stage cost functions, Sennott's conditions are sufficient to guarantee that every AC optimal policy is also SAC optimal. On the other hand, a detailed counterexample is given that shows that the latter result does not extend to the case of unbounded cost functions. >

A Comparison of Alternative Approaches for Numerical Solutions of GI/PH/1 Queues

This paper compares alternative approaches for computing the R matrix of Neuts in numerical solutions of queues of GI/PH/1 type. We consider the relative merits of using two algorithms proposed recently by Latouche, the state reduction procedure proposed in this journal by Kao, and an approach based on the work of Sengupta using the notion of matrix exponential. There are three areas of interest: efficiency, memory accesses, and accuracy. In the empirical investigation, we use a subset of PH/PH/1 problems studied by Ramaswami and Latouche and a set of problems having non-phase-type interarrival times. At the expense of incurring varying degrees of loss of accuracy, the two algorithms proposed by Latouche and the method based on the work of Sengupta call for a lesser amount of computation than that of Kao's. When compared with the three alternatives, this study shows empirically that the R matrices obtained by state reduction—even when the approach is extended to Markov chains with denumerable state spaces—are closer to their respective true but unknown values.