Denumerable State Research Articles

Finite-state approximations for denumerable state discounted markov decision processes

A finite-state iterative scheme introduced by White [9] to approximate the optimal value function of denumerable-state Markov decision processes with bounded rewards, is extended to the case of unbounded rewards. Convergence theorems that, when applied to the case of bounded rewards, give stronger results than those in [9] are proved. Moreover, bounds on the rates of convergence under several assumptions are given and the extended scheme is used to obtain policies with asymptotic optimality properties.

Nonstationary value-iteration and adaptive control of discounted semi-Markov processes

We consider in this paper discounted-reward, denumerable state space, semi-Markov decision processes which depend on unknown parameters. The problems we are interested in are: Given that the true parameter value is unknown, (I) give an iterative scheme to determine the total maximal discounted reward, and (II) find an asymptotically discount optimal (adaptive) policy. Our solutions are inspired by the nonstationary value iteration (NVI) scheme of Federgruen and Schweitzer ( J. Optim. Theory Appl. 34 (1981) , 207–241) combined with the ideas of Schäl (Preprint No. 428, Inst. Angew. Math. Univ. Bonn, 1981) concerning the “principle of estimation and control” for the adaptive control of semi-Markov processes.

Some sufficient conditions for non-ergodicity of markov chains

Some sufficient conditions for non-ergodicity are given for a Markov chain with denumerable state space. These conditions generalize Foster's results, in that unbounded Lyapunov functions are considered. Our criteria directly extend the conditions obtained in Kaplan (1979), in the sense that a class of Lyapunov functions is studied. Applications are presented through some examples; in particular, sufficient conditions for non-ergodicity of a multidimensional Markov chain are given.

Finite state approximation algorithms for average cost denumerable state Markov decision processes

In this paper we study three finite state, value and policy iteration algorithms for denumerable space Markov decision processes with respect to the average cost criterion. The convergence of these algorithms is guaranteed under a scrambling-type recurrency condition and various "tail" conditions on the transition probabilities. With the value iteration schemes we construct nearly optimal policies by concentrating on a finite set of "important" states and controlling them as well as we can. The policy space algorithm consists of a value determination scheme associated with a policy and a policy improvement step where a "better" policy is determined. Thus a sequence of improved policies is constructed which is shown to converge to the optimal average cost policy. Fur Markovsche Entscheidungsprozesse mit abzahlbarem Zustandsraum untersuchen wir fur den Fall des Durchschnittskostenkriteriums drei endliche Wertiterations- und Politikiterations-Algorithmen. Die Konvergenz der Algorithmen wird durch "scramblingtype" Rekurrenzbedingungen und verschiene "tail" Bedingungen an die Ubergangswahrscheinlichkeiten gesichert. Mit den Wertiterationsverfahren konstruieren wir fast optimale Politiken, indem wir uns auf eine endliche Menge von "wichtigen" Zustanden konzentrieren und diese bestmoglich kontrollieren. Der Politikiterations-Algorithmus besteht aus einem Schritt zur Wertbestimmung fur eine Politik und einem Schritt zur Verbesserung der Politik. Auf diese Weise wird eine Folge verbesserter Politiken konstruiert, die Konvergenz zur optimalen Politik wird gezeigt.

Some sufficient conditions for non-ergodicity of markov chains

Some sufficient conditions for non-ergodicity are given for a Markov chain with denumerable state space. These conditions generalize Foster's results, in that unbounded Lyapunov functions are considered. Our criteria directly extend the conditions obtained in Kaplan (1979), in the sense that a class of Lyapunov functions is studied. Applications are presented through some examples; in particular, sufficient conditions for non-ergodicity of a multidimensional Markov chain are given.

On the Existence of Average Optimal Policies in Semiregenerative Decision Models

This paper gives sufficient conditions for the existence of average optimal policies in a decision model, which is a generalization of a denumerable state space semi-Markov decision model. The conditions extend most of those previously known; especially, neither uni-chainedness nor communicatingness is assumed. An average optimal policy can be obtained as a limit of discount optimal policies. If structured policies are discount optimal, there exists a structured average optimal policy. The results are applicable to more general continuous time models such as the one by Yushkevich and Fainberg (Yushkevich, A. A., E. A. Fainberg. 1979. On homogeneous Markov models with continuous time and finite or countable state space. Theory Probab. Appl. 24 156–161.).

Continuous‐time markov decision processes with nonzero terminal reward

AbstractIn this article we consider a continuous‐time Markov decision process with a denumerable state space and nonzero terminal rewards. We first establish the necessary and sufficient optimality condition without any restriction on the cost functions. The necessary condition is derived through the Pontryagin maximum principle and the sufficient condition, by the inherent structure of the problem. We introduce a dynamic programming approximation algorithm for the finite‐horizon problem. As the time between discrete points decreases, the optimal policy of the discretized problem converges to that of the continuous‐time problem in the sense of weak convergence. For the infinite‐horizon problem, a successive approximation method is introduced as an alternative to a policy iteration method.

Technical Note—Mean Drifts and the Non-Ergodicity of Markov Chains

An important question in stochastic modeling is whether or not a denumerable state Markov chain is ergodic. We extend a result of Kaplan giving a condition for the non-ergodicity of a chain based on its drifts. We also clarify the condition under which the mean drift in the stationary chain is zero.

Semi-Markov decision processes with polynomial reward

A semi-Markov decision process, with a denumerable multidimensional state space, is considered. At any given state only a finite number of actions can be taken to control the process. The immediate reward earned in one transition period is merely assumed to be bounded by a polynomial and a bound is imposed on a weighted moment of the next state reached in one transition. It is shown that under an ergodicity assumption there is a stationary optimal policy for the long-run average reward criterion. A queueing network scheduling problem, for which previous criteria are inapplicable, is given as an application.

Modeling of high error rate binary communication channels (Corresp.)

Most of the existing mathematical models for binary communication channels describe low error rate wire lines satisfactorily. For typical high error rate channels, like the ultrahigh frequency (UHF) or very high frequency (VHF) wideband data channel encountered in military uses, finite-state as well as denumerable infinite state Markov chain models do not achieve an accurate characterization. The above models, including some more recent compound models, are compared against data from the actual channel using the multigap distribution as a tool. A modification of a compound model is shown that allows more accurate modeling of a wider class of channels including the high error rate radio channel.

Finite state approximations for denumerable state infinite horizon discounted Markov decision processes with unbounded rewards

Finite state approximations for denumerable state infinite horizon discounted Markov decision processes with unbounded rewards

A positive recurrence criterion associated with multidimensional queueing processes

A criterion is given for positive recurrence of a multidimensional, aperiodic, irreducible Markov chain with a denumerable state space. This criterion extends to the multidimensional case Foster's one-dimensional criterion. The multidimensional criterion consists of several conditions, one for each coordinate of the process. The usefulness of this criterion is shown through a queueing network example.

A positive recurrence criterion associated with multidimensional queueing processes

A criterion is given for positive recurrence of a multidimensional, aperiodic, irreducible Markov chain with a denumerable state space. This criterion extends to the multidimensional case Foster's one-dimensional criterion. The multidimensional criterion consists of several conditions, one for each coordinate of the process. The usefulness of this criterion is shown through a queueing network example.

Denumerable state semi-Markov decision processes with unbounded costs, average cost criterion

This paper establishes a rather complete optimality theory for the average cost semi-Markov decision model with a denumerable state space, compact metric action sets and unbounded one-step costs for the case where the underlying Markov chains have a single ergotic set. Under a condition which, roughly speaking, requires the existence of a finite set such that the supremum over all stationary policies of the expected time and the total expected absolute cost incurred until the first return to this set are finite for any starting state, we shall verify the existence of a finite solution to the average costs optimality equation and the existence of an average cost optimal stationary policy.

On N-person stochastic games by denumerable state space

This paper considers non-cooperative N-person stochastic games with a countable state space and compact metric action spaces. We concentrate upon the average return per unit time criterion for which the existence of an equilibrium policy is established under a number of recurrency conditions with respect to the transition probability matrices associated with the stationary policies. These results are obtained by establishing the existence of total discounted return equilibrium policies, for each discount factor α ∈ [0, 1) and by showing that under each one of the aforementioned recurrency conditions, average return equilibrium policies appear as limit policies of sequences of discounted return equilibrium policies, with discount factor tending to one. Finally, we review and extend the results that are known for the case where both the state space and the action spaces are finite.

The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms

This paper is concerned with the optimality equation for the average costs in a denumerable state semi-Markov decision model. It will be shown that under each of a number of recurrency conditions on the transition probability matrices associated with the stationary policies, the optimality equation has a bounded solution. This solution indeed yields a stationary policy which is optimal for a strong version of the average cost optimality criterion. Besides the existence of a bounded solution to the optimality equation, we will show that both the value-iteration method and the policy-iteration method can be used to determine such a solution. For the latter method we will prove that the average costs and the relative cost functions of the policies generated converge to a solution of the optimality equation.

On N-person stochastic games by denumerable state space

This paper considers non-cooperative N-person stochastic games with a countable state space and compact metric action spaces. We concentrate upon the average return per unit time criterion for which the existence of an equilibrium policy is established under a number of recurrency conditions with respect to the transition probability matrices associated with the stationary policies. These results are obtained by establishing the existence of total discounted return equilibrium policies, for each discount factor α ∈ [0, 1) and by showing that under each one of the aforementioned recurrency conditions, average return equilibrium policies appear as limit policies of sequences of discounted return equilibrium policies, with discount factor tending to one.Finally, we review and extend the results that are known for the case where both the state space and the action spaces are finite.

The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms

An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms

An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms

This paper is concerned with the optimality equation for the average costs in a denumerable state semi-Markov decision model. It will be shown that under each of a number of recurrency conditions on the transition probability matrices associated with the stationary policies, the optimality equation has a bounded solution. This solution indeed yields a stationary policy which is optimal for a strong version of the average cost optimality criterion. Besides the existence of a bounded solution to the optimality equation, we will show that both the value-iteration method and the policy-iteration method can be used to determine such a solution. For the latter method we will prove that the average costs and the relative cost functions of the policies generated converge to a solution of the optimality equation.