Countable Action Spaces Research Articles

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

On the Minimum Pair Approach for Average Cost Markov Decision Processes with Countable Discrete Action Spaces and Strictly Unbounded Costs

We consider average-cost Markov decision processes (MDPs) with Borel state spaces, countable, discrete action spaces, and strictly unbounded one-stage costs. For the minimum pair approach, we introduce a new majorization condition on the state transition stochastic kernel, in place of the commonly required continuity conditions on the MDP model. We combine this majorization condition with Lusin's theorem to prove the existence of a stationary minimum pair, i.e., a stationary policy paired with an invariant probability measure induced on the state space, with the property that the pair attains the minimum long-run average cost over all policies and initial distributions. We also establish other optimality properties of a stationary minimum pair, and for the stationary policy in such a pair, under additional recurrence or regularity conditions, we prove its pathwise optimality and strong optimality. Our results can be applied to a class of countable action space MDPs in which the dynamics and one-stage costs are discontinuous with respect to the state variable.

Semi-Infinite Weighted Markov Decision Processes

In this article, weighted reward Markov decision processes (semi-infinite WMDP) with finite state and countable action spaces are considered. The “weighted reward” refers to an appropriately normalized convex combination of the discounted and long-run average reward criteria. This criterion allows the controller to trade-off short-term rewards versus long-run rewards. We prove that under some conditions, the supremum in the class of general strategies is equivalent to the supremum in the class of relatively simple “ultimately deterministic” strategies. These are strategies that behave just like deterministic stationary strategies, after a certain point in time. We present an iterative algorithm for computing a δ-optimal simple ultimately deterministic strategy. The steps of the algorithm are based on the one developed in Ref. for finite WMDP.

Semi-infinite weighted Markov decision processes with perturbation

In this paper, Weighted reward Perturbed Markov Decision Processes with finite state and countable action spaces (semi-infinite WMDP for short) are considered. The ”weighted reward” refers to appropriately normalized convex combination of the discounted and the long-run average reward criteria. This criterion allows the controller to trade-off short-term rewards versus long-run rewards. In every application where both the discounted and the long-run average criteria have been proposed in the past, there is clearly a rationale for considering the weighted criterion. Of course, as with all Markov decision models, the standard weighted criterion model assumes that all the transition probabilities are known precisely. Since, in most applications this would not be the case, we consider the perturbed version of the weighted reward model. In the case of perturbations, we prove that for many models a nearly optimal strategy can be found in the class of relatively “simple ultimately deterministic” strategies. These are strategies which behave just like deterministic stationary strategies, after a certain point of time.

Strong 0-discount optimal policies in a Markov decision process with a Borel state space

We consider a Markov decision process with a Borel state space, a countable action space, finite action sets, bounded rewards and a bounded transition density satisfying a simultaneous Doeblin condition. The existence of stationary strong 0-discount optimal polices is proved.

Optimal Stationary Policies in General State Space Markov Decision Chains with Finite Action Sets

The result of Sennott [9] on the existence of optimal stationary policies in countable state Markov decision chains with finite action sets is generalized to arbitrary state space Markov decision chains. The assumption of finite action sets occurring in a global countable action space allows a particularly simple theoretical structure for the general state space Markov decision chain. Two examples illustrate the results. Example 1 is a system of parallel queues with stochastic work requirements, a movable server with controllable service rate, and a reject option. Example 2 is a system of parallel queues with stochastic controllable inputs, a movable server with fixed service rates, and a reject option.

Duality in Markov Decision Problems with Countable Action and State Spaces

The recent literature contains several papers which explore mathematical programming formulations of particular Markov sequential decision problems. Each of these papers deals with finite state and action spaces; thus, the corresponding programming formulations yield dual finite linear programs. In this paper these investigations are extended to include countable action and/or state spaces for finite horison problems. Of particular interest are the duality aspects of the mathematical programming formulations. In addition, employing conditions analogous to fundamental concepts of Haar semi-infinite dual programming, we provide sufficient conditions for the existence of optimal rules for countable action spaces. Guided by the semi-infinite duality theory we explore mathematical programming formulations for two cases: 1) Countable action space and finite state space—the result is a pair of dual semi-infinite programs; and 2) Finite action space and countable state space—we obtain a pair of infinite programs. In the latter case we show that no duality gap occurs and obtain duality results comparable to those of finite linear programming.