Total Reward Criterion Research Articles

Denumerable Markov stopping games with risk-sensitive total reward criterion

Denumerable Markov stopping games with risk-sensitive total reward criterion

Formally Verified Solution Methods for Markov Decision Processes

We formally verify executable algorithms for solving Markov decision processes (MDPs) in the interactive theorem prover Isabelle/HOL. We build on existing formalizations of probability theory to analyze the expected total reward criterion on finite and infinite-horizon problems. Our developments formalize the Bellman equation and give conditions under which optimal policies exist. Based on this analysis, we verify dynamic programming algorithms to solve tabular MDPs. We evaluate the formally verified implementations experimentally on standard problems, compare them with state-of-the-art systems, and show that they are practical.

Nash equilibria in a class of Markov stopping games with total reward criterion

Nash equilibria in a class of Markov stopping games with total reward criterion

Markov stopping games with an absorbing state and total reward criterion

Markov stopping games with an absorbing state and total reward criterion

Bi-personal stochastic transient Markov games with stopping times and total reward criterion

Bi-personal stochastic transient Markov games with stopping times and total reward criterion

A Convex Programming Approach for Discrete-Time Markov Decision Processes under the Expected Total Reward Criterion

In this work, we study discrete-time Markov decision processes (MDPs) under constraints with Borel state and action spaces and where all the performance functions have the same form of the expected...

Constrained Markov Decision Processes with Expected Total Reward Criteria

In this paper, we investigate a Markov decision process with constraints on a Borel state space with the expected total reward criterion. Assuming that the transition probability has a density func...

Infinite horizon stopping problems with (nearly) total reward criteria

We study an infinite horizon optimal stopping Markov problem which is either undiscounted (total reward) or with a general Markovian discount rate. Using ergodic properties of the underlying Markov process, we establish the feasibility of the stopping problem and prove the existence of optimal and ε-optimal stopping times. We show the continuity of the value function and its variational characterisation (in the viscosity sense) under different sets of assumptions satisfied by large classes of diffusion and jump–diffusion processes. In the case of a general discounted problem we relax a classical assumption that the discount rate is uniformly separated from zero.

Convergence of a Q-learning variant for continuous states and actions

This paper presents a reinforcement learning algorithm for solving in finite horizon Markov Decision Processes under the expected total discounted reward criterion when both the state and action sp...

Total reward criteria for unconstrained/constrained continuous-time Markov decision processes

This paper studies denumerable continuous-time Markov decision processes with expected total reward criteria. The authors first study the unconstrained model with possible unbounded transition rates, and give suitable conditions on the controlled system’s primitive data under which the authors show the existence of a solution to the total reward optimality equation and also the existence of an optimal stationary policy. Then, the authors impose a constraint on an expected total cost, and consider the associated constrained model. Basing on the results about the unconstrained model and using the Lagrange multipliers approach, the authors prove the existence of constrained-optimal policies under some additional conditions. Finally, the authors apply the results to controlled queueing systems.

Approximate Gradient Methods in Policy-Space Optimization of Markov Reward Processes

We consider a discrete time, finite state Markov reward process that depends on a set of parameters. We start with a brief review of (stochastic) gradient descent methods that tune the parameters in order to optimize the average reward, using a single (possibly simulated) sample path of the process of interest. The resulting algorithms can be implemented online, and have the property that the gradient of the average reward converges to zero with probability 1. On the other hand, the updates can have a high variance, resulting in slow convergence. We address this issue and propose two approaches to reduce the variance. These approaches rely on approximate gradient formulas, which introduce an additional bias into the update direction. We derive bounds for the resulting bias terms and characterize the asymptotic behavior of the resulting algorithms. For one of the approaches considered, the magnitude of the bias term exhibits an interesting dependence on the time it takes for the rewards to reach steady-state. We also apply the methodology to Markov reward processes with a reward-free termination state, and an expected total reward criterion. We use a call admission control problem to illustrate the performance of the proposed algorithms.

Multiple Objective Nonatomic Markov Decision Processes with Total Reward Criteria

We consider a Markov decision process with an uncountable state space and multiple rewards. For each policy, its performance is evaluated by a vector of total expected rewards. Under the standard continuity assumptions and the additional assumption that all initial and transition probabilities are nonatomic, we prove that the set of performance vectors for all policies is equal to the set of performance vectors for (nonrandomized) Markov policies. This result implies the existence of optimal (nonrandomized) Markov policies for nonatomic constrained Markov decision processes with total rewards. We provide two examples of applications of our results to constrained multiple objective problems in inventory control and finance.

Constrained denumerable state non-stationary MDPs with expected total reward criterion

In this paper, we consider constrained denumerable state non-stationary Markov decision processes (MDPs, for short) with expected total reward criterion. By the mechanics of introducing Lagrange multiplier and using the methods of probability and analytics, we prove the existence of constrained optimal policies. Moreover, we prove that a constrained optimal policy may be a Markov policy, or be a randomized Markov policy that randomizes between two Markov policies, that differ in only one state.

Nearly optimal stationary policies in negative dynamic programmin g

This work concerns controlled Markov chains with denumerable state space and discrete time parameter. The reward function is assumed to be≤0 and the performance of a control policy is measured by the expected total-reward criterion. Within this context, sufficient conditions are given so that the existence of a stationary policy which is e-optimal at every state is guaranteed.

Optimal stationary strategies in leavable Markov decision processes

This paper establishes the existence of an optimal stationary strategy in a leavable Markov decision process with countable state space and undiscounted total reward criterion. Besides assumptions of boundedness and continuity, an assumption is imposed on the model which demands the continuity of the mean recurrence times on a subset of the stationary strategies, the so-called ‘good strategies'. For practical applications it is important that this assumption is implied by an assumption about the cost structure and the transition probabilities. In the last part we point out that our results in general cannot be deduced from related works on bias-optimality by Dekker and Hordijk, Wijngaard or Mann.

Pointwise and Uniformly Good Stationary Strategies for Dynamic Programming Models

This paper considers discrete time countable state Markovian decision problems with the total reward criterion. The main results are the following: a method for reducing the size of action sets for the given model without a loss in value, proof of the existence of pointwise good stationary strategies in the given model if such strategies exist in the embedded negative model and proof of the existence of uniformly good stationary strategies if pointwise good stationary strategies exist in the model.

Finite state dynamic programming with the total reward criterion

Dynamic programming problems with the expected total reward criterion and finite state space are considered. Two different value functions, the supremum over all policies and the supremum over all stationary policies, are examined. It is shown that both value functions satisfy the optimality equation and upper and lower bounds as well as conditions for equality for these functions are presented. Using these results characterizations for those policies which achieve the maximum within all policies and for those which achieve the maximum within all stationary policies are derived. Finally a modified version of the Howard policy improvement is proposed which yields an optimal policy for problems with finite action sets.

On iterative optimization ol structured Markov decision processes with discounted rewards

The paper gives a survey on solution techniques for Markov decision processes with respect to the total reward criterion. We will discuss briefly a number of problem structures which guarantee that the optimal policy possesses a specific structure which can be exploited in numerical solution procedures. However, the main emphasis is on iterative methods. It is shown by examples that the effect of a number of modifications of the standard iterative method, which are advocated in the literature, is limited in some realistic situations. Numerical evidence is provided to show that exploiting the structure of a problem under consideration of ten yields a more substantial reduction of the required computational effort than some of the existing acceleration procedures. We advocate that this structure should be analyzed and used in choosing the appropriate solution procedure. The appropriate procedure might be composed on one hand by blending several of the acceleration concepts that are described in literature. ...

On theory and algorithms for Markov decision problems with the total reward criterion

The first part of this survey paper is devoted to derive under rather weak conditions, which don't guarantee contraction, a number of important existency and convergency results in Markov decision theory. In the second part of the paper conditions that guarantee that the contraction mapping approach can be used are analysed. These conditions are rather weak and allow for unbounded rewards. The generation of successive approximation methods for solving Markov decision processes by using action depending stopping times is described at the end of the paper. Im ersten Teil dieser Ubersichtsarbeit werden unter schwachen Voraussetzungen, die keine Kontraktion garantieren, eine Reihe wichtiger Existenz- und Konvergenzaussagen fur Markoffsche Entscheidungs-prozesse gewonnen. Im zweiten Teil der Arbeit werden Bedingungen untersucht, die einen Zugang mit Hilfe kontrahierender Abbildungen erlauben. Diese Bedingungen sind recht schwach, insbesondere sind unbeschrankte Ertrage zugelassen. Am Ende der Arbeit wird beschrieben, wie man mit Hilfe aktionsabhangiger Stopzeiten verschiedene Verfahren der sukzessiven Approximation zur Losung Markoffscher Entscheidungsprozesse gewinnen kann.