Nonnegative Rewards Research Articles

Approximability and efficient algorithms for constrained fixed-horizon POMDPs with durative actions

Partially Observable Markov Decision Process (POMDP) is a fundamental model for probabilistic planning in stochastic domains. More recently, constrained POMDP and chance-constrained POMDP extend the model allowing constraints to be specified on some aspects of the policy in addition to the objective function. Despite their expressive power, these models assume all actions take a fixed duration, which poses a limitation in modeling real-world planning problems. In this work, we propose a unified model for durative POMDP and its constrained extensions. First, we convert these extensions into an Integer Linear Programming (ILP) formulation, which can be solved using existing solvers in the ILP literature. Second, a heuristic search approach is provided that can efficiently prune the search space, guided by solving successive partial ILP programs. Third, we give a theoretical analysis of the problem: unlike short-horizon POMDPs, with policies of a constant depth, which can be solved in polynomial time, the constrained extensions are NP-Hard even with a planning horizon of two and non-negative rewards. To alleviate that, we propose a Fully Polynomial Time Approximation Scheme (FPTAS) that computes (near) optimal deterministic policies in polynomial time. The FPTAS is among the best achievable in theory in terms of approximation ratio. Finally, evaluation results show that our approach is empirically superior to the state-of-the-art fixed-horizon chance-constrained POMDP solver.

Analysis of a generalized Linear Ordering Problem via integer programming

We study a generalized version of the linear ordering problem: Given a collection of partial orders represented by directed trees with unique root and height one, where each tree is associated with a nonnegative reward, the goal is to build a linear order of maximum reward, where the total reward is defined as the sum of the rewards of the trees compatible with the linear order. Each tree has a single root, and includes a distinguished element either as the root or as a leaf. There is a constraint about the position that the distinguished element has to occupy in the final order. The problem is NP-Hard, and has applications in diverse areas such as machine learning, discrete choice theory, and scheduling.Our contribution is two-fold. On the theoretical side, we formulate an integer programming model, establish the dimension of the convex hull of all integer feasible solutions, and infer several families of valid inequalities, including facet defining ones. On the computational side, we develop a branch-and-cut (B&C) algorithm that is competitive with state-of-the-art, generic B&C methods with respect to running time and quality of the solutions obtained. Through an extensive set of numerical studies, we characterize conditions of the model that result in a significant dominance of our B&C proposal.

Local Poisson equations associated with the Varadhan functional

This work concerns Markov chains evolving on a denumerable sate space, which is endowed with a non-negative reward function with finite support. In this context, the problem of determining the Varadhan function, given by the exponential growth rate of the aggregated rewards, is studied. The main results in this direction are expressed in terms of the idea of a system of local Poisson equations, and can be summarized as follows: (i) the Varadhan function is determined by one of such systems, and (ii) if a finite set is accessible form any state, then a system of local Poisson equations exits.

The Transformation Method for Continuous-Time Markov Decision Processes

In this paper, we show that a discounted continuous-time Markov decision process in Borel spaces with randomized history-dependent policies, arbitrarily unbounded transition rates and a non-negative reward rate is equivalent to a discrete-time Markov decision process. Based on a completely new proof, which does not involve Kolmogorov’s forward equation, it is shown that the value function for both models is given by the minimal non-negative solution to the same Bellman equation. A verifiable necessary and sufficient condition for the finiteness of this value function is given, which induces a new condition for the non-explosion of the underlying controlled process.

On real reward testing

We extend the traditional nonnegative reward testing with negative rewards. In this new testing framework, may preorder and must preorder are the inverse of each other. More surprisingly, it turns out that the real reward must testing is no more powerful than the nonnegative reward testing, at least for finite processes. In order to prove that result, we exploit an important property of failure simulation about the inclusion of the testing outcomes between two related processes.

ASSIGNMENT SITUATIONS WITH MULTIPLE OWNERSHIP AND THEIR GAMES

An assignment situation can be considered as a two-sided market consisting of two disjoint sets of objects. A non-negative reward matrix describes the profit if an object of one group is assigned to an object of the other group. Assuming that each object is owned by a different agent, Shapley and Shubik (1972) introduced a class of assignment games. This paper introduces assignment situations with multiple ownership (AMO). In these situations an object can be owned by several agents and an agent can participate in the ownership of more than one object. For AMO situations we introduce the class of k-AMO games. An AMO situation is called balanced if for any choice of the reward matrix the corresponding 1-AMO game is balanced. We provide necessary and sufficient conditions for balancedness of AMO situations. Moreover, sufficient conditions are provided for balancedness of k-AMO games.

An Approximation Algorithm for the Discrete Team Decision Problem

In this paper we study a discrete version of the classical team decision problem. It has been shown previously that the general discrete team decision problem is NP-hard. Here we present an efficient approximation algorithm for this problem. For the maximization version of this problem with nonnegative rewards, this algorithm computes decision rules which are guaranteed to be within a fixed bound of optimal.

Some memoryless bandit policies

We consider a multiarmed bandit problem, where each arm when pulled generates independent and identically distributed nonnegative rewards according to some unknown distribution. The goal is to maximize the long-run average reward per pull with the restriction that any previously learned information is forgotten whenever a switch between arms is made. We present several policies and a peculiarity surrounding them.

Some memoryless bandit policies

We consider a multiarmed bandit problem, where each arm when pulled generates independent and identically distributed nonnegative rewards according to some unknown distribution. The goal is to maximize the long-run average reward per pull with the restriction that any previously learned information is forgotten whenever a switch between arms is made. We present several policies and a peculiarity surrounding them.

Computation of bounds for transient measures of large rewarded Markov models using regenerative randomization

In this paper we generalize a method (called regenerative randomization) for the transient solution of continuous time Markov models. The generalized method allows to compute two transient measures (the expected transient reward rate and the expected averaged reward rate) for rewarded continuous time Markov models with a structure covering bounding models which are useful when a complete, exact model has unmanageable size. The method has the same good properties as the well-known (standard) randomization method: numerical stability, well-controlled computation error, and ability to specify the computation error in advance, and, for large enough models and long enough times, is significantly faster than the standard randomization method. The method requires the selection of a regenerative state and its performance depends on that selection. For a class of models, class C′, including typical failure/repair models with exponential failure and repair time distributions and repair in every state with failed components, a natural selection for the regenerative state exists, and results are available assessing approximately the performance of the method for that natural selection in terms of “visible” model characteristics. Those results can be used to anticipate when the method can be expected to be significantly faster than standard randomization for models in that class. The potentially superior efficiency of the regenerative randomization method compared to standard randomization for models not in class C′ is illustrated using a large performability model of a fault-tolerant multiprocessor system. Scope and purpose Rewarded continuous time Markov models are widely used for performance, dependability and performability analysis of computer and telecommunication systems. Realistic modeling of such systems usually yields Markov models whose size exceeds the available memory resources. An approach to deal with the “largeness” problem is the use of bounding Markov models. However, even those bounding models can have very large state spaces, making important the development of efficient numerical solution techniques. Often, one is interested in transient characteristics of the system which require the transient analysis of the model. This paper generalizes a method, called regenerative randomization, for the transient analysis of Markov models, so that it can be used for the computation of bounds for the expected transient reward rate and the expected averaged reward rate transient measures with general reward rate structures including arbitrary non-negative reward rates associated with the states of the model. Examples of such measures are the unavailability of a fault-tolerant system at time t and the expected interval unavailability of a fault-tolerant system at time t. The method has the same good properties as the well-known standard randomization (also called uniformization) method: numerical stability, well-controlled computation error, and ability to specify the computation error in advance, and, for large models and long mission times, can be significantly faster than that method. Furthermore, for a class of models, class C′, including typical failure/repair models with exponential failure and repair time distributions and repair in every state with failed components, theoretical results are available assessing the performance of the method in terms of “visible” model characteristics. Those results can be used to anticipate when the generalized regenerative randomization method can be expected to be competitive for class C′ models. The generalized regenerative randomization method allows a numerically stable, with well-controlled and specifiable-in-advance error, solution of some large rewarded continuous time Markov models in affordable CPU times.

EXISTENCE OF OPTIMAL STATIONARY POLICIES IN FINITE DYNAMIC PROGRAMS WITH NONNEGATIVE REWARDS

This article concerns Markov decision chains with finite state and action spaces, and a control policy is graded via the expected total-reward criterion associated to a nonnegative reward function. Within this framework, a classical theorem guarantees the existence of an optimal stationary policy whenever the optimal value function is finite, a result that is obtained via a limit process using the discounted criterion. The objective of this article is to present an alternative approach, based entirely on the properties of the expected total-reward index, to establish such an existence result.

Optimal stationary policies inrisk-sensitive dynamic programs with finite state spaceand nonnegative rewards

This work concerns controlled Markov chains with finite state space and nonnegative rewards; it is assumed that the controller has a constant risk-sensitivity, and that the performance ofa control policy is measured by a risk-sensitive expected total-rewa

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

This paper studies the minimizing risk problems in Markov decision processes with countable state space and reward set. The objective is to find a policy which minimizes the probability (risk) that the total discounted rewards do not exceed a specified value (target). In this sort of model, the decision made by the decision maker depends not only on system's states, but also on his target values. By introducing the decision-maker's state, we formulate a framework for minimizing risk models. The policies discussed depend on target values and the rewards may be arbitrary real numbers. For the finite horizon model, the main results obtained are: (i) The optimal value functions are distribution functions of the target, (ii) there exists an optimal deterministic Markov policy, and (iii) a policy is optimal if and only if at each realizable state it always takes optimal action. In addition, we obtain a sufficient condition and a necessary condition for the existence of finite horizon optimal policy independent of targets and we give an algorithm computing finite horizon optimal policies and optimal value functions. For an infinite horizon model, we establish the optimality equation and we obtain the structure property of optimal policy. We prove that the optimal value function is a distribution function of target and we present a new approximation formula which is the generalization of the nonnegative rewards cases. An example which illustrates the mistakes of previous literature shows that the existence of optimal policy has not been proved really. In this paper, we give an existence condition, which is a sufficient and necessary condition for the existence of an infinite horizon optimal policy independent of targets, and we point out that whether there exists an optimal policy remains an open problem in the general case.