Finite Action Sets Research Articles

Two-Person Zero-Sum Stochastic Games with Semicontinuous Payoff

Consider a two-person, zero-sum stochastic game with Borel state space S, finite action sets A,B, and Borel measurable law of motion q. Suppose the payoff is a bounded function f of the infinite history of states and actions that is measurable for the product of the Borel σ-field for S and the σ-fields of all subsets for A and B, and is lower semicontinuous for the product of the discrete topologies on the coordinate spaces. Then the game has a value and player II has a subgame perfect optimal strategy.

PROMETHEE with Precedence Order in the Criteria (PPOC) as a New Group Decision Making Aid: An Application in Urban Water Supply Management

In this paper, a new group Multi-Criteria-Decision-Making (MCDM) method is introduced by combining two “Preference Ranking Organization Method for Enrichment Evaluations (PROMETHEE)” and “Multi-attribute decision making with dominance in the criteria” methods. PROMETHEE family of outranking methods is among the recently developed MCDM methods which have received lots of attention in the recent years because of its capacity in ranking finite set of alternative actions based on conflicting criteria. The second method helps the decision makers to consider ambiguity and imprecision of relative importance of each objective (criterion) without allocating importance weights to them. The proposed method of PROMETHEE with Precedence Order in the Criteria (PPOC) not only can address capabilities of PROMETHEE method just with determination of precedence order of criteria, but also can make it possible to have a group decision making environment with conflicting objectives. Operational management of an urban water supply system is a good example of a set of decision making problems with several objectives and Decision Makers (DMs). In this paper, PPOC method has been applied to the case study of Melbourne water supply system, previously analyzed in the literature, to assess a number of operation rules with respect to eight criteria evaluated under single or group decision-making situations. The satisfaction degree of each DM and the overall group ranking results have also been provided in the paper. The proposed method is applicable for different decision making problems in urban water supply management.

Models of active learning in group-structured state spaces

We investigate the problem of learning the transition dynamics of deterministic, discrete-state environments. We assume that an agent exploring such an environment is able to perform actions (from a finite set of actions) in the environment and to sense the state changes. The question investigated is whether the agent can learn the dynamics without visiting all states. Such a goal is unrealistic in general, hence we assume that the environment has structural properties an agent might exploit. In particular, we assume that the set of all action sequences forms an algebraic group. We introduce a learning model in different variants and study under which circumstances the corresponding “group-structured environments” can be learned efficiently by experimenting with group generators (actions). It turns out that for some classes of such environments the choice of actions given to the agent determines if efficient learning is possible. Negative results are presented, even without efficiency constraints, for rather general classes of groups, showing that even with group structure, learning an environment from partial information is far from trivial. However, positive results for special subclasses of Abelian groups turn out to be a good starting point for the design of efficient learning algorithms based on structured representations.

Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method

We present a method called Generalized Regression with Intensities of Preference (GRIP) for ranking a finite set of actions evaluated on multiple criteria. GRIP builds a set of additive value functions compatible with preference information composed of a partial preorder and required intensities of preference on a subset of actions, called reference actions. It constructs not only the preference relation in the considered set of actions, but it also gives information about intensities of preference for pairs of actions from this set for a given decision maker (DM). Distinguishing necessary and possible consequences of preference information on the considered set of actions, GRIP answers questions of robustness analysis. The proposed methodology can be seen as an extension of the UTA method based on ordinal regression. GRIP can also be compared to the AHP method, which requires pairwise comparison of all actions and criteria, and yields a priority ranking of actions. As for the preference information being used, GRIP can be compared, moreover, to the MACBETH method which also takes into account a preference order of actions and intensity of preference for pairs of actions. The preference information used in GRIP does not need, however, to be complete: the DM is asked to provide comparisons of only those pairs of reference actions on particular criteria for which his/her judgment is sufficiently certain. This is an important advantage comparing to methods which, instead, require comparison of all possible pairs of actions on all the considered criteria. Moreover, GRIP works with a set of general additive value functions compatible with the preference information, while other methods use a single and less general value function, such as the weighted-sum.

Structural properties for a two-state partially observable Markov decision process with an average cost criterion

This paper investigates an optimal replacement problem for a discrete-time Markovian deterioration system. We consider partially observable Markov decision processes with finite core state and observation spaces and finite action set. The problem is to obtain an optimal replacement control-limit policy. Sufficient conditions, are then derived for a bounded solution to the average cost optimality equation to exist. Furthermore, under some reasonable conditions reflecting the practical meaning of the deterioration, structural properties for average costs policies are obtained for a two state replacement problem, and we develop a solution procedure utilizing these structural properties. We have that control permits working directly making it suitable for dealing with real-world processes and applications.

Aspiration level approach in stochastic MCDM problems

The paper considers a discrete stochastic multiple criteria decision making problem. This problem is defined by a finite set of actions A, a set of attributes X and a set of evaluations of actions with respect to attributes E. In stochastic case the evaluation of each action with respect to each attribute takes form of a probability distribution. Thus, the comparison of two actions leads to the comparison of two vectors of probability distributions. In the paper a new procedure for solving this problem is proposed. It is based on three concepts: stochastic dominance, interactive approach, and preference threshold. The idea of the procedure comes from the interactive multiple objective goal programming approach. The set of actions is progressively reduced as the decision maker specifies additional requirements. At the beginning the decision maker is asked to define preference threshold for each attribute. Next, at each iteration the decision maker is confronted with the set of considered actions. If the decision maker is able to make a final choice then the procedure ends, otherwise he/she is asked to specify aspiration level. A didactical example is presented to illustrate the proposed technique.

Extreme point characterizations for infinite network flow problems

AbstractWe study capacitated network flow problems with demands defined on a countably infinite collection of nodes having finite degree. This class of network flow models includes, for example, all infinite horizon deterministic dynamic programs with finite action sets, because these are equivalent to the problem of finding a shortest path in an infinite directed network. We derive necessary and sufficient conditions for flows to be extreme points of the set of feasible flows. Under an additional regularity condition met by all such problems with integer data, we show that a feasible solution is an extreme point if and only if it contains neither a cycle nor a doubly‐infinite path consisting of free arcs (an arc is free if its flow is strictly between its upper and lower bounds). We employ this result to show that the extreme points can be characterized by specifying a basis. Moreover, we establish the integrality of extreme point flows whenever node demands and arc capacities are integer valued. We illustrate our results with an application to an infinite horizon economic lot‐sizing problem. © 2006 Wiley Periodicals, Inc. NETWORKS, Vol. 48(4), 209–222 2006

Generic Uniqueness and Continuity of Rationalizable Strategies

For a finite set of actions and a rich set of fundamentals, consider the rationalizable actions on a universal type space, endowed with the usual product topology. (1) Generically, there exists a unique rationalizable action profile. (2) Every model can be approximately embedded in a dominance-solvable model. (3) For any given rationalizable strategy of any finite model, there exists a nearby finite model with common prior such that the given rationalizable strategy is uniquely rationalizable for nearby types.

How many samples?: a Bayesian nonparametric approach

Summary. The paper considers a Bayesian nonparametric decision theoretic approach to sample size calculations, where the ultimate goal is to make a terminal action from a finite set of actions. This terminal action is made via the maximization of expected utility, the maximization being made with respect to a probability measure on the states of nature. The probability measure depends on the amount of information, i.e. the number of samples collected. It is the prior in the case of no samples and the posterior when samples have been taken.

On Equilibrium in Pure Strategies in Games with Many Players

Motivated by issues of imitation, learning and evolution, we introduce a framework of non-cooperative games, allowing both countable sets of pure actions and player types and player types and demonstrate that for all games with sufficiently many players, every mixed strategy Nash equilibrium can be used to construct a Nash-equilibrium in pure strategies that is equivalent. Our framework introduces and exploits a distinction between crowding attributes of players (their external effects on others) and their taste attributes (their payoff functions and any other attributes that are not directly relevant to other players). The set of crowding attributes is assumed to be compact; this is not required, however, for taste attributes. We stress that for studying issues such as conformity, the case of a finite set of types and actions, while illuminating, cannot yield completely satisfactory results. Our main theorems are based on a new mathematical result, in the spirit of the Shapley-Folkman Theorem but applicable to a countable (not necessarily finite dimensional) strategy space.

Solution to the risk-sensitive average optimality equation in communicating Markov decision chains with finite state space: An alternative approach

This note concerns Markov decision chains with finite state and action sets. The decision maker is assumed to be risk-averse with constant risk sensitive coefficient λ, and the performance of a control policy is measured by the risk-sensitive average cost criterion. In their seminal paper Howard and Matheson established that, when the whole state space is a communicating class under the action of each stationary policy, then there exists a solution to the optimality equation for every λ>0. This paper presents an alternative proof of this fundamental result, which explicitly highlights the essential role of the communication properties in the analysis of the risk-sensitive average cost criterion.

Optimal learning and experimentation in bandit problems

This paper studies how and how much active experimentation is used in discounted or finite-horizon optimization problems with an agent who chooses actions sequentially from a finite set of actions, with rewards depending on unknown parameters associated with the actions. Closed-form approximations are developed for the optimal rules in these ‘multi-armed bandit’ problems. Some refinements and modifications of the basic structure of these approximations also provide a nearly optimal solution to the long-standing problem of incorporating switching costs into multi-armed bandits.

N-Person stochastic games with upper semi-continuous payoffs

Every n-person stochastic game with a countable state space, finite action sets for the players and bounded, upper semi-continuous payoffs has an e-equilibrium for every e>0.

Stay-in-a-set games

There exists a Nash equilibrium (e-Nash equilibrium) for every n-person stochastic game with a finite (countable) state space and finite action sets for the players if the payoff to each player i is one when the process of states remains in a given set of states Gi and is zero otherwise.

Semi-infinite discounted Markov decision processes: Policy improvement and singular perturbations

In this paper, Discounted Markov Decision Processes with finite state and countable action set (semi-infinite DMDP for short) are considered. A policy improvement finite algorithm which finds a nearly optimal deterministic strategy is presented. The steps of the algorithm are based on the classical policy improvement algorithm for finite DMDPs. Singularly perturbed semi-infinite DMDPs are investigated. In case of perturbations, some sufficient condition is given to guarantee that there exists a nearly optimal deterministic strategy which can approximate nearly optimal strategies for a whole family of singularly perturbed semi-infinite DMDP.

Two-player stochastic games I: A reduction

This paper is the first step in the proof of existence of equilibrium payoffs for two-player stochastic games with finite state and action sets. It reduces the existence problem to the class of so-called positive absorbing recursive games. The existence problem for this class is solved in a subsequent paper.

Large Poisson Games

Existence of equilibria is proven for Poisson games with compact type sets and finite action sets. Then three theorems are introduced for characterizing limits of probabilities in Poisson games when the expected number of players becomes large. The magnitude theorem characterizes the rate at which probabilities of events go to zero. The offset theorem characterizes the ratios of probabilities of events that differ by a finite additive translation. The hyperplane theorem estimates probabilities of hyperplane events. These theorems are applied to derive formulas for pivot probabilities in binary elections, and to analyze a voting game that was studied by Ledyard. Journal of Economic Literature Classification Numbers: C63, C70.

Semi-infinite Markov decision processes

In this paper discounted and average Markov decision processes with finite state space and countable action set (semi-infinite MDP for short) are discussed. Without ordinary continuity and compactness conditions, for discounted semi-infinite MDP we have shown that by exploiting the results on semi-infinite linear programming due to Tijs [20] our semi-infinite discounted MDP can be approximated by a sequence of finite discounted MDPs and even in a semi-infinite discounted MDP it is sufficient to restrict ourselves to the class of deterministic stationary strategies. For average reward case we still prove that under some conditions the supremum in the class of general strategies is equivalent to the supremum in the class of deterministic stationary strategies. A counterexample shows that these conditions can not be easily relaxed.

Constrained Discounted Markov Decision Processes and Hamiltonian Cycles

This paper establishes new links between stochastic and discrete optimization. We consider the following three problems for discrete time Markov Decision Processes with finite states and action sets: (i) find an optimal deterministic policy for a discounted problem with constraints, (ii) find an optimal stationary policy for a weighted discounted problem with constraints, (iii) find an optimal deterministic policy for a weighted discounted problem with constraints. We formulate mathematical programs for problems (i)–(iii) and show that the Hamiltonian Cycle Problem is a special case of each of these problems. Therefore problems (i)–(iii) are NP-hard. We also provide new mathematical programming formulations for the Hamiltonian Cycle and Traveling Salesman Problems.

Learning automata algorithms for pattern classification

This paper considers the problem of learning optimal discriminant functions for pattern classification. The criterion of optimality is minimising the probability of misclassification. No knowledge of the statistics of the pattern classes is assumed and the given classified sample may be noisy. We present a comprehensive review of algorithms based on the model of cooperating systems of learning automata for this problem. Both finite action set automata and continuous action set automata models are considered. All algorithms presented have rigorous convergence proofs. We also present algorithms that converge to global optimum. Simulation results are presented to illustrate the effectiveness of these techniques based on learning automata.