Finite Action Sets Research Articles

Existence and Discovery of Average Optimal Solutions in Deterministic Infinite Horizon Optimization

We consider the problem of making a sequence of decisions, each chosen from a finite action set over an infinite horizon, so as to minimize its associated average cost. Both the feasibility and cost of a decision are allowed to depend upon all of the decisions made prior to that decision; moreover, time-varying costs and constraints are allowed. A feasible solution is said to be efficient if it reaches each of the states through which it passes at minimum cost. We show that efficient solutions exist and that, under a state reachability condition, efficient solutions are also average optimal. Exploiting the characterization of efficiency via a solution's short-run as opposed to long-run behavior, a forward algorithm is constructed which recursively discovers the first, second, and subsequent decisions of an efficient, and hence average optimal, infinite horizon solution.

Single Crossing Properties And The Existence Of Pure Strategy Equilibria In Games Of Incomplete Information

This paper derives sufficient conditions for a class of games of incomplete information, such as first price auctions, to have pure strategy Nash equilibria (PSNE). The paper treats games between two or more heterogeneous agents, each with private information about his own type (for example, a bidder's value for an object of a firm's marginal cost of production), and the types are drawn from an atomless joint probability distribution which potentially allows for correlation between types. Agents' utility may depend directly on the realizations of other agents' types, as in Milgrom and Weber's (1982) formulation of the auction. The restriction we consider is that each player's expected payoffs satisfy the following single crossing condition: whenever each opponent uses a nondecreasing strategy (that is, an opponent who has a higher type chooses a higher action), then a player's best response strategy is also nondecreasing in her type. The paper has two main results. The first result shows that, when players are restricted to choose among a finite set of actions (for example, bidding or pricing where the smallest unit is a penny), games where players' objective functions satisfy this single crossing condition will have PSNE. The second result demonstrates that when players' utility functions are continuous, as well as in mineral rights auction games and other games where winning creates a discontinuity in payoffs, the existence result can be extended to the case where players choose from a continuum of actions. The paper then applies the theory to several classes of games, providing conditions on utility functions and joint distributions over types under which each class of games satisfies the single crossing condition. In particular, the single crossing condition is shown to hold in all first-price, private value auctions with potentially heterogeneous, risk-averse bidders, with either independent or affiliated values, and with reserve prices which may differ across bidders; mineral rights auctions with two heterogeneous bidders and affiliated values; a class of pricing games with incomplete information about costs; a class of all-pay auction games; and a class of noisy signaling games. Finally, the formulation of the problem introduced in this paper suggests a straightforward algorithm for numerically computing equilibrium bidding strategies in games such as first price auctions, and we present numerical analyses of several auctions under alternative assumptions about the joint distribution of types.

Colearning in Differential Games

Game playing has been a popular problem area for research in artificial intelligence and machine learning for many years. In almost every study of game playing and machine learning, the focus has been on games with a finite set of states and a finite set of actions. Further, most of this research has focused on a single player or team learning how to play against another player or team that is applying a fixed strategy for playing the game. In this paper, we explore multiagent learning in the context of game playing and develop algorithms for “co-learning” in which all players attempt to learn their optimal strategies simultaneously. Specifically, we address two approaches to colearning, demonstrating strong performance by a memory-based reinforcement learner and comparable but faster performance with a tree-based reinforcement learner.

Blackwell Optimality in Borelian Continuous-in-Action Markov Decision Processes

We prove the existence of stationary Blackwell optimal policies in Markov decision processes with a Borel state space, compact action sets, and continuous-in-action and bounded transition densities and rewards, satisfying a simultaneous Doeblin-type condition. The proof is based on a compactification of the randomized stationary policy space in a weak-strong topology, on the continuity of Laurent coefficients of the discounted rewards in this topology, and on a lexicographical policy improvement. Until now similar results were obtained for the models with a denumerable state space or with a Borel state space and finite action sets.

Map learning with uninterpreted sensors and effectors

This paper presents a set of methods by which a learning agent can learn a sequence of increasingly abstract and powerful interfaces to control a robot whose sensorimotor apparatus and environment are initially unknown. The result of the learning is a rich hierarchical model of the robot's world (its sensorimotor apparatus and environment). The learning methods rely on generic properties of the robot's world such as almost-everywhere smooth effects of motor control signals on sensory features. At the lowest level of the hierarchy, the learning agent analyzes the effects of its motor control signals in order to define a new set of control signals, one for each of the robot's degrees of freedom. It uses a generate-and-test approach to define sensory features that capture important aspects of the environment. It uses linear regression to learn models that characterize context-dependent effects of the control signals on the learned features. It uses these models to define high-level control laws for finding and following paths defined using constraints on the learned features. The agent abstracts these control laws, which interact with the continuous environment, to a finite set of actions that implement discrete state transitions. At this point, the agent has abstracted the robot's continuous world to a finite-state world and can use existing methods to learn its structure. The learning agent's methods are evaluated on several simulated robots with different sensorimotor systems and environments.

A note on asymptotics of discounted value function and strong 0-discount optimality

We consider a Markov decision process with a Borel state space, bounded rewards, and a bounded transition density satisfying a simultaneous Doeblin-Doob condition. An asymptotics for the discounted value function related to the existence of stationary strong 0-discount optimal policies is extended from the case of finite action sets to the case of compact action sets and continuous in action rewards and transition densities.

Maximising a Function Over a Finite Set of Actions—Technical Note

In this technical note we examine a method of extending a problem over a finite set of actions, to a problem of maximising a function over a bi-product set. The solution set is characterised in terms of fixed point solutions and partial solutions, and two improvement algorithms are given.

A Logical Study of Distributed Transition Systems

We extend labelled transition systems to distributed transition systems by labelling the transition relation with a finite set of actions, representing the fact that the actions occur as a concurrent step . We design an addition-based temporal logic in which one can explicity talk about steps. The logic is studied to establish a variety of positive and negative results in terms of axiomatizability and decidability. Our positive results show that the step notion is amenable to logical treatment via standard techniques. They also help us to obtain a logical characterization of two well known models for distributed systems: labelled elementary net systems and labelled prime event structures. Our negative results show that demanding deterministic structures when dealing with a "non-interleaved" notion of transitions is, from a logical standpoint, very expressive. They also show that another well known model of distributed systems called asynchronous transition systems exhibits a surprising amount of expressive power in a natural logical setting.

Strong 0-discount optimal policies in a Markov decision process with a Borel state space

We consider a Markov decision process with a Borel state space, a countable action space, finite action sets, bounded rewards and a bounded transition density satisfying a simultaneous Doeblin condition. The existence of stationary strong 0-discount optimal polices is proved.

Constrained Semi-Markov decision processes with average rewards

This paper deals with constrained average reward Semi-Markov Decision Processes (SMDPs) with finite state and action sets. We consider two average reward criteria. The first criterion is time-average rewards, which equal the lower limits of the expected average rewards per unit time, as the horizon tends to infinity. The second criterion is ratio-average rewards, which equal the lower limits of the ratios of the expected total rewards during the firstn steps to the expected total duration of thesen steps asn → ∞. For both criteria, we prove the existence of optimal mixed stationary policies for constrained problems when the constraints are of the same nature as the objective functions. For unichain problems, we show the existence of randomized stationary policies which are optimal for both criteria. However, optimal mixed stationary policies may be different for each of these critria even for unichain problems. We provide linear programming algorithms for the computation of optimal policies.

Borel Stochastic Games with Lim Sup Payoff

We consider two-person zero-sum stochastic games with limit superior payoff function and Borel measurable state and action spaces. The games are shown to have a value and the value function is calculated by transfinite iteration of an operator and proved to be upper analytic. The paper extends results of our earlier article [17] in which the same class of games was considered for countable state spaces and finite action sets.

The Average Cost Optimality Equation and Critical Number Policies

We consider a Markov decision chain with countable state space, finite action sets, and nonnegative costs. Conditions for the average cost optimality inequality to be an equality are derived. This extends work of Cavazos-Cadena [8]. It is shown that an optimal stationary policy must satisfy the optimality equation at all positive recurrent states. Structural results on the chain induced by an optimal stationary policy are derived. The results are employed in two examples to prove that any optimal stationary policy must be of critical number form.

Optimal Stationary Policies in General State Space Markov Decision Chains with Finite Action Sets

The result of Sennott [9] on the existence of optimal stationary policies in countable state Markov decision chains with finite action sets is generalized to arbitrary state space Markov decision chains. The assumption of finite action sets occurring in a global countable action space allows a particularly simple theoretical structure for the general state space Markov decision chain. Two examples illustrate the results. Example 1 is a system of parallel queues with stochastic work requirements, a movable server with controllable service rate, and a reject option. Example 2 is a system of parallel queues with stochastic controllable inputs, a movable server with fixed service rates, and a reject option.

Comparing recent assumptions for the existence of average optimal stationary policies

We consider discrete time average cost Markov decision processes with countable state space and finite action sets. Conditions recently proposed by Borkar, Cavazos-Cadena, Weber and Stidham, and Sennott for the existence of an expected average cost optimal stationary policy are compared. The conclusion is that the Sennott conditions are the weakest. We also give an example for which the Sennott axioms hold but the others fail.

On the average cost optimality equation and the structure of optimal policies for partially observable Markov decision processes

We consider partially observable Markov decision processes with finite or countably infinite (core) state and observation spaces and finite action set. Following a standard approach, an equivalent completely observed problem is formulated, with the same finite action set but with anuncountable state space, namely the space of probability distributions on the original core state space. By developing a suitable theoretical framework, it is shown that some characteristics induced in the original problem due to the countability of the spaces involved are reflected onto the equivalent problem. Sufficient conditions are then derived for solutions to the average cost optimality equation to exist. We illustrate these results in the context of machine replacement problems. Structural properties for average cost optimal policies are obtained for a two state replacement problem; these are similar to results available for discount optimal policies. The set of assumptions used compares favorably to others currently available.

Value iteration in countable state average cost Markov decision processes with unbounded costs

We deal with countable state Markov decision processes with finite action sets and (possibly) unbounded costs. Assuming the existence of an expected average cost optimal stationary policyf, with expected average costg, when canf andg be found using undiscounted value iteration? We give assumptions guaranteeing the convergence of a quantity related tong−Νn(i), whereΝn(i) is the minimum expectedn-stage cost when the process starts in statei. The theory is applied to a queueing system with variable service rates and to a queueing system with variable arrival parameter.

Optimal strategies for management of Verticillium wilt

Multi-season management of Verticillium wilt, a serious soil borne fungal pathogen in California cotton, is assessed using dynamic programming. The model assumes that maximization of long run present value of profits may be achieved by choosing among a finite set of pre-season pathogen control actions. The time evolution of pathogen density and virulence is the basis in the dynamic model for selecting among treatment alternatives. The treatments differently affect immediate as well as long run profits via their effects on future density and virulence of the pathogen. The model considers both deterministic and stochastic economic environments. The deterministic short run model suggests that no actions are required, while long term policy suggests the use of crop rotation rather than chemical treatments. The effects of uncertainty are examined via the use of hypothetical uncertainty parameters.

Are limits of α-discounted optimal policies Blackwell optimal? A counterexample

In Markov Decision Chains (MDC's) with a finite state space and finite action sets it is a well-known result, that the limits of α-discounted optimal policies, for α tending to 1, are Blackwell optimal. It was conjectured in a recent paper by Cavazos-Cadena and Lasserre, that this property of limiting policies also holds for unichain MDC's. We disprove this conjecture by constructing a non-Blackwell limiting policy in a unichain MDC with finitely many states and compact action sets.

Average Cost Semi-Markov Decision Processes and the Control of Queueing Systems

Semi-Markov decision processes underlie the control of many queueing systems. In this paper, we deal with infinite state semi-Markov decision processes with nonnegative, unbounded costs and finite action sets. Axioms for the existence of an expected average cost optimal stationary policy are presented. These conditions generalize the work in Sennott [22] for Markov decision processes. Verifiable conditions for the axioms to hold are obtained. The theory is applied to control of the M/G/l queue with variable service parameter, with on-off server, and with batch processing, and to control of the G/M/m queue with variable arrival parameter and customer rejection. It is applied to a timesharing network of queues with a single server and finally to optimal routing of Poisson arrivals to parallel exponential servers. The final section extends the existence result to compact action spaces.

On the Evaluation of Government Programs to Reduce Environmental Risk

This paper attempts to identify the correct benefit measure for a government project which might reduce the subjective probability of an environmental risk to health. The evaluation of government projects whose costs or benefits are uncertain occupies a considerable literature including Arrow and Lind, Fisher, and Graham. The value of reducing the probability of an adverse state (outcome) has been examined by Cook and Graham and more recently by Gallagher and Smith. In the latter two papers it was assumed that the individual at risk had access to perfect insurance or contingent claims markets. In such a perfect world the value of reducing the probability of an adverse state could be bounded by measures which Cook and Graham referred to as ransom and compensation and which Gallagher and Smith show to correspond to the more traditional Hicksian notions of compensating and equivalent variation. If the assumption of perfect contingent claims markets does not hold, it is not possible to set upper and lower bounds on the value of reducing the probability of an adverse state. While general forms of insurance are available to individuals at risk from an environmental hazard, specific policies do not allow for the continuous adjustment (reallocation of wealth) assumed to occur when the individual has access to perfect markets. In the next section a more plausible model is constructed. It assumes that the individual has a finite set of self-protective actions which he or she may take in response to a risk to future health. In addition, the government may undertake a project to reduce the risk to health. It is assumed that both the selfprotective action and the government project have the ability to influence the subjective (personal) probability of the future states, continued good health or ill-health. There are three major conclusions which emerge from this model. (a) The value of the government project is an option price (Graham) which may be zero or positive. If positive, the benefits from the project may take the form of cost-savings, risk (probability) reduction, or both. (b) The value of the government project is dependent on the set of self-protective actions and may change as new actions become available or the effectiveness of existing actions is reassessed by the individual. (c) The method of contingent valuation, in theory, would appear capable of eliciting the proper option price estimates for the project's value to the individual. It is even possible to distinguish between the value attributable to cost-savings and risk reduction. In the next section I present the model and demonstrate the above conclusions. In the final section an assessment is made of the in-