Finite Action Spaces Research Articles

Almost Stationary ∈-Equilibria in Zero-Sum Stochastic Games

We show the existence of almost stationary ∈-equilibria, for all ∈ > 0, in zero-sum stochastic games with finite state and action spaces. These are ∈-equilibria with the property that, if neither player deviates, then stationary strategies are played forever with probability almost 1. The proof is based on the construction of specific stationary strategy pairs, with corresponding rewards equal to the value, which can be supplemented with history-dependent δ-optimal strategies, with small δ > 0, in order to obtain almost stationary ∈-equilibria.

Nonstationary denumerable state Markov decision processes – with average variance criterion

. In this paper, we consider the nonstationary Markov decision processes (MDP, for short) with average variance criterion on a countable state space, finite action spaces and bounded one-step rewards. From the optimality equations which are provided in this paper, we translate the average variance criterion into a new average expected cost criterion. Then we prove that there exists a Markov policy, which is optimal in an original average expected reward criterion, that minimizies the average variance in the class of optimal policies for the original average expected reward criterion.

MARKOV STRATEGIES ARE BETTER THAN STATIONARY STRATEGIES

We examine the use of stationary and Markov strategies in zero-sum stochastic games with finite state and action spaces. It is natural to evaluate a strategy for the maximising player, player 1, by the highest reward guaranteed to him against any strategy of the opponent. The highest rewards guaranteed by stationary strategies or by Markov strategies are called the stationary utility or the Markov utility, respectively. Since all stationary strategies are Markov strategies, the Markov utility is always larger or equal to the stationary utility. However, in all presently known subclasses of stochastic games, these utilities turn out to be equal. In this paper, we provide a colourful example in which the Markov utility is strictly larger than the stationary utility and we present several conditions under which the utilities are equal. We also show that each stochastic game has at least one initial state for which the two utilities are equal. Several examples clarify these issues.

A simple suboptimal algorithm for system maintenanceunder partial observability

We suggest a heuristic solution procedure for Partially Observable Markov DecisionProcesses with finite action space and finite state space with infinite horizon. The algorithmis a fast, very simple general heuristic; it is applicable for multiple states (not necessarilyordered) multiple actions and various distribution functions. The quality of the algorithm ischecked in this paper against existing analytical and empirical results for two specific modelsof machine replacement. One model refers to the case of two‐action and two‐system stateswith uniform observations (Grosfeld‐Nir [4]), and the other model refers to a case of manyordered states with binomial observations (Sinuany‐Stern et al. [11]). The paper also presentsthe model realization for various probability distribution functions applied to maintenanceand quality control.

Average-discounted equilibria in stochastic games

In stochastic games with finite state and action spaces, we examine existence of equilibria where player 1 uses the limiting average reward and player 2 a discounted reward for the evaluations of the respective payoff sequences. By the nature of these rewards, the far future determines player 1's reward, while player 2 is rather interested in the near future. This gives rise to a natural cooperation between the players along the course of the play. First we show the existence of stationary ε-equilibria, for all ε>0, in these games. However, besides these stationary ε-equilibria, there also exist ε-equilibria, in terms of only slightly more complex ultimately stationary strategies, which are rather in the spirit of these games because, after a large stage when the discounted game is not interesting any longer, the players cooperate to guarantee the highest feasible reward to player 1. Moreover, we analyze an interesting example demonstrating that 0-equilibria do not necessarily exist in these games, not even in terms of history dependent strategies. Finally, we examine special classes of stochastic games with specific conditions on the transition and payoff structures. Several examples are given to clarify all these issues.

On asymptotic optimization of a class of nonlinear stochastic hybrid systems

We consider the problem of control for continuous time stochastic hybrid systems in finite time horizon. The systems considered are nonlinear: the state evolution is a nonlinear function of both the control and the state. The control parameters change at discrete times according to an underlying controlled Markov chain which has finite state and action spaces. The objective is to design a controller which would minimize an expected nonlinear cost of the state trajectory. We show using an averaging procedure, that the above minimization problem can be approximated by the solution of some deterministic optimal control problem. This paper generalizes our previous results obtained for systems whose state evolution is linear in the control.

Module-Based Reinforcement Learning: Experiments with a Real Robot

The behavior of reinforcement learning (RL) algorithms is best understood in completely observable, discrete-time controlled Markov chains with finite state and action spaces. In contrast, robot-learning domains are inherently continuous both in time and space, and moreover are partially observable. Here we suggest a systematic approach to solve such problems in which the available qualitative and quantitative knowledge is used to reduce the complexity of learning task. The steps of the design process are to:i) decompose the task into subtasks using the qualitative knowledge at hand; ii) design local controllers to solve the subtasks using the available quantitative knowledge and iii) learn a coordination of these controllers by means of reinforcement learning. It is argued that the approach enables fast, semi-automatic, but still high quality robot-control as no fine-tuning of the local controllers is needed. The approach was verified on a non-trivial real-life robot task. Several RL algorithms were compared by ANOVA and it was found that the model-based approach worked significantly better than the model-free approach. The learnt switching strategy performed comparably to a handcrafted version. Moreover, the learnt strategy seemed to exploit certain properties of the environment which were not foreseen in advance, thus supporting the view that adaptive algorithms are advantageous to non-adaptive ones in complex environments.

Asymptotic Optimization of a Nonlinear Hybrid System Governed by a Markov Decision Process

We consider in this paper a continuous time stochastic hybrid control system with finite time horizon. The objective is to minimize a nonlinear function of the state trajectory. The state evolves according to a nonlinear dynamics. The parameters of the dynamics of the system may change at discrete times $l\epsilon$, $l=0,1,...$, according to a controlled Markov chain which has finite state and action spaces. Under the assumption that $\epsilon$ is a small parameter, we justify an averaging procedure allowing us to establish that our problem can be approximated by the solution of some deterministic optimal control problem.

Risk-sensitive optimal control of hidden Markov models: structural results

The authors consider a risk-sensitive optimal control problem for (finite state and action spaces) hidden Markov models (HMM). They present results of an investigation on the nature and structure of risk-sensitive controllers for HMM. Several general structural results are presented, as well as a particular case study of a popular benchmark problem. For the latter, they obtain structural results for the optimal risk-sensitive controller and compare it to that of the risk-neutral controller. Furthermore, they show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation as the risk factor goes to zero. They also study the infinite and general risk aversion cases.

Limited Horizon Forecast in Repeated Alternate Games

In two-player infinite-horizon alternating-move games, a limited forecast (n1, n2)-equilibrium is such that (1) player i chooses actions according to his ni-length forecasts so as to maximise the average payoff over the forthcoming ni periods, and (2) players′ equilibrium forecasts are correct. With finite action spaces, (n1, n2)-solutions always exist and are cyclical, and the memory capacity of the players has no influence on the set of solutions. A solution is hyperstable if it is an (n1, n2)-solution for all n1, n2 sufficiently large. Hyperstable solutions are shown to exist and are characterized for generic repeated alternate-move 2×2 games. Journal of Economic Literature Classification Numbers: C72, D81.

Comparing two algorithms for automatic planning by robots in stochastic environments

SummaryPlanning a sequence of robot actions is especially difficult when the outcome of actions is uncertain, as is inevitable when interacting with the physical environment. In this paper we consider the case of finite state and action spaces where actions can be modeled as Markov transitions. Finding a plan that achieves a desired state with maximum probability is known to be an NP-Complete problem. We consider two algorithms: an exponential-time algorithm that maximizes probability, and a polynomial-time algorithm that maximizes a lower bound on the probability. As these algorithms trade off plan time for plan quality, we compare their performance on a mechanical system for orienting parts. Our results lead us to identify two properties of stochastic actions that can be used to choose between these planning algorithms for other applications.

Linear programming formulation of MDPs in countable state space: The multichain case

We present an Linear Programming formulation of MDPs with countable state and action spaces and no unichain assumption. This is an extension of the Hordijk and Kallenberg (1979) formulation in finite state and action spaces. We provide sufficient conditions for both existence of optimal solutions to the primal LP program and absence of duality gap. Then, existence of a (possibly randomized) average optimal policy is also guaranteed. Existence of a stationary average optimal deterministic policy is also investigated.

Efficient and Nearly-Efficient Partnerships

This paper shows in two ways that the degree to which free-riding diminishes the performance of deterministic partnerships may be less than has been generally thought. First, a necessary and sufficient condition is provided for a partnership to sustain full efficiency. It implies that many non-trivial partnerships sustain efficiency, such as generic ones with finite action spaces, and neoclassical ones with Leontief technologies. Second, approximate efficiency is shown to be achievable in a large class of partnerships, including ones with smooth and monotonic production and disutility functions. Approximate efficiency is achieved by mixed-strategy equilibria: one partner takes, with small probability, an inefficient action. The degree to which efficiency is approximated is restricted only by the amount of liability the partners can bear. Nonetheless, their equilibrium payments are not arbitrarily large.

On Equivariance and the Compound Decision Problem

This paper obtains some extensions of Gilliland and Hannan's results on equivariance and the compound decision problem. Consider a compound decision problem with restricted component risk and component distributions in a norm compact set of mutually absolutely continuous probability measures. Then the method of proof of a theorem of Gilliland and Hannan translates the results of Mashayekhi on symmetrization of product measures into uniform convergence to zero of the excess of the simple envelop over the equivariant envelope. Our envelope results strengthen, among other things, the results of Datta who obtained admissible asymptotically optimal solutions to the compound estimation problem for a large subclass of the real one parameter exponential family under squared error loss. Sufficient conditions for asymptotic optimality of "delete bootstrap" rules are given and, for squared error loss estimation of continuous functions and for finite action space problems with continuous loss functions, the problem of treating the asymptotic excess compound risk of Bayes compound rules is reduced to the question of $L_1$-consistency of certain mixtures. Examples of estimates satisfying the above consistency condition are provided.

Control of a hybrid stochastic system

We consider in this paper a continuous-time stochastic hybrid control system with a finite time horizon. The objective is to minimize a linear function of the expected state trajectory. The state evolves according to a linear dynamics. However, the parameters of the state evolution equation may change at discrete times according to a controlled Markov chain which has finite state and action spaces. We use a procedure similar in form to the maximum principle; this determines a control strategy which is asymptotically optimal as the number of transitions during the finite time horizon grows to infinity.

Variability Sensitive Markov Decision Processes

Considered are time-average Markov Decision Processes (MDPs) with finite state and action spaces. Two definitions of variability are introduced, namely, the expected time-average variability and time-average expected variability. The two criteria are in general different, although they can both be employed to penalize for variance in the stream of rewards. For communicating MDPs, we construct a (randomized) stationary policy that is ε-optimal for both criteria; the policy is optimal and pure for a specific variability function. For general multichain MDPs, a state space decomposition leads to a similar result for the expected time-average variability. We also consider the problem of the decision maker choosing the initial state along with the policy.

An operator solution of stochastic games

A class of zero-sum, two-person stochastic games is shown to have a value which can be calculated by transfinite iteration of an operator. The games considered have a countable state space, finite action spaces for each player, and a payoff sufficiently general to include classical stochastic games as well as Blackwell’s infiniteG δ games of imperfect information.

Two-person zero-sum stochastic games

Two-person zero-sum stochastic games with finite state and action spaces are considered. The expected average payoff criterion is introduced. In the special case of single controller games it is shown that the optimal stationary policies and the value of the game can be obtained from the optimal solutions to a pair of dual programs. For multichain structures, a decomposition algorithm is given which produces such optimal stationary policies for both players. In the case of both players controlling the transitions, a generalized game is obtained, the solution of which gives the optimal policies.

Perfect equilibria in stochastic games

We examine stochastic games with finite state and action spaces. For the β-discounted case, as well as for the irreducible limiting average case, we show the existence of trembling-hand perfect equilibria and give characterizations of those equilibria. In the final section, we give an example which illustrates that the existence of stationary limiting average equilibria in a nonirreducible stochastic game does not imply the existence of a perfect limiting average equilibrium.

Markov decision processes with multiple costs

This note considers finite state and action spaces controlled Markov chains with multiple costs. The set of Pareto-optimal solutions is characterized. A particular utility function is considered and a parallel algorithm is developed to find an optimal solution with respect to that utility function.