Discounted Optimality Criterion Research Articles

Empirical approximation of Nash equilibria in finite Markov games with discounted payoffs

AbstractThis paper deals with finite nonzero‐sum Markov games under a discounted optimality criterion and infinite horizon. The state process evolves according to a stochastic difference equation and depends on players' actions as well as a random disturbance whose distribution is unknown to the players. The actions, the states, and the values of the disturbance are observed by the players, then they use the empirical distribution of the disturbances to estimate the true distribution and make choices based on the available information. In this context, we propose an almost surely convergent procedure—possibly after passing to a subsequence—to approximate Nash equilibria of the Markov game with the true distribution of the random disturbance.

Partially observable queueing systems with controlled service rates under a discounted optimality criterion

Partially observable queueing systems with controlled service rates under a discounted optimality criterion

A Mean Field Approach for Discounted Zero-Sum Games in a Class of Systems of Interacting Objects

The paper deals with systems composed of a large number of N interacting objects (e.g., agents, particles) controlled by two players defining a stochastic zero-sum game. The objects can be classified according to a finite set of classes or categories over which they move randomly. Because N is too large, the game problem is studied following a mean field approach. That is, a zero-sum game model $$\mathcal {GM}_{N}$$ , where the states are the proportions of objects in each class, is introduced. Then, letting $$N\rightarrow \infty $$ (the mean field limit) we obtain a new game model $$\mathcal {GM}$$ , independent on N, which is easier to analyze than $$\mathcal {GM}_{N}$$ . Considering a discounted optimality criterion, our objective is to prove that an optimal pair of strategies in $$\mathcal {GM}$$ is an approximate optimal pair as $$N\rightarrow \infty $$ in the original game model $$\mathcal {GM}_{N}$$ .

Markov games with unknown random state‐actions‐dependent discount factors: Empirical estimation

AbstractThe work deals with a class of discrete‐time zero‐sum Markov games under a discounted optimality criterion with random state‐action‐dependent discount factors of the form , where xn,an,bn, and ξn+1 are the state, the actions of players, and a random disturbance at time n, respectively, taking values in Borel spaces. The one‐stage payoff is assumed to be possibly unbounded. In addition, the process {ξn} is formed by observable, independent, and identically distributed random variables with common distribution θ, which is unknown to players. By using the empirical distribution to estimate θ, we introduce a procedure to approximate the value V∗ of the game; such a procedure yields construction schemes of stationary optimal strategies and asymptotically optimal Markov strategies.

Zero-Sum Markov Games with Random State-Actions-Dependent Discount Factors: Existence of Optimal Strategies

This work deals with a class of discrete-time zero-sum Markov games under a discounted optimality criterion with random state-actions-dependent discount factors of the form $$\tilde{\alpha }(x_{n},a_{n},b_{n},\xi _{n+1})$$ , where $$x_{n}, a_{n}, b_{n}$$ , and $$\xi _{n+1}$$ are the state, the actions of players, and a random disturbance at time n, respectively, taking values in Borel spaces. Assuming possibly unbounded payoff, we prove the existence of a value of the game as well as a stationary pair of optimal strategies.

Control systems of interacting objects modeled as a game against nature under a mean field approach

This paper deals with discrete-time stochastic systems composed of a large number of N interacting objects (a.k.a. agents or particles). There is a central controller whose decisions, at each stage, affect the system behavior. Each object evolves randomly among a finite set of classes, according to a transition law which depends on an unknown parameter. Such a parameter is possibly non observable and may change from stage to stage. Due to the lack of information and to the large number of agents, the control problem under study is rewritten as a game against nature according to the mean field theory; that is, we introduce a game model associated to the proportions of the objects in each class, whereas the values of the unknown parameter are now considered as actions selected by an opponent to the controller (the nature). Then, letting \begin{document} $N \to \infty $ \end{document} (the mean field limit) and considering a discounted optimality criterion, the objective for the controller is to minimize the maximum cost, where the maximum is taken over all possible strategies of the nature.

Discrete-Time Control for Systems of Interacting Objects with Unknown Random Disturbance Distributions: A Mean Field Approach

We are concerned with stochastic control systems composed of a large number of N interacting objects sharing a common environment. The evolution of each object is determined by a stochastic difference equation where the random disturbance density $$\rho $$ź is unknown for the controller. We present the Markov control model (N-model) associated to the proportions of objects in each state, which is analyzed according to the mean field theory. Thus, combining convergence results as $$N\rightarrow \infty $$Nźź (the mean field limit) with a suitable statistical estimation method for $$\rho $$ź, we construct the so-named eventually asymptotically optimal policies for the N-model under a discounted optimality criterion. A consumption-investment problem is analyzed to illustrate our results.

Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach

In this paper, we investigate an optimization problem for continuous-time Markov decision processes with both impulsive and continuous controls. We consider the so-called constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.

Constrained Markov control processes with randomized discounted cost criteria: infinite linear programming approach

SUMMARYIn this paper, we study constrained Markov control processes on Borel spaces with possibly unbounded one‐stage cost, under a discounted optimality criterion with random discount factor and restrictions of the same kind. We prove that the corresponding optimal control problem is equivalent to an infinite‐dimensional linear programming problem. In addition, considering the dual program, we show that there is no duality gap, and moreover, the strong duality condition holds. Hence, both programs are solvable, and their optimal values coincide. Copyright © 2013 John Wiley & Sons, Ltd.