Finite Action Spaces Research Articles

Regularity of the minmax value and equilibria in multiplayer Blackwell games

AbstractA real-valued function φ that is defined over all Borel sets of a topological space is regular if for every Borel set W, φ(W) is the supremum of φ(C), over all closed sets C that are contained in W, and the infimum of φ(O), over all open sets O that contain W.We study Blackwell games with finitely many players. We show that when each player has a countable set of actions and the objective of a certain player is represented by a Borel winning set, that player’s minmax value is regular.We then use the regularity of the minmax value to establish the existence of ε-equilibria in two distinct classes of Blackwell games. One is the class of n-player Blackwell games where each player has a finite action space and an analytic winning set, and the sum of the minmax values over the players exceeds n − 1. The other class is that of Blackwell games with bounded upper semi-analytic payoff functions, history-independent finite action spaces, and history-independent minmax values. For the latter class, we obtain a characterization of the set of equilibrium payoffs.

Reward function design method to achieve system-level objectives in ambulance diversion problem

This study addresses a reward function design problem in multi-agent systems, using ambulance diversion (AD) problem in emergency medical services (EMS) as an application. While Emergency Departments (EDs) may declare AD to alleviate overcrowding, excessive use of this strategy by individual EDs can have negative consequences, such as limited access and delayed care for emergency patients. To eliminate negative consequences caused by individual decisions, we propose a reward function design method to align individual agents’ decisions with socially optimal strategies. This method ensures that an equilibrium status is achieved under socially optimal operation by removing incentives for deviations from socially optimal behavior while maximizing system welfare without extra investment. The proposed method offers an exact solution for designing reward function of a stochastic game model with finite state and action spaces. We conduct numerical experiments for a 2-ED system and demonstrate that the designed reward function properly guides the EDs toward a socially optimal AD strategy.

On existence of Berk-Nash equilibria in misspecified Markov decision processes with infinite spaces

Model misspecification is a critical issue in many areas of economics. In the context of misspecified Markov Decision Processes, Esponda and Pouzo (2021) defined the notion of Berk-Nash equilibrium and established its existence with finite state and action spaces. However, many substantive applications (including two of the three motivating examples presented by Esponda and Pouzo) involve continuous state or action spaces, and are thus not covered by the Esponda-Pouzo existence theorem. We extend the existence of Berk-Nash equilibrium to compact action spaces and sigma-compact state spaces, with possibly unbounded utility functions. A complication arises because Berk-Nash equilibrium depends critically on Radon-Nikodym derivatives, which are bounded in the finite case but typically unbounded in misspecified continuous models. The proofs rely on nonstandard analysis, and draw on novel argumentation traceable to work of the second author on nonstandard representations of Markov processes.

Well-Posedness for Mean Field Games with Finite State and Action Space

The main purpose of this paper is to investigate the well-posedness for mean field game with finite state and action space (FSASMFG) by using nonlinear analysis methods. First, we set up the bounded rationality model of the FSASMFG in the topological space. Next, we study some sufficient conditions of generalized strong well-posedness (GS-wp) and strong well-posedness (S-wp) for a class of FSASMFGs. Finally, we give some characterizations of GS-wp and S-wp for FSASMFGs with the help of set-valued analysis methods. These new results presented in the paper develop and improve the corresponding conclusions in the recent literature.

Sequential Bayesian persuasion

I study a Bayesian persuasion model in which multiple senders sequentially persuade one receiver, after observing signal structures of prior senders and their realizations. I develop a geometric method, recursive concavification, to characterize the Perfect Bayesian Equilibrium paths. I prove the existence of the silent equilibrium, where at most one sender provides nontrivial information. I also show that when there are only two senders and the receiver has a finite action space, it is generically without loss to focus on silent equilibrium. Finally, I show that if there are two senders who have zero-sum payoffs, the truth-telling signal structure is always supported in equilibrium.

Superfair Stochastic Games

A two-person zero-sum stochastic game with a nonnegative stage reward function is superfair if the value of the one-shot game at each state is at least as large as the reward function at the given state. The payoff in the game is the limit superior of the expected stage rewards taken over the directed set of all finite stop rules. If the game has countable state and action spaces and if at least one of the players has a finite action space at each state, then the game has a value. The value of the stochastic game is obtained by a transfinite algorithm on the countable ordinals.

A myopic adjustment process for mean field games with finite state and action space

In this paper, we introduce a natural learning rule for mean field games with finite state and action space, the so-called myopic adjustment process. The main motivation for these considerations is the complexity of the computations necessary to determine dynamic mean field equilibria, which makes it seem questionable whether agents are indeed able to play these equilibria. We prove that the myopic adjustment process converges locally towards strict stationary equilibria under rather broad conditions. Moreover, we also obtain a global convergence result under stronger, yet intuitive conditions.

Potential Game-Based Decision-Making for Autonomous Driving

Decision-making for autonomous driving is challenging, considering the complex interactions among multiple traffic agents (including autonomous vehicles (AVs), human-driven vehicles, and pedestrians) and the computational load needed to evaluate these interactions. This paper develops two general potential game-based frameworks, namely, finite and continuous potential games, for decision-making in autonomous driving. The two frameworks account for the AVs’ two types of action spaces, i.e., finite and continuous action spaces, respectively. The developed frameworks provide theoretical guarantees for the existence of pure-strategy Nash equilibria and for the convergence of the Nash equilibrium (NE) seeking algorithms. The scalability challenge is also addressed. In addition, we provide cost function shaping approaches such that the agents’ cost functions not only reflect common driving objectives but also yield potential games. The performance of the developed algorithms is demonstrated in diverse traffic scenarios, including intersection-crossing and lane-changing scenarios. Statistical comparative studies, including 1) finite potential game vs. continuous potential game, 2) best response dynamics vs. potential function optimization, and 3) potential game vs. reinforcement learning (RL) vs. control barrier function (CBF), are conducted to compare the robustness against various surrounding vehicles’ strategies and to compare the computational efficiency. It is shown that the developed potential game frameworks have better robustness than RL and than CBF if the surrounding vehicles are not safety-conscious, and are computationally feasible for real-time implementation.

Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an epsilon-optimal solution with at most an order of epsilon to -2.5 sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

Zero-sum semi-Markov games with state-action-dependent discount factors

Semi-Markov model is one of the most general models for stochastic dynamic systems. This paper deals with a two-person zero-sum game for semi-Markov processes. We focus on the expected discounted criterion with state-action-dependent discount factors. The state and action spaces are both Polish spaces, and the reward rate function is ω-bounded. We first construct a fairly general model of semi-Markov games under a given semi-Markov kernel and a pair of strategies. Next, based on the standard regularity condition and the continuity-compactness condition for semi-Markov games, we derive a “drift condition” on the semi-Markov kernel and suppose that the discount factors have a positive lower bound, under which the existence of the value function and an optimal pair of stationary strategies of our semi-Markov game are proved by using a general Shapley equation. Moreover, in the scenario of finite state and action spaces, a value iteration-type algorithm for approximating the value function and an optimal pair of stationary strategies is developed. The convergence and the error bound of the algorithm are also proved. Finally, we conduct numerical examples to demonstrate the main results.

Reversible Markov decision processes and the Gaussian free field

A Markov decision problem is called reversible if the stationary controlled Markov chain is reversible under every stationary Markovian strategy. A natural application in which such problems arise is in the control of Metropolis–Hastings type dynamics. We characterize all discrete time reversible Markov decision processes with finite state and action spaces. We show that the policy iteration algorithm for finding an optimal policy can be significantly simplified in Markov decision problems of this type. We also highlight the relation between the finite time evolution of the accrual of reward and the Gaussian free field associated to the controlled Markov chain.

Essential stationary equilibria of mean field games with finite state and action space

Mean field games allow to describe tractable models of dynamic games with a continuum of players, explicit interaction and heterogeneous states. Thus, these models are of great interest for socio-economic applications. A particular class of these models are games with finite state and action space, for which recently in Neumann (2020a) a semi-explicit representation of all stationary equilibria has been obtained. In this paper we investigate whether these stationary equilibria are stable against model perturbations. We prove that the set of all games with only essential equilibria is residual and obtain two characterization results for essential stationary equilibria.

Cooperative Multirelay Network Design With Hybrid Backscatter and Wireless-Powered Relaying

In this article, a wireless multirelay network in which the relays are energy constrained is studied. Especially, in order to consume the harvested energy efficiently at the relays so as to improve the network throughput, a new hybrid relaying protocol is first proposed. In the proposed protocol, each relay can flexibly switch its operation among energy harvesting (EH), information receiving (IR), active information transmission (IT), and two passive backscatter communication (BC) modes according to the channel states as well as its data buffer states and energy states in each transmission block, by which the harvested energy can be efficiently utilized and superior throughput performance can be achieved. However, under the hybrid relaying protocol, it is challenging to achieve a strategy to optimally determine the operation mode for each relay, and the energy and information scheduling at the relays that operate in the IR, IT, and BC modes. To address this issue, the involved optimization problem is formulated as a stochastic optimization problem, which cannot be tackled directly. To make it tractable, the stochastic optimization problem is transformed into a Markov decision process (MDP) with finite state and action spaces. By solving the MDP via a dynamic programming (DP) algorithm, the optimal strategy for the multirelay network is achieved. Furthermore, to reduce the computational complexity in the DP algorithm, an efficient algorithm with low complexity is developed by using a Lyapunov optimization framework. Numerical simulations show that our proposed hybrid relaying strategy can achieve superior throughput performance in wireless multirelay networks.

Reward-Weighted Regression Converges to a Global Optimum

Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.

Learning in nonatomic games, part Ⅰ: Finite action spaces and population games

<p style='text-indent:20px;'>We examine the long-run behavior of a wide range of dynamics for learning in nonatomic games, in both discrete and continuous time. The class of dynamics under consideration includes fictitious play and its regularized variants, the best reply dynamics (again, possibly regularized), as well as the dynamics of dual averaging / "follow the regularized leader" (which themselves include as special cases the replicator dynamics and Friedman's projection dynamics). Our analysis concerns both the actual trajectory of play and its time-average, and we cover potential and monotone games, as well as games with an evolutionarily stable state (global or otherwise). We focus exclusively on games with finite action spaces; nonatomic games with continuous action spaces are treated in detail in Part Ⅱ.</p>

Dynamic Programming Through the Lens of Semismooth Newton-Type Methods

Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods for solving the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys a local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires relatively few iterations to achieve convergence even in presence of a large number of admissible policies. We then show that value iteration is an instance of the fixed-point iteration method and develop a novel locally accelerated version of value iteration with global convergence guarantees and negligible extra computational costs.

Decomposition Methods for Solving Finite‐Horizon Large MDPs

Conventional algorithms for solving Markov decision processes (MDPs) become intractable for a large finite state and action spaces. Several studies have been devoted to this issue, but most of them only treat infinite‐horizon MDPs. This paper is one of the first works to deal with non‐stationary finite‐horizon MDPs by proposing a new decomposition approach, which consists in partitioning the problem into smaller restricted finite‐horizon MDPs, each restricted MDP is solved independently, in a specific order, using the proposed hierarchical backward induction (HBI) algorithm based on the backward induction (BI) algorithm. Next, the sub‐local solutions are combined to obtain a global solution. An example of racetrack problems shows the performance of the proposal decomposition technique.

Efficient Algorithms for Learning to Control Bandits with Unobserved Contexts

Contextual bandits are widely-used in the study of learning-based control policies for finite action spaces. While the problem is well-studied for bandits with perfectly observed context vectors, little is known about the case of imperfectly observed contexts. For this setting, existing approaches are inapplicable and new conceptual and technical frameworks are required. We present an implementable posterior sampling algorithm for bandits with imperfect context observations and study its performance for learning optimal decisions. The numerical results relate the performance of the algorithm to different quantities of interest, including the number of arms, dimensions, observation matrices, posterior rescaling factors, and signal-to-noise ratios. In general, the proposed algorithm exposes efficiency in learning from the noisy imperfect observations and taking actions accordingly. Enlightening understandings the analyses provide as well as interesting future directions it points to, are discussed as well.

Uniformly supported approximate equilibria in families of games

This paper considers uniformly bounded classes of non-zero-sum strategic-form games with large finite or compact action spaces. The central class of games considered is assumed to be defined via a semi-algebraic condition. We show that for each ɛ>0, the support size required for ɛ-equilibrium can be taken to be uniform over the entire class. As a corollary, the value of zero-sum games, as a function of a single-variable, is well-behaved in the limit. More generally, the result only requires that the collection of payoff functions considered, as functions of other players actions, have finite pseudo-dimension.

Zero-sum semi-Markov games with a probability criterion

In this paper, we consider a semi-Markov zero-sum stochastic game with a general state and finite action spaces. The performance is analysed via a probability criterion. Under suitable assumptions, we establish the existence of the value of the game and also characterize it through an optimality equation. In the process, we also prescribe a saddle point equilibrium.