Mean-Field Games of Optimal Stopping: Master Equation and Weak Equilibria

In this paper we establish a new connection between a class of two-player nonzero-sum games of optimal stopping and certain two-player nonzero-sum games of singular control. We show that whenever a Nash equilibrium in the game of stopping is attained by hitting times at two separate boundaries, then such boundaries also trigger a Nash equilibrium in the game of singular control. Moreover, a differential link between the players' value functions holds across the two games.

This paper continues the study of the mean field game (MFG) convergence problem: In what sense do the Nash equilibria of $n$-player stochastic differential games converge to the mean field game as $n\rightarrow\infty $? Previous work on this problem took two forms. First, when the $n$-player equilibria are open-loop, compactness arguments permit a characterization of all limit points of $n$-player equilibria as weak MFG equilibria, which contain additional randomness compared to the standard (strong) equilibrium concept. On the other hand, when the $n$-player equilibria are closed-loop, the convergence to the MFG equilibrium is known only when the MFG equilibrium is unique and the associated “master equation” is solvable and sufficiently smooth. This paper adapts the compactness arguments to the closed-loop case, proving a convergence theorem that holds even when the MFG equilibrium is nonunique. Every limit point of $n$-player equilibria is shown to be the same kind of weak MFG equilibrium as in the open-loop case. Some partial results and examples are discussed for the converse question, regarding which of the weak MFG equilibria can arise as the limit of $n$-player (approximate) equilibria.

The goal of this paper is to provide a selection principle for potential mean field games on a finite state space and, in this respect, to show that equilibria that do not minimize the corresponding mean field control problem should be ruled out. Our strategy is a tailor-made version of the vanishing viscosity method for partial differential equations. Here, the viscosity has to be understood as the square intensity of a common noise that is inserted in the mean field game or, equivalently, as the diffusivity parameter in the related parabolic version of the master equation. As established in the recent contribution (Bayraktar et al., 2021, J. Math. Pures Appl. 147:98–162), the randomly forced mean field game becomes indeed uniquely solvable for a relevant choice of a Wright-Fisher common noise, the counterpart of which in the master equation is a Kimura operator on the simplex. We here elaborate on (Bayraktar et al., 2021, J. Math. Pures Appl. 147:98–162) to make the mean field game with common noise both uniquely solvable and potential, meaning that its unique solution is in fact equal to the unique minimizer of a suitable stochastic mean field control problem. Taking the limit as the intensity of the common noise vanishes, we obtain a rigorous proof of the aforementioned selection principle. As a byproduct, we get that the classical solution to the viscous master equation associated with the mean field game with common noise converges to the gradient of the value function of the mean field control problem without common noise. We hence select a particular weak solution of the master equation of the original mean field game. Lastly, we establish an intrinsic uniqueness criterion for this solution within a suitable class of weak solutions to the master equation satisfying a weak one-sided Lipschitz inequality.

These notes are an introduction to Mean Field Game (MFG) theory, which models differential games involving infinitely many interacting players. We focus here on the Partial Differential Equations (PDEs) approach to MFGs. The two main parts of the text correspond to the two emblematic equations in MFG theory: the first part is dedicated to the MFG system, while the second part is devoted to the master equation. The MFG system describes Nash equilibrium configurations in the mean field approach to differential games with infinitely many players. It consists in the coupling between a backward Hamilton-Jacobi equation (for the value function of a single player) and a forward Fokker-Planck equation (for the distribution law of the individual states). We discuss the existence and the uniqueness of the solution to the MFG system in several frameworks, depending on the presence or not of a diffusion term and on the nature of the interactions between the players (local or nonlocal coupling). We also explain how these different frameworks are related to each other. As an application, we show how to use the MFG system to find approximate Nash equilibria in games with a finite number of players and we discuss the asymptotic behavior of the MFG system. The master equation is a PDE in infinite space dimension: more precisely it is a kind of transport equation in the space of measures. The interest of this equation is that it allows to handle more complex MFG problems as, for instance, MFG problems involving a randomness affecting all the players. To analyse this equation, we first discuss the notion of derivative of maps defined on the space of measures; then we present the master equation in several frameworks (classical form, case of finite state space and case with common noise); finally we explain how to use the master equation to prove the convergence of Nash equilibria of games with finitely many players as the number of players tends to infinity. As the works on MFGs are largely inspired by P.L. Lions’ courses held at the Collège de France in the years 2007–2012, we complete the text with an appendix describing the organization of these courses.

Mean Field Game (MFG) systems describe equilibrium configurations in games with infinitely many interacting controllers. We are interested in the behavior of this system as the horizon becomes large, or as the discount factor tends to $0$. We show that, in the two cases, the asymptotic behavior of the Mean Field Game system is strongly related with the long time behavior of the so-called master equation and with the vanishing discount limit of the discounted master equation, respectively. Both equations are nonlinear transport equations in the space of measures. We prove the existence of a solution to an ergodic master equation, towards which the time-dependent master equation converges as the horizon becomes large, and towards which the discounted master equation converges as the discount factor tends to $0$. The whole analysis is based on the obtention of new estimates for the exponential rates of convergence of the time-dependent MFG system and the discounted MFG system.

Minimal solutions of master equations for extended mean field games

This book describes the latest advances in the theory of mean field games, which are optimal control problems with a continuum of players, each of them interacting with the whole statistical distribution of a population. While it originated in economics, this theory now has applications in areas as diverse as mathematical finance, crowd phenomena, epidemiology, and cybersecurity. Because mean field games concern the interactions of infinitely many players in an optimal control framework, one expects them to appear as the limit for Nash equilibria of differential games with finitely many players as the number of players tends to infinity. The book rigorously establishes this convergence, which has been an open problem until now. The limit of the system associated with differential games with finitely many players is described by the so-called master equation, a nonlocal transport equation in the space of measures. After defining a suitable notion of differentiability in the space of measures, the authors provide a complete self-contained analysis of the master equation. Their analysis includes the case of common noise problems in which all the players are affected by a common Brownian motion. They then go on to explain how to use the master equation to prove the mean field limit. The book presents two important new results in mean field games that contribute to a unified theoretical framework for this exciting and fast-developing area of mathematics.

We develop a splitting method to prove the well-posedness, in short time, of solutions for two master equations in mean field game (MFG) theory: the second order master equation, describing MFGs with a common noise, and the system of master equations associated with MFGs with a major player. Both problems are infinite-dimensional equations stated in the space of probability measures. Our new approach simplifies and generalizes previous existence results for second order master equations and provides the first existence result for systems associated with MFG problems with a major player.

**Read paper on the following link:** https://ifaamas.org/Proceedings/aamas2022/pdfs/p926.pdf **Abstract:** Recent advances in multiagent learning have seen the introduction of a family of algorithms that revolve around the population-based training method PSRO, showing convergence to Nash, correlated and coarse correlated equilibria. Notably, when the number of agents increases, learning best-responses becomes exponentially more difficult, and as such hampers PSRO training methods. The field of mean-field games provides an asymptotic solution to this problem when the considered games are anonymous-symmetric. Unfortunately, the mean-field approximation introduces non-linearities which prevent a straightforward adaptation of PSRO. Building upon optimization and adversarial regret minimization, this paper sidesteps this issue and introduces mean-field PSRO, an adaptation of PSRO which learns Nash, coarse correlated and correlated equilibria in mean-field games. The key is to replace the exact distribution computation step by newly-defined mean-field no-adversarial-regret learners, or by black-box optimization. We compare the asymptotic complexity of the approach to standard PSRO, greatly improve empirical bandit convergence speed by compressing temporal mixture weights, and ensure it is theoretically robust to payoff noise. Finally, we illustrate the speed and accuracy of mean-field PSRO on several mean-field games, demonstrating convergence to strong and weak equilibria.

**Read paper on the following link:** https://ifaamas.org/Proceedings/aamas2022/pdfs/p926.pdf **Abstract:** Recent advances in multiagent learning have seen the introduction of a family of algorithms that revolve around the population-based training method PSRO, showing convergence to Nash, correlated and coarse correlated equilibria. Notably, when the number of agents increases, learning best-responses becomes exponentially more difficult, and as such hampers PSRO training methods. The field of mean-field games provides an asymptotic solution to this problem when the considered games are anonymous-symmetric. Unfortunately, the mean-field approximation introduces non-linearities which prevent a straightforward adaptation of PSRO. Building upon optimization and adversarial regret minimization, this paper sidesteps this issue and introduces mean-field PSRO, an adaptation of PSRO which learns Nash, coarse correlated and correlated equilibria in mean-field games. The key is to replace the exact distribution computation step by newly-defined mean-field no-adversarial-regret learners, or by black-box optimization. We compare the asymptotic complexity of the approach to standard PSRO, greatly improve empirical bandit convergence speed by compressing temporal mixture weights, and ensure it is theoretically robust to payoff noise. Finally, we illustrate the speed and accuracy of mean-field PSRO on several mean-field games, demonstrating convergence to strong and weak equilibria.

We develop the fictitious play algorithm in the context of the linear programming approach for mean field games of optimal stopping and mean field games with regular control and absorption. This algorithm allows to approximate the mean field game population dynamics without computing the value function by solving linear programming problems associated with the distributions of the players still in the game and their stopping times/controls. We show the convergence of the algorithm using the topology of convergence in measure in the space of subprobability measures, which is needed to deal with the lack of continuity of the flows of measures. Numerical examples are provided to illustrate the convergence of the algorithm.

The lion’s share of this chapter is devoted to the construction of equilibria for mean field games with a common noise. We develop a general two-step strategy for the search of weak solutions. The first step is to apply Schauder’s theorem in order to prove the existence of strong solutions to mean field games driven by a discretized version of the common noise. The second step is to make use of a general stability property of weak equilibria in order to pass to the limit along these discretized equilibria. We also present several criteria for strong uniqueness, in which cases weak equilibria are known to be strong.

We present here results obtained in the joint work with Delarue, Lasry and Lions [4] on the convergence, as N tends to infinity, of a system of N coupled Hamilton-Jacobi equations, the Nash system. This system arises in differential game theory. The limit problem can be expressed in terms of the “Mean Field Game” system (coupling a Hamilton-Jacobi equation with a Fokker-Planck equation), or, alternatively, in terms of the “master equation” (a kind of second order partial differential equation stated on the space of probability measures). We also discuss the behavior of the optimal trajectories, for which we show a propagation of chaos property.

We present a new notion of solution for mean field games master equations. This notion allows us to work with solutions which are merely continuous. We first prove results of uniqueness and stability for such solutions. It turns out that this notion is helpful to characterize the value function of mean field games of optimal stopping or impulse control and this is the topic of the second half of this paper. The notion of solution we introduce is only useful in the monotone case. In this article we focus on the finite state space case.

The need for clean water is expected to substantially increase while further reductions of water availability in sufficient quantity and quality are projected owing to climate change and anthropogenic activities. Accordingly, the debate on water security has recently intensified and reached the intergovernmental arena. Industry is, in particular, one of the largest (non-consumptive) water users, accountable for massive toxic wastewater discharges and facing stringent and costly environmental oversight. However, the management of reservoirs is intricate and operational research must be further expanded to design tools that enhance water security while improving operators’ profitability. We therefore consider a game-theoretic framework to study the strategies adopted by a large group of similar producers sharing a water reservoir for their manufacturing activities. Each operator faces random demand for its outputs and chooses the optimal time to invest in a technology that ends its reliance on the reservoir. This technology introduces cost saving opportunities for the operator and benefits for the environment. Each producer therefore solves a so-called optimal stopping problem, and all problems are coupled through the reservoir level. We formulate the problem of finding a Nash equilibrium as a mean-field game (MFG) of optimal stopping. We then apply the model to the paper milling industry, an extensive water user facing a tightening of environmental regulations. This paper provides fresh insights into how to rethink the problem of technological change and water management, by offering an innovative application of operational research that builds on recent mathematical developments made in MFG theory.