Intrinsic Regularization by Noise for 1d Mean Field Games

The goal of this paper is to provide a selection principle for potential mean field games on a finite state space and, in this respect, to show that equilibria that do not minimize the corresponding mean field control problem should be ruled out. Our strategy is a tailor-made version of the vanishing viscosity method for partial differential equations. Here, the viscosity has to be understood as the square intensity of a common noise that is inserted in the mean field game or, equivalently, as the diffusivity parameter in the related parabolic version of the master equation. As established in the recent contribution (Bayraktar et al., 2021, J. Math. Pures Appl. 147:98–162), the randomly forced mean field game becomes indeed uniquely solvable for a relevant choice of a Wright-Fisher common noise, the counterpart of which in the master equation is a Kimura operator on the simplex. We here elaborate on (Bayraktar et al., 2021, J. Math. Pures Appl. 147:98–162) to make the mean field game with common noise both uniquely solvable and potential, meaning that its unique solution is in fact equal to the unique minimizer of a suitable stochastic mean field control problem. Taking the limit as the intensity of the common noise vanishes, we obtain a rigorous proof of the aforementioned selection principle. As a byproduct, we get that the classical solution to the viscous master equation associated with the mean field game with common noise converges to the gradient of the value function of the mean field control problem without common noise. We hence select a particular weak solution of the master equation of the original mean field game. Lastly, we establish an intrinsic uniqueness criterion for this solution within a suitable class of weak solutions to the master equation satisfying a weak one-sided Lipschitz inequality.

Mean field game (MFG) theory where there is a major agent and many minor agents (MM-MFG) has been formulated for both the linear quadratic Gaussian (LQG) case and for the case of nonlinear state dynamics and nonlinear cost functions. In this framework, even asymptotically (as the population size $N$ approaches infinity), and in contrast to the situation without major agents, the mean field term becomes stochastic due to the stochastic evolution of the state of the major agent; furthermore, the best response control actions of the minor agents depend on the state of the major agent as well as the stochastic mean field. In a decentralized environment, one is led to consider the situation where the agents are provided only with partial information on the major agent's state; in this work such a scenario is considered for systems with nonlinear dynamics and cost functions, and an $\epsilon$-Nash MFG theory is developed for this MM-MFG setup. The approach to the problem of partially observed MM-MFG systems adopted in this work is to follow the procedure of constructing the associated completely observed system via the application of nonlinear filtering theory; consequently, as a first step, nonlinear filtering equations are obtained for partially observed stochastic dynamical systems whose state equations contain a measure term corresponding to the distribution of the solution of a state process. Stochastic control theory for systems with random parameters is next generalized to the partially observed case by lifting the analysis to the infinite-dimensional domain. To achieve this, the Itô--Kunita lemma is first generalized to processes taking values in a subset of $L^1$ consisting of the space of solutions of the conditional density process generated by the filtering equations. The existence and uniqueness of solutions to the MFG system of equations is next established by a fixed point argument in the Wasserstein space of random probability measures in which the robustness property of nonlinear filtering theory is used. Finally, the $\epsilon$-Nash property of such a solution is analyzed in this setting where the state consists of finite- and infinite-dimensional (density) valued stochastic processes.

Power markets are modelled as dynamic large population games where suppliers and consumers submit their bids in real-time. The agents are coupled in their dynamics and cost functions through the price process. Here, a common unpredictable partially observed major agent is added to the system which represents common unpredictable disturbance factors (e.g. wind) and exogenous market factors (e.g. competing energy resource prices), etc. In previous work, the Mean Field Game (MFG) methodology was used to study the limit (Le., infinite population) behaviour of large population market systems without a major agent; this results in a decentralized algorithm where agents submit their bids solely using statistical information on the dynamics of the entire population. When a major agent is absent, the system exhibits the standard counter intuitive property of MFG solutions that agents need not observe the behaviour (i.e., inputs and state trajectories, market price evolution, etc.) of any other agent (individually or collectively) in order that simple decentralized control actions achieve a mass ϵ-Nash equilibrium (with ϵ vanishing as the population goes to infinity) and individual L <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> stability. The contribution of this paper is the extension of the MFG theory to cover the addition of a major agent to the power market problem. In general, the addition of a major agent in the MFG framework makes the mean field stochastic in contrast to the situation with purely minor agents where the mean field is deterministic. In the general situation of sporadic noisy observations of the mean field and the state of the major agent, the extended MFG theory (with estimation of the mean field and the major agent state) yields simple decentralized control laws which achieve a mass ϵ-Nash equilibrium (with ϵ vanishing as the population goes to infinity) and individual L <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> stability. In this paper, this is carried out for the MFG formulation of the power market problem in order to fit the situation where sporadic noisy observations of the state of the major agent and of the market price are available for recursive mean field state estimation.

In this paper, we show existence and uniqueness of solutions of the infinite horizon McKean–Vlasov FBSDEs using two different methods, which lead to two different sets of assumptions. We use these results to solve the infinite horizon mean field type control problems and mean field games.

In this paper, the optimal tracking control for large-scale multi-agent systems (MAS) under constraints has been investigated. The Mean Field Game (MFG) theory is an emerging technique to solve the &#x201C;curse of dimensionality&#x201D; problem in large-scale multi-agent decision-making problems. Specifically, the MFG theory can calculate the optimal strategy based on one unified fix-dimension probability density function (PDF) instead of the high-dimensional large-scale MAS information collected from all the individual agents. However, the MFG theory has stringent limitations by assuming all the agents operate in a predefined unlimited space, which is often too ideal for practical applications due to complex environments. In this paper, the original MFG theory has been extended by considering two practical state constraints caused by the environment, i.e., boundary and density constraints. Moreover, to solve the extended MFG type control online, the actor-critic reinforcement learning mechanism is utilized and further extended to a novel actor-critic-mass (ACM) algorithm. Finally, a series of numerical simulations are conducted to demonstrate the effectiveness of the developed schemes.

These notes are an introduction to Mean Field Game (MFG) theory, which models differential games involving infinitely many interacting players. We focus here on the Partial Differential Equations (PDEs) approach to MFGs. The two main parts of the text correspond to the two emblematic equations in MFG theory: the first part is dedicated to the MFG system, while the second part is devoted to the master equation. The MFG system describes Nash equilibrium configurations in the mean field approach to differential games with infinitely many players. It consists in the coupling between a backward Hamilton-Jacobi equation (for the value function of a single player) and a forward Fokker-Planck equation (for the distribution law of the individual states). We discuss the existence and the uniqueness of the solution to the MFG system in several frameworks, depending on the presence or not of a diffusion term and on the nature of the interactions between the players (local or nonlocal coupling). We also explain how these different frameworks are related to each other. As an application, we show how to use the MFG system to find approximate Nash equilibria in games with a finite number of players and we discuss the asymptotic behavior of the MFG system. The master equation is a PDE in infinite space dimension: more precisely it is a kind of transport equation in the space of measures. The interest of this equation is that it allows to handle more complex MFG problems as, for instance, MFG problems involving a randomness affecting all the players. To analyse this equation, we first discuss the notion of derivative of maps defined on the space of measures; then we present the master equation in several frameworks (classical form, case of finite state space and case with common noise); finally we explain how to use the master equation to prove the convergence of Nash equilibria of games with finitely many players as the number of players tends to infinity. As the works on MFGs are largely inspired by P.L. Lions’ courses held at the Collège de France in the years 2007–2012, we complete the text with an appendix describing the organization of these courses.

Minimal solutions of master equations for extended mean field games

This paper studies large population dynamic games involving nonlinear stochastic dynamical systems with agents of the following mixed types: (i) a major agent and (ii) a population of $N$ minor agents where $N$ is very large. The major and minor agents are coupled via both (i) their individual nonlinear stochastic dynamics and (ii) their individual finite time horizon nonlinear cost functions. This problem is analyzed by the so-called $\epsilon$-Nash mean field game theory. A distinct feature of the mixed agent mean field game problem is that even asymptotically (as the population size $N$ approaches infinity) the noise process of the major agent causes random fluctuation of the mean field behavior of the minor agents. To deal with this, the overall asymptotic ($N \rightarrow \infty$) mean field game problem is decomposed into (i) two nonstandard stochastic optimal control problems with random coefficient processes which yield forward adapted stochastic best response control processes determined from the solution of (backward in time) stochastic Hamilton--Jacobi--Bellman (SHJB) equations and (ii) two stochastic coefficient McKean--Vlasov (SMV) equations which characterize the state of the major agent and the measure determining the mean field behavior of the minor agents. This yields a stochastic mean field game (SMFG) system which is in contrast to the deterministic mean field game systems of standard MFG problems with only minor agents. Existence and uniqueness of the solutions to SMFG systems (SHJB and SMV equations) is established by a fixed point argument in the Wasserstein space of random probability measures. In the case where minor agents are coupled to the major agent only through their cost functions, the $\epsilon_N$-Nash equilibrium property of the SMFG best responses is shown for a finite $N$ population system where $\epsilon_N=O(1/\sqrt N)$.

Mean field games are limit models for symmetric $N$-player games with interaction of mean field type as $N\to\infty$. The limit relation is often understood in the sense that a solution of a mean field game allows to construct approximate Nash equilibria for the corresponding $N$-player games. The opposite direction is of interest, too: When do sequences of Nash equilibria converge to solutions of an associated mean field game? In this direction, rigorous results are mostly available for stationary problems with ergodic costs. Here, we identify limit points of sequences of certain approximate Nash equilibria as solutions to mean field games for problems with It{\^o}-type dynamics and costs over a finite time horizon. Limits are studied through weak convergence of associated normalized occupation measures and identified using a probabilistic notion of solution for mean field games.

Data ecosystems are becoming larger and more complex, while privacy concerns are threatening to erode their potential benefits. Recently, users have developed obfuscation techniques that issue fake search engine queries, undermine location tracking algorithms, or evade government surveillance. These techniques raise one conflict between each user and the machine learning algorithms which track the users, and one conflict between the users themselves. We use game theory to capture the first conflict with a Stackelberg game and the second conflict with a mean field game. Both are combined into a bi-level framework which quantifies accuracy using empirical risk minimization and privacy using differential privacy. We identify necessary and sufficient conditions under which 1) each user is incentivized to obfuscate if other users are obfuscating, 2) the tracking algorithm can avoid this by promising a level of privacy protection, and 3) this promise is incentive-compatible for the tracking algorithm.

Adaptive Distributed Formation Control for Multi-Group Large-Scale Multi-Agent Systems: A Hybrid Game Approach

In this paper we study mean field games with possibly multiple mean field equilibria. Instead of focusing on the individual equilibria, we propose to study the set of values over all possible equilibria, which we call the set value of the mean field game. When the mean field equilibrium is unique, typically under certain monotonicity conditions, our set value reduces to the singleton of the standard value function which solves the master equation. The set value is by nature unique, and we shall establish two crucial properties: (i) the dynamic programming principle, also called time consistency; and (ii) the convergence of the set values of the corresponding N N -player games, which can be viewed as a type of stability result. To our best knowledge, this is the first work in the literature which studies the dynamic value of mean field games without requiring the uniqueness of mean field equilibria. We emphasize that the set value is very sensitive to the type of the admissible controls. In particular, for the convergence one has to restrict to corresponding types of equilibria for the N-player game and for the mean field game. We shall illustrate this point by investigating three cases, two in finite state space models and the other in a continuous time model with controlled diffusions.

Cooperative multiaccess edge computing (MEC) is a promising paradigm for the next-generation mobile networks. However, when the number of users explodes, the computational complexity of the existing optimization or learning-based task placement approaches in the cooperative MEC can increase significantly, which leads to intolerable MEC decision-making delay. In this article, we propose a mean field game (MFG) guided deep reinforcement learning (DRL) approach for the task placement in the cooperative MEC, which can help servers make timely task placement decisions, and significantly reduce average service delay. Instead of applying MFG or DRL separately, we jointly leverage MFG and DRL for task placement, and let the equilibrium of MFG guide the learning directions of DRL. We also ensure that the MFG and DRL approaches are consistent with the same goal. Specifically, we novelly define a mean field guided Q -value (MFG-Q), which is an estimation of the Q -value with the Nash equilibrium gained by MFG. We evaluate the proposed method's performance using real-world user distribution. Through extensive simulations, we show that the proposed scheme is effective in making timely decisions and reducing the average service delay. Besides, the convergence rates of our proposed method outperform the pure DR-based approaches.

Mean field games (MFGs) and mean field control (MFC) have been introduced to study large populations of strategic players. They correspond respectively to non-cooperative or cooperative scenarios, where the aim is to find the Nash equilibrium and social optimum. They approximate finite player problems and have many applications in economics, biology, machine learning. This paper studies how players can pass from a non-cooperative to a cooperative regime, and vice-versa. The first direction is reminiscent of mechanism design, in which the game's definition is modified so that non-cooperative players reach an outcome similar to a cooperative scenario. The second direction studies how initially cooperative players gradually deviate from social optimum to reach Nash equilibrium when they optimize their individual cost very much in the spirit of the free-rider phenomenon. To formalize these connections, we introduce and theoretically analyze two new classes of games which lie between MFG and MFC: -interpolated MFGs, in which the cost of an individual player is an interpolation of the MFG and the MFC costs, and p-partial MFGs, in which a proportion of the population deviates from social optimum by behaving non-cooperatively. We conclude by providing an algorithm for myopic players to learn a p-partial mean field equilibrium.

We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The same algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space where the agent not only provides an action to the environment but also a distribution of the state in order to take into account the mean field feature of the problem. Importantly, we assume that the agent cannot observe the population’s distribution and needs to estimate it in a model-free manner. The asymptotic MFG and MFC problems are also presented in continuous time and space, and compared with classical (non-asymptotic or stationary) MFG and MFC problems. They lead to explicit solutions in the linear-quadratic (LQ) case that are used as benchmarks for the results of our algorithm.