Mean-field Equilibrium Research Articles

Reinforcement Learning for Mean-Field Game

Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously. After taking an action, the state of every agent updates to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often difficult when the number of agents becomes large. This paper focuses on finding a mean-field equilibrium (MFE) in an action-coupled stochastic game setting in an episodic framework. It is assumed that an agent can approximate the impact of the other agents’ by the empirical distribution of the mean of the actions. All agents know the action distribution and employ lower-myopic best response dynamics to choose the optimal oblivious strategy. This paper proposes a posterior sampling-based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions. We show that the policy and action distributions converge to the optimal oblivious strategy and the limiting distribution, respectively, which constitute an MFE.

Relative performance evaluation for dynamic contracts in a large competitive market

In this article, we develop a novel dynamic model to study the role and effect of relative performance evaluation (RPE) in a delegated portfolio management framework, where a non-zero-sum game is introduced among managers in addition to the hierarchical Stackelberg game between the shareholders and managers. The characterization of the optimal contracts in the generic finite-population case commonly involves a complicated system of coupled nonlinear equations for which the existence of a solution may not be established. Alternatively, an approximation of the large population case is still viable. Indeed, we derive an analytically tractable mean field equilibrium as the population size goes to infinity and conclude that: (1) it is the risk sharing between shareholders and managers and the presence of correlated investment opportunities across firms that lead to the inclusion of RPE in contracts; (2) these contracts incorporating RPE, in return, trigger the managers to invest more in the correlated assets with the desire to reduce the volatility of compensation payments. To this end, we echo the finding that the incorporation of RPE increases the systemic risk, which was first observed in Albuquerque et al. (Rev. Financ. Stud. 32(11): 4304–4342, 2019). In addition, we show that the systemic risk increases with the aggregate Sharpe Ratios of securities and the risk seeking appetites of managers.

Mean-Field Game Theory Based Optimal Caching Control in Mobile Edge Computing

Mobile edge computing (MEC) can use wireless access network (RAN) to provide users with nearby information technology (IT) services and cloud computing functions, which creates a high-performance and low latency service environment. By caching the popular content at small base station (SBS) can reduce the heavy backhaul load and the content retransmission. However, the time-varying and dynamic of the content requests may lead to the base station to cache the useless contents. In this paper, we study a distributed caching optimization problem in edge networks (ENs) with the spatio-temporal requirements. In the considered ENs, the cache control is described as a stochastic differential game (SDG) in which each SBS defines a caching strategy to reduce the total cost in terms of the service delay and backhaul link load. To reduce the computational complexity, the original optimization problem is transformed into a mean field game (MFG). We propose a distributed caching iterative control algorithm that decouples the information interaction between the general SBS and others through the mean field distribution. In addition, we obtain the optimal edge caching control strategy, while the existence and uniqueness of the mean field equilibrium (MFE) can also be guaranteed. Simulation results demonstrate that our proposed caching control algorithm can average reduce 27.12% storage cost and achieve better performance than other existing schemes.

Time-Consistent Investment and Reinsurance Strategies for Mean–Variance Insurers in N-Agent and Mean-Field Games

In this study, we investigate the competition among insurers under the mean–variance criterion. The optimization problems are formulated for finite and infinite insurers. The surplus processes of the insurers are characterized by jump-diffusion processes with common and idiosyncratic insurance risks. The insurers can purchase a reinsurance business to divide the insurance risk. In the financial market, the insurers decide the proportion of their fund to be retained as cash and to be invested in a stock characterized by a jump-diffusion process with common and idiosyncratic financial risks. The insurers compete with each other and are concerned with the relative performance. By the extended Hamilton-Jacobi-Bellman equation, the explicit forms of the n-agent equilibrium and mean-field equilibrium (MFE) are obtained in the n-agent case and mean-field case, respectively. Our results show that the MFE of the reinsurance strategy is composed of two parts, one part associated with the individual preference and the other associated with the common insurance shock. Meanwhile, the MFE of the investment strategy is composed of three parts: the individual preference, common market risks, and common shocks. Numerical examples are presented at the end of this article to demonstrate the effects of different parameters on the MFE. The results reveal that the insurers become convergent in a competitive environment.

Multi-UAV Trajectory and Power Optimization for Cached UAV Wireless Networks With Energy and Content Recharging-Demand Driven Deep Learning Approach

In this paper, we propose a novel joint trajectory and communication scheduling scheme for multiple unmanned aerial vehicles (UAVs) enabled wireless caching networks. To exploit the favorable propagation of air-to-ground channels, we consider an ultra dense UAVs enabled content-centric wireless transmission network, where massive UAVs are deployed to transmit cached contents to a group of random distributed ground users. We formulate the problem as an infinite horizon ergodic stochastic differential game (SDG) for optimizing the users' quality-of-experience (QoE). In particular, stochastic dynamics of channel states, UAVs' mobility, energy queues and content request queues are modeled in this game. To deal with the state coupling between the UAVs, we consider a limiting problem for large number of UAV based on mean field analysis. A reduced-complexity decentralized solution can be obtained through mean-field equilibrium analysis. To further reduce the solution complexity on each UAV, we propose a model-specific deep neural network (DNN) to learn the optimal control solution in an online manner. The DNN is not arbitrarily generated but tailored to the structural properties of the value function and stationary distribution based on the homotopy perturbation method analysis. Finally, simulation results are provided to show that the proposed solution can achieve significant gain over the existing baselines.

Mean Field Equilibrium: Uniqueness, Existence, and Comparative Statics

The Uniqueness of a Mean Field Equilibrium

Continuous-Time Mean Field Games with Finite State Space and Common Noise

We formulate and analyze a mathematical framework for continuous-time mean field games with finitely many states and common noise, including a rigorous probabilistic construction of the state process and existence and uniqueness results for the resulting equilibrium system. The key insight is that we can circumvent the master equation and reduce the mean field equilibrium to a system of forward-backward systems of (random) ordinary differential equations by conditioning on common noise events. In the absence of common noise, our setup reduces to that of Gomes, Mohr and Souza (Appl Math Optim 68(1): 99–143, 2013) and Cecchin and Fischer (Appl Math Optim 81(2):253–300, 2020).

Origin-to-destination network flow with path preferences and velocity controls: A mean field game-like approach

<p style='text-indent:20px;'>In this paper we consider a mean field approach to modeling the agents flow over a transportation network. In particular, beside a standard framework of mean field games, with controlled dynamics by the agents and costs mass-distribution dependent, we also consider a path preferences dynamics obtained as a generalization of the so-called noisy best response dynamics. We introduce this last dynamics to model the fact that the agents choose their path on the basis of both the network congestion state and the observation of the agents' decision that have preceded them. We prove the existence of a mean field equilibrium obtained as a fixed point of a map over a suitable set of time-varying mass-distributions, defined edge by edge in the network. We also address the case where the admissible set of controls is suitably bounded depending on the mass-distribution on the edge itself.</p>

Mean Field Contest with Singularity

We formulate a mean field game where each player stops a privately observed Brownian motion with absorption. Players are ranked according to their level of stopping and rewarded as a function of their relative rank. There is a unique mean field equilibrium and it is shown to be the limit of associated $n$-player games. Conversely, the mean field strategy induces $n$-player $\varepsilon$-Nash equilibria for any continuous reward function -- but not for discontinuous ones. In a second part, we study the problem of a principal who can choose how to distribute a reward budget over the ranks and aims to maximize the performance of the median player. The optimal reward design (contract) is found in closed form, complementing the merely partial results available in the $n$-player case. We then analyze the quality of the mean field design when used as a proxy for the optimizer in the $n$-player game. Surprisingly, the quality deteriorates dramatically as $n$ grows. We explain this with an asymptotic singularity in the induced $n$-player equilibrium distributions.

A MEAN FIELD GAME ANALYSIS OF SIR DYNAMICS WITH VACCINATION

We analyze a mean field game model of SIR dynamics (Susceptible, Infected, and Recovered) where players choose when to vaccinate. We show that this game admits a unique mean field equilibrium (MFE) that consists in vaccinating at a maximal rate until a given time and then not vaccinating. The vaccination strategy that minimizes the total cost has the same structure as the MFE. We prove that the vaccination period of the MFE is always smaller than the one minimizing the total cost. This implies that, to encourage optimal vaccination behavior, vaccination should always be subsidized. Finally, we provide numerical experiments to study the convergence of the equilibrium when the system is composed by a finite number of agents ($N$) to the MFE. These experiments show that the convergence rate of the cost is$1/N$and the convergence of the switching curve is monotone.

Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games

In this paper, we study a class of discrete-time mean-field games under the infinite-horizon risk-sensitive optimality criterion. Risk sensitivity is introduced for each agent (player) via an exponential utility function. In this game model, each agent is coupled with the rest of the population through the empirical distribution of the states, which affects both the agent’s individual cost and its state dynamics. Under mild assumptions, we establish the existence of a mean-field equilibrium in the infinite-population limit as the number of agents (N) goes to infinity, and we then show that the policy obtained from the mean-field equilibrium constitutes an approximate Nash equilibrium when N is sufficiently large.

Dynamic Pricing of New Products in Competitive Markets: A Mean-Field Game Approach

Dynamic pricing of new products has been extensively studied in monopolistic and oligopolistic markets. But, the optimal control and differential game tools used to investigate the pricing behavior on markets with a finite number of firms are not well-suited to model competitive markets with an infinity of firms. Using a mean-field games approach, this paper examines dynamic pricing policies in competitive markets, where no firm exerts market power. The theoretical setting is based on a diffusion modei a la Bass. We prove both the existence and the uniqueness of a mean-field game equilibrium, and we investigate mean tendencies and firms dispersion in the market. Numerical simulations show that the competitive market splits into two separate groups of firms depending on their production experience. The two groups differ in price and profit. Thus, high prices and profits do not have to signal anticompetitive practices, stimulating the debate on market regulation.

Value iteration algorithm for mean-field games

In the literature, existence of mean-field equilibria has been established for discrete-time mean field games under both the discounted cost and the average cost optimality criteria. In this paper, we provide a value iteration algorithm to compute stationary mean-field equilibrium for both the discounted cost and the average cost criteria, whose existence proved previously. We establish that the value iteration algorithm converges to the fixed point of a mean-field equilibrium operator. Then, using this fixed point, we construct a stationary mean-field equilibrium. In our value iteration algorithm, we use Q -functions instead of value functions.

Optimal Trading with Differing Trade Signals

We consider the problem of maximizing portfolio value when an agent has a subjective view on asset value which differs from the traded market price. The agent's trades will have a price impact which affect the price at which the asset is traded. In addition to the agent's trades affecting the market price, the agent may change his view on the asset's value if its difference from the market price persists. We also consider a situation of several agents interacting and trading simultaneously when they have a subjective view on the asset value. Two cases of the subjective views of agents are considered, one in which they all share the same information, and one in which they all have an individual signal correlated with price innovations. To study the large agent problem we take a mean-field game approach which remains tractable. After classifying the mean-field equilibrium we compute the cross-sectional distribution of agents' inventories and the dependence of price distribution on the amount of shared information among the agents.

Optimal Trading with Differing Trade Signals

ABSTRACT We consider the problem of maximizing portfolio value when an agent has a subjective view on asset value which differs from the traded market price. The agent’s trades will have a price impact which affects the price at which the asset is traded. In addition to the agent’s trades affecting the market price, the agent may change his view on the asset’s value if its difference from the market price persists. We also consider a situation of several agents interacting and trading simultaneously when they have a subjective view of the asset value. Two cases of the subjective views of agents are considered: one in which they all share the same information, and one in which they all have an individual signal correlated with price innovations. To study the large agent problem we take a mean-field game approach which remains tractable. After classifying the mean-field equilibrium we compute the cross-sectional distribution of agents’ inventories and the dependence of price distribution on the amount of shared information among the agents.

Millimeter-Wave Networking in the Sky: A Machine Learning and Mean Field Game Approach for Joint Beamforming and Beam-Steering

In unmanned aerial vehicle (UAV)-assisted massive multi-input multi-output (MIMO) millimeter-wave (mmWave) networks, beam-steering guarantees reliable and steady connection between flying base stations and ground users with the challenge of strict angular deviation. In this paper, we investigate a joint optimization problem of beamforming and beam-steering in the multi-UAV mmWave networks, considering line-of-sight (LoS) communication for UAVs. For the hybrid beamforming optimization of massive MIMO mmWave, we propose a hybrid beamforming scheme based on the cross-entropy estimation with the robustness algorithm inspired by machine learning, which aims to optimize the hybrid precoding matrix. For the beam-steering optimization, we propose a novel mean field game (MFG)-based massive MIMO angle control scheme to model the optimal mmWave channel optimization problem between UAVs and ground users. In addition, when dealing with the problem of initial sensitivity and difficulty to solve the partial differential equations in the MFG, we utilize reinforcement learning to achieve the mean field equilibrium, which is described as the mean field learning game algorithm. Finally, a joint beamforming and beam-steering optimization algorithm is proposed to maximize the system sum-rate. Simulation results show the significant improvements in sum-rate, energy efficiency, and spectral efficiency, which verify the effectiveness of the proposed algorithm.

A Mean Field Game Approach to Scheduling in Cellular Systems

There has been much work done on designing cellular scheduling algorithms. These algorithms are set up in the manner of a direct mechanism in which the queues reveal their states (backlog), and the scheduler chooses an allocation. In policies such as longest queue first (LQF), scheduling can be shown to yield short queue lengths. However, these algorithms are reliant on truthful declarations of state. Our goal in this article is to determine if a Vickrey–Clarke–Groves (VCG)-type mechanism (a second price auction) that elicits truthful value will also possess the desired LQF-like behavior in a system of dynamically evolving queues. Our approach is to use the concept of a mean field game under which at each time instant, a queue chooses its bid as a best response to its belief that the bids of the others will be drawn independently from a common bid distribution. If this best response is itself a sample from the belief bid distribution, a mean field equilibrium (MFE) is said to exist. We show the existence of the MFE and find it using its structure that the results of the allocation policy are the same as LQF. Thus, the desired LQF-like behavior arises naturally under the second-price auction mechanism. We also present results on the accuracy of the model as the number of agents (queues) becomes large. Finally, using simulations with a large number of queues, we show that computation of the MFE is straightforward.

Linearly Solvable Mean-Field Traffic Routing Games

We consider a dynamic traffic routing game over an urban road network involving a large number of drivers in which each driver selecting a particular route is subject to a penalty that is affine in the logarithm of the number of drivers selecting the same route. We show that the mean-field approximation of such a game leads to the so-called linearly solvable Markov decision process, implying that its mean-field equilibrium (MFE) can be found simply by solving a finite-dimensional linear system backward in time. Based on this backward-only characterization, it is further shown that the obtained MFE has the notable property of strong time-consistency. A connection between the obtained MFE and a particular class of fictitious play is also discussed.

A Mean Field Game-Based Distributed Edge Caching in Fog Radio Access Networks

In this paper, the edge caching optimization problem in fog radio access networks (F-RANs) is investigated. Taking into account time-variant user requests and ultra-dense deployment of fog access points (F-APs), we propose a distributed edge caching scheme to jointly minimize the request service delay and fronthaul traffic load. Considering the interactive relationship among F-APs, we model the optimization problem as a stochastic differential game (SDG) which captures the dynamics of F-AP states. To address both the intractability problem of the SDG and the caching capacity constraint, we propose to solve the optimization problem in a distributive manner. Firstly, a mean field game (MFG) is converted from the original SDG by exploiting the ultra-dense property of F-RANs, and the states of all F-APs are characterized by a mean field distribution. Then, an iterative algorithm is developed that enables each F-AP to obtain the mean field equilibrium and caching control without extra information exchange with other F-APs. Secondly, a fractional knapsack problem is formulated based on the mean field equilibrium, and a greedy algorithm is developed that enables each F-AP to obtain the final caching policy subject to the caching capacity constraint. Simulation results show that the proposed scheme outperforms the baselines.

From the master equation to mean field game limit theory: Large deviations and concentration of measure

We study a sequence of symmetric $n$-player stochastic differential games driven by both idiosyncratic and common sources of noise, in which players interact with each other through their empirical distribution. The unique Nash equilibrium empirical measure of the $n$-player game is known to converge, as $n$ goes to infinity, to the unique equilibrium of an associated mean field game. Under suitable regularity conditions, in the absence of common noise, we complement this law of large numbers result with nonasymptotic concentration bounds for the Wasserstein distance between the $n$-player Nash equilibrium empirical measure and the mean field equilibrium. We also show that the sequence of Nash equilibrium empirical measures satisfies a weak large deviation principle, which can be strengthened to a full large deviation principle only in the absence of common noise. For both sets of results, we first use the master equation, an infinite-dimensional partial differential equation that characterizes the value function of the mean field game, to construct an associated McKean–Vlasov interacting $n$-particle system that is exponentially close to the Nash equilibrium dynamics of the $n$-player game for large $n$, by refining estimates obtained in our companion paper. Then we establish a weak large deviation principle for McKean–Vlasov systems in the presence of common noise. In the absence of common noise, we upgrade this to a full large deviation principle and obtain new concentration estimates for McKean–Vlasov systems. Finally, in two specific examples that do not satisfy the assumptions of our main theorems, we show how to adapt our methodology to establish large deviations and concentration results.