Linear Quadratic Mean Field Games Research Articles

Exploration Noise for Learning Linear-Quadratic Mean Field Games

The goal of this paper is to demonstrate that common noise may serve as an exploration noise for learning the solution of a mean field game. This concept is here exemplified through a toy linear-quadratic model, for which a suitable form of common noise has already been proven to restore existence and uniqueness. We here go one step further and prove that the same form of common noise may force the convergence of the learning algorithm called fictitious play, and this without any further potential or monotone structure. Several numerical examples are provided to support our theoretical analysis. Funding: F. Delarue acknowledges the financial support of the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme [AdG ELISA project, Grant 101054746]. A. Vasileiadis acknowledge the financial support of French ANR project ANR-19-P3IA-0002-3IA Côte d'Azur-Nice-Interdisciplinary Institute for Artificial Intelligence.

A Unified Relation Analysis of Linear-Quadratic Mean-Field Game, Team, and Control

A Unified Relation Analysis of Linear-Quadratic Mean-Field Game, Team, and Control

Linear quadratic mean-field game with volatility uncertainty

In this paper, we consider a linear quadratic (LQ) mean field game (MFG) with uncertain volatility. Applying a robust optimization approach and based on mean-field approximation, we firstly obtain the worst uncertainty by dealing with an LQ MFG with so-called soft-constraint. Then with the help of convex analysis theory, we derive the auxiliary optimal control problems for the agents and construct the decentralized strategies. Finally, by virtue of estimates of (forward) backward stochastic differential equations, the asymptotical optimality of the decentralized strategy is verified. A numerical example is given to illustrate the above theoretical results.

A mean field game approach for a class of linear quadratic discrete choice problems with congestion avoidance

This paper approaches large multi-agent dynamic discrete choice problems, such as those arising in crowd evacuation or micro-robotic-based exploration of unknown terrain, via a linear quadratic mean field games framework (LQ-MFG). Two particular features distinguish the proposed LQ-MFG: (i) agents have to reach within a fixed finite time one of a predefined set of possible destinations, which depend on their initial position; (ii) agent running costs can become negative as agents accrue a reward the further they remain from other agents aiming for the same set of destinations, so as to simulate crowd avoidance with limited surroundings awareness. The desirable tractability of the LQ-MFG setup must be balanced in this case by the fact that feature (ii) of the model can lead to agents escaping to infinity in finite time. An upper bound on the time horizon is derived to guarantee that the finite escape time behavior is avoided. The existence of a Nash equilibrium for the infinite population is then established, and it is shown that this equilibrium remains ɛ-Nash for the more realistic case of a large but finite population of agents. Finally, the model behavior is explored via simulations for different scenarios.

Model-free policy iteration approach to NCE-based strategy design for linear quadratic Gaussian games

This paper presents a novel completely model-free method to solve continuous-time linear quadratic (LQ) mean field games (MFGs). We consider the setting where agents can only access their own states and are coupled via their individual infinite-horizon discounted costs. For such games, a set of Nash certainty equivalence-based strategies, determined by two gain matrices and the aggregate population effect, i.e. mean field (MF) state, are developed, providing an ɛ-Nash equilibrium under certain assumptions. Note that the gain matrices are solved from two algebraic Riccati equations (AREs), respectively, and the MF state is given by the linear ordinary differential equation (ODE). All the solutions of these three equations require the exact model of system dynamics. To implement the NCE-based strategy with completely unknown system dynamics, the proposed approach develops a single-agent policy iteration (PI) and a single-agent MF generator to solve the above parameters. Specifically, first, model-based iterative equations with guaranteed convergence are developed to approximate two gain matrices. Then, the requirement of dynamical model is removed from the iteration process via the integral reinforcement learning (IRL) technique. A model-free algorithm is conducted by using measured data from a selected agent as reinforcement signals over a certain time interval. Finally, based on the obtained gain matrices, the MF state is computed offline depending on samples collected from the same agent. A numerical example is given to verify the effectiveness of the proposed method.

Decentralized ϵ‐Nash strategy for linear quadratic mean field games using a successive approximation approach

AbstractThis paper presents a successive approximation method for decentralized strategy design in the large‐scale linear quadratic (LQ) Gaussian game. The strategy consists of transforming the original mean field game (MFG) problem into solving decoupled ordinary differential equations (ODEs) whose numerical solutions are obtained by a new two‐loop iteration algorithm. It should be noted that we employ the augmented model technique and the LQ framework to derive these low‐dimensional solvable ODEs, which is the cornerstone of constructing the decentralized ‐Nash strategy. In addition, the quadratic ODEs contained therein are approximately solved for by a sequence of iterative linear ordinary differential equations (LODEs) with guaranteed convergence. A numerical example is given to show the effectiveness of the proposed algorithm.

An addendum to the problem of zero-sum LQ stochastic mean-field dynamic games

In this paper, we first address a linear quadratic mean-field game problem with a leader–follower structure. By adopting a Riccati-type approach, we show how one can obtain a state-feedback representation of the pairs of strategies which achieve an open-loop Stackelberg equilibrium in terms of the global solutions of a system of coupled matrix differential Riccati-type equations. In the second part of this paper, we obtain necessary and sufficient conditions for the solvability of the involved coupled generalized Riccati equations.

Linear-quadratic mean field games of controls with non-monotone data

In this paper, we study a class of linear-quadratic (LQ) mean field games of controls with common noises and their corresponding N N -player games. The theory of mean field game of controls considers a class of mean field games where the interaction is via the joint law of both the state and control. By the stochastic maximum principle, we first analyze the limiting behavior of the representative player and obtain his/her optimal control in a feedback form with the given distributional flow of the population and its control. The mean field equilibrium is determined by the Nash certainty equivalence (NCE) system. Thanks to the common noise, we do not require any monotonicity conditions for the solvability of the NCE system. We also study the master equation arising from the LQ mean field game of controls, which is a finite-dimensional second-order parabolic equation. It can be shown that the master equation admits a unique classical solution over an arbitrary time horizon without any monotonicity conditions. Beyond that, we can solve the N N -player game directly by further assuming the non-degeneracy of the idiosyncratic noises. As byproducts, we prove the quantitative convergence results from the N N -player game to the mean field game and the propagation of chaos property for the related optimal trajectories.

A linear-quadratic mean-field game of backward stochastic differential equation with partial information and common noise

This paper studies a linear-quadratic (LQ) mean-field game of stochastic large-population system with partial information and common noise, where the large-population system satisfies a class of backward stochastic differential equations (BSDEs), and both coupling structure and ki (a part of solution of BSDE) enter state equations and cost functionals. By virtue of stochastic maximum principle and optimal filter technique, we obtain a Hamiltonian system first, which is a fully-coupled forward-backward stochastic differential equation (FBSDE). Decoupling the Hamiltonian system, we obtain three ordinary differential equations (ODEs), a forward and a backward optimal filtering equations, which enable us to derive an optimal control of an auxiliary limiting control problem. Next, we verify that a decentralized control strategy is an ϵ-Nash equilibrium of the LQ game. Finally, we solve a product pricing problem and a network security control problem in applications. We give a near-optimal price strategy and a near-optimal control strategy with numerical simulations, respectively.

Decentralized Open-Loop Strategies of Linear Quadratic Mean Field Games: From Finite to Infinite Population

This paper studies the decentralized open-loop asymptotic Nash equilibria for linear quadratic mean field games following the direct approach. In the first step of the direct approach, the necessary and sufficient condition for the existence of centralized open-loop Nash equilibria is obtained by using the variational analysis. The Nash equilibria are characterized by an adapted solution to a system of forward-backward stochastic differential equations. A feedback representation of the open-loop Nash equilibrium is obtained by using the solution to a system of Riccati equations. In the second step of the direct approach, the decentralized asymptotic Nash equilibria are designed by considering the limit of the centralized Nash equilibria with a standard Riccati equation and a non-symmetric Riccati equation. The results show that the decentralized open-loop asymptotic Nash equilibria coincide with the feedback solutions which are designed by the fixed-point approach, provided both exist.

Model-free Computation Method in First-order Linear Quadratic Mean Field Games

This paper considers first-order continuous-time systems of mean field games in which agents are coupled via their dynamics and individual cost functions. For such games, a mean field equilibrium is developed by using the state space augmentation technique and the Nash certainty equivalence (NCE) principle. This decentralized strategy consists of two constant gains and a mean field state corresponding to the optimal strategy, where the coefficients are obtained by solving two algebraic Riccati equations and the time-varying mean field is solved from the ordinary differential equation. Note that solving these equations requires agents’ models. In order to eliminate the use of system information, two-agents model-free method is proposed. First, model-based iterative equations are proposed to approximate the feedback constant gain. By the integral reinforcement learning (IRL) technique, the system dynamics in the iterative equations are replaced by real-time data. Moreover, based on the learned solution, the coefficient of input can be calculated. Then, a model-free computational equation is developed by embedding the same collected data. Finally, the mean field state is computed depending on the obtained gains and the observed state and input trajectories from agents.

Leader–follower mean field LQ games: A direct method

AbstractThis paper investigates a linear quadratic mean field game with a leader and a large number of followers. The leader first gives its strategy, and then each follower optimizes its own cost. We first solve a mean field game problem, which gives the best response of followers to the leader's strategy. After applying the followers' strategies, the leader is faced to an optimal control problem driven by a high‐dimensional forward‐backward stochastic differential equations (FBSDEs). By decoupling the high‐dimensional Hamiltonian system with mean field approximations, we construct a set of decentralized strategies for all the players, which are further shown to be an ‐Stackelberg equilibrium. ‐

Optimal scheduling management of the parking lot and decentralized charging of electric vehicles based on Mean Field Game

As an intermediary for the interaction between the grid and electric vehicles (EVs), the parking lot aggregator not only facilitates the exchange of electricity between EVs and the grid, but also brings benefits to all participants. This paper proposes a linear quadratic (LQ) Mean Field Game (MFG) theory with a major player to optimal scheduling management of the parking lot and formulate optimal decentralized charging control strategies for a large number of EVs, to achieve the minimization of EVs charging cost while maximizing the profit of the parking lot, but these two problems are a set of coupled control problems. In addition to modeling the interaction between the parking lot and the EVs as a finite-time dynamic game problem, the Nash Certainty Equivalence (NCE) of related optimization problems is also proposed, and the corresponding solution algorithm is designed. The control effects of the proposed dynamic game problem on the charging cycle, as well as the effects of parameters change and electricity price fluctuations on charging control are illustrated through numerical simulations.

Reinforcement Learning for Non-stationary Discrete-Time Linear–Quadratic Mean-Field Games in Multiple Populations

Reinforcement Learning for Non-stationary Discrete-Time Linear–Quadratic Mean-Field Games in Multiple Populations

Linear Quadratic Mean Field Games: Decentralized O(1/N)-Nash Equilibria

This paper studies an asymptotic solvability problem for linear quadratic (LQ) mean field games with controlled diffusions and indefinite weights for the state and control in the costs. The authors employ a rescaling approach to derive a low dimensional Riccati ordinary differential equation (ODE) system, which characterizes a necessary and sufficient condition for asymptotic solvability. The rescaling technique is further used for performance estimates, establishing an O(1/N)-Nash equilibrium for the obtained decentralized strategies.

Pareto efficiency in the infinite horizon mean-field type cooperative stochastic differential game

This paper discusses the linear quadratic (LQ) mean-field type stochastic differential game with non-homogeneous terms in infinite horizon. Employing the equivalent description of Pareto efficiency, necessary conditions for the existence of Pareto solutions are presented under an assumption on the Lagrange multiplier set. Two conditions are introduced to guarantee that the assumption is established. Further, sufficient conditions for a control to be Pareto efficient are put forward in terms of the necessary conditions, the convexity condition on the homogeneous weighted sum cost functional and a transversality condition. The characterization of Pareto solutions is also studied for the homogeneous case. If the system is MF-L2-stabilizable, then the solvability of the related generalized algebraic Riccati equations (GAREs) provides a sufficient condition under which the cost functionals are convex and all Pareto efficient strategies can be obtained by the weighted sum minimization method. In addition, by introducing two algebraic Lyapunov equations (ALEs), we derive all Pareto solutions.

Linear-quadratic mean-field type stackelberg differential games for stochastic jump-diffusion systems

<p style='text-indent:20px;'>In this paper, we consider linear-quadratic (LQ) leader-follower Stackelberg differential games for mean-field type stochastic systems with jump diffusions, where the system includes mean-field variables, i.e., the expected value of state and control variables. We first solve the LQ mean-field type control problem of the follower using the stochastic maximum principle and obtain the state-feedback representation of the open-loop optimal solution in terms of the coupled integro-Riccati differential equations (CIRDEs) via the Four-Step Scheme. Next, we solve the problem of the leader, which is the LQ control problem subject to the mean-field type forward-backward stochastic system with jump diffusions, where the constraint characterizes the rational behavior of the follower. Using the variational approach, we obtain the (mean-field type) stochastic maximum principle. However, to obtain the state-feedback representation of the open-loop optimal solution of the leader, there is a technical challenge due to the jump process. We consider two different cases, in which the state-feedback type control in terms of the CIRDEs can be characterized by generalizing the Four-Step Scheme. We finally show that the state-feedback type controls of the open-loop optimal solutions for the leader and the follower constitute the Stackelberg equilibrium.</p>

Backward-forward linear-quadratic mean-field Stackelberg games

This paper studies a controlled backward-forward linear-quadratic-Gaussian (LQG) large population system in Stackelberg games. The leader agent is of backward state and follower agents are of forward state. The leader agent is dominating as its state enters those of follower agents. On the other hand, the state-average of all follower agents affects the cost functional of the leader agent. In reality, the leader and the followers may represent two typical types of participants involved in market price formation: the supplier and producers. This differs from standard MFG literature and is mainly due to the Stackelberg structure here. By variational analysis, the consistency condition system can be represented by some fully-coupled backward-forward stochastic differential equations (BFSDEs) with high dimensional block structure in an open-loop sense. Next, we discuss the well-posedness of such a BFSDE system by virtue of the contraction mapping method. Consequently, we obtain the decentralized strategies for the leader and follower agents which are proved to satisfy the ε-Nash equilibrium property.

Linear-quadratic mean field games with a major player: Nash certainty equivalence versus master equations

Mean field games with a major player were introduced in (Huang, 2010) within a linear-quadratic (LQ) modeling framework. Due to the rich structure of major-minor player models, the past ten years have seen significant research efforts for different solution notions and analytical techniques. For LQ models, we address the relation between three solution frameworks: the Nash certainty equivalence (NCE) approach in (Huang, 2010), master equations, and asymptotic solvability, which have been developed starting with different ideas. We establish their equivalence relationships.

Partially-observed decentralized optimal control for large population two-wheeled vehicles: A differential game approach

In this paper, we consider the decentralized noncooperative optimal control problem for a large population of two-wheeled unmanned vehicles with partial observations. Specifically, an individual vehicle is controlled only by its local noisy information measured from a local sensor in order to follow the average behavior (mean field) of the entire population while achieving the overall optimal control performance. We solve the partially-observed linear-quadratic mean field game to obtain decentralized optimal controls for two-wheeled vehicles. These controls are decentralized since they are functions of the local estimated state from the local Kalman filter. We show that the set of the decentralized optimal controls constitutes an ϵ-Nash equilibrium, where ϵ converges to zero as the number of vehicles becomes large. Finally, the theoretical results are validated through simulations and experiments with various operation scenarios for a large population of two-wheeled vehicles.