Mean-field Control Research Articles

Mean field control of droplet dynamics with high-order finite-element computations

Liquid droplet dynamics are widely used in biological and engineering applications, which contain complex interfacial instabilities and pattern formation such as droplet merging, splitting and transport. This paper studies a class of mean field control formulations for these droplet dynamics, which can be used to control and manipulate droplets in applications. We first formulate the droplet dynamics as gradient flows of free energies in modified optimal transport metrics with nonlinear mobilities. We then design an optimal control problem for these gradient flows. As an example, a lubrication equation for a thin volatile liquid film laden with an active suspension is developed, with control achieved through its activity field. Lastly, we apply the primal–dual hybrid gradient algorithm with high-order finite-element methods to simulate the proposed mean field control problems. Numerical examples, including droplet formation, bead-up/spreading, transport, and merging/splitting on a two-dimensional spatial domain, demonstrate the effectiveness of the proposed mean field control mechanism.

Numerical Analysis of the Projection Dynamics and Their Associated Mean Field Control

Numerical Analysis of the Projection Dynamics and Their Associated Mean Field Control

On optimal control of coupled mean-field forward-backward stochastic equations

Abstract We consider a control problem for a mean-field coupled forward-backward stochastic differential equations, called also McKean–Vlasov equation (MF-FBSDE). For this type of equations, the coefficients depend not only on the state of the system, but also on its marginal distributions. They arise naturally in mean-field control problems and mean-field games. We consider the relaxed control problem where admissible controls are measure-valued processes. We prove the existence of a relaxed optimal control by using a suitable form of Skorokhod representation theorem and Jakubowski’s topology, on the space of càdlàg functions. We use martingale measure to define the relaxed state process. Our results extend to MF-FBSDEs those already known for forward and backward stochastic equations of Itô type.

On the optimal rate for the convergence problem in mean field control

The goal of this work is to obtain (nearly) optimal rates for the convergence problem in mean field control. Our analysis covers cases where the solutions to the limiting problem may not be unique nor stable. Equivalently the value function of the limiting problem might not be differentiable on the entire space. Our main result is then to derive sharp rates of convergence in two distinct regimes. First, when the data is sufficiently regular, we obtain rates proportional to N−1/2, with N being the number of particles, and we verify that N−1/2 is indeed optimal in this setting. Second, when the data is merely Lipschitz and semi-concave with respect to the first Wasserstein distance, we obtain rates proportional to N−2/(3d+6). We do not expect this second estimate to be optimal, but it improves substantially on the existing literature. Moreover, we construct an example showing that the optimal rate is no faster than N−1/d, and we conjecture that the optimal rate should indeed be exactly N−1/d (at least when d≥3). The key argument in our approach consists in mollifying the value function of the limiting problem in order to produce functions that are almost classical sub-solutions to the limiting Hamilton-Jacobi equation (which is a PDE set on the space of probability measures). These sub-solutions can be projected onto finite dimensional spaces and then compared with the value functions associated with the particle systems. In the end, this comparison is used to prove the most demanding bound in the estimates. The key challenge therein is thus to exhibit an appropriate form of mollification. We do so by employing sup-convolution within a convenient functional Hilbert space. To make the whole easier, we limit ourselves to the periodic setting. We also provide some examples to show that our results are sharp up to some extent.

An optimal advertising model with carryover effect and mean field terms

We consider a class of optimal advertising problems under uncertainty for the introduction of a new product into the market, on the line of the seminal papers of Vidale and Wolfe (Oper Res 5:370–381, 1957) and Nerlove and Arrow (Economica 29:129–142, 1962). The main features of our model are that, on one side, we assume a carryover effect (i.e. the advertisement spending affects the goodwill with some delay); on the other side we introduce, in the state equation and in the objective, some mean field terms that take into account the presence of other agents. We take the point of view of a planner who optimizes the average profit of all agents, hence we fall into the family of the so-called “Mean Field Control” problems. The simultaneous presence of the carryover effect makes the problem infinite dimensional hence belonging to a family of problems which are very difficult in general and whose study started only very recently, see Cosso et al. [Ann Appl Probab 33(4):2863–2918, 2023]. Here we consider, as a first step, a simple version of the problem providing the solutions in a simple case through a suitable auxiliary problem.

Correlated Equilibria for Mean Field Games with Progressive Strategies

In a discrete space and time framework, we study the mean field game limit for a class of symmetric N-player games based on the notion of correlated equilibrium. We give a definition of correlated solution that allows us to construct approximate N-player correlated equilibria that are robust with respect to progressive deviations. We illustrate our definition by way of an example with explicit solutions. Funding: O. Bonesini acknowledges financial support from Engineering and Physical Sciences Research Council [Grant EP/T032146/1]. M. Fischer acknowledges partial support through the University of Padua [Research Project BIRD229791 “Stochastic mean field control and the Schrödinger problem”].

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

The goal of this paper is to prove a comparison principle for viscosity solutions of semilinear Hamilton–Jacobi equations in the space of probability measures. The method involves leveraging differentiability properties of the 2-Wasserstein distance in the doubling of variables argument, which is done by introducing a further entropy penalization that ensures that the relevant optima are achieved at positive, Lipschitz continuous densities with finite Fischer information. This allows to prove uniqueness and stability of viscosity solutions in the class of bounded Lipschitz continuous (with respect to the 1-Wasserstein distance) functions. The result does not appeal to a mean field control formulation of the equation, and, as such, applies to equations with nonconvex Hamiltonians and measure-dependent volatility. For convex Hamiltonians that derive from a potential, we prove that the value function associated with a suitable mean-field optimal control problem with nondegenerate idiosyncratic noise is indeed the unique viscosity solution.

A Global Optimality Principle for Fully Coupled Mean-field Control Systems

A Global Optimality Principle for Fully Coupled Mean-field Control Systems

Special issue on “Mean Field Games and Mean Field Control”

Special issue on “Mean Field Games and Mean Field Control”

From NeurODEs to AutoencODEs: A mean-field control framework for width-varying neural networks

Abstract The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks, which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modelling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularisation, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularisation may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples.

Deep learning for mean field optimal transport

Mean field control (MFC) problems have been introduced to study social optima in very large populations of strategic agents. The main idea is to consider an infinite population and to simplify the analysis by using a mean field approximation. These problems can also be viewed as optimal control problems for McKean-Vlasov dynamics. They have found applications in a wide range of fields, from economics and finance to social sciences and engineering. Usually, the goal for the agents is to minimize a total cost which consists in the integral of a running cost plus a terminal cost. In this work, we consider MFC problems in which there is no terminal cost but, instead, the terminal distribution is prescribed as in optimal transport problem. By analogy with MFC, we call such problems mean field optimal transport problems (or MFOT for short) since they can be viewed as a generalization of classical optimal transport problems when mean field interactions occur in the dynamics or the running cost function. We propose three numerical methods based on neural networks. The first one is based on directly learning an optimal control. The second one amounts to solve a forward-backward PDE system characterizing the solution. The third one relies on a primal-dual approach. We illustrate these methods with numerical experiments conducted on two families of examples.

A decentralized algorithm for a mean field control problem of piecewise deterministic Markov processes

This paper provides a decentralized approach for the control of a population of N agents to minimize an aggregate cost. Each agent evolves independently according to a Piecewise Deterministic Markov dynamics controlled via unbounded jumps intensities. The N-agent high dimensional stochastic control problem is approximated by the limiting mean field control problem. A Lagrangian approach is proposed. Although the mean field control problem is not convex, it is proved to achieve zero duality gap. A stochastic version of the Uzawa algorithm is shown to converge to the primal solution. At each dual iteration of the algorithm, each agent solves its own small dimensional sub problem by means of the Dynamic Programming Principal, while the dual multiplier is updated according to the aggregate response of the agents. Finally, this algorithm is used in a numerical simulation to coordinate the charging of a large fleet of electric vehicles in order to track a target consumption profile.

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems

On mean-field control problems for backward doubly stochastic systems

This article is concerned with stochastic control problems for backward doubly stochastic differential equations of mean-field type, where the coefficient functions depend on the joint distribution of the state process and the control process. We obtain the stochastic maximum principle which serves as a necessary condition for an optimal control, and we also prove its sufficiency under proper conditions. As a byproduct, we prove the well-posedness for a type of mean-field fully coupled forward-backward doubly stochastic differential equation arising naturally from the control problem, which is of interest in its own right. Some examples are provided to illustrate the applications of our results to control problems in the types of scalar interaction and first order interaction.

Master Bellman equation in the Wasserstein space: Uniqueness of viscosity solutions

We study the Bellman equation in the Wasserstein space arising in the study of mean field control problems, namely stochastic optimal control problems for McKean-Vlasov diffusion processes. Using the standard notion of viscosity solution à la Crandall-Lions extended to our Wasserstein setting, we prove a comparison result under general conditions on the drift and reward coefficients, which coupled with the dynamic programming principle, implies that the value function is the unique viscosity solution of the Master Bellman equation. This is the first uniqueness result in such a second-order context. The classical arguments used in the standard cases of equations in finite-dimensional spaces or in infinite-dimensional separable Hilbert spaces do not extend to the present framework, due to the awkward nature of the underlying Wasserstein space. The adopted strategy is based on finite-dimensional approximations of the value function obtained in terms of the related cooperative n n -player game, and on the construction of a smooth gauge-type function, built starting from a regularization of a sharp estimate of the Wasserstein metric; such a gauge-type function is used to generate maxima/minima through a suitable extension of the Borwein-Preiss generalization of Ekeland’s variational principle on the Wasserstein space.

In-context operator learning with data prompts for differential equation problems

This paper introduces the paradigm of "in-context operator learning" and the corresponding model "In-Context Operator Networks" to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, rather than a solution/operator approximator, we can not only get rid of retraining (even fine-tuning) the neural network for new problems but also leverage the commonalities shared across operators so that only a few examples in the prompt are needed when learning a new operator. Our numerical results show the capability of a single neural network as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations, partial differential equations, and mean-field control problems, and also show that it can generalize its learning capability to operators beyond the training distribution.

A posteriori error estimates for fully coupled McKean–Vlasov forward-backward SDEs

Abstract Fully coupled McKean–Vlasov forward-backward stochastic differential equations (MV-FBSDEs) arise naturally from large population optimization problems. Judging the quality of given numerical solutions for MV-FBSDEs, which usually require Picard iterations and approximations of nested conditional expectations, is typically difficult. This paper proposes an a posteriori error estimator to quantify the $L^2$-approximation error of an arbitrarily generated approximation on a time grid. We establish that the error estimator is equivalent to the global approximation error between the given numerical solution and the solution of a forward Euler discretized MV-FBSDE. A crucial and challenging step in the analysis is the proof of stability of this Euler approximation to the MV-FBSDE, which is of independent interest. We further demonstrate that, for sufficiently fine time grids, the accuracy of numerical solutions for solving the continuous MV-FBSDE can also be measured by the error estimator. The error estimates justify the use of residual-based algorithms for solving MV-FBSDEs. Numerical experiments for MV-FBSDEs arising from mean field control and games confirm the effectiveness and practical applicability of the error estimator.

From Optimal Control to Mean Field Optimal Transport via Stochastic Neural Networks

In this paper, we derive a unified perspective for Optimal Transport (OT) and Mean Field Control (MFC) theories to analyse the learning process for Neural Network algorithms in a high-dimensional framework. We consider a Mean Field Neural Network in the context of MFC theory referring to the mean field formulation of OT theory that may allow the development of efficient algorithms in a high-dimensional framework while providing a powerful tool in the context of explainable Artificial Intelligence.

Graphon mean-field control for cooperative multi-agent reinforcement learning

The marriage between mean-field theory and reinforcement learning has shown a great capacity to solve large-scale control problems with homogeneous agents. To break the homogeneity restriction of mean-field theory, a recent interest is to introduce graphon theory to the mean-field paradigm. In this paper, we propose a graphon mean-field control (GMFC) framework to approximate cooperative heterogeneous multi-agent reinforcement learning (MARL) with nonuniform interactions and heterogeneous reward functions and state transition functions among agents and show that the approximate order is of O(1N), with N the number of agents. By discretizing the graphon index of GMFC, we further introduce a smaller class of GMFC called block GMFC, which is shown to well approximate cooperative MARL in terms of the value function and the policy. Finally, we design a Proximal Policy Optimization based algorithm for block GMFC that converges to the optimal policy of cooperative MARL. Our empirical studies on several examples demonstrate that our GMFC approach is comparable with the state-of-art MARL algorithms while enjoying better scalability.

A stochastic maximum principle for partially observed general mean-field control problems with only weak solution

In this paper we focus on a general type of mean-field stochastic control problem with partial observation, in which the coefficients depend in a non-linear way not only on the state process Xt and its control ut but also on the conditional law E[Xt|FtY] of the state process conditioned with respect to the past of observation process Y. We first deduce the well-posedness of the controlled system by showing weak existence and uniqueness in law. Neither supposing convexity of the control state space nor differentiability of the coefficients with respect to the control variable, we study Peng’s stochastic maximum principle for our control problem. The novelty and the difficulty of our work stem from the fact that, given an admissible control u, the solution of the associated control problem is only a weak one. This has as consequence that also the probability measure in the solution Pu=LTuQ depends on u and has a density LTu with respect to a reference measure Q. So characterizing an optimal control leads to the differentiation of non-linear functions f(Pu∘{EPu[Xt|FtY]}−1) with respect to (LTu,Xt). This has as consequence for the study of Peng’s maximum principle that we get a new type of first and second order variational equations and adjoint backward stochastic differential equations, all with new mean-field terms and with coefficients which are not Lipschitz. For their estimates and for those for the Taylor expansion new techniques have had to be introduced and rather technical results have had to be established. The necessary optimality condition we get extends Peng’s one with new, non-trivial terms.