Stochastic learning in multi-agent optimization: Communication and payoff-based approaches

Abstract

Game theory serves as a powerful tool for distributed optimization in multi-agent systems in different applications. In this paper we consider multi-agent systems that can be modeled by means of potential games whose potential function coincides with a global objective function to be maximized. In this approach, the agents correspond to the strategic decision makers and the optimization problem is equivalent to the problem of learning a potential function maximizer in the designed game. The paper deals with two different information settings in the system. Firstly, we consider systems, where agents have the access to the gradient of their utility functions. However, they do not possess the full information about the joint actions. Thus, to be able to move along the gradient toward a local optimum, they need to exchange the information with their neighbors by means of communication. The second setting refers to a payoff-based approach. Here, we assume that at each iteration agents can only observe their own played actions and experienced payoffs. In both cases, the paper studies unconstrained non-concave optimization with a differentiable objective function. To develop the corresponding algorithms guaranteeing convergence to a local maximum of the potential function in absence of saddle points, we utilize the idea of the well-known Robbins–Monro procedure based on the theory of stochastic approximation.