Multi-agent Setting Research Articles

A dynamical systems approach to optimal foraging

Foraging for resources in an environment is a fundamental activity that must be addressed by any biological agent. Modelling this phenomenon in simulations can enhance our understanding of the characteristics of natural intelligence. In this work, we present a novel approach to model foraging in-silico using a continuous coupled dynamical system. The dynamical system is composed of three differential equations, representing the position of the agent, the agent’s control policy, and the environmental resource dynamics. Crucially, the control policy is implemented as a parameterized differential equation which allows the control policy to adapt in order to solve the foraging task. Using this setup, we show that when these dynamics are coupled and the controller parameters are optimized to maximize the rate of reward collected, adaptive foraging emerges in the agent. We further show that the internal dynamics of the controller, as a surrogate brain model, closely resemble the dynamics of the evidence accumulation mechanism, which may be used by certain neurons of the dorsal anterior cingulate cortex region in non-human primates, for deciding when to migrate from one patch to another. We show that by modulating the resource growth rates of the environment, the emergent behaviour of the artificial agent agrees with the predictions of the optimal foraging theory. Finally, we demonstrate how the framework can be extended to stochastic and multi-agent settings.

The SocialAI school: a framework leveraging developmental psychology toward artificial socio-cultural agents.

Developmental psychologists have long-established socio-cognitive abilities as fundamental to human intelligence and development. These abilities enable individuals to enter, learn from, and contribute to a surrounding culture. This drives the process of cumulative cultural evolution, which is responsible for humanity's most remarkable achievements. AI research on social interactive agents mostly concerns the emergence of culture in a multi-agent setting (often without a strong grounding in developmental psychology). We argue that AI research should be informed by psychology and study socio-cognitive abilities enabling to enter a culture as well. We draw inspiration from the work of Michael Tomasello and Jerome Bruner, who studied socio-cognitive development and emphasized the influence of a cultural environment on intelligence. We outline a broader set of concepts than those currently studied in AI to provide a foundation for research in artificial social intelligence. Those concepts include social cognition (joint attention, perspective taking), communication, social learning, formats, and scaffolding. To facilitate research in this domain, we present The SocialAI school-a tool that offers a customizable parameterized suite of procedurally generated environments. This tool simplifies experimentation with the introduced concepts. Additionally, these environments can be used both with multimodal RL agents, or with pure-text Large Language Models (LLMs) as interactive agents. Through a series of case studies, we demonstrate the versatility of the SocialAI school for studying both RL and LLM-based agents. Our motivation is to engage the AI community around social intelligence informed by developmental psychology, and to provide a user-friendly resource and tool for initial investigations in this direction. Refer to the project website for code and additional resources: https://sites.google.com/view/socialai-school.

Does Supply Concentration Encourage Cooperation? Evidence from Airlines

One of a firm’s key strategic decisions is whether to concentrate its input purchases in a small number of suppliers versus spreading them among many suppliers. We propose that supplier concentration solves an interfirm free-riding problem: By internalizing externalities between suppliers, it incentivizes buyer-supplier cooperation. Guided by a simple theoretical model, we investigate this hypothesis on slot exchanges between major airlines and their outsourced regional airline partners during inclement weather, a setting where externalities between suppliers are ubiquitous. We find robust evidence that a regional airline engages in more frequent slot exchanges with its major airline partner when it operates a larger share of the major’s outsourced flights. We also find that this positive effect of concentration on mutual cooperation increases in the size of the externalities between regionals. Our results suggest that, in contrast with Porter’s classic five forces framework, supplier concentration can serve as a governance instrument for buyer-supplier collaborations. More broadly, our paper provides novel evidence on task concentration as a tool to solve free-riding problems in multiagent settings. This paper was accepted by Joshua Gans, business strategy. Funding: This paper received financial support from the Spanish Research Agency (MICIU/AEI) [Grants PID2022-136983NB-I00, MCIN/AEI /10.13039/501100011033]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.03455 .

Costly information providing in binary contests

Contests are commonly used as a mechanism for eliciting effort and participation in multi-agent settings. Naturally, and much like with various other mechanisms, the information provided to the agents prior to and throughout the contest fundamentally influences its outcomes. In this paper we study the problem of information providing whenever the contest organizer does not initially hold the information and obtaining it is potentially costly. As the underlying contest mechanism for our model we use the binary contest, where contestants’ strategy is captured by their decision whether or not to participate in the contest in the first place. Here, it is often the case that the contest organizer can proactively obtain and provide contestants information related to their expected performance in the contest. We provide a comprehensive equilibrium analysis of the model, showing that even when such information is costless, it is not necessarily the case that the contest organizer will prefer to obtain and provide it to all agents, let alone when the information is costly.

Multi-agent reinforcement learning via distributed MPC as a function approximator

This paper presents a novel approach to multi-agent reinforcement learning (RL) for linear systems with convex polytopic constraints. Existing work on RL has demonstrated the use of model predictive control (MPC) as a function approximator for the policy and value functions. The current paper is the first work to extend this idea to the multi-agent setting. We propose the use of a distributed MPC scheme as a function approximator, with a structure allowing for distributed learning and deployment. We then show that Q-learning updates can be performed distributively without introducing nonstationarity, by reconstructing a centralized learning update. The effectiveness of the approach is demonstrated on a numerical example.

Depthwise Convolution for Multi-Agent Communication With Enhanced Mean-Field Approximation.

Multi-Agent settings remain a fundamental challenge in the reinforcement learning (RL) domain due to the partial observability and the lack of accurate real-time interactions across agents. In this article, we propose a new method based on local communication learning to tackle the multi-agent RL (MARL) challenge within a large number of agents coexisting. First, we design a new communication protocol that exploits the ability of depthwise convolution to efficiently extract local relations and learn local communication between neighboring agents. To facilitate multi-agent coordination, we explicitly learn the effect of joint actions by taking the policies of neighboring agents as inputs. Second, we introduce the mean-field approximation into our method to reduce the scale of agent interactions. To more effectively coordinate behaviors of neighboring agents, we enhance the mean-field approximation by a supervised policy rectification network (PRN) for rectifying real-time agent interactions and by a learnable compensation term for correcting the approximation bias. The proposed method enables efficient coordination as well as outperforms several baseline approaches on the adaptive traffic signal control (ATSC) task and the StarCraft II multi-agent challenge (SMAC).

Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback

Feasible Online Learning with Gradient Feedback Online gradient descent (OGD) is well-known to be doubly optimal under strong convexity or monotonicity assumptions: (1) in the single-agent setting, it achieves an optimal regret of [Formula: see text] for strongly convex cost functions, and (2) in the multiagent setting of strongly monotone games with each agent employing OGD we obtain last-iterate convergence of the joint action to a unique Nash equilibrium at an optimal rate of [Formula: see text]. Whereas these finite-time guarantees highlight its merits, OGD has the drawback that it requires knowing the strong convexity/monotonicity parameters. In “Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback,” M. Jordan, T. Lin, and Z. Zhou design a fully adaptive OGD algorithm, AdaOGD, that does not require a priori knowledge of these parameters. In the single-agent setting, the algorithm achieves [Formula: see text] regret under strong convexity, which is optimal up to a log factor. Further, if each agent employs AdaOGD in strongly monotone games, the joint action converges in a last-iterate sense to a unique Nash equilibrium at a rate of [Formula: see text], again optimal up to log factors. The algorithms are illustrated in a learning version of the classic newsvendor problem, in which, because of lost sales, only (noisy) gradient feedback can be observed. The results immediately yield the first feasible and near-optimal algorithm for both the single-retailer and multiretailer settings.

Initial Goal Allocation for Multi-agent Systems

In the multi-agent environment, a human expert engages inthe allocation of objectives among individual agents. How-ever, autonomous agents need to determine and allocate ob-jectives without external intervention from humans. There-fore, in this research, we attempt to solve initial goal distri-bution challenges in multi-agent settings by developing twogoal allocation algorithms. The primary objectives are to findcost-effective goal solution sets and distribute them evenlyamong available agents. We introduce two algorithms whenthe goals are structured in a hierarchical goal tree structure,and then test their efficiency across a variety of baseline al-location methods. Both algorithms were able to increase theperformance of agents in multi-agent settings by finding themost optimal distributions of goals and allowing agents to actindependently from human intervention.

Adaptive mean field multi-agent reinforcement learning

Large-scale Multi-Agent Reinforcement Learning (MARL) is fundamentally a challenge due to the curse of dimensionality. In a homogeneous multi-agent setting, mean field theory gives an effective way of scalable MARL by abstracting other agents to a virtual mean agent, assuming that the influence between agents is equal and infinitesimal. However, in some real scenarios, only several neighboring agents, rather than all agents, affect the decision-making of an agent, and different neighboring agents may have varying degrees of influence on the agent's decision-making. In this paper, not restricted to a homogeneous setting, we propose adaptive mean field MARL, which is based on the attention mechanism and can be used to deal with many-agent scenarios where there may be different influence relationships among agents. Specifically, we first derive the mean field approximation with adaptive weight and give the error bound of the approximation. Then, we propose adaptive mean field Q-Learning and describe how to obtain the adaptive weight. In addition, we discuss the differences between the proposed approach and existing mean-field MARL methods. Finally, we conduct experiments on simulation platforms, and the results show that the performance of the proposed approach outperforms that of the state-of-the-art method.

Finite-Time Frequentist Regret Bounds of Multi-Agent Thompson Sampling on Sparse Hypergraphs

We study the multi-agent multi-armed bandit (MAMAB) problem, where agents are factored into overlapping groups. Each group represents a hyperedge, forming a hypergraph over the agents. At each round of interaction, the learner pulls a joint arm (composed of individual arms for each agent) and receives a reward according to the hypergraph structure. Specifically, we assume there is a local reward for each hyperedge, and the reward of the joint arm is the sum of these local rewards. Previous work introduced the multi-agent Thompson sampling (MATS) algorithm and derived a Bayesian regret bound. However, it remains an open problem how to derive a frequentist regret bound for Thompson sampling in this multi-agent setting. To address these issues, we propose an efficient variant of MATS, the epsilon-exploring Multi-Agent Thompson Sampling (eps-MATS) algorithm, which performs MATS exploration with probability epsilon while adopts a greedy policy otherwise. We prove that eps-MATS achieves a worst-case frequentist regret bound that is sublinear in both the time horizon and the local arm size. We also derive a lower bound for this setting, which implies our frequentist regret upper bound is optimal up to constant and logarithm terms, when the hypergraph is sufficiently sparse. Thorough experiments on standard MAMAB problems demonstrate the superior performance and the improved computational efficiency of eps-MATS compared with existing algorithms in the same setting.

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

Policy gradient methods enjoy strong practical performance in numerous tasks in reinforcement learning. Their theoretical understanding in multiagent settings, however, remains limited, especially beyond two-player competitive and potential Markov games. In this paper, we develop a new framework to characterize optimistic policy gradient methods in multi-player Markov games with a single controller. Specifically, under the further assumption that the game exhibits an equilibrium collapse, in that the marginals of coarse correlated equilibria (CCE) induce Nash equilibria (NE), we show convergence to stationary epsilon-NE in O(1/epsilon^2) iterations, where O suppresses polynomial factors in the natural parameters of the game. Such an equilibrium collapse is well-known to manifest itself in two-player zero-sum Markov games, but also occurs even in a class of multi-player Markov games with separable interactions, as established by recent work. As a result, we bypass known complexity barriers for computing stationary NE when either of our assumptions fails. Our approach relies on a natural generalization of the classical Minty property that we introduce, which we anticipate to have further applications beyond Markov games.

Factored Online Planning in Many-Agent POMDPs

In centralized multi-agent systems, often modeled as multi-agent partially observable Markov decision processes (MPOMDPs), the action and observation spaces grow exponentially with the number of agents, making the value and belief estimation of single-agent online planning ineffective. Prior work partially tackles value estimation by exploiting the inherent structure of multi-agent settings via so-called coordination graphs. Additionally, belief estimation methods have been improved by incorporating the likelihood of observations into the approximation. However, the challenges of value estimation and belief estimation have only been tackled individually, which prevents existing methods from scaling to settings with many agents. Therefore, we address these challenges simultaneously. First, we introduce weighted particle filtering to a sample-based online planner for MPOMDPs. Second, we present a scalable approximation of the belief. Third, we bring an approach that exploits the typical locality of agent interactions to novel online planning algorithms for MPOMDPs operating on a so-called sparse particle filter tree. Our experimental evaluation against several state-of-the-art baselines shows that our methods (1) are competitive in settings with only a few agents and (2) improve over the baselines in the presence of many agents.

A Survey of Deep Reinforcement Learning Based on Multi-Particle Environments

This paper extensively explores the application of deep reinforcement learning within multi-agent environments, a domain marked by intricate challenges and burgeoning opportunities. Multi-agent environments inherently entail the presence of multiple interacting intelligent agents, thereby amplifying the complexity of decision-making and control. Traditional reinforcement learning methods, when confronted with these intricate settings, manifest limitations that have prompted a compelling shift towards the fusion of deep learning and reinforcement learning. The journey commences by elucidating the multifaceted challenges and intricacies pervasive in multi-agent environments. It systematically delineates the constraints of conventional reinforcement learning techniques, underscoring their inadequacies in effectively addressing the multifarious complexities presented by these environments. As the narrative unfolds, it places a spotlight on the transformative potential of deep learning in surmounting the formidable challenges associated with decision-making in multi-agent settings. A critical examination ensues, encompassing a thorough review of pioneering applications of methodologies such as value functions, policy gradients, and actor-critic approaches within the realm of multi-agent systems. Through an exhaustive exploration of existing literature, this article endeavors to unveil the vanguard developments and persisting challenges within the domain of deep reinforcement learning in multi-agent environments. Furthermore, this paper conducts a comparative analysis of the performance of two prominent algorithms, namely Q-learning and the Deep Q Network (DQN) algorithm. This analysis serves as a compass, elucidating the trajectory of development within the sphere of deep reinforcement learning. Ultimately, this paper acts as a compass, guiding future research and innovation in the dynamic and promising field of deep reinforcement learning within multi-agent environments.

A Comprehensive Analysis of Game theory on Multi-Agent Reinforcement

In recent years, reinforcement learning has gradually emerged as a popular research field in artificial intelligence. However, for multi-agent systems, due to their complex environment and an abundance of agents, the learning target of maximizing the anticipated value of cumulative rewards for specific agents frequently fails to converge. Introducing game theory into reinforcement learning can effectively address the interactions among intelligent agents, provide a rationale for convergence points corresponding to strategies. To address the issue of non-existence of optimal solutions for certain tasks in multi-agent settings, this paper takes a game-theoretic perspective and summarizes classical reinforcement learning algorithms developed in recent years. The basic theory of multi-agent reinforcement learning, essential game theory, categorization of reinforcement learning using multiple agents game strategies, primary worries are addressed in the study. It also analyzes the challenges that game-theoretic multi-agent reinforcement learning algorithms may encounter in the future, along with the relevant optimization directions.

Residual Q-Networks for Value Function Factorizing in Multiagent Reinforcement Learning.

Multiagent reinforcement learning (MARL) is useful in many problems that require the cooperation and coordination of multiple agents. Learning optimal policies using reinforcement learning in a multiagent setting can be very difficult as the number of agents increases. Recent solutions such as value decomposition networks (VDNs), QMIX, QTRAN, and QPLEX adhere to the centralized training and decentralized execution (CTDE) scheme and perform factorization of the joint action-value functions. However, these methods still suffer from increased environmental complexity, and at times fail to converge in a stable manner. We propose a novel concept of residual Q-networks (RQNs) for MARL, which learns to transform the individual Q -value trajectories in a way that preserves the individual-global-max (IGM) criteria, but is more robust in factorizing action-value functions. The RQN acts as an auxiliary network that accelerates convergence and will become obsolete as the agents reach the training objectives. The performance of the proposed method is compared against several state-of-the-art techniques such as QPLEX, QMIX, QTRAN, and VDN, in a range of multiagent cooperative tasks. The results illustrate that the proposed method, in general, converges faster, with increased stability, and shows robust performance in a wider family of environments. The improvements in results are more prominent in environments with severe punishments for noncooperative behaviors and especially in the absence of complete state information during training time.

MARLens: Understanding Multi-Agent Reinforcement Learning for Traffic Signal Control Via Visual Analytics.

The issue of traffic congestion poses a significant obstacle to the development of global cities. One promising solution to tackle this problem is intelligent traffic signal control (TSC). Recently, TSC strategies leveraging reinforcement learning (RL) have garnered attention among researchers. However, the evaluation of these models has primarily relied on fixed metrics like reward and queue length. This limited evaluation approach provides only a narrow view of the model's decision-making process, impeding its practical implementation. Moreover, effective TSC necessitates coordinated actions across multiple intersections. Existing visual analysis solutions fall short when applied in multi-agent settings. In this study, we delve into the challenge of interpretability in multi-agent reinforcement learning (MARL), particularly within the context of TSC. We propose MARLens, a visual analytics system tailored to understand MARL-based TSC. Our system serves as a versatile platform for both RL and TSC researchers. It empowers them to explore the model's features from various perspectives, revealing its decision-making processes and shedding light on interactions among different agents. To facilitate quick identification of critical states, we have devised multiple visualization views, complemented by a traffic simulation module that allows users to replay specific training scenarios. To validate the utility of our proposed system, we present three comprehensive case studies, incorporate insights from domain experts through interviews, and conduct a user study. These collective efforts underscore the feasibility and effectiveness of MARLens in enhancing our understanding of MARL-based TSC systems and pave the way for more informed and efficient traffic management strategies.

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Deep reinforcement learning (DRL) faces significant challenges in addressing hard-exploration tasks with sparse or deceptive rewards and large state spaces. These challenges severely limit the practical application of DRL. Most previous exploration methods relied on complex architectures to estimate state novelty or introduced sensitive hyperparameters, resulting in instability. To mitigate these issues, we propose an efficient adaptive trajectory-constrained exploration strategy for DRL. The proposed method guides the agent’s policy away from suboptimal solutions by regarding previous offline demonstrations as references. Specifically, this approach gradually expands the exploration scope of the agent and strives for optimality in a constrained optimization manner. Additionally, we introduce a novel policy-gradient-based optimization algorithm that utilizes adaptive clipped trajectory-distance rewards for both single- and multi-agent reinforcement learning. We provide a theoretical analysis of our method, including a deduction of the worst-case approximation error bounds, highlighting the validity of our approach for enhancing exploration. To evaluate the effectiveness of the proposed method, we conducted experiments on two large 2D grid world mazes and several MuJoCo tasks. The extensive experimental results demonstrated the significant advantages of our method in achieving temporally extended exploration and avoiding myopic and suboptimal behaviors in both single- and multi-agent settings. Notably, the specific metrics and quantifiable results further support these findings. The code used in the study is available at https://github.com/buaawgj/TACE.

Maximizing Local Rewards on Multi-Agent Quantum Games through Gradient-Based Learning Strategies.

This article delves into the complex world of quantum games in multi-agent settings, proposing a model wherein agents utilize gradient-based strategies to optimize local rewards. A learning model is introduced to focus on the learning efficacy of agents in various games and the impact of quantum circuit noise on the performance of the algorithm. The research uncovers a non-trivial relationship between quantum circuit noise and algorithm performance. While generally an increase in quantum noise leads to performance decline, we show that low noise can unexpectedly enhance performance in games with large numbers of agents under some specific circumstances. This insight not only bears theoretical interest, but also might have practical implications given the inherent limitations of contemporary noisy intermediate-scale quantum (NISQ) computers. The results presented in this paper offer new perspectives on quantum games and enrich our understanding of the interplay between multi-agent learning and quantum computation. Both challenges and opportunities are highlighted, suggesting promising directions for future research in the intersection of quantum computing, game theory and reinforcement learning.

MARNet: Backdoor Attacks Against Cooperative Multi-Agent Reinforcement Learning

Recent works have revealed that backdoor attacks against Deep Reinforcement Learning (DRL) could lead to abnormal action selections of the agent, which may result in failure or even catastrophe in crucial decision processes. However, existing attacks only consider single-agent RL systems, in which the only agent can observe the global state and have full control of the decision process. In this paper, we explore a new backdoor attack paradigm in cooperative multi-agent reinforcement learning (CMARL) scenarios, where a group of agents coordinate with each other to achieve a common goal, while each agent can only observe the local state. In the proposed MARNet attack framework, we carefully design a pipeline of trigger design, action poisoning, and reward hacking modules to accommodate the cooperative multi-agent settings. In particular, as only a subset of agents can observe the triggers in their local observations, we maneuver their actions to the worst actions suggested by an expert policy model. Since the global reward in CMARL is aggregated by individual rewards from all agents, we propose to modify the reward in a way that boosts the bad actions of poisoned agents (agents who observe the triggers) but mitigates the influence on non-poisoned agents. We conduct extensive experiments on three classical CMARL algorithms VDN, COMA, and QMIX, in two popular CMARL games Predator Prey and SMAC. The results show that the baselines extended from single-agent DRL backdoor attacks seldom work in CMARL problems while MARNet performs well by reducing the utility under attack by nearly 100%. We apply fine-tuning as a potential defense against MARNet and demonstrate that fine-tuning cannot entirely eliminate the effect of the attack.

Deep reinforcement learning in service of air traffic controllers to resolve tactical conflicts

Dense and complex air traffic requires higher levels of automation than those exhibited by tactical conflict detection and resolution (CD&R) tools that air traffic controllers (ATCOs) use today: AI tools can act on their own initiative, increasing the capacity of ATCOs to control higher volumes of traffic. However, given that the air traffic control (ATC) domain is safety critical, requires AI systems to which ATCOs are comfortable to relinquishing control, guaranteeing operational integrity and automation adoption. Two major factors towards this goal are quality of solutions and operational transparency. ResoLver, the system that this article presents, addresses these challenges using an enhanced graph convolutional reinforcement learning method operating in a multiagent setting where each agent – representing a flight – performs a CD&R task, jointly with other agents. We show that ResoLver can provide high-quality solutions with respect to stakeholders interests (air traffic controllers and airspace users), addressing also operational transparency issues, which have been validated by ATCOs in simulated real-world settings.