Multi-agent Teams Research Articles

The Harmony Index: Evaluating, Predicting, and Visualizing Effectiveness in Multi-Agent Team Dynamics

Team-based games are a keystone pillar of the gaming industry. Sadly, the understanding of team dynamics—and the recommendations for both human and AI-based teammates—are based on a rudimentary understanding of human-AI teaming. We propose a superior metric, which provides information about team effectiveness in an efficient and easily-replicable manner. Without an accurate and effective metric for team evaluation, it is nigh-impossible to provide feedback to players and game designers to improve team balance. We provide such a metric. The Harmony Index, a novel algorithm using real-world data, provides simpler and more accurate actionable directives to improve game design across MOBAs and other game genres. We prove its predictive power in a separate analysis and make recommendations for its use in assessing team effectiveness as well as its future use in additional domains.

Abstract 4138722: Large Language Model based multi-agent Transcatheter Aortic Valve Implantation team to augment multidisciplinary meetings - proof of concept.

Introduction: Multidisciplinary team (MDT) discussions are integral to Transcatheter Aortic Valve Implantation (TAVI) decision making. Large language model (LLM) ubiquity and low-code no-code platforms have enabled clinician lead solution development. Specialised chatbots or ‘agents’ have evolved into multi-agent systems that can personify human collaboration. We assess the performance of an artificial intelligence (AI) multi-agent TAVI MDT. Methods: Four de-identified TAVI cases from two metropolitan Australian hospitals were assessed by a mock human TAVI MDT (h-MDT) and an AI multi-agent TAVI MDT (ai-MDT). The ai-MDT was created with Agentflow within Flowise AITM and had a hierarchical multi-agent architecture suited to complex reasoning required for TAVI MDT simulation (figure). LLM limitations necessitated the ai-MDT rely on imaging reports rather than clinical images. The h-MDT and ai-MDT consisted of similar team members. Outputs from the h-MDT and ai-MDT was adjudicated by a panel of four blinded TAVI doctors that determined if output was human vs AI and assigned a SMIC score (4-12, 4=good, 12=poor) that assessed structure, missing information, incorrect information and clinical utility. Time durations for h-MDT and ai-MDT were recorded. Results: Adjudicators differentiated human vs. AI output 100% of the time and ai-MDT output had better SMIC scores than h-MDT (U-stat 213, p=0.0011). ai-MDT outperformed h-MDT in the domains of structure, missing information and clinical utility but was not statistically different in the incorrect information domain (U-stat 132, p=0.88). The average time for each case in h-MDT was 15 minutes and 45 seconds compared to 97 seconds for ai-MDT. Conclusion: This demonstrates the potential of using LLM based multi-agent systems as a clinical adjunct in highly specialized multidisciplinary clinical meetings. AI responses were superior for structure, clinical utility and missing information and non-inferior for incorrect information compared to humans, which highlights that hallucinations remain an issue with generative AI. Time was saved but image interpretation still requires human input, for now. Cognitive AI continues to require human supervision for implementation.

Thematic Analysis of Interviews With U.S. Space Force Guardians on Multi-Agent System Teams in Space

Thematic Analysis of Interviews With U.S. Space Force Guardians on Multi-Agent System Teams in Space

Unveiling Multi-Agent Strategies: A Data-Driven Approach for Extracting and Evaluating Team Tactics from Football Event and Freeze-Frame Data

Studying collective behavior in opposing multi-agent teams is crucial across game theory, robotics, and sports analytics. In sports, especially football, team tactics involve intricate strategic spatial and action behaviors displayed as event sequences during possession. Understanding and analyzing these tactics is essential for successful training, strategic planning, and on-field success. While traditional approaches, such as notational and statistical analyses, offer valuable insights into team tactics, they often lack a comprehensive consideration of contextual information, thereby limiting the holistic evaluation of teams’ performances. To bridge this gap and capture the nuanced intricacies of team tactics, we employed advanced methodologies. The sequential pattern mining algorithm PrefixSpan was utilized to extract tactical patterns from possession sequences, enabling a deeper understanding of how teams strategize and adapt during play. Additionally, the neural marked spatio temporal point process (NMSTPP) model was leveraged to model and predict team behaviors, facilitating a fair comparison among teams. The evaluation of team possessions was further enhanced through the innovative holistic possession utilization score metrics, providing a more nuanced assessment of performance. In our experimental exploration, we identified and classified five distinct team tactics, validated the efficacy of the NMSTPP model when integrating StatsBomb 360 data, and conducted a comprehensive analysis of English Premier League teams during the 2022/2023 season. The results were visualized using radar plots and scatter plots with mean shift clustering. Lastly, the potential applications to RoboCup were discussed.

The impact of sociality regimes on heterogeneous cooperative-competitive multi-agent reinforcement learning: a study with the predator-prey game

ABSTRACT The performance in multi-agent reinforcement learning (MARL) scenarios has usually been analysed in homogeneous teams with a few choices for the sociality regime (selfish, egalitarian, or altruistic). In this paper we analyse both homogeneous and heterogeneous teams in a variation of sociality regimes in the predator-prey game, using a novel normalisation of the weights so that the sum of all rewards is independent of the sociality regime. We find that the selfish regime is advantageous for both predator and prey teams, and for both homogeneous and heterogeneous teams. In particular, rewards are about 100% higher for the predator team when switching from the egalitarian to selfish regime and more than 400% higher from the altruistic regime. For the prey, the increase is around 40% and 100% respectively. The results are similar for homogeneous and heterogeneous situations. The takeaway message is that any study of homogeneous and heterogeneous cooperative-competitive multi-agent reinforcement learning teams should also take into account the sociality regimes before making conclusions on the preference of any algorithm.

Reinforcement learning for encouraging cooperation in a multiagent system

Success in cooperative tasks may be compromised if cooperative stagnation and failure from information sharing occur. However, information sharing may require excessive memory. Other methods must be developed to encourage agents to take actions in accordance with the needs of the team. In this study, a method called the cooperative tendency model using Q-learning (CTM-Q) is proposed for a partial-communication multiagent team. Each agent maintains and records its tendency values (encouraging cooperation) and Q-values (encouraging goal-seeking) as input for a payoff function that is used to select actions. Each agent selects the action with the highest payoff value for the current state. The method improves learning performance, enabling agents to rapidly reach a consensus. In simulations, the proposed method accelerated learning for multiagent cooperative applications and outperformed competing methods in solution speed, convergence time, and stability.

Robust Distributed Containment Control with Adaptive Performance and Collision Avoidance for Multi-Agent Systems

This paper deals with the containment control problem for multi-agent systems. The objective is to develop a distributed control scheme that leads a sub-group of the agents, called followers, within the convex hull that is formed by the leaders, which operate autonomously. Towards this direction, we propose a twofold approach comprising the following: (i) a cyber layer, where the agents establish, through the communication network, a consensus on a reference trajectory that converges exponentially fast within the convex hull of the leaders and (ii) a physical layer, where each agent tracks the aforementioned trajectory while avoiding collisions with other members of the multi-agent team. The main contributions of this work lie in the robustness of the proposed framework in both the trajectory estimation and the tracking control tasks, as well as the guaranteed collision avoidance, despite the presence of dynamic leaders and bounded but unstructured disturbances. A simulation study of a multi-agent system composed of five followers and four leaders demonstrates the applicability of the proposed scheme and verifies its robustness against both external disturbances that act on the follower model and the dynamic motion of the leaders. A comparison with a related work is also included to outline the strong properties of the proposed approach.

Penalizing Shirking in Discovering Unsuitability

Abstract We study the optimal penalty scheme for an expert firm and a layman client against shirking in their costly interaction that helps the firm discover the unsuitable transaction, i.e. one that yields a loss for the client. The market solution to the bilateral hidden action problem fails to incentivize sufficient effort and leads to an inefficient outcome, which creates scope for government action. By contrast, private contracts obtain efficiency in an alternative framework with just the firm’s unilateral effort, highlighting the role of the client’s effort in rationalizing legal intervention. Under the unique first-best policy, the firm not only refunds the client but also pays the client’s loss to the government if the firm fails to stop a transaction that ends up being unsuitable. The client receives the refund but submits the firm’s production cost to the government. The optimal punitive penalty is their total contingent loss and is robust to private contracts. The optimal penalty generalizes to a multi-agent team by making each agent internalize all others’ contingent losses.

Steps towards Satisficing Distributed Dynamic Team Trust

Defining and measuring trust in dynamic, multiagent teams is important in a range of contexts, particularly in defense and security domains. Team members should be trusted to work towards agreed goals and in accordance with shared values. In this paper, our concern is with the definition of goals and values such that it is possible to define ‘trust’ in a way that is interpretable, and hence usable, by both humans and robots. We argue that the outcome of team activity can be considered in terms of ‘goal’, 'individual/team values', and ‘legal principles’. We question whether alignment is possible at the level of 'individual/team values', or only at the 'goal' and ‘legal principles’ levels. We argue for a set of metrics to define trust in human-robot teams that are interpretable by human or robot team members, and consider an experiment that could demonstrate the notion of 'satisficing trust' over the course of a simulated mission.

Interpretable Learned Emergent Communication for Human–Agent Teams

Learning interpretable communication is essential for multi-agent and human-agent teams (HATs). In multi-agent reinforcement learning for partially-observable environments, agents may convey information to others via learned communication, allowing the team to complete its task. Inspired by human languages, recent works study discrete (using only a finite set of tokens) and sparse (communicating only at some time-steps) communication. However, the utility of such communication in human-agent team experiments has not yet been investigated. In this work, we analyze the efficacy of sparse-discrete methods for producing emergent communication that enables high agent-only and human-agent team performance. We develop agent-only teams that communicate sparsely via our scheme of Enforcers that sufficiently constrain communication to any budget. Our results show no loss or minimal loss of performance in benchmark environments and tasks. In human-agent teams tested in benchmark environments, where agents have been modeled using the Enforcers, we find that a prototype-based method produces meaningful discrete tokens that enable human partners to learn agent communication faster and better than a one-hot baseline. Additional HAT experiments show that an appropriate sparsity level lowers the cognitive load of humans when communicating with teams of agents and leads to superior team performance.

Evaluating Emergent Coordination in Multi-Agent Task Allocation Through Causal Inference and Sub-Team Identification

Coordination in multi-agent systems is a vital component in teaming effectiveness. In dynamically changing situations, agent decisions depict emergent coordination strategies from following pre-defined rules to exploiting incentive-driven policies. While multi-agent reinforcement learning shapes team behaviors from experience, interpreting learned coordination strategies offers benefits in understanding complex agent dynamics and further improvement in developing adaptive strategies for evolving and unexpected situations. In this work, we develop an approach to quantitatively measure team coordination by collecting decision time series data, detecting causality between agents, and identifying statistically high coordinated sub-teams in decentralized multi-agent task allocation operations. We focus on multi-agent systems with homogeneous agents and homogeneous tasks as the strategy formation is more ambiguous and challenging than heterogeneous teams with specialized capabilities. Emergent team coordination is then analyzed using rule-based and reinforcement learning-based strategies for task allocation in operations at different demand stages (stress) levels. We also investigate correlation vs. causation and agent over- or under-estimating demand levels.

Trust-based Performance Optimization for Human-Swarm Collaboration

This work develops a trust inference model to address scenarios where agents in a swarm collaborate to achieve the coverage control task. To gather empirical data from human subjects for the probabilistic model development, we build various simulation tools and user interfaces. Using our visual training tool, we train a single-agent model and then extend that to create our multi-agent model. These models utilize a dynamic Bayesian network and produce stochastic predictions. We then apply these models to our Voronoi-based area coverage problem in real time, where agents adjust their behavior to maximize the team performance and hence human trust. As a result of this research, multi-agent teams will be able to increase their individual trust levels thereby enhancing team performance and efficiency.

The Effect of Trust-based Management Strategy on Performance of Human-Machine Collaborative Team: A Dynamic Computational Model

In human-machine collaborative teams, trust has an important effect on team performance. However, precisely how trust and trust-based management strategy influence team performance have remained elusive. This paper develops a method of trust-based team management to understand which managerial strategy has what impact on performance, and to discern decision alternatives. The method concentrates on providing a dynamic model of trust considering trust relationship initialization, updating trust based on experience and determining what trust should have an effect on. The model is used to explain how the trust-based management strategy caused the team to evolve in particular directions rather than others. Some prescriptions are put forward for the proper management of team. We argue that a human-machine collaborative team is a multi-agent system and team members are autonomous agents. Three computational experiments are conducted under different internal and external conditions for the artificial team, yielding the following results. (1) Under different difficulty of tasks, trust may produce either positive or negative effect on performance. (2) Trust-based management strategy dose has effect on performance under difficult task. (3) The results demonstrate the different effect of three trust-based management strategies on performance. The study method and findings presented herein are appropriate for other studies focusing on dynamic effects on team, laying the foundations for new ideas for studying team building and team development.

Formalizing Distributed Situation Awareness in Multi-Agent Networks

The version of distributed situation awareness (DSA) used in this article originated in human factors/ergonomics. Typically, this has involved the study of human operators working in teams and has used various forms of concept maps to qualitatively describe the information that team members use. In this article, we apply the concept of DSA to multi-agent teams (where team members might be human or automation) and extend the concept using the formal properties of Bayesian belief networks. In particular, we show how the Bayesian belief network can define DSA and how such a network can (using expectation maximization) adapt, even on the basis of limited information. The approach was considered in terms of situation awareness levels of perception (i.e., sensor state values conversion to belief) and projection (in terms of prediction and mission missing values estimation accuracy).

I See You: Examining the Role of Spatial Information in Human-Agent Teams

Awareness, and specifically, spatial awareness, has long played a pivotal role in Computer-Supported Cooperative Work research in both theory and design. This significant background gives awareness the ability to answer challenges facing human-agent teams in communication and shared understanding. As such, the current study investigates the effects of spatial information level (low, high) on the development of team cognition and its outcomes in varying compositions of human-agent teams (human-human-agent, human-agent-agent) versus human-only (human-human-human) teams. The mixed-methods study had teams complete several rounds of the NeoCITIES emergency response management simulation and complete various team cognition and perception measures, followed by qualitative free-response questions. The study found that human-only teams did not perform at the same level as human-agent teams, with multi-agent human-agent teams having the best performance. A significant interaction, though with inconclusive simple main effects, displayed the trend that human-agent teams had better team mental model similarity when spatial awareness was high rather than low, while human-only teams experienced the reverse trend. Qualitative findings identified that high spatial awareness jump-started team cognition development, fostered more accurate shared mental models, enhanced the explainability of the agent, and helped the iterative development of team cognition over time.

ET-HF: A novel information sharing model to improve multi-agent cooperation

Many real-world multi-agent systems require agents to cooperate with each other. However, it is challenging to generate optimal cooperative strategies (e.g., location coordination or speed coordination in the encirclement task) in partially observable environments. Information sharing and temporal experience are two main ways to alleviate partial observability. The main idea of information sharing is observation-sharing scheme which is under-utilization of the topological information, hence agents obtain incomplete environmental information, resulting in a weak information sharing ability between agents. Our idea is to combine observation information and topology information to form a more comprehensive information sharing scheme. Inspired by this, we propose a novel information sharing model in multi-agent reinforcement learning (MARL), named the Entity-Teammate Hierarchical Fusion (ET-HF) model. It comprises three modules (1) message generation, using graph neural networks (GNNs) to combine local observations and topology information to generate two types of messages; (2) message interaction, transmitting messages, and adopting a hierarchical attention fusion mechanism to integrate the above two types of messages and finish information sharing; (3) cooperative policy optimization, inputting integrated messages into the proximal policy optimization (PPO) algorithm to generate the cooperative strategy that adapts to partial observability. Through ET-HF, agents can improve their understanding of the environment and reduce the loss of information caused by partial observability to generate a superior collaborative strategy. We train the multi-agent teams in a fully decentralized framework and the empirical results show that our model outperforms the baselines in cooperative tasks such as coverage control, formation control and line control.

Developing, evaluating and scaling learning agents in multi-agent environments

The Game Theory & Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.

Efficient Exploration for Multi-Agent Reinforcement Learning via Transferable Successor Features

In multi-agent reinforcement learning (MARL), the behaviors of each agent can influence the learning of others, and the agents have to search in an exponentially enlarged joint-action space. Hence, it is challenging for the multi-agent teams to explore in the environment. Agents may achieve suboptimal policies and fail to solve some complex tasks. To improve the exploring efficiency as well as the performance of MARL tasks, in this paper, we propose a new approach by transferring the knowledge across tasks. Differently from the traditional MARL algorithms, we first assume that the reward functions can be computed by linear combinations of a shared feature function and a set of task-specific weights. Then, we define a set of basic MARL tasks in the source domain and pre-train them as the basic knowledge for further use. Finally, once the weights for target tasks are available, it will be easier to get a well-performed policy to explore in the target domain. Hence, the learning process of agents for target tasks is speeded up by taking full use of the basic knowledge that was learned previously. We evaluate the proposed algorithm on two challenging MARL tasks: cooperative box-pushing and non-monotonic predator-prey. The experiment results have demonstrated the improved performance compared with state-of-the-art MARL algorithms.

Multi-Agent Team Learning in Virtualized Open Radio Access Networks (O-RAN).

Starting from the concept of the Cloud Radio Access Network (C-RAN), continuing with the virtual Radio Access Network (vRAN) and most recently with the Open RAN (O-RAN) initiative, Radio Access Network (RAN) architectures have significantly evolved in the past decade. In the last few years, the wireless industry has witnessed a strong trend towards disaggregated, virtualized and open RANs, with numerous tests and deployments worldwide. One unique aspect that motivates this paper is the availability of new opportunities that arise from using machine learning, more specifically multi-agent team learning (MATL), to optimize the RAN in a closed-loop where the complexity of disaggregation and virtualization makes well-known Self-Organized Networking (SON) solutions inadequate. In our view, Multi-Agent Systems (MASs) with MATL can play an essential role in the orchestration of O-RAN controllers, i.e., near-real-time and non-real-time RAN Intelligent Controllers (RIC). In this article, we first provide an overview of the landscape in RAN disaggregation, virtualization and O-RAN, then we present the state-of-the-art research in multi-agent systems and team learning as well as their application to O-RAN. We present a case study for team learning where agents are two distinct xApps: power allocation and radio resource allocation. We demonstrate how team learning can enhance network performance when team learning is used instead of individual learning agents. Finally, we identify challenges and open issues to provide a roadmap for researchers in the area of MATL based O-RAN optimization.

Evolution of Combined Arms Tactics in Heterogeneous Multi-Agent Teams

Multi-agent systems research is concerned with the emergence of system-level behaviors from relatively simple agent interactions. Multi-agent systems research to date is primarily concerned with systems of homogeneous agents, with member agents both physically and behaviorally identical. Systems of heterogeneous agents with differing physical or behavioral characteristics may be able to accomplish tasks more efficiently than homogeneous teams, via cooperation between mutually complementary agent types. In this article, we compare the performance of homogeneous and heterogeneous teams in combined arms situations. Combined arms theory proposes that the application of heterogeneous forces, en masse, can generate effects far greater than outcomes achieved by homogeneous forces or the serial use of individual arms. Results from experiments show that combined arms tactics can emerge from simple agent interactions.