Team Of Agents Research Articles

Preliminary Experimental Results of Context-Aware Teams of Multiple Autonomous Agents Operating under Constrained Communications

This work presents and experimentally tests the framework used by our context-aware, distributed team of small Unmanned Aerial Systems (SUAS) capable of operating in real time, in an autonomous fashion, and under constrained communications. Our framework relies on a three-layered approach: (1) an operational layer, where fast temporal and narrow spatial decisions are made; (2) a tactical layer, where temporal and spatial decisions are made for a team of agents; and (3) a strategical layer, where slow temporal and wide spatial decisions are made for the team of agents. These three layers are coordinated by an ad hoc, software-defined communications network, which ensures sparse but timely delivery of messages amongst groups and teams of agents at each layer, even under constrained communications. Experimental results are presented for a team of 10 small unmanned aerial systems tasked with searching for and monitoring a person in an open area. At the operational layer, our use case presents an agent autonomously performing searching, detection, localization, classification, identification, tracking, and following of the person, while avoiding malicious collisions. At the tactical layer, our experimental use case presents the cooperative interaction of a group of multiple agents that enables the monitoring of the targeted person over wider spatial and temporal regions. At the strategic layer, our use case involves the detection of complex behaviors, i.e., the person being followed enters a car and runs away, or the person being followed exits the car and runs away, which require strategic responses to successfully accomplish the mission.

A distributed constraint multi-agent model for water and reclaimed wastewater allocation in urban areas: Application of a modified ADOPT algorithm.

Distributed Constraint Optimization (DCOP)-based approaches, as the distributed version of constraint optimization, provide a framework for coordinated decision making by a team of agents. In this paper, an agent-based DCOP model is developed to allocate water and reclaimed wastewater to demands considering the conflicting interests of involved stakeholders. One of the well-known DCOP algorithms, ADOPT1, is modified to incorporate an agent responsible for monitoring and conserving water resources. This new algorithm considers the social characteristics of agents and a new form of interaction between agents. For the first time in the literature, a real-world water and reclaimed wastewater allocation problem is formulated as a DCOP and solved using the Modified ADOPT (MADOPT) algorithm. To evaluate the MADOPT algorithm, it is applied to a water and reclaimed wastewater allocation problem in Tehran, Iran. The results illustrate the applicability and efficiency of the proposed methodology in dealing with large-scale multi-agent water resources systems. It is also shown that agents' selfishness and social relationships could affect their water use policies.

Polynomial-Time Algorithms for Multiagent Minimal-Capacity Planning

In this article, we study the problem of minimizing the resource capacity of autonomous agents that cooperate to achieve a shared task. More specifically, we consider high-level planning for a team of homogeneous agents that operate under resource constraints in stochastic environments and share a common goal: given a set of target locations, ensure that each location is visited infinitely often by some agents almost surely. We formalize the dynamics of agents by the so-called consumption Markov decision processes. In a consumption Markov decision process, the agent has a resource of limited capacity. Each action of the agent may consume some amount of the resource. To avoid exhaustion, the agent can replenish its resource to full capacity in designated reload states. The resource capacity restricts the capabilities of the agent. The objective is to assign target locations to agents, and each agent is only responsible for visiting the assigned subset of target locations repeatedly. Moreover, the assignment must ensure that the agents can carry out their tasks with minimal resource capacity. We reduce the problem to an equivalent graph-theoretical problem. We develop an algorithm that solves this graph problem in time that is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">polynomial</i> in the number of agents, target locations, and size of the consumption Markov decision process. We demonstrate the applicability and scalability of the algorithm in a scenario where hundreds of unmanned underwater vehicles monitor hundreds of locations in environments with stochastic ocean currents.

From motor control to team play in simulated humanoid football.

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.

Adversarial Attacks on Heterogeneous Multi-Agent Deep Reinforcement Learning System with Time-Delayed Data Transmission

This paper studies the gradient-based adversarial attacks on cluster-based, heterogeneous, multi-agent, deep reinforcement learning (MADRL) systems with time-delayed data transmission. The structure of the MADRL system consists of various clusters of agents. The deep Q-network (DQN) architecture presents the first cluster’s agent structure. The other clusters are considered as the environment of the first cluster’s DQN agent. We introduce two novel observations in data transmission, termed on-time and time-delay observations. The proposed observations are considered when the data transmission channel is idle, and the data is transmitted on time or delayed. By considering the distance between the neighboring agents, we present a novel immediate reward function by appending a distance-based reward to the previously utilized reward to improve the MADRL system performance. We consider three types of gradient-based attacks to investigate the robustness of the proposed system data transmission. Two defense methods are proposed to reduce the effects of the discussed malicious attacks. We have rigorously shown the system performance based on the DQN loss and the team reward for the entire team of agents. Moreover, the effects of the various attacks before and after using defense algorithms are demonstrated. The theoretical results are illustrated and verified with simulation examples.

Strategic maneuver and disruption with reinforcement learning approaches for multi-agent coordination

Reinforcement learning (RL) approaches can illuminate emergent behaviors that facilitate coordination across teams of agents as part of a multi-agent system (MAS), which can provide windows of opportunity in various military tasks. Technologically advancing adversaries pose substantial risks to a friendly nation’s interests and resources. Superior resources alone are not enough to defeat adversaries in modern complex environments because adversaries create standoff in multiple domains against predictable military doctrine-based maneuvers. Therefore, as part of a defense strategy, friendly forces must use strategic maneuvers and disruption to gain superiority in complex multi-faceted domains, such as multi-domain operations (MDOs). One promising avenue for implementing strategic maneuver and disruption to gain superiority over adversaries is through coordination of MAS in future military operations. In this paper, we present overviews of prominent works in the RL domain with their strengths and weaknesses for overcoming the challenges associated with performing autonomous strategic maneuver and disruption in military contexts.

Scalable and Robust Algorithms for Task-Based Coordination From High-Level Specifications (ScRATCHeS)

Many existing approaches for coordinating heterogeneous teams of robots either consider small numbers of agents, are application-specific, or do not adequately address common real-world requirements, e.g., strict deadlines or intertask dependencies. We introduce scalable and robust algorithms for task-based coordination from high-level specifications (ScRATCHeS) to coordinate such teams. We define a specification language, capability temporal logic, to describe rich, temporal properties involving tasks requiring the participation of multiple agents with multiple capabilities, e.g., sensors or end effectors. Arbitrary missions and team dynamics are jointly encoded as constraints in a mixed integer linear program, and solved efficiently using commercial off-the-shelf solvers. ScRATCHeS optionally allows optimization for maximal robustness to agent attrition at the penalty of increased computation time. We include an online replanning algorithm that adjusts the plan after an agent has dropped out. The flexible specification language, fast solution time, and optional robustness of ScRATCHeS provide a first step toward a multipurpose on-the-fly planning tool for tasking large teams of agents with multiple capabilities enacting missions with multiple tasks. We present randomized computational experiments to characterize scalability and hardware demonstrations to illustrate the applicability of our methods.

Doubly Stochastic Pairwise Interactions for Agreement and Alignment

Random pairwise encounters often occur in large populations, or groups of mobile agents, and various types of local interactions that happen at encounters account for emergent global phenomena. In particular, in the fields of swarm robotics, sociobiology, and social dynamics, several types of local pairwise interactions were proposed and analysed leading to spatial gathering or clustering and agreement in teams of robotic agents coordinated motion, in animal herds, or in human societies. We here propose a very simple stochastic interaction at encounters that leads to agreement or geometric alignment in swarms of simple agents, and analyse the process of converging to consensus. Consider a group of agents whose evolve in time by pairwise interactions: the state of an agent is either a real value (a randomly initialised position within an interval) or a vector that is either unconstrained (e.g. the location of the agent in the plane) or constrained to have unit length (e.g. the direction of the agent's motion). The interactions are doubly stochastic, in the sense that, at discrete time steps, pairs of agents are randomly selected and their new states are independently and uniformly set at random in (local) domains or intervals defined by the states of the interacting pair. We show that such processes lead, in finite expected time (measured by the number of interactions that occurred) to agreement in case of unconstrained states and alignment when the states are unit vectors.

Leveraging Experience in Lifelong Multi-Agent Pathfinding

In Lifelong Multi-Agent Path Finding (L-MAPF) a team of agents performs a stream of tasks consisting of multiple locations to be visited by the agents on a shared graph while avoiding collisions with one another. L-MAPF is typically tackled by partitioning it into multiple consecutive, and hence similar, "one-shot" MAPF queries, as in the Rolling-Horizon Collision Resolution (RHCR) algorithm. Therefore, a solution to one query informs the next query, which leads to similarity with respect to the agents' start and goal positions, and how collisions need to be resolved from one query to the next. Thus, experience from solving one MAPF query can potentially be used to speedup solving the next one. Despite this intuition, current L-MAPF planners solve consecutive MAPF queries from scratch. In this paper, we introduce a new RHCR-inspired approach called exRHCR, which exploits experience in its constituent MAPF queries. In particular, exRHCR employs an extension of Priority-Based Search (PBS), a state-of-the-art MAPF solver. The extension, which we call exPBS, allows to warm-start the search with the priorities between agents used by PBS in the previous MAPF instances. We demonstrate empirically that exRHCR solves L-MAPF instances up to 39% faster than RHCR, and has the potential to increase system throughput for given task streams by increasing the number of agents a planner can cope with for a given time budget.

DIMASS: A Delaunay-Inspired, Hybrid Approach to a Team of Agents Search Strategy.

This article describes an approach for multiagent search planning for a team of agents. A team of UAVs tasked to conduct a forest fire search was selected as the use case, although solutions are applicable to other domains. Fixed-path (e.g., parallel track) methods for multiagent search can produce predictable and structured paths, with the main limitation being poor management of agents’ resources and limited adaptability (i.e., based on predefined geometric paths, e.g., parallel track, expanding square, etc.). On the other hand, pseudorandom methods allow agents to generate well-separated paths; but methods can be computationally expensive and can result in a lack of coordination of agents’ activities. We present a hybrid solution that exploits the complementary strengths of fixed-pattern and pseudorandom methods, i.e., an approach that is resource-efficient, predictable, adaptable, and scalable. Our approach evolved from the Delaunay triangulation of systematically selected waypoints to allocate agents to explore a specific region while optimizing a given set of mission constraints. We implement our approach in a simulation environment, comparing the performance of the proposed algorithm with fixed-path and pseudorandom baselines. Results proved agents’ resource utilization, predictability, scalability, and adaptability of the developed path. We also demonstrate the proposed algorithm’s application on real UAVs.

Anytime Multi-Agent Path Finding via Machine Learning-Guided Large Neighborhood Search

Multi-Agent Path Finding (MAPF) is the problem of finding a set of collision-free paths for a team of agents in a common environment. MAPF is NP-hard to solve optimally and, in some cases, also bounded-suboptimally. It is thus time-consuming for (bounded-sub)optimal solvers to solve large MAPF instances. Anytime algorithms find solutions quickly for large instances and then improve them to close-to-optimal ones over time. In this paper, we improve the current state-of-the-art anytime solver MAPF-LNS, that first finds an initial solution fast and then repeatedly replans the paths of subsets of agents via Large Neighborhood Search (LNS). It generates the subsets of agents for replanning by randomized destroy heuristics, but not all of them increase the solution quality substantially. We propose to use machine learning to learn how to select a subset of agents from a collection of subsets, such that replanning increases the solution quality more. We show experimentally that our solver, MAPF-ML-LNS, significantly outperforms MAPF-LNS on the standard MAPF benchmark set in terms of both the speed of improving the solution and the final solution quality.

Assignment and Prioritization of Tasks with Uncertain Durations for Satisfying Makespans in Decentralized Execution

Task assignment under execution uncertainty and temporal/resource constraints is a standard problem for many organizations. Existing approaches in the AI planning &amp; scheduling and operations research literature predominantly focus on dynamic controllability, and non-preemptive execution of tasks. Such solutions are appropriate for teams of agents under tight control requirements. However, in most organizations with human teams, once tasks have been assigned, humans tend to execute their assignments without a constant central oversight (which is needed for dynamic controllability). In this paper we define a problem in which execution of tasks is distributed (without central oversight), and assumes humans can preempt their tasks when other tasks of higher priority are ready to be worked on. We present two algorithms based on Tabu search and Monte Carlo Tree Search to assign and prioritize tasks for such problems. Experimental results show the improved efficacy of these approaches for this problem setting over non-preemptive strategies.

Teammates Instead of Tools: The Impacts of Level of Autonomy on Mission Performance and Human-Agent Teaming Dynamics in Multi-Agent Distributed Teams.

Human–agent teaming (HAT) is becoming more commonplace across industry, military, and consumer settings. Agents are becoming more advanced, more integrated, and more responsible for tasks previously assigned to humans. In addition, the dyadic human–agent teaming nature is evolving from a one–one pair to one–many, in which the human is working with numerous agents to accomplish a task. As capabilities become more advanced and humanlike, the best method for humans and agents to effectively coordinate is still unknown. Therefore, current research must start diverting focus from how many agents can a human manage to how can agents and humans work together effectively. Levels of autonomy (LOAs), or varying levels of responsibility given to the agents, implemented specifically in the decision-making process could potentially address some of the issues related to workload, stress, performance, and trust. This study sought to explore the effects of different LOAs on human–machine team coordination, performance, trust, and decision making in hand with assessments of operator workload and stress in a simulated multi-unmanned aircraft vehicle (UAV) intelligence surveillance and reconnaissance (ISR) task. The results of the study can be used to identify human factor roadblocks to effective HAT and provide guidance for future designs of HAT. Additionally, the unique impacts of LOA and autonomous decision making by agents on trust are explored.

Knowledge-based strategies for multi-agent teams playing against Nature

We study teams of agents that play against Nature towards achieving a common objective. The agents are assumed to have imperfect information due to partial observability, and have no communication during the play of the game. We propose a natural notion of higher-order knowledge of agents. Based on this notion, we define a class of knowledge-based strategies, and consider the problem of synthesis of strategies of this class. We introduce a multi-agent extension, MKBSC, of the well-known knowledge-based subset construction applied to such games. Its iterative applications turn out to compute higher-order knowledge of the agents. We show how the MKBSC can be used for the design of knowledge-based strategy profiles, and investigate the transfer of existence of such strategies between the original game and in the iterated applications of the MKBSC, under some natural assumptions. We also relate and compare the “intensional” view on knowledge-based strategies based on explicit knowledge representation and update, with the “extensional” view on finite memory strategies based on finite transducers and show that, in a certain sense, these are equivalent.

Trust Development in Military and Civilian Human–Agent Teams: The Effect of Social-Cognitive Recovery Strategies

Autonomous agents (AA) will increasingly be deployed as teammates instead of tools. In many operational situations, flawless performance from AA cannot be guaranteed. This may lead to a breach in the human’s trust, which can compromise collaboration. This highlights the importance of thinking about how to deal with error and trust violations when designing AA. The aim of this study was to explore the influence of uncertainty communication and apology on the development of trust in a Human–Agent Team (HAT) when there is a trust violation. Two experimental studies following the same method were performed with (I) a civilian group and (II) a military group of participants. The online task environment resembled a house search in which the participant was accompanied and advised by an AA as their artificial team member. Halfway during the task, an incorrect advice evoked a trust violation. Uncertainty communication was manipulated within-subjects, apology between-subjects. Our results showed that (a) communicating uncertainty led to higher levels of trust in both studies, (b) an incorrect advice by the agent led to a less severe decline in trust when that advice included a measure of uncertainty, and (c) after a trust violation, trust recovered significantly more when the agent offered an apology. The two latter effects were only found in the civilian study. We conclude that tailored agent communication is a key factor in minimizing trust reduction in face of agent failure to maintain effective long-term relationships in HATs. The difference in findings between participant groups emphasizes the importance of considering the (organizational) culture when designing artificial team members.

The role of shared mental models in human-AI teams: a theoretical review

Mental models are knowledge structures employed by humans to describe, explain, and predict the world around them. Shared Mental Models (SMMs) occur in teams whose members have similar mental models of their task and of the team itself. Research on human teaming has linked SMM quality to improved team performance. Applied understanding of SMMs should lead to improvements in human-AI teaming. Yet, it remains unclear how the SMM construct may differ in teams of human and AI agents, how and under what conditions such SMMs form, and how they should be quantified. This paper presents a review of SMMs and the associated literature, including their definition, measurement, and relation to other concepts. A synthesized conceptual model is proposed for the application of SMM literature to the human-AI setting. Several areas of AI research are identified and reviewed that are highly relevant to SMMs in human-AI teaming but which have not been discussed via a common vernacular. A summary of design considerations to support future experiments regarding Human-AI SMMs is presented. We find that while current research has made significant progress, a lack of consistency in terms and of effective means for measuring Human-AI SMMs currently impedes realization of the concept.

Intent integration for human‐agent teaming

AbstractKnowledge of intent is critical in high performing human teams. The fundamental question addressed by this research is, how should intent be integrated into future human‐artificial agent (AA) teams to improve coordination among team members? A brief review of the use of intent for improving performance within human‐human teams is conducted to provide a better understanding of this term. This review differentiates intent estimation from intent application, as well as the differentiation of “why,” “what” and “how” based intent. A taxonomy of intent‐based systems is then developed through a review of existing examples in the literature. Together these reviews demonstrate that intent has been modeled in a variety of ways without a cohesive understanding of intent and its different forms. Based upon these reviews and our understanding of multi‐agent system architectures, we propose “operationalized intent” as a method of modeling intent regarding “how” the operators would like to execute the team's tasks. We propose including an Intent Agent (IA) dedicated to estimating intent of each operator and embedding knowledge of how to execute within the Functional Agents (FAs) of a multi‐agent system. The proposed Operationalized Intent Ontology provides a means of modeling human‐agent teams as an intent informed system.

Fast Decomposition of Temporal Logic Specifications for Heterogeneous Teams

In this work, we focus on decomposing large multi-agent path planning problems with global temporal logic goals (common to all agents) into smaller sub-problems that can be solved and executed independently. Crucially, the sub-problems' solutions must jointly satisfy the common global mission specification. The agents' missions are given as Capability Temporal Logic (CaTL) formulas, a fragment of signal temporal logic, that can express properties over tasks involving multiple agent capabilities (sensors, e.g., camera, IR, and effectors, e.g., wheeled, flying, manipulators) under strict timing constraints. The approach we take is to decompose both the temporal logic specification and the team of agents. We jointly reason about the assignment of agents to subteams and the decomposition of formulas using a satisfiability modulo theories (SMT) approach. The output of the SMT is then distributed to subteams and leads to a significant speed up in planning time. We include computational results to evaluate the efficiency of our solution, as well as the trade-offs introduced by the conservative nature of the SMT encoding.

ForMIC: Foraging via Multiagent RL With Implicit Communication

Multi-agent foraging (MAF) involves distributing a team of agents to search an environment and extract resources from it. Nature provides several examples of highly effective foragers, where individuals within the foraging collective use biological markers (e.g., pheromones) to communicate critical information to others via the environment. In this work, we propose ForMIC, a distributed reinforcement learning MAF approach that endows agents with implicit communication abilities via their shared environment. However, learning efficient policies with stigmergic interactions is highly nontrivial, since agents need to perform well to send each other useful signals, but also need to sense others’ signals to perform well. In this work, we develop several key learning techniques for training policies with stigmergic interactions, where such a circular dependency is present. By relying on clever curriculum learning design, action filtering, and the introduction of non-learning agents to increase the agent density at training time at low computational cost, we develop a minimal learning framework that leads to the stable training of efficient stigmergic policies. We present simulation results which demonstrate that our learned policy outperforms existing state-of-the-art MAF algorithms in a set of experiments that vary team size, number and placement of resources, and key environmental dynamics not seen at training time.

Knowledge Reuse of Multi-Agent Reinforcement Learning in Cooperative Tasks.

With the development and appliance of multi-agent systems, multi-agent cooperation is becoming an important problem in artificial intelligence. Multi-agent reinforcement learning (MARL) is one of the most effective methods for solving multi-agent cooperative tasks. However, the huge sample complexity of traditional reinforcement learning methods results in two kinds of training waste in MARL for cooperative tasks: all homogeneous agents are trained independently and repetitively, and multi-agent systems need training from scratch when adding a new teammate. To tackle these two problems, we propose the knowledge reuse methods of MARL. On the one hand, this paper proposes sharing experience and policy within agents to mitigate training waste. On the other hand, this paper proposes reusing the policies learned by original teams to avoid knowledge waste when adding a new agent. Experimentally, the Pursuit task demonstrates how sharing experience and policy can accelerate the training speed and enhance the performance simultaneously. Additionally, transferring the learned policies from the N-agent enables the (N+1)–agent team to immediately perform cooperative tasks successfully, and only a minor training resource can allow the multi-agents to reach optimal performance identical to that from scratch.