Large-scale Multi-agent Systems Research Articles

Adaptive Anytime Multi-Agent Path Finding Using Bandit-Based Large Neighborhood Search

Anytime multi-agent path finding (MAPF) is a promising approach to scalable path optimization in large-scale multi-agent systems. State-of-the-art anytime MAPF is based on Large Neighborhood Search (LNS), where a fast initial solution is iteratively optimized by destroying and repairing a fixed number of parts, i.e., the neighborhood of the solution, using randomized destroy heuristics and prioritized planning. Despite their recent success in various MAPF instances, current LNS-based approaches lack exploration and flexibility due to greedy optimization with a fixed neighborhood size which can lead to low-quality solutions in general. So far, these limitations have been addressed with extensive prior effort in tuning or offline machine learning beyond actual planning. In this paper, we focus on online learning in LNS and propose Bandit-based Adaptive LArge Neighborhood search Combined with Exploration (BALANCE). BALANCE uses a bi-level multi-armed bandit scheme to adapt the selection of destroy heuristics and neighborhood sizes on the fly during search. We evaluate BALANCE on multiple maps from the MAPF benchmark set and empirically demonstrate performance improvements of at least 50% compared to state-of-the-art anytime MAPF in large-scale scenarios. We find that Thompson Sampling performs particularly well compared to alternative multi-armed bandit algorithms.

Cascaded Attention: Adaptive and Gated Graph Attention Network for Multiagent Reinforcement Learning.

Modeling the interactive relationships of agents is critical to improving the collaborative capability of a multiagent system. Some methods model these by predefined rules. However, due to the nonstationary problem, the interactive relationship changes over time and cannot be well captured by rules. Other methods adopt a simple mechanism such as an attention network to select the neighbors the current agent should collaborate with. However, in large-scale multiagent systems, collaborative relationships are too complicated to be described by a simple attention network. We propose an adaptive and gated graph attention network (AGGAT), which models the interactive relationships between agents in a cascaded manner. In the AGGAT, we first propose a graph-based hard attention network that roughly filters irrelevant agents. Then, normal soft attention is adopted to decide the importance of each neighbor. Finally, gated attention further refines the collaborative relationship of agents. By using cascaded attention, the collaborative relationship of agents is precisely learned in a coarse-to-fine style. Extensive experiments are conducted on a variety of cooperative tasks. The results indicate that our proposed method outperforms state-of-the-art baselines.

A Decomposition Approach to Multiagent Systems With Bernoulli Packet Loss

In this paper, we extend the decomposable systems framework to multi-agent systems with Bernoulli distributed packet loss with uniform probability. The proposed sufficient analysis conditions for mean-square stability and $H_2$-performance -- which are expressed in the form of linear matrix inequalities -- scale linearly with increased network size and thus allow to analyse even very large-scale multi-agent systems. A numerical example demonstrates the potential of the approach by application to a first-order consensus problem.

A Decentralized Communication Framework based on Dual-Level Recurrence for Multi-Agent Reinforcement Learning

Designing communication channels for multi-agent is a feasible method to conduct decentralized learning, especially in partially observable environments or large-scale multi-agent systems. In this work, a communication model with dual-level recurrence is developed to provide a more efficient communication mechanism for the multi-agent reinforcement learning field. The communications are conducted by a gated-attention-based recurrent network, in which the historical states are taken into account and regarded as the second-level recurrence.We separate communication messages from memories in the recurrent model so that the proposed communication flow can adapt changeable communication objects in the case of limited communication, and the communication results are fair to every agent. We provide a sufficient discussion about our method in both partially observable and fully observable environments. The results of several experiments suggest our method outperforms the existing decentralized communication frameworks and the corresponding centralized training method.

SATF: A Scalable Attentive Transfer Framework for Efficient Multiagent Reinforcement Learning.

It is challenging to train an efficient learning procedure with multiagent reinforcement learning (MARL) when the number of agents increases as the observation space exponentially expands, especially in large-scale multiagent systems. In this article, we proposed a scalable attentive transfer framework (SATF) for efficient MARL, which achieved goals faster and more accurately in homogeneous and heterogeneous combat tasks by transferring learned knowledge from a small number of agents (4) to a large number of agents (up to 64). To reduce and align the dimensionality of the observed state variations caused by increasing numbers of agents, the proposed SATF deployed a novel state representation network with a self-attention mechanism, known as dynamic observation representation network (DorNet), to extract the dominant observed information with excellent cost-effectiveness. The experiments on the MAgent platform showed that the SATF outperformed the distributed MARL (independent Q-learning (IQL) and A2C) in task sequences from 8 to 64 agents. The experiments on StarCraft II showed that the SATF demonstrated superior performance relative to the centralized training with decentralized execution MARL (QMIX) by presenting shorter training steps, achieving a desired win rate of up to approximately 90% when increasing the number of agents from 4 to 32. The findings of our study showed the great potential for enhancing the efficiency of MARL training in large-scale agent combat missions.

Distributed Adaptive Flocking Control for Large-Scale Multiagent Systems.

This article presents a novel distributed flocking control method for large-scale multiagent systems (LS-MASs) operating in uncertain environments. When dealing with a massive number of flocking agents in uncertain environments, existing flocking methods encounter the problem of communication complexity and "Curse of dimensionality" caused by the exponential growth of agent interactions while solving PDE-based optimal flocking control for large-scale systems. The mean field game (MFG) method addresses this issue by transforming interactions among all agents into the interaction of each individual agent with average effects represented by a probability density function (pdf) of other agents. However, relying solely on a pdf term to consider other agents' states can result in inefficient flocking performance due to the absence of a proficient coordination mechanism encompassing all agents involved in flocking. To overcome these difficulties and achieve the desired flocking performance for LS-MASs, the agents are decomposed into a finite number of subgroups. Each subgroup comprises a leader and followers, and a hybrid game theory is developed to manage both inter-and intragroup interactions. The method incorporates a cooperative game that links leaders from different groups to formulate distributed flocking control, a Stackelberg game that teams up leaders and followers within the same group to extend collective flocking behavior, and an MFG for followers to address the challenges of LS-MASs. Furthermore, to achieve distributed adaptive flocking using the hybrid game structure, we propose a hierarchical actor-critic-mass-based reinforcement learning technique. This approach incorporates a multiactor-critic method for leaders and an actor-critic-mass algorithm for followers, enabling adaptive flocking control in a distributed manner for large-scale agents. Finally, numerical simulation including comparison study and Lyapunov analysis demonstrates the effectiveness of the developed method.

Fast and Adaptive Multi-Agent Planning under Collaborative Temporal Logic Tasks via Poset Products

Efficient coordination and planning is essential for large-scale multi-agent systems that collaborate in a shared dynamic environment. Heuristic search methods or learning-based approaches often lack the guarantee on correctness and performance. Moreover, when the collaborative tasks contain both spatial and temporal requirements, e.g., as linear temporal logic (LTL) formulas, formal methods provide a verifiable framework for task planning. However, since the planning complexity grows exponentially with the number of agents and the length of the task formula, existing studies are mostly limited to small artificial cases. To address this issue, a new planning paradigm is proposed in this work for system-wide temporal task formulas that are released online and continually. It avoids two common bottlenecks in the traditional methods, i.e., (a) the direct translation of the complete task formula to the associated Büchi automaton and (b) the synchronized product between the Büchi automaton and the transition models of all agents. Instead, an adaptive planning algorithm is proposed, which computes the product of relaxed partially ordered sets (R-posets) on-the-fly and assigns these subtasks to the agents subject to the ordering constraints. It is shown that the first valid plan can be derived with a polynomial time and memory complexity with respect to the system size and the formula length. Our method can take into account task formulas with a length of more than 400 and a fleet with more than 400 agents, while most existing methods fail at the formula length of 25 within a reasonable duration. The proposed method is validated on large fleets of service robots in both simulation and hardware experiments.

NVIF: Neighboring Variational Information Flow for Cooperative Large-Scale Multiagent Reinforcement Learning.

Communication-based multiagent reinforcement learning (MARL) has shown promising results in promoting cooperation by enabling agents to exchange information. However, the existing methods have limitations in large-scale multiagent systems due to high information redundancy, and they tend to overlook the unstable training process caused by the online-trained communication protocol. In this work, we propose a novel method called neighboring variational information flow (NVIF), which enhances communication among neighboring agents by providing them with the maximum information set (MIS) containing more information than the existing methods. NVIF compresses the MIS into a compact latent state while adopting neighboring communication. To stabilize the overall training process, we introduce a two-stage training mechanism. We first pretrain the NVIF module using a randomly sampled offline dataset to create a task-agnostic and stable communication protocol, and then use the pretrained protocol to perform online policy training with RL algorithms. Our theoretical analysis indicates that NVIF-proximal policy optimization (PPO), which combines NVIF with PPO, has the potential to promote cooperation with agent-specific rewards. Experiment results demonstrate the superiority of our method in both heterogeneous and homogeneous settings. Additional experiment results also demonstrate the potential of our method for multitask learning.

Adaptive Dynamic Event-Triggered Distributed Output Observer for Leader-Follower Multiagent Systems Under Directed Graphs.

Leader-follower consensus problem for multiagent systems (MASs) is an important research hotspot. However, the existing methods take the leader system matrix as a priori knowledge for each agent to design the controller and use the leader's state information. In fact, only the output information may be available in some practical applications. On this basis, this article first designs a novel adaptive distributed dynamic event-triggered observer for each follower to learn the minimum polynomial coefficients of the leader system matrix instead of the leader system matrix. The proposed method is scalable and suitable for large-scale MASs and can reduce the information transmission dimension in observer design. Then, an adaptive dynamic event-triggered compensator based on the observer and leader output information is designed for each follower, thereby solving the leader-follower consensus problem. Finally, several simulation examples are given to verify the effectiveness of the proposed scheme.

Hierarchical Multiagent Formation Control Scheme via Actor-Critic Learning.

This article presents a nearly optimal solution to the cooperative formation control problem for large-scale multiagent system (MAS). First, multigroup technique is widely used for the decomposition of the large-scale problem, but there is no consensus between different subgroups. Inspired by the hierarchical structure applied in the MAS, a hierarchical leader-following formation control structure with multigroup technique is constructed, where two layers and three types of agents are designed. Second, adaptive dynamic programming technique is conformed to the optimal formation control problem by the establishment of performance index function. Based on the traditional generalized policy iteration (PI) algorithm, the multistep generalized policy iteration (MsGPI) is developed with the modification of policy evaluation. The novel algorithm not only inherits the advantages of high convergence speed and low computational complexity in the generalized PI algorithm but also further accelerates the convergence speed and reduces run time. Besides, the stability analysis, convergence analysis, and optimality analysis are given for the proposed multistep PI algorithm. Afterward, a neural network-based actor-critic structure is built for approximating the iterative control policies and value functions. Finally, a large-scale formation control problem is provided to demonstrate the performance of our developed hierarchical leader-following formation control structure and MsGPI algorithm.

Distributed zeroth-order optimization: Convergence rates that match centralized counterpart

Zeroth-order optimization has become increasingly important in complex optimization and machine learning when cost functions are impossible to be described in closed analytical forms. The key idea of zeroth-order optimization lies in the ability for a learner to build gradient estimates by queries sent to the cost function, and then traditional gradient descent algorithms can be executed replacing gradients by the estimates. For optimization over large-scale multi-agent systems with decentralized data and costs, zeroth-order optimization can continue to be utilized to develop scalable and distributed algorithms. In this paper, we aim at understanding the trend in performance transitioning from centralized to distributed zeroth-order algorithms in terms of convergence rates, and focus on multi-agent systems with time-varying communication networks. We establish a series of convergence rates for distributed zeroth-order subgradient algorithms under both one-point and two-point zeroth-order oracles. Apart from the additional node-to-node communication cost due to the distributed nature of algorithms, the established rates in convergence are shown to match their centralized counterpart. We also propose a multi-stage distributed zeroth-order algorithm that better utilizes the learning rates, reduces the computational complexity, and attains even faster convergence rates for compact decision set.

Distributed control for geometric pattern formation of large-scale multirobot systems.

Introduction: Geometric pattern formation is crucial in many tasks involving large-scale multi-agent systems. Examples include mobile agents performing surveillance, swarms of drones or robots, and smart transportation systems. Currently, most control strategies proposed to achieve pattern formation in network systems either show good performance but require expensive sensors and communication devices, or have lesser sensor requirements but behave more poorly. Methods and result: In this paper, we provide a distributed displacement-based control law that allows large groups of agents to achieve triangular and square lattices, with low sensor requirements and without needing communication between the agents. Also, a simple, yet powerful, adaptation law is proposed to automatically tune the control gains in order to reduce the design effort, while improving robustness and flexibility. Results: We show the validity and robustness of our approach via numerical simulations and experiments, comparing it, where possible, with other approaches from the existing literature.

Hierarchical game theoretical distributed adaptive control for large scale multi‐group multi‐agent system

AbstractThis paper introduces a distributed adaptive formation control for large‐scale multi‐agent systems (LS‐MAS) that addresses the heavy computational complexity and communication traffic challenges while directly extending conventional distributed control from small scale to large scale. Specifically, a novel hierarchical game theoretic algorithm is developed to provide a feasible theory foundation for solving LS‐MAS distributed optimal formation problem by effectively integrating the mean‐field game (MFG), the Stackelberg game, and the cooperative game. In particular, LS‐MAS is divided into multiple groups geographically with each having one group leader and a significant amount of followers. Then, a cooperative game is used among multi‐group leaders to formulate distributed inter‐group formation control for leaders. Meanwhile, an MFG is adopted for a large number of intra‐group followers to achieve the collective intra‐group formation while a Stackelberg game is connecting the followers with their corresponding leader within the same group to achieve the overall LS‐MAS multi‐group formation behavior. Moreover, a hybrid actor–critic‐based reinforcement learning algorithm is constructed to learn the solution of the hierarchical game‐based optimal distributed formation control. Finally, to show the effectiveness of the presented schemes, numerical simulations and Lyapunov analysis is performed.

Massive Shape Formation in Grid Environments

Shape formation mechanism plays an essential role in many natural processes, involving the formation and evolution of living or non-living structures, and shows potential applications in many emerging domains. In existing research and practice, there still lacks a shape formation mechanism that manifests efficiency, scalability, and stability at the same time. Inspired by phototaxis observed in nature, we propose a self-organized approach for the massive formation of connected shapes in grid environments. The key component of this approach is an artificial light field superimposed on a grid environment, which is determined by the positions of all agents and at the same time drives all agents to change their positions, forming a dynamic mutual feedback process. To evaluate the effectiveness of this approach, we conduct a set of simulations, involving <inline-formula> <tex-math notation="LaTeX">$156$</tex-math> </inline-formula> shapes from <inline-formula> <tex-math notation="LaTeX">$16$</tex-math> </inline-formula> categories, comparing with four baseline methods. The results show that: (1) our approach outperforms the three semi-/decentralized non-optimal baselines in efficiency, scalability, and stability; (2) compared to the centralized optimal baseline, our approach exhibits considerable decreases in the absolute completion time on diverse shape formation tasks, indicating a better efficiency and scalability of our approach. <i>Note to Practitioners</i>&#x2014;In nature, shape formation phenomena emerge from collective behaviors of swarms based on chemical or physical signals. These natural phenomena provide valuable insights to build large-scale multi-agent collaboration systems using software-defined digital signals. This work proposes a phototaxis-inspired computational approach for shape formation that enables a massive swarm of agents to form arbitrary connected shapes in grid environments based on a digital signal called artificial light field. The significance of this work is twofold: 1. it could contribute to a deep understanding of shape formation mechanisms; 2. it would motivate new research on advanced multi-agent algorithms, massive collaboration mechanisms, and artificial collective intelligence systems and facilitate their practical applications. Specifically, the shape formation mechanism has promising applications, including smart warehouses, autonomous cooperation of UAVs, and intelligent transportation systems. A possible realistic application scenario of our method in intelligent transportation systems is bike sharing systems, in which the designated parking areas for shared bicycles near the work area are often overcrowded and difficult to park in during the morning peak period, so dynamic parking route guidance for users is required. Our method can directly apply to this scenario by utilizing the light field to represent the parking state of nearby bicycles and guiding the moving direction of each user.

Automatic Curriculum Learning for Large-Scale Cooperative Multiagent Systems

Recently, a lot of works have been devoted to researching how agents can learn efficient cooperation in multiagent systems. However, it still remains challenging in large-scale multiagent systems (MASs) due to the complex dynamics between the agents and environment and the dimension explosion of state-action space. In this paper, we propose a novel MultiAgent Automatic Curriculum Learning method (MA-ACL) to solve learning problems of large-scale cooperative MASs by beginning from learning on a multiagent scenario with a few agents and automatically progressively increasing the number of agents. An evaluation mechanism based on self-supervised learning is innovatively designed to automatically generate appropriate curricula with a progressively increasing number of agents. Moreover, since the observation state dimension of agents varies across curricula and the learned policy knowledge needs to be effectively encoded, we design a new Distributed Transferable Relation-modeling Policy network structure (DTRP) to handle the dynamic size of the network input and model relational knowledge between agents and their surrounding environment. Simulation results show that the proposed MA-ACL using DTRP can significantly improve the performance of large-scale multiagent learning compared with manual or non curriculum learning methods, and DTRP greatly boosts the performance of MA-ACL.

FEMA-CL: Fair Efficient Multi-Agent Course learning

Sociology shows that blindly pursuing the fairness of resource distribution will significantly reduce people’s enthusiasm for work, which is not conducive to the increase of total social material resources. Promoting the fairness of the social system in stages, that is, achieving fairness on the premise of a certain material basis, can not only ensure the efficiency of the total social material accumulation, but also promote the fairness of the social system. Therefore, inspired by the above, we introduced multi-stage curriculum learning into learning fair policy of multi-agent systems, and proposed a novel Fair and Effective Multi-Agent Curriculum Learning (FEMA-CL). The multi-stage course learning progressively promotes the learning fairness and efficiency of large-scale multi-agent systems through three stages: selfish stage, soft fair stage and global fair stage. Our method is easy to learn fairness and efficiency, and has carried out extensive experiments in three typical multi-agent scenarios. Compared with the current popular work, our method has superior performance.

Adaptive Distributed Formation Control for Multi-Group Large-Scale Multi-Agent Systems: A Hybrid Game Approach

This paper presents a distributed adaptive formation control for large-scale multi-agent systems(LS-MAS) that addresses the “Curse of Dimensionality”. A novel hybrid game theoretic algorithm that effectively integrates the mean field, Stackelberg, and cooperative game seamlessly has been developed. In particular, LS-MAS has been separated into a multiple number of groups, each has one group leader and a significant amount of followers. Next, a cooperative game is utilized for the inter-group formation of leaders, the mean-field game is adopted for intra-group followers, and a Stackelberg game is connecting the leader and followers in same group. A hybrid reinforcement learning algorithm learns the solution of the hybrid game optimal distributed formation. It comprises a multi-actor-critic to obtain the optimal formation control among inter-group leaders in a distributed manner, and a Stackelberg game-based actor-critic-mass algorithm to obtain the followers’ adaptive formation control. Lastly, to demonstrate the efficacy of the proposed approaches, numerical simulations have been conducted.

Local Convergence of Multi-Agent Systems Toward Rigid Lattices

Geometric pattern formation is an important emergent behavior in many applications involving large-scale multi-agent systems, such as sensor networks deployment and collective transportation. Attraction/ repulsion virtual forces are the most common control approach to achieve such behavior in a distributed and scalable manner. Nevertheless, for most existing solutions only numerical and/or experimental evidence of their convergence is available. Here, we revisit the problem of achieving pattern formation in spaces of any dimension, giving sufficient conditions to prove analytically that under the influence of appropriate virtual forces, a large-scale multi-agent swarming system locally converges towards a stable and robust rigid lattice configuration. Our theoretical results are complemented by exhaustive numerical simulations confirming their effectiveness and estimating the region of asymptotic stability of the rigid lattice configuration.

Continuification Control of Large-Scale Multiagent Systems in a Ring

In this paper, we propose a method to control large-scale multiagent systems swarming in a ring. Specifically, we use a continuification-based approach that transforms the microscopic, agent-level description of the system dynamics into a macroscopic, continuum-level representation, which we employ to synthesize a control action towards a desired distribution of the agents. The continuum-level control action is then discretized at the agent-level in order to practically implement it. To confirm the effectiveness and the robustness of the proposed approach, we complement theoretical derivations with a series of numerical simulations.

Continuification Control of Large-Scale Multiagent Systems Under Limited Sensing and Structural Perturbations

Continuification Control of Large-Scale Multiagent Systems Under Limited Sensing and Structural Perturbations