Large Multi-agent Systems Research Articles

Stability Analysis for Large-Scale Multi-Agent Molecular Communication Systems.

Molecular communication (MC) is recently featured as a novel communication tool to connect individual biological nanorobots. It is expected that a large number of nanorobots can form large multi-agent MC systems through MC to accomplish complex and large-scale tasks that cannot be achieved by a single nanorobot. However, most previous models for MC systems assume a unidirectional diffusion communication channel and cannot capture the feedback between each nanorobot, which is important for multi-agent MC systems. In this paper, we introduce a system theoretic model for large-scale multi-agent MC systems using transfer functions, and then propose a method to analyze the stability for multi-agent MC systems. The proposed method decomposes the multi-agent MC system into multiple single-input and single-output (SISO) systems, which facilitates the application of simple analysis technique for SISO systems to the large-scale multi-agent MC system. Finally, we demonstrate the proposed method by analyzing the stability of a specific large-scale multi-agent MC system and clarify a parameter region to synchronize the states of nanorobots, which is important to make cooperative behaviors at a population level.

Partially Centralized Model-Predictive Mean Field Games for controlling multi-agent systems

Fast, high-performance control of large Multi-Agent Systems (MASs) is a major challenge. We address this problem using Mean Field Games (MFGs), which deduce the macroscopic dynamics of the density distribution of the agent population from the microscopic dynamics of each agent. To control MASs using the MFG, two main problems need to be solved: preventing control performance degradation caused by distribution estimation errors and ensuring the scalability of communication. To overcome these issues, we develop a new control method called the Partially Centralized Model-Predictive MFG (PCMP-MFG). The proposed method solves the first issue by repeating distribution estimation and model-predictive control at each time; it solves the second by introducing broadcast control. Our numerical results show that the proposed PCMP-MFG method outperforms the conventional MFG-based method in a wide parameter range.

Priority tagged Artificial Potential Field Swarm Formation Control through Narrow Corridors

Cooperative missions carried out by multiagent systems consisting of aerial agents such as UAV or MAV have immense application in surveillance, reconnaissance, exploration, search-and-rescue, etc. Some of these applications require these swarms to navigate through complicated environments such as urban areas consisting of narrow corridors or channels. The formation control of the multiagent system has to accommodate for tangibility in the formation structure in order to navigate through such tight spaces without much congestion amongst the individuals of the swarm. A congested swarm is prone to interagent collisions or collisions with the environment leading to complete failure of the mission. This paper proposes a congestion-relieving method for formation control of a swarm using priority tags on individuals based on Artificial Potential Fields (APF). Based on the formation equilibrium study for the swarm, a minimum distance is calculated as a threshold to classify the swarm as dense or sparse and detect congestion due to narrow corridor navigation. The key feature of the proposed formation control is its decentralized nature which allows scalability and flexibility on large multiagent systems as it requires only limited local information to generate the formation structure. Numerical simulations are carried out to evaluate the performance of priority-tagged APF while navigating through narrow corridors and the results are discussed by comparing performance without priority tags.

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning

With the development of unmanned aerial vehicle (UAV) technology, UAV swarm confrontation has attracted many researchers’ attention. However, the situation faced by the UAV swarm has substantial uncertainty and dynamic variability. The state space and action space increase exponentially with the number of UAVs, so that autonomous decision-making becomes a difficult problem in the confrontation environment. In this paper, a multiagent reinforcement learning method with macro action and human expertise is proposed for autonomous decision-making of UAVs. In the proposed approach, UAV swarm is modeled as a large multiagent system (MAS) with an individual UAV as an agent, and the sequential decision-making problem in swarm confrontation is modeled as a Markov decision process. Agents in the proposed method are trained based on the macro actions, where sparse and delayed rewards, large state space, and action space are effectively overcome. The key to the success of this method is the generation of the macro actions that allow the high-level policy to find a near-optimal solution. In this paper, we further leverage human expertise to design a set of good macro actions. Extensive empirical experiments in our constructed swarm confrontation environment show that our method performs better than the other algorithms.

Decentralized Adaptive Optimal Tracking Control for Massive Autonomous Vehicle Systems With Heterogeneous Dynamics: A Stackelberg Game.

In this article, a decentralized optimal tracking control problem has been studied for a large-scale autonomous vehicle system with heterogeneous system dynamics. Due to the ultralarge number of agents, the notorious "curse of dimension" problem as well as the unrealistic assumption of the existence of reliable very large-scale communication links in uncertain environments have challenged the traditional multiagent system (MAS) algorithms for decades. The emerging mean-field game (MFG) theory has recently been widely adopted to generate a decentralized control method that deals with those challenges by encoding the large scale MASs' information into a novel time-varying probability density functions (PDF) which can be obtained locally. However, the traditional MFG methods assume all agents are homogeneous, which is unrealistic in practical industrial applications, e.g., Internet of Things (IoTs), and so on. Therefore, a novel mean-field Stackelberg game (MFSG) is formulated based on the Stackelberg game, where all the agents have been classified as two different categories where one major leader's decision dominates the other minor agents. Moreover, a hierarchical structure that treats all minor agents as a mean-field group is developed to tackle the assumption of homogeneous agents. Then, the actor-actor-critic-critic-mass ( A2C2M ) algorithm with five neural networks is designed to learn the optimal policies by solving the MFSG. The Lyapunov theory is utilized to prove the convergence of A2C2M neural networks and the closed-loop system's stability. Finally, a series of numerical simulations are conducted to demonstrate the effectiveness of the developed method.

Multiagent Decision Making For Maritime Traffic Management

We address the problem of maritime traffic management in busy waterways to increase the safety of navigation by reducing congestion. We model maritime traffic as a large multiagent systems with individual vessels as agents, and VTS authority as the regulatory agent. We develop a maritime traffic simulator based on historical traffic data that incorporates realistic domain constraints such as uncertain and asynchronous movement of vessels. We also develop a traffic coordination approach that provides speed recommendation to vessels in different zones. We exploit the nature of collective interactions among agents to develop a scalable policy gradient approach that can scale up to real world problems. Empirical results on synthetic and real world problems show that our approach can significantly reduce congestion while keeping the traffic throughput high.

Composition of Least Restrictive Controllers, With Application to Collision Avoidance in Multiagent Systems

A supervisor (of a continuous-time or hybrid system) is a controller in charge of modifying the input assigned by a user or set of users to a system, in order to enforce a given specification. This paper describes conditions under which multiple supervisors, designed to enforce different specifications, can be composed to obtain a supervisor enforcing the union of those specifications. As an application, we propose the composition of two supervisors, one enforcing collision avoidance of a large multiagent system, the other enforcing a second property, called sparsity, that allows efficient computation of the collision avoidance conditions.

Expert Level Control of Ramp Metering Based on Multi-Task Deep Reinforcement Learning

This paper shows how the recent breakthroughs in reinforcement learning (RL) that have enabled robots to learn to play arcade video games, walk, or assemble colored bricks, can be used to perform other tasks that are currently at the core of engineering cyberphysical systems. We present the first use of RL for the control of systems modeled by discretized non-linear partial differential equations (PDEs) and devise a novel algorithm to use non-parametric control techniques for large multi-agent systems. Cyberphysical systems (e.g., hydraulic channels, transportation systems, the energy grid, and electromagnetic systems) are commonly modeled by PDEs, which historically have been a reliable way to enable engineering applications in these domains. However, it is known that the control of these PDE models is notoriously difficult. We show how neural network-based RL enables the control of discretized PDEs whose parameters are unknown, random, and time-varying. We introduce an algorithm of mutual weight regularization (MWR), which alleviates the curse of dimensionality of multi-agent control schemes by sharing experience between agents while giving each agent the opportunity to specialize its action policy so as to tailor it to the local parameters of the part of the system it is located in. A discretized PDE, such as the scalar Lighthill–Whitham–Richards PDE can indeed be considered as a macroscopic freeway traffic simulator and which presents the most salient challenges for learning to control large cyberphysical system with multiple agents. We consider two different discretization procedures and show the opportunities offered by applying deep reinforcement for continuous control on both. Using a neural RL PDE controller on a traffic flow simulation based on a Godunov discretization of the San Francisco Bay Bridge, we are able to achieve precise adaptive metering without model calibration thereby beating the state of the art in traffic metering. Furthermore, with the more accurate BeATS simulator, we manage to achieve a control performance on par with ALINEA, a state-of-the-art parametric control scheme, and show how using MWR improves the learning procedure.

Implicit crowds

Large multi-agent systems such as crowds involve inter-agent interactions that are typically anticipatory in nature, depending strongly on both the positions and the velocities of agents. We show how the nonlinear, anticipatory forces seen in multi-agent systems can be made compatible with recent work on energy-based formulations in physics-based animation, and propose a simple and effective optimization-based integration scheme for implicit integration of such systems. We apply this approach to crowd simulation by using a state-of-the-art model derived from a recent analysis of human crowd data, and adapting it to our framework. Our approach provides, for the first time, guaranteed collision-free motion while simultaneously maintaining high-quality collective behavior in a way that is insensitive to simulation parameters such as time step size and crowd density. These benefits are demonstrated through simulation results on various challenging scenarios and validation against real-world crowd data.

Collective Multiagent Sequential Decision Making Under Uncertainty

Multiagent sequential decision making has seen rapid progress with formal models such as decentralized MDPs and POMDPs. However, scalability to large multiagent systems and applicability to real world problems remain limited. To address these challenges, we study multiagent planning problems where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our work exploits recent advances in graphical models for modeling and inference with a population of individuals such as collective graphical models and the notion of finite partial exchangeability in lifted inference. We develop a collective decentralized MDP model where policies can be computed based on counts of agents in different states. As the policy search space over counts is combinatorial, we develop a sampling based framework that can compute open and closed loop policies. Comparisons with previous best approaches on synthetic instances and a real world taxi dataset modeling supply-demand matching show that our approach significantly outperforms them w.r.t. solution quality.

Reachability-Based Safety and Goal Satisfaction of Unmanned Aerial Platoons on Air Highways

Recently, there has been immense interest in using unmanned aerial vehicles for civilian operations. As a result, unmanned aerial systems traffic management is needed to ensure the safety and goal satisfaction of potentially thousands of unmanned aerial vehicles flying simultaneously. Currently, the analysis of large multi-agent systems cannot tractably provide these guarantees if the agents’ set of maneuvers is unrestricted. In this paper, platoons of unmanned aerial vehicles flying on air highways is proposed to impose an airspace structure that allows for tractable analysis. For the air highway placement problem, the fast marching method is used to produce a sequence of air highways that minimizes the cost of flying from an origin to any destination. The placement of air highways can be updated in real time to accommodate sudden airspace changes. Within platoons traveling on air highways, each vehicle is modeled as a hybrid system. Using Hamilton–Jacobi reachability, safety and goal satisfaction are guaranteed for all mode transitions. For a single altitude range, the proposed approach guarantees safety for one safety breach per vehicle; in the unlikely event of multiple safety breaches, safety can be guaranteed over multiple altitude ranges. This paper demonstrates the platooning concept through simulations of three representative scenarios.

Distributed User Association in Energy Harvesting Dense Small Cell Networks: A Mean-Field Multi-Armed Bandit Approach

The emerging Internet of Things (IoT)-driven ultra-dense small cell networks (UD-SCNs) will need to combat a variety of challenges. On one hand, massive number of devices sharing the limited wireless resources will render centralized control mechanisms infeasible due to the excessive cost of information acquisition and computations. On the other hand, to reduce energy consumption from fixed power grid and/or battery, many IoT devices may need to depend on the energy harvested from the ambient environment (e.g., from RF transmissions, environmental sources). However, due to the opportunistic nature of energy harvesting, this will introduce uncertainty in the network operation. In this article, we study the distributed cell association problem for energy harvesting IoT devices in UD-SCNs. After reviewing the state-of-the-art research on the cell association problem in small cell networks, we outline the major challenges for distributed cell association in IoT-driven UD-SCNs where the IoT devices will need to perform cell association in a distributed manner in presence of uncertainty (e.g., limited knowledge on channel/network) and limited computational capabilities. To this end, we propose an approach based on mean-field multi-armed bandit games to solve the uplink cell association problem for energy harvesting IoT devices in a UD-SCN. This approach is particularly suitable to analyze large multi-agent systems under uncertainty and lack of information. We provide some theoretical results as well as preliminary performance evaluation results for the proposed approach.

Combining reward shaping and hierarchies for scaling to large multiagent systems

AbstractCoordinating the actions of agents in multiagent systems presents a challenging problem, especially as the size of the system is increased and predicting the agent interactions becomes difficult. Many approaches to improving coordination within multiagent systems have been developed including organizational structures, shaped rewards, coordination graphs, heuristic methods, and learning automata. However, each of these approaches still have inherent limitations with respect to coordination and scalability. We explore the potential of synergistically combining existing coordination mechanisms such that they offset each others’ limitations. More specifically, we are interested in combining existing coordination mechanisms in order to achieve improved performance, increased scalability, and reduced coordination complexity in large multiagent systems.In this work, we discuss and demonstrate the individual limitations of two well-known coordination mechanisms. We then provide a methodology for combining the two coordination mechanisms to offset their limitations and improve performance over either method individually. In particular, we combine shaped difference rewards and hierarchical organization in the Defect Combination Problem with up to 10 000 sensing agents. We show that combining hierarchical organization with difference rewards can improve both coordination and scalability by decreasing information overhead, structuring agent-to-agent connectivity and control flow, and improving the individual decision-making capabilities of agents. We show that by combining hierarchies and difference rewards, the information overheads and computational requirements of individual agents can be reduced by as much as 99% while simultaneously increasing the overall system performance. Additionally, we demonstrate the robustness of this approach to handling up to 25% agent failures under various conditions.

Scalable Agent Modeling for Large Multiagent Systems

In a heterogeneous multiagent system it can be useful to have knowledge about the different types of agents in the system. Agent modeling develops agent models based on interactions between agents, then predicts agent actions. This approach is effective in small domains but does not scale well. We develop an approach where an agent can learn using an abstract model identification or stereotype rather than an explicit and unique model for each agent. We associate each agent with a stereotype and learn a policy incorporating this knowledge. The benefits of this approach are that it is simple, scalable, and degrades gracefully with misidentification.

Role and member selection in team formation using resource estimation for large-scale multi-agent systems

We propose an efficient team formation method for multi-agent systems consisting of self-interested agents in task-oriented domains where agents have no prior knowledge of the resources/abilities of the other agents. Internet services based on services computing and cloud computing, which have been rapidly increasing, are usually achieved by combining a number of service elements that are distributed over the Internet. We modelled the executions of these elements as teams of agents with the resources and abilities required in the corresponding service elements. This team formation method with the appropriate agents for the service elements makes the entire system efficient. Our proposed method is based on our previous parameter learning method that enables agents to identify their roles in forming a team but requires prior knowledge of all others' resources. This restricts the applicability to real systems. The contribution of this paper is twofold. First, we extended our original method by adding a resource estimation method. Second, we further improved the first extension for large scale multi-agent systems by introducing purviews, which are a relatively small set of agents that are potential members of the teams, for practical computational time and required memory size. We experimentally evaluated our first method by comparing it with the previous method and the task allocation using the contract net protocol (CNP). Then, after increasing the number of agents, we evaluated our second extended method and investigated how the number of agents and the size of the purview affected the overall performances. Results showed that the learning speed was faster in the proposed method so it outperformed other methods in a practical sense even though it did not require prior knowledge of resources in other agents in busy, large-scale, multi-agent systems.

Modeling and Simulation of Complex Network Attributes on Coordinating Large Multiagent System

With the expansion of distributed multiagent systems, traditional coordination strategy becomes a severe bottleneck when the system scales up to hundreds of agents. The key challenge is that in typical large multiagent systems, sparsely distributed agents can only communicate directly with very few others and the network is typically modeled as an adaptive complex network. In this paper, we present simulation testbed CoordSim built to model the coordination of network centric multiagent systems. Based on the token-based strategy, the coordination can be built as a communication decision problem that agents make decisions to target communications and pass them over to the capable agents who will potentially benefit the team most. We have theoretically analyzed that the characters of complex network make a significant difference with both random and intelligent coordination strategies, which may contribute to future multiagent algorithm design.

Discovering the influences of complex network effects on recovering large scale multiagent systems.

Building efficient distributed coordination algorithms is critical for the large scale multiagent system design, and the communication network has been shown as a key factor to influence system performance even under the same coordination protocol. Although many distributed algorithm designs have been proved to be feasible to build their functions in the large scale multiagent systems as claimed, the performances may not be stable if the multiagent networks were organized with different complex network topologies. For example, if the network was recovered from the broken links or disfunction nodes, the network topology might have been shifted. Therefore, their influences on the overall multiagent system performance are unknown. In this paper, we have made an initial effort to find how a standard network recovery policy, MPLS algorithm, may change the network topology of the multiagent system in terms of network congestion. We have established that when the multiagent system is organized as different network topologies according to different complex network attributes, the network shifts in different ways. Those interesting discoveries are helpful to predict how complex network attributes influence on system performance and in turn are useful for new algorithm designs that make a good use of those attributes.

A coordination model for ultra-large scale systems of systems

The ultra large multi-agent systems are becoming increasingly popular due to quick decay of the individual production costs and the potential of speeding up the solving of complex problems. Examples include nano-robots, or systems of nano-satellites for dangerous meteorite detection, or cultures of stem cells for organ regeneration or nerve repair. The topics associated with these systems are usually dealt within the theories of intelligent swarms or biologically inspired computation systems. Stochastic models play an important role and they are based on various formulations of the mechanical statistics. In these cases, the main assumption is that the swarm elements have a simple behaviour and that some average properties can be deduced for the entire swarm. In contrast, complex systems in areas like aeronautics are formed by elements with sophisticated behaviour, which are even autonomous. In situations like this, a new approach to swarm coordination is necessary. We present a stochastic model where the swarm elements are communicating autonomous systems, the coordination is separated from the component autonomous activity and the entire swarm can be abstracted away as a piecewise deterministic Markov process, which constitutes one of the most popular model in stochastic control. Keywords: ultra large multi-agent systems, system of systems, autonomous systems, stochastic hybrid systems.

Robust design of multi-agent system interactions: A testing approach based on pattern matching

The definition of protocols between agents is not enough for guaranteeing the absence of undesirable communication in organizations and the presence of desirable ones in large multi-agent systems (MASs). This is a consequence of the complex system nature of MASs, which cause sophisticated behaviors to arise out of a multiplicity of relatively simple interactions among the independent agents composing them. With this motivation, this paper presents an approach for testing communication in MAS architectures. In this approach, designers are not only recommended to specify the desired communication protocols, but also the undesired patterns and organization structures in the agents’ communications, allowing designers to define robust communication structures. For this purpose, this work presents (1) a language to define such patterns; (2) a set of already defined desired and undesired patterns which usually appear in general MASs; (3) a tool that allows developers to automatically detect these patterns in logs of MAS executions; and (4) a guideline that takes developers through the testing of the communications in MASs. The current approach is experienced with a case study, and the results show that the application of the current approach and the suppression of detected undesired patterns improve the effectiveness and efficiency of the corresponding MAS.

Model checking of emergent behaviour properties of robot swarms

This paper presents a case study on scalability of explicit state model checking. Three state space reduction methods - state vector compression, bit state hashing, and symmetry reduction - were applied on an exercise with the objective of verifying a distributed coordination algorithm for robot swarms. Based on the analysis results, the feasibility of using explicit state model checking to prove properties of large multi-agent systems is questioned and the limitations faced in verifying a dynamic cleaning algorithm are outlined.