Tasks In Multi-agent Systems Research Articles

PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning

Despite many breakthroughs in recent years, it is still hard for MultiAgent Reinforcement Learning (MARL) algorithms to directly solve complex tasks in MultiAgent Systems (MASs) from scratch. In this work, we study how to use Automatic Curriculum Learning (ACL) to reduce the number of environmental interactions required to learn a good policy. In order to solve a difficult task, ACL methods automatically select a sequence of tasks (i.e., curricula). The idea is to obtain maximum learning progress towards the final task by continuously learning on tasks that match the current capabilities of the learners. The key question is how to measure the learning progress of the learner for better curriculum selection. We propose a novel ACL framework, PrOgRessive mulTiagent Automatic curricuLum (PORTAL), for MASs. PORTAL selects curricula according to two critera: 1) How difficult is a task, relative to the learners’ current abilities? 2) How similar is a task, relative to the final task? By learning a shared feature space between tasks, PORTAL is able to characterize different tasks based on the distribution of features and select those that are similar to the final task. Also, the shared feature space can effectively facilitate the policy transfer between curricula. Experimental results show that PORTAL can train agents to master extremely hard cooperative tasks, which can not be achieved with previous state-of-the-art MARL algorithms.

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

Human-inspired strategies to solve complex joint tasks in multi agent systems

Abstract In this paper we propose a methodology to integrate human expertise with effective control laws to drive artificial agents in a complex joint task. We use Supervised Machine Learning to derive human-inspired strategies that succeed in task performance independently from the operating conditions of the samples provided in the training phase. Numerical simulations validate the efficiency of the proposed human-inspired strategies against simpler yet computationally expensive rule-based strategies.

Learning adversarial policy in multiple scenes environment via multi-agent reinforcement learning

ABSTRACT Learning adversarial policy aims to learn behavioural strategies for agents with different goals, is one of the most significant tasks in multi-agent systems. Multi-agent reinforcement learning (MARL), as a state-of-the-art learning-based model, employs centralised or decentralised control methods to learn behavioural strategies by interacting with environments. It suffers from instability and slowness in the training process. Considering that parallel simulation or computation is an effective way to improve training performance, we propose a novel MARL method called Multiple scenes multi-agent proximal Policy Optimisation (MPO) in this paper. In MPO, we first simulate multiple parallel scenes in the training environment. Multiple policies control different agents in the same scene, and each policy also controls several identical agents from multiple scenes. Then, we expand proximal policy optimisation (PPO) with an improved actor-critic network, ensuring the stability of training in multi-agent tasks. The actor network only uses local information for decision making, and the critic network uses global information for training. Finally, effective training trajectories are computed with two criteria from multiple parallel scenes rather than single to accelerate the learning process. We evaluate our approach in two simulated 3D environments, one of which is Unity's official open-source soccer game, and the other is unmanned surface vehicles (USVs) built by Unity. Experiments demonstrate that MPO converges more stable and faster than benchmark methods in model training, and demonstrates excellent adversarial policy compared with benchmark models.

Distributed task allocation in multi-agent environments using cellular learning automata

People’s safety is threatened by the repetition of critical events. Many people lose their lives due to unprofessional rescue operation as well as time pressure of the rescue operation. A key problem in urban search and rescue teams, considering the severe turbulence and complexity of the environments which are hit by a crisis, is the coordination between the team members. In order to solve this problem, an effective plan would be the provision of measures where human works with intelligent assistant agents to assign the tasks in any way. Dynamic tasks are identified by the human agent of the rescue team in the crisis environment and are characterized by spatial–temporal characteristics assigned to the appropriate rescue team by the intelligent assistant agents who apply intelligent decision-making techniques. The objective of this study is to propose a new approach for allocating spatial–temporal tasks in multi-agent systems through cellular learning automata as the decision-making technique. Results obtained here indicate that this proposed model can significantly improve the rescue time and space. Rescue teams could cover all critical areas by going through the minimum distance to make maximum use of time.

Multiagent Resource Allocation for Dynamic Task Arrivals with Preemption

In this article, we present a distributed algorithm for allocating resources to tasks in multiagent systems, one that adapts well to dynamic task arrivals where new work arises at short notice. Our algorithm is designed to leverage preemption if it is available, revoking resource allocations to tasks in progress if new opportunities arise that those resources are better suited to handle. Our multiagent model assigns a task agent to each task that must be completed and a proxy agent to each resource that is available. Preemption occurs when a task agent approaches a proxy agent with a sufficiently compelling need that the proxy agent determines the newcomer derives more benefit from the proxy agent’s resource than the task agent currently using that resource. Task agents reason about which resources to request based on a learning of churn and congestion. We compare to a well-established multiagent resource allocation framework that permits preemption under more conservative assumptions and show through simulation that our model allows for improved allocations through more permissive preemption. In all, we offer a novel approach for multiagent resource allocation that is able to cope well with dynamic task arrivals.

Modeling multi-robot task allocation with limited information as global game

Continuous response threshold functions to coordinate collaborative tasks in multi-agent systems are commonly employed models in a number of fields including ethology, economics, and swarm robotics. Although empirical evidence exists for the response threshold model in predicting and matching swarm behavior for social insects, there has been no formal argument as to why natural swarms use this approach and why it should be used for engineering artificial ones. In this paper, we show, by formulating task allocation as a global game, that continuous response threshold functions used for communication-free task assignment result in system level Bayesian Nash equilibria. Building up on these results, we show that individual agents not only do not need to communicate with each other, but also do not need to model each other’s behavior, which makes this coordination mechanism accessible to very simple agents, suggesting a reason for their prevalence in nature and motivating their use in an engineering context.

Consensus Computation in Unreliable Networks: A System Theoretic Approach

This paper addresses the problem of ensuring trustworthy computation in a linear consensus network. A solution to this problem is relevant for several tasks in multi-agent systems including motion coordination, clock synchronization, and cooperative estimation. In a linear consensus network, we allow for the presence of misbehaving agents, whose behavior deviate from the nominal consensus evolution. We model misbehaviors as unknown and unmeasurable inputs affecting the network, and we cast the misbehavior detection and identification problem into an unknown-input system theoretic framework. We consider two extreme cases of misbehaving agents, namely faulty (non-colluding) and malicious (Byzantine) agents. First, we characterize the set of inputs that allow misbehaving agents to affect the consensus network while remaining undetected and/or unidentified from certain observing agents. Second, we provide worst-case bounds for the number of concurrent faulty or malicious agents that can be detected and identified. Precisely, the consensus network needs to be 2 <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> +1 (resp. <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> +1) connected for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> malicious (resp. faulty) agents to be generically detectable and identifiable by every well behaving agent. Third, we quantify the effect of undetectable inputs on the final consensus value. Fourth, we design three algorithms to detect and identify misbehaving agents. The first and the second algorithm apply fault detection techniques, and affords complete detection and identification if global knowledge of the network is available to each agent, at a high computational cost. The third algorithm is designed to exploit the presence in the network of weakly interconnected subparts, and provides local detection and identification of misbehaving agents whose behavior deviates more than a threshold, which is quantified in terms of the interconnection structure.

Proposal of a Method of Effective Team Formation Using Dynamic Reorganization and Its Evaluation

We propose an effective method of dynamic reorganization using reinforcement learning for the team formation in multi-agent systems (MAS). A task in MAS usually consists of a number of subtasks that require their own resources, and it has to be processed in the appropriate team whose agents have the sufficient resources. The resources required for tasks are often unknown \ extit{a priori} and it is also unknown whether their organization is appropriate to form teams for the given tasks or not. Therefore, their organization should be adopted according to the environment where agents are deployed. In this paper, we investigated how the structures of network and the number of tasks affect team formations of the agents. We will show that the utility and the success of the team formation is deeply affected by depth of the tree structure and number of tasks.

On the role of multiply sectioned bayesian networks to cooperative multiagent systems

We consider a common task in multiagent systems where agents need to estimate the state of an uncertain domain so that they can act accordingly. If each agent only has partial knowledge about the domain and local observations, how can the agents accomplish the task with a limited amount of communication? Multiply sectioned Bayesian networks (MSBNs) provide an effective and exact framework for such a task but also impose a set of constraints. Are there simpler frameworks with the same performance but with less constraints? We identify a small set of high level choices which logically imply the key representational choices leading to MSBNs. The result addresses the necessity of constraints of the framework. It facilitates comparisons with related frameworks and provides guidance to potential extensions of the framework.

On handling component and transaction failures in multi agent systems

Multi agent systems are being used for various practical applications like e-commerce, e-auctions and gathering information from the web. Thus there is a need for these systems to be robust. However, agents can fail due to component failures. The atomic tasks taken up by the agents might also fail. So, agents need to recover to a correct state after a failure. This paper deals with the logging required and hence forth the recovery protocol to recover the agent to a correct state. An agent is modeled as a colored Petrinet. By logging the state information of the agent (Petrinet) an agent recovers from failures. For atomicity of tasks in multi-agent systems, linear two-phase commit protocol with some enhancements is proposed