Multi-agent Cooperation Research Articles

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

MARP: A Cooperative Multiagent DRL System for Connected Autonomous Vehicle Platooning

MARP: A Cooperative Multiagent DRL System for Connected Autonomous Vehicle Platooning

Resolving social dilemmas with minimal reward transfer

Social dilemmas present a significant challenge in multi-agent cooperation because individuals are incentivised to behave in ways that undermine socially optimal outcomes. Consequently, self-interested agents often avoid collective behaviour. In response, we formalise social dilemmas and introduce a novel metric, the general self-interest level, to quantify the disparity between individual and group rationality in such scenarios. This metric represents the maximum proportion of their individual rewards that agents can retain while ensuring that a social welfare optimum becomes a dominant strategy. Our approach diverges from traditional concepts of altruism, instead focusing on strategic reward redistribution. By transferring rewards among agents in a manner that aligns individual and group incentives, rational agents will maximise collective welfare while pursuing their own interests. We provide an algorithm to compute efficient transfer structures for an arbitrary number of agents, and introduce novel multi-player social dilemma games to illustrate the effectiveness of our method. This work provides both a descriptive tool for analysing social dilemmas and a prescriptive solution for resolving them via efficient reward transfer contracts. Applications include mechanism design, where we can assess the impact on collaborative behaviour of modifications to models of environments.

Evolutionary game analysis of multi-agent cooperation strategy analysis in agricultural water conservancy PPP project under digitization background

This study constructed a system dynamics (SD) evolutionary game model, from which we analyzed the conditions that need to be met by multi-agents to achieve cooperation in agricultural water conservancy (AWC) Public-Private Partnership (PPP) projects. Simultaneously, this study used numerical simulation to depict the impact of initial participation willingness, reward, punishment, income, and other parameters of local governments, project enterprises, and farmers on the evolution strategy. The results firstly showed that the basic conditions for tripartite cooperation are to strengthen the rewards and punishments for project enterprises and improve the participation benefits of farmers. Secondly, it showed that increasing the rewards for farmers and improving the project dividend benefits can effectively motivate farmers to participate. Thirdly, it showed that enhancing the incentive subsidies and reputation benefits for project enterprise’s participation, as well as strengthening the supervision of local government, boosts the motivation of project enterprises significantly. However, it is vital to ensure that the reward amounts are not too high to allow local government to play its guiding role efficiently. Lastly, the results indicated that digital technology can help reduce the participation cost of local government, project enterprises and farmers, and improve the external reputation benefits of each participant, thus forming a win-win situation for all three participants. Analyzing the cooperation strategies of the three parties in AWC PPP projects provides theoretical support and a decision-making basis for the efficient operation of AWC PPP projects under the digital background, and also provides practical policy recommendations for the high-quality development of AWC.

Multi-agent, human–agent and beyond: A survey on cooperation in social dilemmas

The study of cooperation within social dilemmas has long been a fundamental topic across various disciplines, including computer science and social science. Recent advancements in Artificial Intelligence (AI) have significantly reshaped this field, offering fresh insights into understanding and enhancing cooperation. This survey examines three key areas at the intersection of AI and cooperation in social dilemmas. First, focusing on multi-agent cooperation, we review the intrinsic and external motivations that support cooperation among rational agents, and the methods employed to develop effective strategies against diverse opponents. Second, looking into human–agent cooperation, we discuss the current AI algorithms for cooperating with humans and the human biases towards AI agents. Third, we review the emergent field of leveraging AI agents to enhance cooperation among humans. We conclude by discussing future research avenues, such as using large language models, establishing unified theoretical frameworks, revisiting existing theories of human cooperation, and exploring multiple real-world applications.

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

Integrated Sensing and Communication for Effective Multi-Agent Cooperation Systems

Integrated Sensing and Communication for Effective Multi-Agent Cooperation Systems

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

Temporal graph convolutional network for multi-agent reinforcement learning of action detection

Most action detection techniques process untrimmed action videos using temporal context, without explicitly consider the inherent spatio-temporal context information, leading to limited accuracy in the cases with big spatial complexity and temporal redundancy. To this end, by intuitively dealing with the problem following a two-stage ”clip selection + clip classification” scheme, this paper proposes to formulate action detection as a Markov process and builds up a multi-agent reinforcement learning framework capturing global structural relationships of videos to optimize the selection and classification, simultaneously and progressively. In particular, a temporal graph convolutional network is constructed to represent the spatio-temporal correlations of video clips, which are initialized by evenly sampling, and further adjusted via learning the rewards adaptively for multi-agent cooperation. Multi-head dot-product attention mechanism is adopted to integrate the relations of latent CNN features of interacting agents. Our framework is jointly learnt by fusing the objectives of clip selection policy and clip recognition. The proposed method comprises a novel graph convolutional network based spatio-temporal semantic observation module which captures topological features among nearby agents, and a new policy module that segments actions according to the rewards from the objectives of action recognition. Extensive experiments are conducted on ActivityNet v1.3 and THUMOS14, with 30.13% and 55.4% mAP obtained, demonstrate the applicability and superiority of our approach.

CycLight: Learning traffic signal cooperation with a cycle-level strategy

This study introduces CycLight, a novel cycle-level deep reinforcement learning (RL) approach for network-level adaptive traffic signal control (NATSC) systems. Traditional RL-based step-by-step traffic signal controllers, while precise and responsive, demand high computational resources for frequent data exchanges and can increase traffic accident risks due to disordered phase switching. Different from the step-by-step control strategy aforementioned, the proposed CycLight adopts a cycle-level strategy, optimizing cycle length and splits simultaneously using Parameterized Deep Q-Networks (PDQN) algorithm. This cycle-level approach effectively reduces the computational burden associated with frequent data communication, meanwhile enhancing the practicality and safety of real-world applications. Moreover, a decentralized framework is formulated for multi-agent cooperation, while attention mechanism is integrated to accurately assess the impact of the surroundings on the current intersection. CycLight is tested in a large synthetic traffic grid using the microscopic traffic simulation tool, Simulation of Urban MObility (SUMO). Experimental results not only demonstrate the superiority of CycLight over other state-of-the-art approaches but also showcase its robustness against information transmission delays.

Efficient multi-agent cooperation: Scalable reinforcement learning with heterogeneous graph networks and limited communication

This paper addresses the challenge of scalable multi-agent reinforcement learning (MARL) under partial observability and communication constraints. An Efficient Multi-Agent Cooperation (EMAC) approach is proposed, which comprises a Heterogeneous Graph Network (HGN) and a Frame-wise Communication Reduction Algorithm (FCRA). HGN can effectively extract semantic information from local observations, while FCRA utilizes historical embeddings to achieve an approximation of multi-hop communication with 1-hop communication. Through comparative experiments, EMAC showed a 30% improvement in efficiency over the best-performing benchmark, scalability to arbitrary numbers of agents, and adaptation to unstructured environments. A t-SNE analysis of agent embeddings further reveals EMAC’s capability to understand and leverage semantic similarities among agents. These results highlight EMAC’s potential as a robust framework for decentralized multi-agent decision-making in complex and dynamic environments.

A Local-and-Global Attention Reinforcement Learning Algorithm for Multiagent Cooperative Navigation.

The cooperative navigation algorithm is the crucial technology for multirobot systems to accomplish autonomous collaborative operations, and it is still a challenge for researchers. In this work, we propose a new multiagent reinforcement learning algorithm called multiagent local-and-global attention actor-critic (MLGA2C) for multiagent cooperative navigation. Inspired by the attention mechanism, we design the local-and-global attention module to dynamically extract and encode critical environmental features. Meanwhile, based on the centralized training and decentralized execution (CTDE) paradigm, we extend a new actor-critic method to handle feature encoding and make navigation decisions. We also evaluate the proposed algorithm in two cooperative navigation scenarios: static target navigation and dynamic pedestrian target tracking. The multiple experimental results show that our algorithm performs well in cooperative navigation tasks with increasing agents.

Cooperative Multiagent Transfer Learning With Coalition Pattern Decomposition

Cooperative Multiagent Transfer Learning With Coalition Pattern Decomposition

Locally Centralized Execution for Less Redundant Computation in Multi-Agent Cooperation

Decentralized execution is a widely used framework in multi-agent reinforcement learning. However, it has a well-known but neglected shortcoming, redundant computation, that is, the same/similar computation is performed redundantly in different agents owing to their overlapping observations. This study proposes a novel method, the locally centralized team transformer (LCTT), to address this problem. This method first proposes a locally centralized execution framework that autonomously determines some agents as leaders that generate instructions and other agents as workers to act according to the received instructions without running their policy networks. For the LCTT, we subsequently propose the team-transformer (T-Trans) structure, which enables leaders to generate targeted instructions for each worker, and the leadership shift, which enables agents to determine those that should instruct or be instructed by others. The experimental results demonstrated that the proposed method significantly reduces redundant computations without decreasing rewards and achieves faster learning convergence.

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

Autonomous agents operating in real-world environments frequently encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. Even when agents can execute their primary task optimally when operating in isolation, their training may not account for potential negative interactions that arise in the presence of other agents. We frame the challenge of minimizing NSEs as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but recognize that addressing negative side effects creates a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks---up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

Optimization of Energy Efficiency for Uplink mURLLC Over Multiple Cells Using Cooperative Multiagent Reinforcement Learning

Optimization of Energy Efficiency for Uplink mURLLC Over Multiple Cells Using Cooperative Multiagent Reinforcement Learning

Decentralized multi-agent cooperation via adaptive partner modeling

Multi-agent reinforcement learning encounters a non-stationary challenge, where agents concurrently update their policies, leading to changes in the environment. Existing approaches have tackled this challenge through communication among agents to obtain their partners’ actions, but this introduces computational complexity known as partner sample complexity. An alternative approach is to develop partner models that generate samples instead of direct communication to mitigate this complexity. However, a discrepancy arises between the real policies distribution and the policy of partner models, termed as model bias, which can significantly impact performance when heavily relying on partner models. In order to achieve a trade-off between sample complexity and performance, a novel multi-agent model-based reinforcement learning algorithm called decentralized adaptive partner modeling (DAPM) is proposed, which utilizes fictitious self play (FSP) to construct partner models and update policies. Model bias is addressed by establishing an upper bound to restrict the usage of partner models. Coupled with that, an adaptive rollout approach is introduced, enabling real agents to dynamically communicate with partner models based on their quality, ensuring that agent performance can progressively improve with partner model samples. The effectiveness of DAPM is exhibited in two multi-agent tasks, showing that DAPM outperforms existing model-free algorithms in terms of partner sample complexity and training stability. Specifically, DAPM requires 28.5% fewer communications compared to the best baseline and exhibits reduced fluctuations in the learning curve, indicating superior performance.

Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning

When agents need to collaborate without previous coordination, the multi-agent cooperation problem transforms into an ad hoc teamwork (AHT) problem. Mainstream research on AHT is divided into type-based and type-free methods. The former depends on known teammate types to infer the current teammate type, while the latter does not require them at all. However, in many real-world applications, the complete absence and sufficient knowledge of known types are both impractical. Thus, this research focuses on the challenge of AHT with limited known types. To this end, this paper proposes a method called a Few typE-based Ad hoc Teamwork via meta-reinforcement learning (FEAT), which effectively adapts to teammates using a small set of known types within a single episode. FEAT enables agents to develop a highly adaptive policy through meta-reinforcement learning by employing limited priors about known types. It also utilizes this policy to generate a diverse type repository automatically. During the ad hoc cooperation, the agent can autonomously identify known teammate types followed by directly utilizing the pre-trained optimal cooperative policy or swiftly updating the meta policy to respond to teammates of unknown types. Comprehensive experiments in the pursuit domain validate the effectiveness of the algorithm and its components.

Cooperative Multiagent Reinforcement Learning Coupled With A* Search for Ship Multicabin Equipment Layout Considering Pipe Route

_ The paper presents a novel approach of cooperative multiagent reinforcement learning (CMARL) combined with A* search to address ship multicabin equipment layout considering pipe route, aiming to minimize pipe cost while considering practical requirements. The formulation is established through equipment simplification and grid marking, and A* search is utilized to value the pipe route. By designing equipment states, the equipment layout in each cabin is solved using a CMARL approach that involves three actions. Subsequently, comparative experiments were conducted on an engine room case by CMARL against genetic algorithm and single multiagent reinforcement learning methods under the condition of coupling with A* search. The parameter values for these methods were sampled using Latin Hypercube. The findings demonstrate that CMARL has superior combination properties. Keywords ship equipment layout; multicabin layout; cooperative multiagent reinforcement learning; A* search; pipe route

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Deep reinforcement learning has become a prominent area of research in artificial intelligence in recent years. Its application in solving complex tasks and game environments has garnered significant attention. This study aims to develop a deep reinforcement learning algorithm based on multi-agent parallelism to enhance intelligent decision-making in game environments. The algorithm combines a deep Q-network with a multi-agent cooperation strategy. Through parallel training of multiple agents, the learning process is accelerated, and decision accuracy is improved. The experimental results indicated that the Actor-Critic algorithm, when combined with precision rate, recall rate, and average fitness of multi-agent parallel, achieves a relatively high accuracy rate index, which stabilizes above 0.95. The recall rate index was also above 0.8, and the average fitness was in a relatively high range. The research shows that the deep reinforcement learning algorithm based on multi-agent parallelism performs better and is more effective in game environments. It can learn the optimal strategy faster and obtain higher rewards. This not only provides a new technical means for game development but also offers a useful reference for the application of multi-agent systems in complex environments.