Cooperative Agents Research Articles

Message Action Adapter Framework in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has demonstrated significant potential in enabling cooperative agents. The communication protocol, which is responsible for message exchange between agents, is crucial in cooperation. However, communicative MARL systems still face challenges due to the noisy messages in complex multi-agent decision processes. This issue often stems from the entangled representation of observations and messages in policy networks. To address this, we propose the Message Action Adapter Framework (MAAF), which first trains individual agents without message inputs and then adapts a residual action based on message components. This separation isolates the effect of messages on action inference. We explore how training the MAAF framework with model-agnostic message types and varying optimization strategies influences adaptation performance. The experimental results indicate that MAAF achieves competitive performance across multiple baselines despite utilizing only half of the available communication, and shows an average improvement of 7.6% over the full attention-based communication approach. Additional findings suggest that different message types result in significant performance variations, emphasizing the importance of environment-specific message types. We demonstrate how the proposed architecture separates communication channels, effectively isolating message contributions.

Employing artificial intelligence to steer exascale workflows with colmena

Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. Our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.

Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility

This paper investigates the use of deep multi-agent reinforcement learning (MARL) for the coordination of residential energy flexibility. Particularly, we focus on achieving cooperation between homes in a way that is fully privacy-preserving, scalable, and that allows for the management of distribution network voltage constraints. Previous work demonstrated that MARL-based distributed control can be achieved with no sharing of personal data required during execution. However, previous cooperative MARL-based approaches impose an ever greater training computational burden as the size of the system increases, limiting scalability. Moreover, they do not manage their impact on distribution network constraints. We therefore adopt a deep multi-agent actor–critic method that uses a centralised but factored critic to rehearse coordination ahead of execution, such that homes can successfully cooperate at scale, with only first-order growth in computational time as the system size increases. Training times are thus 34 times shorter than with a previous state-of-the-art reinforcement learning approach without the factored critic for 30 homes. Moreover, experiments show that the cooperation of agents allows for a decrease of 47.2% in the likelihood of under-voltages. The results indicate that there is significant potential value for management of energy user bills, battery depreciation, and distribution network voltage management, with minimal information and communication infrastructure requirements, no interference with daily activities, and no sharing of personal data.

Multi-agent, human–agent and beyond: A survey on cooperation in social dilemmas

The study of cooperation within social dilemmas has long been a fundamental topic across various disciplines, including computer science and social science. Recent advancements in Artificial Intelligence (AI) have significantly reshaped this field, offering fresh insights into understanding and enhancing cooperation. This survey examines three key areas at the intersection of AI and cooperation in social dilemmas. First, focusing on multi-agent cooperation, we review the intrinsic and external motivations that support cooperation among rational agents, and the methods employed to develop effective strategies against diverse opponents. Second, looking into human–agent cooperation, we discuss the current AI algorithms for cooperating with humans and the human biases towards AI agents. Third, we review the emergent field of leveraging AI agents to enhance cooperation among humans. We conclude by discussing future research avenues, such as using large language models, establishing unified theoretical frameworks, revisiting existing theories of human cooperation, and exploring multiple real-world applications.

Periodic update rule with Q-learning promotes evolution of cooperation in game transition with punishment mechanism

Cooperative behavior assumes a critical role in resolving conflicts arising between collective and individual interests, while punishment measures serve as a robust deterrent against opportunistic free-riding. Within this context, evolutionary game theory (EGT) emerges as an indispensable paradigm for addressing this multifaceted issue. When it comes to introspection behaviors, reinforcement learning (RL) methods exhibit remarkable capabilities to capture agents’ cognitive processes. Nonetheless, previous research has often focused on a static and time-invariant update rule, neglecting the dynamic nature of real-world scenarios where individuals can flexibly transit between strategies in periodic time-dependent patterns. Here, we propose periodic update rules with Q-learning algorithm and game transition model with a punishment mechanism that grants cooperative agents the autonomy to exercise discretion in deciding whether to initiate punishment actions. The agents display dynamic rules periodically through game model transitions, thus ensuring EGT’s inherent adaptability. By employing Monte Carlo (MC) simulations, we analyze the emergence of cooperation that underscores the substantial enhancement of cooperative behavior through the proposed periodic update rules with Q-learning algorithm and game transitions in the presence of punishment. Our study highlights the indispensable significance of appropriate periodic intervals for updating rules and determining optimal punishment costs in the game transition model as critical elements for fostering the evolution of cooperation in real-world scenarios.

Navigating ambiguity: A novel neutrosophic cubic shapley normalized weighted Bonferroni Mean aggregation operator with application in the investment environment

The Neutrosophic Cubic Shapley Normalized Bonferroni (NC-SNWBM) method represents a cutting-edge approach to decision making theory, combining three distinct mathematical frameworks the neutrosophic cubic sets (NCS), Shapley values, and the Bonferroni aggregation operator. This innovative method addresses the challenges posed by uncertainty, vagueness, and imprecision in decision-making (DM) processes, offering a comprehensive and versatile tool for handling complex and dynamic scenarios. Neutrosophic cubic sets offers a strong platform to handle ambiguous and vague data due to three components Membership Grade (MG), Non-Membership Grade (NMG) and Indeterminancy Grade (IG) in data. By adding Shapley Fuzzy Measures (SFM), which come from cooperative game theory, distribute values among cooperative agents equally and to account for each agent's contributions to all potential coalitions. The Bonferroni aggregation operator—a statistical aggregative tool that regulates the likelihood of many types in error in statistical tests and the interdependence of the input arguments by allowing different values to parameters involved. These values are further improved by normalization in the framework of the NC-SNWBM approach in order to consider the various degrees of impact that agents exert in various circumstances. This operator is smoothly combined with normalized Shapley values and neutrosophic cubic sets in the NC-SNWBM approach to enable the aggregation of data with different levels of imprecision and uncertainty from various sources using NCS. The MG, NMG and IG connected to NCS are important elements of the NC-SNWBM approach. To evaluate each element's contribution to the overall value distribution SFM are used, and the Bonferroni aggregation operator maintains a careful balance between conservatism and significance. Together, these components provide a thorough framework that successfully tackles the problems caused by ambiguity, imprecision, and uncertainty in scenarios involving decision-making. The NC-SNWBM operator is applied to a numerical problem as an application in investment environment and sensitive and comparative analysis are conducted. The recommendation based on sensitive and comparative analysis proposed.

Multi-Agent Collaborative Path Planning Algorithm with Multiple Meeting Points

Traditional multi-agent path planning algorithms often lead to path overlap and excessive energy consumption when dealing with cooperative tasks due to the single-agent-single-task configuration. For this reason, the “many-to-one” cooperative planning method has been proposed, which, although improved, still faces challenges in the vast search space for meeting points and unreasonable task handover locations. This paper proposes the Cooperative Dynamic Priority Safe Interval Path Planning with a multi-meeting-point and single-meeting-point solving mode switching (Co-DPSIPPms) algorithm to achieve multi-agent path planning with task handovers at multiple or single meeting points. First, the initial priority is set based on the positional relationships among agents within the cooperative group, and the improved Fermat point method is used to locate multiple meeting points quickly. Second, considering that agents must pick up sub-tasks or conduct task handovers midway, a segmented path planning strategy is proposed to ensure that cooperative agents can efficiently and accurately complete task handovers. Finally, an automatic switching strategy between multi-meeting-point and single-meeting-point solving modes is designed to ensure the algorithm’s success rate. Tests show that Co-DPSIPPms outperforms existing algorithms in 1-to-1 and m-to-1 cooperative tasks, demonstrating its efficiency and practicality.

Optimal management of cooperative energy communities: impact of agent cooperation

In this work we analyse the profitability of the agents participating in a Local Energy Community (LEC); the agents in these communities can be either passive or active. The main difference is that active agents own generating units that they need to dispatch, whereas passive agents do not own any generating units. The LEC is operated and managed by a coordinating agent that is in charge of all energy exchanges between the agents in the LEC and the external Public Electricity Network (PEN). We propose a general optimization-based mathematical model to study the interactions among all the participants and to foresee strategies and optimal behaviour within the rules of the LEC. With this model we can compute optimal costs, profits, energy purchases and energy sales, assessing the best configuration for the LEC as a whole and for each individual participant.

Enhancing Reflective and Conversational User Engagement in Argumentative Dialogues with Virtual Agents

In their process of information seeking, human users tend to selectively ignore information that contradicts their pre-existing beliefs or opinions. These so-called “self-imposed filter bubbles” (SFBs) pose a significant challenge for argumentative conversational agents aiming to facilitate critical, unbiased opinion formation on controversial topics. With the ultimate goal of developing a system that helps users break their self-imposed filter bubbles (SFBs), this paper aims to investigate the role of co-speech gestures, specifically examining how these gestures significantly contribute to achieving this objective. This paper extends current research by examining methods to engage users in cooperative discussions with a virtual human-like agent, encouraging a deep reflection on arguments to disrupt SFBs. Specifically, we investigate the agent’s non-verbal behavior in the form of co-speech gestures. We analyze whether co-speech gestures, depending on the conveyed information, enhance motivation, and thus conversational user engagement, thereby encouraging users to consider information that could potentially disrupt their SFBs. The findings of a laboratory study with 56 participants highlight the importance of non-verbal agent behaviors, such as co-speech gestures, in improving users’ perceptions of the interaction and the conveyed content. This effect is particularly notable when the content aims to challenge the user’s SFB. Therefore, this research offers valuable insights into enhancing user engagement in the design of multimodal interactions with future cooperative argumentative virtual agents.

Development of a Zr-Based Metal-Organic Framework (UiO-66) for a Cooperative Flame Retardant in the PC/ABS.

Polycarbonate/acrylonitrile butadiene styrene (PC/ABS) blends are widely used as engineering plastic alloys; however, they have a low fire safety level. To improve the flame-retardant property of PC/ABS, a zirconium-based metal-organic framework material (UiO-66) was synthesized with zirconium chloride and terephthalic acid and used as a flame-retardant cooperative agent. Its flame-retardant performance and mode of action in the PC/ABS blends were carefully investigated. The results showed that UiO-66 had good thermal stability and delayed the pyrolysis of the materials, thus significantly enhancing the efficiency of intumescent flame retardants. By compounding 7.0 wt% hexaphenyloxy-cyclotri-phosphazene (HPCTP) with 3.0 wt% UiO-66, the PC/ABS blends reached a limiting oxygen index value of 27.0% and V0 rating in the UL-94 test, showing significantly improved resistance to combustion dripping. In addition, UiO-66 enhanced the smoke and heat suppression characteristics of the intumescent flame-retardant materials. Finally, the flame-retardant mode of action in the blends was indicative of UiO-66 having a cooperative effect on the flame-retardant performance of PC/ABS/HPCTP materials. This work provides good ideas for further development of the flame-retardant ABS/PC.

A hybrid multi-agent metaheuristic for the offshore wind farm cable routing problem

A crucial task concerning offshore wind farm designs is connecting previously placed turbines to a substation, such that the total cable cost is minimal. An NP-hard optimization problem named Wind Farm Cable Routing Problem (WFCRP) models this task. Previous studies have proposed Mixed Integer Linear Programming formulations (MILP) and metaheuristics for the WFCRP. However, they faced difficulties in solving instances with many turbines. Hybridization is a widely-explored approach that has produced better solutions as it combines the best features of individual algorithms. This paper proposes a hybrid algorithm for the WFCRP that combines four state-of-the-art metaheuristics and a MILP formulation. The hybridization technique uses multi-agent and Particle Swarm Optimization concepts, where particles are cooperative agents that work together to find high-quality solutions. Experimental results on instances with up to 120 turbines show that the proposed hybrid algorithm outperforms the best algorithm from the WFCRP literature.

Resilient leaderless and leader–follower consensus over random networks through [formula omitted]-hop communication

This paper studies the resilient consensus problems under ℓ-hop communication over directed random networks. We develop novel multi-hop protocols to tackle both leaderless and leader–follower consensus with a single leader when the multiagent system is under both attack from Byzantine agents and failure of communication edges. Our unified framework features a multiple-input multiple-output system design, and generalizes the weighted mean subsequence reduced algorithms, in which extremal values in the ℓ-hop neighborhood of a cooperative agent are censored based on the notion of minimum message cover. We establish conditions on dynamical network structures, under which secure consensus with vector states can be achieved in the sense of almost sure convergence.

Multi-agent system-assisted fault section location scheme for distribution networks with DFIG-based wind DGs

This paper proposes a multi-agent system (MAS)-assisted relaying scheme to identify the fault section in a distribution network (DN) that incorporates doubly-fed induction generators (DFIG)-based wind distributed generations (DGs). Firstly, to mitigate the adverse effect of the uncertain fault behavior of DFIGs on conventional directional elements, the fault direction identifiers (FDIs) are assigned at the I-type and II-type buses in the DN using the high-frequency component principle. Furthermore, a MAS-assisted relaying structure is devised to collect and integrate these identifiers and then accomplish the fault section location. On the one hand, since the setting of FDIs only requires local current information and the logical signals are solely exchanged within each predetermined associated region, the proposed fault section location scheme can be implemented in an immature DN with a limited number of voltage transformers or under weak synchronization conditions. On the other hand, the autonomy and cooperation of intelligent Agents also enable the proposed scheme to determine the fault section in a decentralized decision-making (DDM) mode, eliminating the dependence on strong central control and reducing redundant data transmission in a multi-bus distribution system. Simulation results under representative faulty scenarios validate the expected performance of the protection scheme.

Pedestrian evacuations with imitation of cooperative behavior.

We analyze the dynamics of room evacuation for mixed populations that include both competitive and cooperative individuals through numerical simulations using the social force model. Cooperative agents represent well-trained individuals who know how to behave in order to reduce risks within high-density crowds. We consider that competitive agents can imitate cooperative behavior when they are in close proximity to cooperators. We study the effects of the imitation of cooperative behavior on the duration and safety of evacuations, analyzing evacuation time and other quantities of interest for varying parameters such as the proportions of mixing, the aspect ratio of the room, and the parameters characterizing individual behaviors. Our main findings reveal that the addition of a relatively small number of cooperative agents into a crowd can reduce evacuation time and the density near the exit door, making the evacuation faster and safer despite an increase in the total number of agents. In particular, for long spaces such as corridors, a small number of added cooperative agents can significantly facilitate the evacuation process. We compare our results with those of systems without imitation and also study the general role of cooperation, providing further analysis for homogeneous populations. Our main conclusions emphasize the potential relevance of training people how to behave in high-density crowds.

Enhancing hierarchical remote preparation of an arbitrary n-qudit equatorial state by weak measurement and measurement reversal

Based on the demand for safer and more efficient quantum network communication, we propose two deterministic protocols to achieve the remote preparation of an arbitrary n-qudit equatorial states among multiple agents by selecting appropriate entanglement resources. The level of the agents varies depending on their authorities to restore the secret state. The superior agent requires the cooperation of the residual superior agents and any one of the subordinate agents, while the subordinate agent needs the help of all the other agents. To be emphasized, the recovery operations are deduced by general formulas that distinctly elucidate their relevance with the measurement outcomes. Additionally, we discuss the impact of amplitude damping noise and utilize weak measurement and measurement reversal to suppress noise.

Adaptive bipartite output containment control of heterogeneous multi‐agent systems against FDI attacks

AbstractThis paper researches the adaptive bipartite output containment control of heterogeneous multi‐agent systems (HMASs) against false data injection (FDI) attacks. The malicious adversaries could inject attacks on the actuator to maximize damage to the HMASs, which in turn causes the compromised followers to be unable to track the output states of the leaders. To address this issue, a resilient layer is introduced that prevents attack signals from affecting the states. Moreover, an adaptive compensator is ingeniously devised to counteract these attacks, thereby ensuring that both cooperative and antagonistic agents adhere to distinct convex hulls. Leveraging the output regulation approach, conducting Lyapunov stability analysis, and drawing upon other pertinent methodologies, we have systematically constructed a robust framework for the realization of bipartite output containment control. Ultimately, the validity of the analytical findings is confirmed through a series of rigorous numerical simulations.

Synergistic Humidification and Chemical Agglomeration to Improve Capturing the Fine Particulate Matter by Electrostatic Precipitator

The wet electrostatic precipitator (WESP) overcomes the shortcomings of traditional electrostatic precipitators, such as dust re-entrainment and back corona. It can effectively remove high-specific-resistivity dust, with a good removal effect on PM2.5. It is proposed to adopt chemical agglomeration and humidification agglomeration technology in the wet electrostatic precipitators to achieve ultra-low dust emissions from coal-fired power plants. The results show that the addition of chemical agglomerates, surfactants, and water vapor all affect the dust diameter of coal-fired power plants. After adding sesbania gum (SG), the D50 of dust particles increases from 28.29 μm to 48.22 μm. And the D50 of dust particles is 36.46 μm when spraying 3.6 kg/h water vapor only. With the cooperation of chemical agglomeration agents and water vapor, the dust agglomeration effect and removal efficiency can be further improved. When 10 mg/L SG is synergistically combined with 2.9 kg/h water vapor, the D50 is 64.75 μm, and the dust removal efficiency reaches 97.88%. On this basis, by adding 5 mg/L of Hexadecyltrimethylammonium bromide (CTAB), the D50 is 83.06 μm, and the dust removal efficiency increases to 98.62%. The synergistic effect of chemical agglomeration and humidification agglomeration promotes the aggregation of dust from coal-fired power plants. It can improve the removal efficiency of WESP for fine particulate matter but has little impact on the operation of existing equipment. The synergistic effects of multiple agglomeration technologies are also the direction for future research on the removal efficiency of fine particulate matter.

ProAgent: Building Proactive Cooperative Agents with Large Language Models

Building agents with adaptive behavior in cooperative tasks stands as a paramount goal in the realm of multi-agent systems. Current approaches to developing cooperative agents rely primarily on learning-based methods, whose policy generalization depends heavily on the diversity of teammates they interact with during the training phase. Such reliance, however, constrains the agents' capacity for strategic adaptation when cooperating with unfamiliar teammates, which becomes a significant challenge in zero-shot coordination scenarios. To address this challenge, we propose ProAgent, a novel framework that harnesses large language models (LLMs) to create proactive agents capable of dynamically adapting their behavior to enhance cooperation with teammates. ProAgent can analyze the present state, and infer the intentions of teammates from observations. It then updates its beliefs in alignment with the teammates' subsequent actual behaviors. Moreover, ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various of coordination scenarios. Experimental evaluations conducted within the Overcooked-AI environment unveil the remarkable performance superiority of ProAgent, outperforming five methods based on self-play and population-based training when cooperating with AI agents. Furthermore, in partnered with human proxy models, its performance exhibits an average improvement exceeding 10% compared to the current state-of-the-art method. For more information about our project, please visit https://pku-proagent.github.io.

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means sub-optimal actions by some agents will affect other agent's policy learning. While using individual critics for policy updates can avoid this issue, they severely limit cooperation among agents. To address this issue, we propose an agent topology framework, which decides whether other agents should be considered in policy gradient and achieves compromise between facilitating cooperation and alleviating the CDM issue. The agent topology allows agents to use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. To constitute the agent topology, various models are studied. We propose Topology-based multi-Agent Policy gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove the policy improvement theorem for stochastic TAPE and give a theoretical explanation for the improved cooperation among agents. Experiment results on several benchmarks show the agent topology is able to facilitate agent cooperation and alleviate CDM issue respectively to improve performance of TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm are devised to show the efficacy of the agent topology.

Exercise Promotion System for Single Households Based on Agent-Oriented IoT Architecture.

People living alone encounter well-being challenges due to unnoticed personal situations. Thus, it is essential to monitor their activities and encourage them to adopt healthy lifestyle habits without imposing a mental burden, aiming to enhance their overall well-being. To realize such a support system, its components should be simple and loosely coupled to handle various internet of things (IoT)-based smart home applications. In this study, we propose an exercise promotion system for individuals living alone to encourage them to adopt good lifestyle habits. The system comprises autonomous IoT devices as agents and is realized using an agent-oriented IoT architecture. It estimates user activity via sensors and offers exercise advice based on recognized conditions, surroundings, and preferences. The proposed system accepts user feedback to improve status estimation accuracy and offers better advice. The proposed system was evaluated from three perspectives through experiments with subjects. Initially, we demonstrated the system's operation through agent cooperation. Then, we showed it adapts to user preferences within two weeks. Third, the users expressed satisfaction with the detection accuracy regarding their stay-at-home status and the relevance of the advice provided. They were also motivated to engage in exercise based on a subjective evaluation, as indicated by preliminary results.