Multi-agent Reinforcement Learning Research Articles

ARLP: Automatic multi-agent transformer reinforcement learning pruner for one-shot neural network pruning

Overparameterized Neural Networks demonstrate state-of-the-art performance; however, the escalating demand for more compact and energy-efficient neural networks has arisen to facilitate the deployment of machine learning applications on devices with limited computational resources. A prevalent approach employs various pruning techniques. However, hyperparameters, such as the pruning ratio for each layer in pruning techniques, are typically set by human experts and often lack optimization. In this paper, we therefore propose a novel method named “Automatic Multi-Agent Transformer Reinforcement Learning Pruner” (ARLP). ARLP leverages a transformer-based multi-agent reinforcement learning controller to autonomously prune networks at initialization by extracting network meta-features. This autonomous process eliminates the need for human intervention in determining the optimal pruning ratio for each layer. The meta-features are derived from various zero-cost pruning-at-initialization proxies to perform One-shot Pruning. Extensive experimental results demonstrate that ARLP outperforms other state-of-the-art methods, establishing its efficacy in achieving superior performance.

Learning Agents in Robot Navigation: Trends and Next Challenges

Multiagent reinforcement learning performs well in multiple situations such as social simulation and data mining. It particularly stands out in robot control. In this approach, artificial agents behave in a system and learn their policies for their own satisfaction and that of others. Robots encode policies to simulate the performance. Therefore, learning should maintain and improve system performance. Previous studies have attempted various approaches to outperform control robots. This paper provides an overview of multiagent reinforcement learning work, primarily on navigation. Specifically, we discuss current achievements and limitations, followed by future challenges.

A Study for Comparative Analysis of Dueling DQN and Centralized Critic Approaches in Multi-Agent Reinforcement Learning

In this study, we introduce a deep Q-network agent utilizing a dueling architecture to refine the valuation of actions through separate estimations of the state-value and action-value functions, adapted to facilitate concurrent multi-agent operations within a shared environment. Inspired by the self-organized, decentralized cooperation observed in natural swarms, this study uniquely integrates a centralized mechanism, or a centralized critic. This enhances performance and coherence in decision-making within the multi-agent system. This hybrid approach enables agents to execute informed and optimized decisions by considering the actions of their counterparts while maintaining an element of collective and flexible task-information sharing, thereby presenting a groundbreaking framework for cooperation and information sharing in swarm robot systems. To augment the communication capabilities, we employ low-power wide-area networks, or Long Range (LoRa), which are characterized by their low power consumption and long-range communication abilities, facilitating the sharing of task information and reducing the load on individual robots. The aim is to leverage LoRa as a communication platform to construct a cooperative algorithm that enables efficient task-information sharing among groups. This can provide innovative solutions and promote effective cooperation and communication within multi-agent systems, with significant implications for industrial and exploratory robots. In conclusion, by integrating a centralized system into the proposed model, this approach successfully enhances the performance of multi-agent systems in real-world applications, offering a balanced synergy between decentralized flexibility and centralized control.

A review of research on reinforcement learning algorithms for multi-agents

In recent years, multi-agent reinforcement learning techniques have been widely used and evolved in the field of artificial intelligence. However, traditional reinforcement learning methods have limitations such as long training time, large sample data requirements, and highly delayed rewards. Therefore, this paper systematically and specifically studies the MARL algorithm. Firstly, this paper uses Citespace software to visually analyze the existing literature on multi-agent reinforcement learning and briefly indicates the research hotspots and key research directions in this field. Secondly, the applications of traditional reinforcement learning algorithms under two task objects, namely single-agent and multi-agent systems, are described in detail. Then, the paper highlights the diverse applications, challenges, and corresponding solutions of MARL algorithmic techniques in the field of MAS. Finally, the paper points out future research directions based on the existing limitations of the algorithm. Through this paper, readers will gain a systematic and in-depth understanding of MARL algorithms and how they can be utilized to better address the various challenges posed by MAS.

Efficient multi-agent cooperation: Scalable reinforcement learning with heterogeneous graph networks and limited communication

This paper addresses the challenge of scalable multi-agent reinforcement learning (MARL) under partial observability and communication constraints. An Efficient Multi-Agent Cooperation (EMAC) approach is proposed, which comprises a Heterogeneous Graph Network (HGN) and a Frame-wise Communication Reduction Algorithm (FCRA). HGN can effectively extract semantic information from local observations, while FCRA utilizes historical embeddings to achieve an approximation of multi-hop communication with 1-hop communication. Through comparative experiments, EMAC showed a 30% improvement in efficiency over the best-performing benchmark, scalability to arbitrary numbers of agents, and adaptation to unstructured environments. A t-SNE analysis of agent embeddings further reveals EMAC’s capability to understand and leverage semantic similarities among agents. These results highlight EMAC’s potential as a robust framework for decentralized multi-agent decision-making in complex and dynamic environments.

MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIs

As the number of selectable items increases, point-and-click interfaces rapidly become complex, leading to a decrease in usability. Adaptive user interfaces can reduce this complexity by automatically adjusting an interface to only display the most relevant items. A core challenge for developing adaptive interfaces is to infer user intent and chose adaptations accordingly. Current methods rely on tediously hand-crafted rules or carefully collected user data. Furthermore, heuristics need to be recrafted and data regathered for every new task and interface. To address this issue, we formulate interface adaptation as a multi-agent reinforcement learning problem. Our approach learns adaptation policies without relying on heuristics or real user data, facilitating the development of adaptive interfaces across various tasks with minimal adjustments needed. In our formulation, a user agent mimics a real user and learns to interact with an interface via point-and-click actions. Simultaneously, an interface agent learns interface adaptations, to maximize the user agent's efficiency, by observing the user agent's behavior. For our evaluation, we substituted the simulated user agent with actual users. Our study involved twelve participants and concentrated on automatic toolbar item assignment. The results show that the policies we developed in simulation effectively apply to real users. These users were able to complete tasks with fewer actions and in similar times compared to methods trained with real data. Additionally, we demonstrated our method's efficiency and generalizability across four different interfaces and tasks.

Dense Multiagent Reinforcement Learning Aided Multi-UAV Information Coverage for Vehicular Networks

Dense Multiagent Reinforcement Learning Aided Multi-UAV Information Coverage for Vehicular Networks

The impact of sociality regimes on heterogeneous cooperative-competitive multi-agent reinforcement learning: a study with the predator-prey game

ABSTRACT The performance in multi-agent reinforcement learning (MARL) scenarios has usually been analysed in homogeneous teams with a few choices for the sociality regime (selfish, egalitarian, or altruistic). In this paper we analyse both homogeneous and heterogeneous teams in a variation of sociality regimes in the predator-prey game, using a novel normalisation of the weights so that the sum of all rewards is independent of the sociality regime. We find that the selfish regime is advantageous for both predator and prey teams, and for both homogeneous and heterogeneous teams. In particular, rewards are about 100% higher for the predator team when switching from the egalitarian to selfish regime and more than 400% higher from the altruistic regime. For the prey, the increase is around 40% and 100% respectively. The results are similar for homogeneous and heterogeneous situations. The takeaway message is that any study of homogeneous and heterogeneous cooperative-competitive multi-agent reinforcement learning teams should also take into account the sociality regimes before making conclusions on the preference of any algorithm.

Energy management based on safe multi-agent reinforcement learning for smart buildings in distribution networks

The rapid urbanization and increasing use of distributed renewable energy resources have imposed a significant burden on power networks. Smart buildings equipped with artificial intelligence technology can play a pivotal role in energy management, ultimately enhancing energy efficiency and voltage quality. However, ensuring voltage stability within large-scale smart building systems presents challenges due to the coexistence of diverse energy sources and the fluctuating nature of renewable energy. This paper proposes a safe multi-energy management framework achieved by online decentralized execution and centralized training for large scale smart buildings in distribution networks. The energy management problem is formulated as a safety-augmented Markov decision process, presenting intractability for dynamic programming due to its extensive continuous state space. To solve this issue and improve the convergence speed and training process stability, a safety-augmented constrained multi-agent reinforcement learning algorithm based on reward extrapolation is proposed. In this algorithm, hazard values are introduced to enhance non-safe multi-agent reinforcement learning algorithms and meet safety constraints. A novel reward network is designed by imitating expert underlying intentions to ensure the rationality of the reward function for multi-objective tasks. Additionally, the loss function for estimating the Q-network is redesigned during training process to guarantee effective convergence. Theoretical analysis is conducted to provide the convergence guarantee. Numerical case studies based on actual data are performed to validate the effectiveness and scalability of our approach, showing that smart buildings can achieve superior energy management performance while ensuring voltage safety for distribution networks. The source code of the proposed algorithm is available at https://github.com/SYiyun/CMARL-EX.

Competitive pricing for ride-sourcing platforms with MARL

With the advancement of autonomous driving, the competition between shared autonomous vehicles (SAVs) and the existing human-driven vehicles (HVs) ride-sourcing platforms has emerged as a prominent research area within the fields of transportation and management. Solving the pricing problem, considering the competition between HV and SAV platforms, is critical for the platform’s operation. The competitive pricing problem is hard to solve since it is a partially observable Markov decision process (POMDP), that is, only the strategies of other platforms (such as price and wage) are observable, while their states are unobservable. It is also a multivariable sequential problem pricing since spatial-temporal pricing varies with region and time. To tackle this complex problem, this paper implements the multi-agent reinforcement learning (MARL) approach of multi-agent deep deterministic policy gradient (MADDPG) to an mesoscopic simulation system. The simulation replicates the intricate dynamics within a competitive environment where SAV and HV platforms coexist. MADDPG effectively resolves the competitive pricing issue using data derived from simulation. The findings indicate that the SAV platform secures greater profits with a smaller fleet, as it eliminates the need to compensate drivers. Meanwhile, for the same reason, even with fewer vehicles, the SAV platform can set lower prices to attract more passengers and thereby generate higher profits. We further evaluate the efficacy of dynamic pricing versus spatial-temporal pricing and discover that spatial-temporal pricing yields greater profits.

An overview: Attention mechanisms in multi-agent reinforcement learning

In recent years, in the field of Multi-Agent Systems (MAS), significant progress has been made in the research of algorithms that combine Reinforcement Learning (RL) with Attention Mechanism (AM). However, there is a lack of comprehensive reviews in this field. Based on this, this paper does the following work. Firstly, it reviews the classical algorithms of RL and AM; Secondly, it systematically introduces the combination of RL and AM; Thirdly, it sorts out their application in the field of single-agent and multi-agent, and pays attention to and looks forward to the challenges and future research prospects in this field; The last part offers a comprehensive analysis of the challenges encountered by research in this field and anticipates future research paths. The research offers a conceptual understanding and theoretical foundation for future applications of RL using AM and facilitates further in-depth study in this field for researchers.

Cournot Policy Model: Rethinking centralized training in multi-agent reinforcement learning

This work studies Centralized Training and Decentralized Execution (CTDE), which is a powerful mechanism to ease multi-agent reinforcement learning. Although the centralized evaluation ensures unbiased estimates of Q-value, peers with unknown policies make the decentralized policy far from the expectation. To make progress in more stabilized and effective joint policy, we develop a novel game framework, termed Cournot Policy Model, to enhance the CTDE-based multi-agent learning. Combining the game theory and reinforcement learning, we regard the joint decision-making in a single time step as a Cournot duopoly model, and then design a Hetero Variational Auto-Encoder to model the policies of peers in the decentralized execution. With a conditional policy, each agent is guided to a stable mixed-strategy equilibrium even though the joint policy evolves over time. We further demonstrate that such an equilibrium must exist in the case of centralized evaluation. We investigate the improvement of our method on existing centralized learning methods. The experimental results on a comprehensive collection of benchmarks indicate our approach consistently outperforms baseline methods.

Long-short-view aware multi-agent reinforcement learning for signal snippet distillation in delirium movement detection

Automatic movement analysis utilizing surveillance video is believed to be an important and convenient way for timely delirium detection in an Intensive Care Unit (ICU). However, video-based delirium movement detection (DMD) faces inherent challenges: 1) Irregular movements with large differences in the four limbs of a patient; 2) similar movements in delirium and normal situations, and large movement variations between patients experiencing delirium. To address the challenges 1, this paper proposes a Long-Short-View Aware Multi-Agent Reinforcement Learning (LS-MARL) method to identify the most representative movement snippet for DMD, considering that the global state provided by the long-view is important for guiding the agent's decision-making, but ignored in existing MARL methods. The proposed LS-MARL has two novel designs. First, a novel Teacher Auxiliary Policy (TAP) is developed for direction preperception of the representative movement snippet. Second, a new reward mechanism, Team Intrinsic Reward (TIR), is introduced to quantify the contribution of each agent. Experiments demonstrate that the proposed LS-MARL method outperforms state-of-the-art methods. Furthermore, to handle the second challenge, a new Self-Adjusting Ensemble Learning (SAEL) strategy is built to adaptively integrate an optimal classifier combination from multidomain features, which further improves the performance of classification tasks in the proposed LS-MARL method.

HyperComm: Hypergraph-based communication in multi-agent reinforcement learning

In the realm of fully cooperative multi-agent reinforcement learning (MARL), effective communication can induce implicit cooperation among agents and improve overall performance. In current communication strategies, agents are allowed to exchange local observations or latent embeddings, which can augment individual local policy inputs and mitigate uncertainty in local decision-making processes. Unfortunately, in previous communication schemes, agents may potentially receive irrelevant information, which increases training difficulty and leads to poor performance in complex settings. Furthermore, most existing works lack the consideration of the impact of small coalitions formed by agents in the multi-agent system. To address these challenges, we propose HyperComm, a novel framework that uses the hypergraph to model the multi-agent system, improving the accuracy and specificity of communication among agents. Our approach brings the concept of hypergraph for the first time in multi-agent communication for MARL. Within this framework, each agent can communicate more effectively with other agents within the same hyperedge, leading to better cooperation in environments with multiple agents. Compared to those state-of-the-art communication-based approaches, HyperComm demonstrates remarkable performance in scenarios involving a large number of agents.

Research on UAV-assisted Vehicle networking task unloading strategy based on multi-agent reinforcement learning

Research on UAV-assisted Vehicle networking task unloading strategy based on multi-agent reinforcement learning

Age of information minimization in UAV-assisted data harvesting networks by multi-agent deep reinforcement curriculum learning

Unmanned Aerial Vehicles (UAVs) are increasingly being used for data harvesting from Wireless Sensor Nodes (SNs). This study aims to minimize the Age of Information (AoI) during data collection, while also considering the energy sustainability of the UAVs. Addressing the challenge of optimizing performance in Multi-Agent Reinforcement Learning (MARL) systems as the number of agents increases, our study introduces a MARL approach combined with curriculum learning and evolutionary strategy. This innovation specifically targets the performance issue in traditional MARL setups when scaling up the number of agents. By applying curriculum learning and evolutionary strategies, our method not only enhances MARL scalability but also integrates energy-efficient charging mechanisms, effectively enhancing system performance in large-scale deployments. Numerical results show that our proposed algorithm outperforms baselines in terms of AoI and charging, proving its effectiveness in managing the complexities of large-scale MARL systems.

Fully distributed multi-agent processing strategy applied to vehicular networks

This work explores distributed processing techniques, together with recent advances in multi-agent reinforcement learning (MARL) to implement a fully decentralized reward and decision-making scheme to efficiently allocate resources (spectrum and power). The method targets processes with strong dynamics and stringent requirements such as cellular vehicle-to-everything networks (C-V2X). In our approach, the C-V2X is seen as a strongly connected network of intelligent agents which adopt a distributed reward scheme in a cooperative and decentralized manner, taking into consideration their channel conditions and selected actions in order to achieve their goals cooperatively. The simulation results demonstrate the effectiveness of the developed algorithm, named Distributed Multi-Agent Reinforcement Learning (DMARL), achieving performances very close to that of a centralized reward design, with the advantage of not having the limitations and vulnerabilities inherent to a fully or partially centralized solution.

Multi-agent deep reinforcement learning with enhanced collaboration for distribution network voltage control

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses.

Blocklength Allocation and Power Control in UAV-Assisted URLLC System via Multi-agent Deep Reinforcement Learning

Integration of unmanned aerial vehicles (UAVs) with ultra-reliable and low-latency communication (URLLC) systems can improve the real-time communication performance for various industrial internet of things (IIoT) applications. Designing an intelligent resource allocation system is one of the challenges posed by an energy-constrained UAV communication system. Therefore, we formulate a sum rate maximization problem, subject to the UAVs’ energy by optimizing the blocklength allocation and the power control jointly in the uplink UAV-assisted URLLC systems, in which the probabilistic channel model between UAV and users is adopted. The problem is difficult to solve due to the non-convex objective function and the energy constraints, and also challenging to make fast decision in the complex communication environment. Thus, we propose a deep reinforcement learning (DRL)-based scheme to optimize the blocklength allocation and power control jointly. First, transform the original problem into the multi-agent reinforcement learning process, where each subcarrier is regarded as the agent that optimizes its individual blocklength allocation and power control. Then, each agent makes the intelligent decision to obtain the maximum reward value depending on the weighted segmented reward function, which is related to the UAV energy consumption and user rates to improve the rate performance. Finally, the simulation results show that the proposed scheme outperforms the benchmark schemes and has the stable convergence in different settings, such as the learning rate, the error probability, the subcarrier spacing, and the number of users.

Attention-Based Intrinsic Reward Mixing Network for Credit Assignment in Multiagent Reinforcement Learning

Credit assignment is a critical problem in cooperative Multi-Agent Reinforcement Learning (MARL). To address this problem, current studies mainly rely on the intrinsic reward, which is directly summed with the global reward to generate a total reward. However, such kinds of intrinsic reward functions ignore the dependence among agents and inevitably limit the adaptivity and effectiveness of MARL methods. In this paper, we propose a novel method, Attention-based Intrinsic Reward Mixing Network (AIRMN), for credit assignment in MARL. Specifically, we design a new intrinsic reward network on the basis of the attention mechanism, in order to enhance the effectiveness of teamwork. Besides, we devise a new mixing network that combines the intrinsic and extrinsic rewards in a nonlinear and dynamic manner, so as to adapt the total reward to the variation of the environment. Experimental results on the battle games of StarCraft II demonstrate that AIRMN outperforms the state-of-the-art methods in terms of the average test win rate, and also validate that AIRMN can dynamically return the precise intrinsic reward to each agent based on their contributions to the team cooperation, thereby better dealing with the credit assignment problem.