Multi-agent Reinforcement Learning Research Articles

Navigating Trust and Safety in Multi-Agent Reinforcement Learning: Attacks, Defenses, and Future Directions

Multi-Agent Reinforcement Learning (MARL) is a key framework for building intelligent systems where multiple agents operate within a shared environment, with applications spanning autonomous driving, robotics, and distributed control systems. However, real-world deployment of MARL brings significant trust and safety challenges, as these systems are susceptible to a range of attacks that can compromise their robustness and reliability. This paper provides a comprehensive review of trust and safety attacks in MARL, categorizing various types of attacks and their implications. We explore existing defense mechanisms designed to mitigate these threats, highlighting their strengths and limitations. Additionally, we identify open challenges that remain unaddressed and propose potential future research directions to enhance the robustness and security of MARL systems.

A Multi-Agent Reinforcement Learning Blockchain Framework for Improving Vehicular Internet of Things Cybersecurity

The Vehicular Internet of Things (VIoT) is a novel idea in the field of connected transportation systems that defines a new paradigm. Nevertheless, even the most modern and complex system will require additional and more powerful layers to protect the conversation from interception and the data from leakage. The centralized models have problems like trust problems and that there might be vulnerabilities and that is why there are attempts to integrate decentralization in operation. The first challenge in the VIoT networks is that the security and openness of such a connection and data transfer are still not well developed. Another issue is the security and dependability of the interaction between the vehicles and the infrastructure, although this issue is magnified by the size of the VIoT network. This research is a blockchain and game theory base research that uses Multi-Agent Reinforcement Learning (MARL) to improve the security and efficiency of the VIoT ecosystem. The technology of blockchain gives a distributed ledger where data cannot be altered or erased. Moreover, the MARL architecture allows for the realisation of better decisions for each of the members of the network. To this, the set that was made up of the smart contracts, Vehicle Units (VUs) and the decentralized servers that form the proposed architecture would be added to allow for the right flow and processing of the data. The blockchain’s decentralized nature provides a guarantee for all secure, immutable data transfers and transparent transactions throughout the network of the VIoT. MARL enables agents to learn and acquire the best strategies as they pass through time, which leads to secure and effective communication among entities. Besides, the implementation of lightweight cryptography techniques and strategic selections according to game theory help to protect and improve the performance of the security system of the VIoT ecosystem.

Energy-Saving Control Strategy for Ultra-Dense Network Base Stations Based on Multi-Agent Reinforcement Learning

Aiming at the problem of mobile data traffic surge in 5G networks, this paper proposes an effective solution combining massive multiple-input multiple-output techniques with Ultra-Dense Network (UDN) and focuses on solving the resulting challenge of increased energy consumption. A base station control algorithm based on Multi-Agent Proximity Policy Optimization (MAPPO) is designed. In the constructed 5G UDN model, each base station is considered as an agent, and the MAPPO algorithm enables inter-base station collaboration and interference management to optimize the network performance. To reduce the extra power consumption due to frequent sleep mode switching of base stations, a sleep mode switching decision algorithm is proposed. The algorithm reduces unnecessary power consumption by evaluating the network state similarity and intelligently adjusting the agent's action strategy. Simulation results show that the proposed algorithm reduces the power consumption by 24.61% compared to the no-sleep strategy and further reduces the power consumption by 5.36% compared to the traditional MAPPO algorithm under the premise of guaranteeing the quality of service of users.

CLlight: Enhancing representation of multi-agent reinforcement learning with contrastive learning for cooperative traffic signal control

Multi-agent reinforcement learning has shown great potential for coordinating multi-intersection traffic signals due to its powerful adaptive capabilities, treating each intersection as an agent. However, in the real world, different intersections possess differentiating characteristics such as unique vehicle distributions and traffic patterns. Most existing methods directly add neighboring intersection states to local intersections and optimize the cooperative policy network based on synthesized global features. This indirect optimization approach makes it difficult to thoroughly explore the mutual interactions among different intersection agents, preventing agents from truly learning features with cooperative awareness. To resolve these challenges, we introduce contrastive learning as representation task to the multi-intersection traffic signal control approach named CLlight for two-stage policy network updating. In the first stage, we utilize policy-based or actor-critic-based reinforcement learning methods such as A2C, SAC, and PPO to train policy networks with certain representational capabilities. In the second stage, by extracting pre- and post-masked features and reconstructing the post-masked features, the agents are encouraged to learn the similarities and differences between different intersection policies, which in turn enhances the cooperative and individual representation capabilities of the policy network. To the best of our knowledge, this is the first application of contrastive learning in the field of traffic signal control. Experimental results demonstrate, compared to other state-of-the-art traffic signal control methods, superior average travel time and average waiting time performance under various scenarios, tested on synthetic and real-world datasets.

Reconfigurable Intelligent Surface Assisted VEC Based on Multi-Agent Reinforcement Learning

Reconfigurable Intelligent Surface Assisted VEC Based on Multi-Agent Reinforcement Learning

Multi-objective two-stage robust optimization of wind/PV/thermal power system based on meta multi-agent reinforcement learning

The integration of renewable energy into the power grid poses significant challenges for optimization and scheduling of the power system. In recent years, methods based on deep reinforcement learning have surpassed traditional methods on the high complexity and long-term decision-making of power system optimization and scheduling. However, faced with the inherent uncertainty of renewable energy generation and the different optimization objectives in power system, the deep reinforcement learning methods are unable to effectively address them. This paper proposes a method that combines meta reinforcement learning with multi-agent reinforcement learning to solve the multi-objective two-stage robust optimization of wind/PV/thermal power system. We conducts optimization and scheduling experiments on the IEEE39 bus system. The results indicate that our method not only enhances the robustness of the scheduling strategy, but also outperforms baseline methods in terms of convergence, diversity, and uniformity of the Pareto frontier.

Reinforcement learning-based anatomical maps for pancreas subregion and duct segmentation.

The pancreas is a complex abdominal organ with many anatomical variations, and therefore automated pancreas segmentation from medical images is a challengingapplication. In this paper, we present a framework for segmenting individual pancreatic subregions and the pancreatic duct from three-dimensional (3D) computed tomography (CT) images. A multiagent reinforcement learning (RL) network was used to detect landmarks of the head, neck, body, and tail of the pancreas, and landmarks along the pancreatic duct in a selected target CT image. Using the landmark detection results, an atlas of pancreases was nonrigidly registered to the target image, resulting in anatomical probability maps for the pancreatic subregions and duct. The probability maps were augmented with multilabel 3D U-Net architectures to obtain the final segmentationresults. To evaluate the performance of our proposed framework, we computed the Dice similarity coefficient (DSC) between the predicted and ground truth manual segmentations on a database of 82 CT images with manually segmented pancreatic subregions and 37 CT images with manually segmented pancreatic ducts. For the four pancreatic subregions, the mean DSC improved from 0.38, 0.44, and 0.39 with standard 3D U-Net, Attention U-Net, and shifted windowing (Swin) U-Net architectures, to 0.51, 0.47, and 0.49, respectively, when utilizing the proposed RL-based framework. For the pancreatic duct, the RL-based framework achieved a mean DSC of 0.70, significantly outperforming the standard approaches and existing methods on different datasets. The resulting accuracy of the proposed RL-based segmentation framework demonstrates an improvement against segmentation with standard U-Netarchitectures.

Bidirectional Influence and Interaction for Multiagent Reinforcement Learning

Bidirectional Influence and Interaction for Multiagent Reinforcement Learning

Safe robust multi-agent reinforcement learning with neural control barrier functions and safety attention mechanism

In this paper, a novel safe robust multi-agent reinforcement learning method integrated with decentralized robust neural control barrier functions (CBFs) and a safety attention mechanism (SAM) is proposed for the safety-critical multi-agent system (MAS). Safety is fundamental in the safety-critical MAS but can be affected by factors such as modeling errors, external unknown disturbances, and time-varying observable agents. Several appropriate measures are implemented to address these issues. First, modeling errors and external disturbances are regarded as an adversary for each agent. The agent learns a policy that is robust to disturbances created by the adversary. Accordingly, decentralized robust neural CBFs are introduced to maintain the safety of the MAS, particularly when the general handcrafted CBFs are difficult to construct. The SAM, in combination with the robust neural CBFs, provides a control policy with the capacity to handle time-varying observable agents and increases its attention to dangerous events. The online fine-tuning procedure further enhances the safety. Finally, experiments demonstrate the safety and effectiveness of the proposed method.

Matching Combined Heterogeneous Multi-Agent Reinforcement Learning for Resource Allocation in NOMA-V2X Networks

Matching Combined Heterogeneous Multi-Agent Reinforcement Learning for Resource Allocation in NOMA-V2X Networks

Distributed Safe Multi-Agent Reinforcement Learning: Joint Design of THz-Enabled UAV Trajectory and Channel Allocation

Distributed Safe Multi-Agent Reinforcement Learning: Joint Design of THz-Enabled UAV Trajectory and Channel Allocation

Quantum Multi-Agent Reinforcement Learning is All You Need: Coordinated Global Access in Integrated TN/NTN Cube-Satellite Networks

Quantum Multi-Agent Reinforcement Learning is All You Need: Coordinated Global Access in Integrated TN/NTN Cube-Satellite Networks

Decentralized Multiagent Reinforcement Learning Based State-of-Charge Balancing Strategy for Distributed Energy Storage System

Decentralized Multiagent Reinforcement Learning Based State-of-Charge Balancing Strategy for Distributed Energy Storage System

A Fault-Tolerant Multi-Agent Reinforcement Learning Framework for Unmanned Aerial Vehicles–Unmanned Ground Vehicle Coverage Path Planning

In this paper, we introduce a fault-tolerant multi-agent reinforcement learning framework called SERT-DQN to optimize the operations of UAVs with UGV central control in coverage path planning missions. Our approach leverages dual learning systems that combine individual agent autonomy with centralized strategic planning, thus enhancing the efficiency of cooperative path planning missions. This framework is designed for high performance in environments with fault uncertainty detected and operational challenges such as interruptions in connectivity and compromised sensor reliability. With the integration of an innovative communication system between agents, our system appropriately handles both static and dynamic environments. Also, we introduce similarity-based shared experience replay to attain faster convergence and sample efficiency in the multi-agent system. The architecture is specially designed to respond adaptively to such irregularities by effectively showing enhanced resilience in scenarios where data integrity is impaired due to faults or the UAV faces disruptions. Simulation results indicate that our fault tolerance algorithms are very resilient and do indeed improve mission outcomes, especially under dynamic and highly uncertain operating conditions. This approach becomes critical for the most recent sensor-based research in autonomous systems.

Impulsive maneuver strategy for multi-agent orbital pursuit-evasion game under sparse rewards

To address the subjectivity of dense reward designs for the orbital pursuit-evasion game with multiple optimization objectives, this paper proposes the reinforcement learning method with a hierarchical network structure to guide game strategies under sparse rewards. Initially, to overcome the convergence challenges in the reinforcement learning training process under sparse rewards, a hierarchical network structure is proposed based on the hindsight experience replay. Subsequently, considering the strict constraints imposed by orbital dynamics on spacecraft state space, the reachable domain method is introduced to refine the subgoal space in the hierarchical network, further facilitating the achievement of subgoals. Finally, by adopting the centralized training-layered execution approach, a complete multi-agent reinforcement learning method with the hierarchical network structure is established, enabling networks at each level to learn effectively in parallel within sparse reward environments. Numerical simulations indicate that, under the single-agent reinforcement learning framework, the proposed method exhibits superior stability in the late training stage and enhances exploration efficiency in the early stage by 38.89% to 55.56% to the baseline method. Under the multi-agent reinforcement learning framework, as the relative distance decreases, the subgoals generated by the hierarchical network transition from long-term to short-term, aligning with human behavioral logic.

Three-dimensional cooperative guidance with impact angle constraints via value-policy decomposed multi-agent reinforcement learning

For a three-dimensional impact angle-constrained time-coordination guidance problem, a value-policy decomposed multi-agent twin delayed deep deterministic (VPD-MATD3) policy gradient algorithm is proposed in this paper, and a temporal-spatial cooperative guidance law is designed consequently. The derived cooperative engagement kinematics is formulated as a value-policy decomposed partially observable Markov decision process. Value decomposition and policy decomposition are correspondingly proposed to address the two bottlenecks of reward coupling and policy coupling in 3D guidance, thus improving the training process. The training environment with multiple uncertainties, such as observation noise, maneuverability perturbation, autopilot time lag, switching communication topology, and random initialization, is subsequently constructed. Time-energy type reward functions are designed, incorporating energy consumption as a direct optimization indicator via a penalty term. A long short-term memory network layer is inserted inside the policy networks to suppress the negative effect of observation noise. The policy training curves verify the significant effect of the proposed algorithm in improving training convergence. The policy testing results illustrate the robustness and generalization of the VPD-MATD3 cooperative guidance law and its ability to cope with randomly switched communication topologies. Moreover, the comparative testing further reveals its time-energy suboptimal property.

Multi-agent reinforcement learning for task offloading with hybrid decision space in multi-access edge computing

Multi-access Edge Computing (MEC) has become a significant technology for supporting the computation-intensive and time-sensitive applications on the Internet of Things (IoT) devices. However, it is challenging to jointly optimize task offloading and resource allocation in the dynamic wireless environment with constrained edge resource. In this paper, we investigate a multi-user and multi-MEC servers system with varying task request and stochastic channel condition. Our purpose is to minimize the total energy consumption and time delay by optimizing the offloading decision, offloading ratio and computing resource allocation simultaneously. As the users are geographically distributed within an area, we formulate the problem of task offloading and resource allocation in MEC system as a partially observable Markov decision process (POMDP) and propose a novel multi-agent deep reinforcement learning (MADRL) -based algorithm to solve it. In particular, two aspects have been modified for performance enhancement: (1) To make fine-grained control, we design a novel neural network structure to effectively handle the hybrid action space arisen by the heterogeneous variables. (2) An adaptive reward mechanism is proposed to reasonably evaluate the infeasible actions and to mitigate the instability caused by manual configuration. Simulation results show the proposed method can achieve 7.12%−20.97% performance enhancements compared with the existing approaches.

An Effective Training Method for Counterfactual Multi-Agent Policy Network Based on Differential Evolution Algorithm

Due to the advantages of a centralized critic to estimate the Q-function value and decentralized actors to optimize the agents’ policies, counterfactual multi-agent (COMA) stands out in most multi-agent reinforcement learning (MARL) algorithms. The sharing of policy parameters can improve sampling efficiency and learning effectiveness, but it may lead to a lack of policy diversity. Hence, to balance parameter sharing and diversity among agents in COMA has been a persistent research topic. In this paper, an effective training method for a COMA policy network based on a differential evolution (DE) algorithm is proposed, named DE-COMA. DE-COMA introduces individuals in a population as computational units to construct the policy network with operations such as mutation, crossover, and selection. The average return of DE-COMA is set as the fitness function, and the best individual of policy network will be chosen for the next generation. By maintaining better parameter sharing to enhance parameter diversity, multi-agent strategies will become more exploratory. To validate the effectiveness of DE-COMA, experiments were conducted in the StarCraft II environment with 2s_vs_1sc, 2s3z, 3m, and 8m battle scenarios. Experimental results demonstrate that DE-COMA significantly outperforms the traditional COMA and most other multi-agent reinforcement learning algorithms in terms of win rate and convergence speed.

Deep Reinforcement Learning for Fluid Mechanics: Control, Optimization, and Automation

A comprehensive review of recent advancements in applying deep reinforcement learning (DRL) to fluid dynamics problems is presented. Applications in flow control and shape optimization, the primary fields where DRL is currently utilized, are thoroughly examined. Moreover, the review introduces emerging research trends in automation within computational fluid dynamics, a promising field for enhancing the efficiency and reliability of numerical analysis. Emphasis is placed on strategies developed to overcome challenges in applying DRL to complex, real-world engineering problems, such as data efficiency, turbulence, and partial observability. Specifically, the implementations of transfer learning, multi-agent reinforcement learning, and the partially observable Markov decision process are discussed, illustrating how these techniques can provide solutions to such issues. Finally, future research directions that could further advance the integration of DRL in fluid dynamics research are highlighted.

A multi-agent reinforcement learning based approach for intelligent traffic signal control

A multi-agent reinforcement learning based approach for intelligent traffic signal control