Multi-agent Reinforcement Learning Method Research Articles

Multi-UAV Escape Target Search: A Multi-Agent Reinforcement Learning Method.

The multi-UAV target search problem is crucial in the field of autonomous Unmanned Aerial Vehicle (UAV) decision-making. The algorithm design of Multi-Agent Reinforcement Learning (MARL) methods has become integral to research on multi-UAV target search owing to its adaptability to the rapid online decision-making required by UAVs in complex, uncertain environments. In non-cooperative target search scenarios, targets may have the ability to escape. Target probability maps are used in many studies to characterize the likelihood of a target's existence, guiding the UAV to efficiently explore the task area and locate the target more quickly. However, the escape behavior of the target causes the target probability map to deviate from the actual target's position, thereby reducing its effectiveness in measuring the target's probability of existence and diminishing the efficiency of the UAV search. This paper investigates the multi-UAV target search problem in scenarios involving static obstacles and dynamic escape targets, modeling the problem within the framework of decentralized partially observable Markov decision process. Based on this model, a spatio-temporal efficient exploration network and a global convolutional local ascent mechanism are proposed. Subsequently, we introduce a multi-UAV Escape Target Search algorithm based on MAPPO (ETS-MAPPO) for addressing the escape target search difficulty problem. Simulation results demonstrate that the ETS-MAPPO algorithm outperforms five classic MARL algorithms in terms of the number of target searches, area coverage rate, and other metrics.

Knowledge graph-enhanced multi-agent reinforcement learning for adaptive scheduling in smart manufacturing

AbstractSelf-organizing manufacturing network has emerged as a viable solution for adaptive manufacturing control within the mass personalization paradigm. This approach involves three critical elements: system modeling and control architecture, interoperable communication, and adaptive manufacturing control. However, current research often separates interoperable communication from adaptive manufacturing control as isolated areas of study. To address this gap, this paper introduces Knowledge Graph-enhanced Multi-Agent Reinforcement Learning (MARL) method that integrates interoperable communication via Knowledge Graphs with adaptive manufacturing control through Reinforcement Learning. We hypothesize that implicit domain knowledge obtained from historical production job allocation records can guide each agent to learn more effective scheduling policies with accelerated learning rates. This is based on the premise that machine assignment preferences effectively could reduce the Reinforcement Learning search space. Specifically, we redesign machine agents with new observation, action, reward, and cooperation mechanisms considering the preference of machines, building upon our previous MARL base model. The scheduling policies are trained under extensive simulation experiments that consider manufacturing requirements. During the training process, our approach demonstrates improved training speed compared with individual Reinforcement Learning methods under the same training hyperparameters. The obtained scheduling policies generated by our Knowledge Graph-enhanced MARL also outperform both individual Reinforcement Learning methods and heuristic rules under dynamic manufacturing settings.

Safe robust multi-agent reinforcement learning with neural control barrier functions and safety attention mechanism

In this paper, a novel safe robust multi-agent reinforcement learning method integrated with decentralized robust neural control barrier functions (CBFs) and a safety attention mechanism (SAM) is proposed for the safety-critical multi-agent system (MAS). Safety is fundamental in the safety-critical MAS but can be affected by factors such as modeling errors, external unknown disturbances, and time-varying observable agents. Several appropriate measures are implemented to address these issues. First, modeling errors and external disturbances are regarded as an adversary for each agent. The agent learns a policy that is robust to disturbances created by the adversary. Accordingly, decentralized robust neural CBFs are introduced to maintain the safety of the MAS, particularly when the general handcrafted CBFs are difficult to construct. The SAM, in combination with the robust neural CBFs, provides a control policy with the capacity to handle time-varying observable agents and increases its attention to dangerous events. The online fine-tuning procedure further enhances the safety. Finally, experiments demonstrate the safety and effectiveness of the proposed method.

Impulsive maneuver strategy for multi-agent orbital pursuit-evasion game under sparse rewards

To address the subjectivity of dense reward designs for the orbital pursuit-evasion game with multiple optimization objectives, this paper proposes the reinforcement learning method with a hierarchical network structure to guide game strategies under sparse rewards. Initially, to overcome the convergence challenges in the reinforcement learning training process under sparse rewards, a hierarchical network structure is proposed based on the hindsight experience replay. Subsequently, considering the strict constraints imposed by orbital dynamics on spacecraft state space, the reachable domain method is introduced to refine the subgoal space in the hierarchical network, further facilitating the achievement of subgoals. Finally, by adopting the centralized training-layered execution approach, a complete multi-agent reinforcement learning method with the hierarchical network structure is established, enabling networks at each level to learn effectively in parallel within sparse reward environments. Numerical simulations indicate that, under the single-agent reinforcement learning framework, the proposed method exhibits superior stability in the late training stage and enhances exploration efficiency in the early stage by 38.89% to 55.56% to the baseline method. Under the multi-agent reinforcement learning framework, as the relative distance decreases, the subgoals generated by the hierarchical network transition from long-term to short-term, aligning with human behavioral logic.

Multiple Unmanned Aerial Vehicle (multi-UAV) Reconnaissance and Search with Limited Communication Range Using Semantic Episodic Memory in Reinforcement Learning

Unmanned Aerial Vehicles (UAVs) have garnered widespread attention in reconnaissance and search operations due to their low cost and high flexibility. However, when multiple UAVs (multi-UAV) collaborate on these tasks, a limited communication range can restrict their efficiency. This paper investigates the problem of multi-UAV collaborative reconnaissance and search for static targets with a limited communication range (MCRS-LCR). To address communication limitations, we designed a communication and information fusion model based on belief maps and modeled MCRS-LCR as a multi-objective optimization problem. We further reformulated this problem as a decentralized partially observable Markov decision process (Dec-POMDP). We introduced episodic memory into the reinforcement learning framework, proposing the CNN-Semantic Episodic Memory Utilization (CNN-SEMU) algorithm. Specifically, CNN-SEMU uses an encoder–decoder structure with a CNN to learn state embedding patterns influenced by the highest returns. It extracts semantic features from the high-dimensional map state space to construct a smoother memory embedding space, ultimately enhancing reinforcement learning performance by recalling the highest returns of historical states. Extensive simulation experiments demonstrate that in reconnaissance and search tasks of various scales, CNN-SEMU surpasses state-of-the-art multi-agent reinforcement learning methods in episodic rewards, search efficiency, and collision frequency.

Graph-based multi-agent reinforcement learning for collaborative search and tracking of multiple UAVs

This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

Joint optimization of communication and mission performance for multi-UAV collaboration network: A multi-agent reinforcement learning method

In emergency rescue, target search and other mission scenarios with Unmanned Aerial Vehicles (UAVs), the Relay UAVs (RUs) and Mission UAVs (MUs) can collaborate to accomplish tasks in unknown environments. In this paper, we investigate the problem of trajectory planning and power control for the MU and RU collaboration. Firstly, considering the characteristics of multi-hop data transmission between the MU and Ground Control Station, a multi-UAV collaborative coverage model is designed. Meanwhile, a UAV control algorithm named MUTTO is proposed based on multi-agent reinforcement learning. In order to solve the problem of the unknown information about the number and locations of targets, the geographic coverage rate is used to replace the target coverage rate for decision making. Then, the reward functions of two types of UAVs are designed separately for the purpose of better cooperation. By simultaneously planning the trajectory and transmission power of the RU and MU, the mission target coverage rate and network transmission rate are maximized while the energy consumption of the UAV is minimized. Finally, numerical simulations results show that MUTTO can solve the UAV network control problem in an efficient way and has better performance than the benchmark method.

Environmental-Impact-Based Multi-Agent Reinforcement Learning

To promote cooperation and strengthen the individual impact on the collective outcome in social dilemmas, we propose the Environmental-impact Multi-Agent Reinforcement Learning (EMuReL) method where each agent estimates the “environmental impact” of every other agent, that is, the difference in the current environment state compared to the hypothetical environment in the absence of that other agent. Inspired by the inequity aversion model, the agent then compares its own reward with that of its fellows multiplied by their environmental impacts. If its reward exceeds the scaled reward of one of its fellows, the agent takes “social responsibility” toward that fellow by reducing its own reward. Therefore, the less influential an agent is in reaching the current state, the more social responsibility is taken by other agents. Experiments in the Cleanup (resp. Harvest) test environment demonstrated that agents trained based on EMuReL learned to cooperate more effectively and obtained 54% (39%) and 20% (44%) more total rewards while preserving the same cooperation levels compared to when they were trained based on the two state-of-the-art reward reshaping methods: inequity aversion and social influence.

Generative subgoal oriented multi-agent reinforcement learning through potential field

Multi-agent reinforcement learning (MARL) effectively improves the learning speed of agents in sparse reward tasks with the guide of subgoals. However, existing works sever the consistency of the learning objectives of the subgoal generation and subgoal reached stages, thereby significantly inhibiting the effectiveness of subgoal learning. To address this problem, we propose a novel Potential field Subgoal-based Multi-Agent reinforcement learning (PSMA) method, which introduces the potential field (PF) to unify the two-stage learning objectives. Specifically, we design a state-to-PF representation model that describes agents’ states as potential fields, allowing easy measurement of the interaction effect for both allied and enemy agents. With the PF representation, a subgoal selector is designed to automatically generate multiple subgoals for each agent, drawn from the experience replay buffer that contains both individual and total PF values. Based on the determined subgoals, we define an intrinsic reward function to guide the agent to reach their respective subgoals while maximizing the joint action-value. Experimental results show that our method outperforms the state-of-the-art MARL method on both StarCraft II micro-management (SMAC) and Google Research Football (GRF) tasks with sparse reward settings.

Securing demand–response in smart grids against false pricing attacks

Two-way communication systems in smart grids help to engage consumers in demand–response programs, which can bring many benefits but also make smart grids vulnerable to cyber attacks. In this paper, a cyber attack which aims to disturb the demand–response process by injecting false electricity price signals is considered. A real-time pricing model where the operator has incomplete knowledge of the private demand–response behaviors of the customers is proposed. The vulnerability of the power system to false pricing attacks is analyzed by a Markov decision process, and the dynamic interaction between the attacker and the defender is modeled as a zero-sum Markov game where neither player has full information of the game model. For the solution of the Markov game, a model-free multi-agent reinforcement learning method is proposed to find the Nash Equilibrium policies for both players. The proposed method is applied to the IEEE 34 Node Test Feeder, in which the effect of the defense on mitigating the impact of the attack is demonstrated and different policies of the players given various resources are analyzed. The results shows that the studied cyber attack can cause a maximum of 1.96% unsatisfied load and the proposed defense measure can reduce the unsatisfied load by 50%–70% compared with the cases without defense measures.

A multi-agent reinforcement learning method for distribution system restoration considering dynamic network reconfiguration

Extreme weather, chain failures, and other events have increased the probability of wide-area blackouts, which highlights the importance of rapidly and efficiently restoring the affected loads. This paper proposes a multi-agent reinforcement learning method for distribution system restoration. Firstly, considering that the topology of the distribution system may change during network reconfiguration, a dynamic agent network (DAN) architecture is designed to address the challenge of input dimensions changing in neural network. Two encoders are created to capture observations of the environment and other agents respectively, and an attention mechanism is used to aggregate an arbitrary-sized neighboring agent feature set. Then, considering the operation constraints of the DSR, an action mask mechanism is implemented to filter out invalid actions, ensuring the security of the strategy. Finally, an IEEE 123-node test system is used for validation, and the experimental results showed that the proposed algorithm can effectively assist agents in accomplishing collaborative DSR tasks.

Long-short-view aware multi-agent reinforcement learning for signal snippet distillation in delirium movement detection

Automatic movement analysis utilizing surveillance video is believed to be an important and convenient way for timely delirium detection in an Intensive Care Unit (ICU). However, video-based delirium movement detection (DMD) faces inherent challenges: 1) Irregular movements with large differences in the four limbs of a patient; 2) similar movements in delirium and normal situations, and large movement variations between patients experiencing delirium. To address the challenges 1, this paper proposes a Long-Short-View Aware Multi-Agent Reinforcement Learning (LS-MARL) method to identify the most representative movement snippet for DMD, considering that the global state provided by the long-view is important for guiding the agent's decision-making, but ignored in existing MARL methods. The proposed LS-MARL has two novel designs. First, a novel Teacher Auxiliary Policy (TAP) is developed for direction preperception of the representative movement snippet. Second, a new reward mechanism, Team Intrinsic Reward (TIR), is introduced to quantify the contribution of each agent. Experiments demonstrate that the proposed LS-MARL method outperforms state-of-the-art methods. Furthermore, to handle the second challenge, a new Self-Adjusting Ensemble Learning (SAEL) strategy is built to adaptively integrate an optimal classifier combination from multidomain features, which further improves the performance of classification tasks in the proposed LS-MARL method.

Joint Spectrum and Power Allocation in Wireless Network: A Two-Stage Multi-Agent Reinforcement Learning Method

Joint Spectrum and Power Allocation in Wireless Network: A Two-Stage Multi-Agent Reinforcement Learning Method

Multi-agent reinforcement learning method for cutting parameters optimization based on simulation and experiment dual drive environment

Improving production efficiency while ensuring product surface quality is a constant focus of manufacturers. Cutting parameter optimization is an important technique for ensuring high-efficiency and high-quality production. In this paper, a novel method for cutting parameter optimization that integrates multi-agent reinforcement learning with a dual-drive virtual machining environment is proposed. First, a feature extraction, fusion and generation model for cutting simulation and experimental data is proposed to solve the problem of incomplete data acquisition in the production process. Second, a Markov decision model for optimizing cutting parameters is defined, and a virtual machining environment driven by both simulation and experimental data is constructed. Third, a novel multi-agent reinforcement learning method called Q-MIX-MATD3, in which a twin delay deep deterministic policy gradient, value function decomposition and a teacher model are combined, is proposed to explore the cutting parameter optimization policy by interacting with the virtual machining environment. Finally, the proposed method is verified on a commutator production line. Moreover, the results show that the accuracy of the virtual machining environment driven by both simulation and experiment increases by more than 5 %, response efficiency increases by 31 %, and Q-MIX-MATD3-based cutting parameter optimization method reduces time cost by 98 % and achieves the optimization effect of the classical optimization method.

A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning

Faced with the challenges of security strategy design brought about by the complexity of attack behavior and the dynamism of network structure, dynamic hierarchical intelligent defense methods have shown their effectiveness. However, in complex network environments, their application requires a higher level of coordination mechanisms. Therefore, this paper proposes a hierarchical multi-agent reinforcement learning network attack and defense game and cooperative defense decision-making method, which autonomously and efficiently completes the formulation of defense strategies and defense behavior responses. Firstly, we construct a Stackelberg hypergame model of cyberspace conflicts, and under the condition of information loss, characterize the multi-layer dynamic defense coordination response mechanism. Secondly, By utilizing a hierarchical multi-agent reinforcement learning method as the driving force for game evolution, we sequentially solve the Nash equilibrium of the game, and form a dynamic autonomous defense strategy. Finally, we construct a hierarchical multi-agent reinforcement learning framework, which decouples the defense decision problem, reduces the dimension of the defense action space and the exploration difficulty of the strategy space, and learns coordinated defense strategies more efficiently. We used the CybORG (Cyber Operations Research Gym) environment for simulation. We compared and analyzed the autonomously generated cyber defense strategies with other related works, verifying the superior coordination performance of our method in defense strategy generation and control.

Population Game-Assisted Multi-Agent Reinforcement Learning Method for Dynamic Multi-Vehicle Route Selection

Population Game-Assisted Multi-Agent Reinforcement Learning Method for Dynamic Multi-Vehicle Route Selection

A smart inventory management system with medication demand dependencies in a hospital supply chain: A multi-agent reinforcement learning approach

In light of the intense need for quality health and well-being, the healthcare industry must improve its operations by strengthening its service delivery and reducing overall costs. A smart inventory management system for managing medicines in a hospital supply chain (HSC) is one of the best solutions as its advantages ensure availability of affordable medicines as well as inventory cost reduction. Significant effort is made on inventory management from literature; however, smart inventory systems for HSC considering the various complexities and uniqueness of healthcare systems are limited. This paper attempts to fill this gap by developing a stochastic semi-Markov decision process model which is solved by a multi-agent reinforcement learning method. Data collected from a multispecialty hospital in India’s eastern region serves to validate the proposed model. The results present the optimal order quantities and optimal inventory control policy for the HSC considering the uncertain medication demand at the hospital pharmacy and multiple point-of-care units, and demand dependencies among medicines prescribed to the patients admitted in the hospital. The study also has significant managerial implications for enhancing HSC’s inventory performance, improving the healthcare service delivery, reducing healthcare cost, thereby ensuring the affordable, accessible, and quality healthcare to the society.

Cooperative Multiagent Reinforcement Learning Coupled With A* Search for Ship Multicabin Equipment Layout Considering Pipe Route

_ The paper presents a novel approach of cooperative multiagent reinforcement learning (CMARL) combined with A* search to address ship multicabin equipment layout considering pipe route, aiming to minimize pipe cost while considering practical requirements. The formulation is established through equipment simplification and grid marking, and A* search is utilized to value the pipe route. By designing equipment states, the equipment layout in each cabin is solved using a CMARL approach that involves three actions. Subsequently, comparative experiments were conducted on an engine room case by CMARL against genetic algorithm and single multiagent reinforcement learning methods under the condition of coupling with A* search. The parameter values for these methods were sampled using Latin Hypercube. The findings demonstrate that CMARL has superior combination properties. Keywords ship equipment layout; multicabin layout; cooperative multiagent reinforcement learning; A* search; pipe route

Multi-agent cooperative strategy with explicit teammate modeling and targeted informative communication

The mainstream Multi-Agent Reinforcement Learning (MARL) methods introduce the teammate modeling or the communication mechanism into Centralized Training Decentralized Execution (CTDE) paradigm, which can improve coordination performance. However, the existing teammate modeling methods predict either actions or local observations, limiting their applicability. In addition, the traditional communication mechanism only considers the quantity of the communication links while ignoring the quality of retained communication links, leading to inefficient and redundant communication. To solve the above problems, this paper proposes a novel Multi-Agent Cooperative Strategy with Explicit Teammate Modeling and Targeted Informative Communication (MACS), which can generate and send the more informative message with the higher communication efficiency, further improving the coordination performance. Specifically, the Variational Auto-Encoder (VAE) is leveraged to allow each agent to simultaneously predict the observations and actions of teammates, thus generating more comprehensive communication message. Then, we propose a new Mutual Information (MI) between the communication message and teammate Q-value, which can obtain the informative message, ensuring the exploration and stability of the method. In addition, a targeted dynamic informative communication graph is established by the Graph Neural Network (GNN) which can reduce the redundant communication link through hypothetical analysis, further improving the overall communication efficiency. Eventually, we conduct experiments in StarCraft II, Collaborative Navigation, and Multi-Target Multi-Sensor Coverage environments. Experimental results show that the proposed approach is superior to the state-of-the-art in terms of coordination performance and communication efficiency.