Multi-agent Deep Reinforcement Learning Research Articles

A clustering-aided multi-agent deep reinforcement learning for multi-objective parallel batch processing machines scheduling in semiconductor manufacturing

Batch processing machines are often the bottleneck in semiconductor manufacturing and their scheduling plays a key role in production management. Pioneer researches on multi-objective batch machines scheduling mainly focus on evolutionary algorithms, failing to meet the online scheduling demand. To deal with the challenges confronted by incompatible job families, dynamic job arrivals, capacitated machines and multiple objectives, we propose a clustering-aided multi-agent deep reinforcement learning approach (CA-MADRL) for the scheduling problem. Specifically, to achieve diverse nondominated solutions, an offline multi-objective scheduling algorithm named Multi-Subpopulation fast elitist Non-Dominated Sorting Genetic Algorithm (MS-NSGA-II) is firstly developed to obtain the Pareto Fronts, and a clustering algorithm based on cosine distance is employed to analyze the distribution of Pareto frontier solution, which would be used to guide reward functions design in multi-agent deep reinforcement learning. To realize multi-objective optimization, several reinforcement learning base models are trained for different optimization directions, each of which composed of batch forming agent and batch scheduling agent. To alleviate time complexity of model training, a parameter sharing strategy is introduced between different reinforcement learning base model. By validating the proposed approach with 16 instances designed based on actual production data from a semiconductor manufacturing company, it has been demonstrated that the approach not only meets the high-frequency scheduling requirements of manufacturing systems for parallel batch processing machines but also effectively reduces the total job tardiness and machine energy consumption.

Bio-inspired distributed load frequency control in Islanded Microgrids: A multi-agent deep reinforcement learning approach

To prevent errors in load frequency control (LFC) decision-making in an islanded microgrid and reduce the waste of frequency regulation resources, a fully distributed “starfish” load frequency control method is proposed. For this method, a novel deep reinforcement learning algorithm called the distributed decomposed multirole multiagent deep deterministic policy gradient (DDMR-MADDPG) algorithm is introduced. This algorithm treats each unit as an agent, enabling centralized training of all the agents. Through the coordination and control of multiple agents, a global optimal strategy can be obtained. During online application, the algorithm employs a distributed implementation strategy, enabling each unit to make decisions autonomously. This decision-making strategy is akin to the distributed neural network of a starfish, and it can fundamentally improve the efficiency and correctness of decision-making. In addition, it employs multiple techniques, including value function decomposition, multirole learning, imitation learning, and curriculum learning approaches. The performance of the proposed method compared with 24 existing methods is demonstrated on a simulated LFC model of the Zhuzhou Island microgrid in China.

Utility-Driven End-to-End Network Slicing for Diverse IoT Users in MEC: A Multi-Agent Deep Reinforcement Learning Approach.

Mobile Edge Computing (MEC) is crucial for reducing latency by bringing computational resources closer to the network edge, thereby enhancing the quality of services (QoS). However, the broad deployment of cloudlets poses challenges in efficient network slicing, particularly when traffic distribution is uneven. Therefore, these challenges include managing diverse resource requirements across widely distributed cloudlets, minimizing resource conflicts and delays, and maintaining service quality amid fluctuating request rates. Addressing this requires intelligent strategies to predict request types (common or urgent), assess resource needs, and allocate resources efficiently. Emerging technologies like edge computing and 5G with network slicing can handle delay-sensitive IoT requests rapidly, but a robust mechanism for real-time resource and utility optimization remains necessary. To address these challenges, we designed an end-to-end network slicing approach that predicts common and urgent user requests through T distribution. We formulated our problem as a multi-agent Markov decision process (MDP) and introduced a multi-agent soft actor-critic (MAgSAC) algorithm. This algorithm prevents the wastage of scarce resources by intelligently activating and deactivating virtual network function (VNF) instances, thereby balancing the allocation process. Our approach aims to optimize overall utility, balancing trade-offs between revenue, energy consumption costs, and latency. We evaluated our method, MAgSAC, through simulations, comparing it with the following six benchmark schemes: MAA3C, SACT, DDPG, S2Vec, Random, and Greedy. The results demonstrate that our approach, MAgSAC, optimizes utility by 30%, minimizes energy consumption costs by 12.4%, and reduces execution time by 21.7% compared to the closest related multi-agent approach named MAA3C.

Automated architectural spatial composition via multi-agent deep reinforcement learning for building renovation

This paper focuses on utilizing deep reinforcement learning technology to explore how to achieve automated architectural space composition under established built environment conditions to address the extensive demands of old building renovation. A reinforcement learning platform has been develop0065d, centered around the multi-agent deep deterministic policy gradient (MADDPG) algorithm. The paper leverages functional blocks as agents within Grasshopper to simulate existing architectural structures and their interactions, establishing a task-based reward system to guide the design process. Furthermore, data exchange between algorithmic and modeling software is facilitated through UDP communication. Training results using typical frame structure spaces indicate a significant increase in cumulative rewards around the 650,000th training step, effectively achieving the given design tasks. The paper demonstrates the potential of automated architectural space composition methods, based on deep reinforcement learning, to enhance design efficiency, optimize spatial arrangements, and promote human-machine collaborative design in building renovations.

Residential demand response online optimization based on multi-agent deep reinforcement learning

Demand side management in the context of smart grids exploits the value stream of load flexibility, promoting stable and economic operation of the power grid. To address multi-uncertainties such as device characteristics, distributed renewable generation, and residential electricity demand, this paper proposes a model-free framework based on multi-agent deep reinforcement learning to achieve efficient online optimization of demand response strategies. Firstly, the demand side management is formulated as a partially observable Markov game. Then, a multi-agent soft actor-critic algorithm is proposed, combining a temporal electricity price feature extraction technology to improve learning efficiency. Reward correction mechanism is designed to avoid new charging and discharging peaks in the demand response. Finally, case studies demonstrate the effectiveness of the proposed method in reducing the electricity cost and aggregate peak demand of residents. The proposed method reduces the daily average cost by 21.40%, 8.09%, and 5.88% compared to MAPPO, MADDPG, and SUMADRL, and exhibits the highest training efficiency and scalability against the benchmarks.

Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties

Improving the efficiency of home energy management (HEM) is of great significance in reducing resource waste and prompting renewable energy consumption. With the development of advanced HEM systems and internet-of-things technology, more residents are willing to participate in the electricity market through a demand response mechanism, which can not only reduce the electricity bill but also stabilize the operation of the main grid through peak shaving and valley filling. However, the uncertainties from household appliance parameters, user behaviors, penetrated renewable energy, and electricity prices render the HEM problem a non-trivial task. Although previous deep reinforcement learning (DRL) methods have no requirement for system dynamics due to their model-free and data-driven merits, the unrestricted action space may violate the physical constraints of various devices, and few works have concentrated on this area before. To solve the above challenges, this paper proposes a novel interior-point policy optimization-based multi-agent DRL algorithm to optimize the home energy scheduling procedure while guaranteeing safety during its operation. Besides, a time-series prediction model based on a non-stationary Transformer neural network is proposed to predict the future trend of solar generation and electricity price, which can provide the agents with abundant information to make better decisions. The superior performance of our proposed predictive-control coupled method is demonstrated through extensive numerical experiments, which achieves more precise prediction results, near-zero constraint violations, and better trade-offs between cost and safety than several benchmark methods.

Reliability-assured service function chain migration strategy in edge networks using deep reinforcement learning

With the widespread adoption of edge computing and the rollout of 5G technology, the edge network is experiencing rapid growth. Edge computing enables the execution of certain computational tasks on edge devices, fostering more efficient resource utilization. However, the reliability of the edge network is constrained by its network connections. Network instability can significantly compromise service quality. An effective service function chain (SFC) migration algorithm is essential to optimize resource utilization, enhance service quality. This paper begins by analyzing the current research landscape of edge networks and SFC migration algorithms. Subsequently, the challenges associated with edge network and SFC migration are formally articulated, leading to the proposal of a SFC migration algorithm based on deep reinforcement learning (DRL) with a focus on reliability assurance (RA-SFCM). The algorithm leverages multi-agent deep reinforcement learning to dynamically perceive changes in the edge network environment. It introduces an advantage function to evaluate the performance of each agent relative to the average level and incorporates a central attention mechanism with multiple attention heads to better capture the interdependencies and relationships among different agents. Additionally, this paper innovatively defines and quantifies the reliability of the migration process. By introducing a reliability penalty mechanism based on the migration target nodes and link capacity, it enhances the reliability of the migration schemes. The experimental results conclusively demonstrate the remarkable advantages of the RA-SFCM algorithm in terms of real-time performance, resource utilization efficiency, and reliability. Compared to algorithms such as Sa-VNFM, ROVM, and DLTSAC, RA-SFCM exhibits superior performance. For RA-SFCM, the optimized deployment migration strategy enhances real-time performance, precise resource management improves utilization efficiency, and advanced fault tolerance mechanisms strengthen reliability.

Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach

In the context of the expanding diversity of energy demands, an increasing number of heterogeneous Multi-energy Microgrids (MEMGs) are engaging in the collaborative framework of the Multi-energy Multi-microgrid System (MEMMG). However, following this trend, the existing centralized Integrated Energy Management System (IEMS) control strategy is unreliable for future energy systems, characterized by more complex optimization control and a flexible system structure. This paper introduces a hierarchical Multi-agent Deep Reinforcement Learning (HMADRL) approach for distributed IEMS in MEMMG. Firstly, by employing a hierarchical approach, this method simplifies control complexity by segmenting the overarching control challenge into tasks of collaborative planning and action control, which are distributed across varied multi-agent scenes. Considering both macro and microeconomic factors, alongside carbon emissions, the optimal operation of MEMMG is realized through a bottom-up edge multi-agent control approach, in contrast to traditional top-down centralized methods. Secondly, in the phase of the inter-MEMG collaborative strategy, the Centralized Training Decentralized Execution (CTDE) framework is adopted to overcome the problems of unstable training environments and large-scale agent training, and each heterogeneous microgrid can develop local strategies independently with the assurance that their internal data will not be overly exposed. Thirdly, within each MEMG, the Trust-Region (TR) model is introduced for multi-agent action control, adeptly addressing the effects of mutual exclusion in decision-making time series. Simultaneously, an initialized Hot Experience Pool (HEP) is implemented, efficiently reducing exploration in complex, high-dimensional spaces. Finally, the off-time agent model is integrated into the HMADRL environment and undergoes secondary training based on real interactions, thereby deriving the optimal energy management policy. The proposed method markedly reduces reliance on exact physical modeling systems. Comparative simulations validate the proposed control scheme’s efficacy.

GLIDE: Multi-Agent Deep Reinforcement Learning for Coordinated UAV Control in Dynamic Military Environments

Unmanned aerial vehicles (UAVs) are widely used for missions in dynamic environments. Deep Reinforcement Learning (DRL) can find effective strategies for multiple agents that need to cooperate to complete the task. In this article, the challenge of controlling the movement of a fleet of UAVs is addressed by Multi-Agent Deep Reinforcement Learning (MARL). The collaborative movement of the UAV fleet can be controlled centrally and also in a decentralized fashion, which is studied in this work. We consider a dynamic military environment with a fleet of UAVs, whose task is to destroy enemy targets while avoiding obstacles like mines. The UAVs inherently come with a limited battery capacity directing our research to focus on the minimum task completion time. We propose a continuous-time-based Proximal Policy Optimization (PPO) algorithm for multi-aGent Learning In Dynamic Environments (GLIDE). In GLIDE, the UAVs coordinate among themselves and communicate with the central base to choose the best possible action. The action control in GLIDE can be controlled in a centralized and decentralized way, and two algorithms called Centralized-GLIDE (C-GLIDE), and Decentralized-GLIDE (D-GLIDE) are proposed on this basis. We developed a simulator called UAV SIM, in which the mines are placed at randomly generated 2D locations unknown to the UAVs at the beginning of each episode. The performance of both the proposed schemes is evaluated through extensive simulations. Both C-GLIDE and D-GLIDE converge and have comparable performance in target destruction rate for the same number of targets and mines. We observe that D-GLIDE is up to 68% faster in task completion time compared to C-GLIDE and could keep more UAVs alive at the end of the task.

Multi-Agent Deep Reinforcement Learning Based Dynamic Task Offloading in a Device-to-Device Mobile-Edge Computing Network to Minimize Average Task Delay with Deadline Constraints.

Device-to-device (D2D) is a pivotal technology in the next generation of communication, allowing for direct task offloading between mobile devices (MDs) to improve the efficient utilization of idle resources. This paper proposes a novel algorithm for dynamic task offloading between the active MDs and the idle MDs in a D2D-MEC (mobile edge computing) system by deploying multi-agent deep reinforcement learning (DRL) to minimize the long-term average delay of delay-sensitive tasks under deadline constraints. Our core innovation is a dynamic partitioning scheme for idle and active devices in the D2D-MEC system, accounting for stochastic task arrivals and multi-time-slot task execution, which has been insufficiently explored in the existing literature. We adopt a queue-based system to formulate a dynamic task offloading optimization problem. To address the challenges of large action space and the coupling of actions across time slots, we model the problem as a Markov decision process (MDP) and perform multi-agent DRL through multi-agent proximal policy optimization (MAPPO). We employ a centralized training with decentralized execution (CTDE) framework to enable each MD to make offloading decisions solely based on its local system state. Extensive simulations demonstrate the efficiency and fast convergence of our algorithm. In comparison to the existing sub-optimal results deploying single-agent DRL, our algorithm reduces the average task completion delay by 11.0% and the ratio of dropped tasks by 17.0%. Our proposed algorithm is particularly pertinent to sensor networks, where mobile devices equipped with sensors generate a substantial volume of data that requires timely processing to ensure quality of experience (QoE) and meet the service-level agreements (SLAs) of delay-sensitive applications.

Transformer-Based Reinforcement Learning for Multi-Robot Autonomous Exploration.

A map of the environment is the basis for the robot's navigation. Multi-robot collaborative autonomous exploration allows for rapidly constructing maps of unknown environments, essential for application areas such as search and rescue missions. Traditional autonomous exploration methods are inefficient due to the repetitive exploration problem. For this reason, we propose a multi-robot autonomous exploration method based on the Transformer model. Our multi-agent deep reinforcement learning method includes a multi-agent learning method to effectively improve exploration efficiency. We conducted experiments comparing our proposed method with existing methods in a simulation environment, and the experimental results showed that our proposed method had a good performance and a specific generalization ability.

Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Networks

Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Networks

Dynamic Pricing for Vehicle Dispatching in Mobility-as-a-Service Market via Multi-Agent Deep Reinforcement Learning

Dynamic Pricing for Vehicle Dispatching in Mobility-as-a-Service Market via Multi-Agent Deep Reinforcement Learning

Satellite-Terrestrial Coordinated Multi-Satellite Beam Hopping Scheduling Based on Multi-Agent Deep Reinforcement Learning

Satellite-Terrestrial Coordinated Multi-Satellite Beam Hopping Scheduling Based on Multi-Agent Deep Reinforcement Learning

Deep Reinforcement Learning-based Collaborative Multi-UAV Coverage Path Planning

Abstract The coverage path planning problem has gained significant attention in research due to its wide applicability and practical value in various fields such as logistics and distribution, smart homes, and unmanned vehicles. This paper focuses on studying the coverage path planning problem under multi-UAV collaboration to maximize the coverage of the mission area within a given time. To address this problem, we propose a multi-objective optimization model and reformulate it with the framework of Decentralized Partially Observable Markov Decision Process (Dec-POMDP). We then employ a multi-agent deep reinforcement learning (MADRL) method to solve the problem. Specifically, we introduce the ε—Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (ε—MADT3), which incorporates an exploration coefficient based on MATD3. This coefficient gradually decays with the number of iterations, allowing for a balance between exploration and exploitation. Numerous simulation results demonstrate that ε—MADT3 outperforms the baseline algorithm in terms of coverage rate and number of collisions.

Collision-Avoiding Flocking With Multiple Fixed-Wing UAVs in Obstacle-Cluttered Environments: A Task-Specific Curriculum- Based MADRL Approach.

Multiple unmanned aerial vehicles (UAVs) are able to efficiently accomplish a variety of tasks in complex scenarios. However, developing a collision-avoiding flocking policy for multiple fixed-wing UAVs is still challenging, especially in obstacle-cluttered environments. In this article, we propose a novel curriculum-based multiagent deep reinforcement learning (MADRL) approach called task-specific curriculum-based MADRL (TSCAL) to learn the decentralized flocking with obstacle avoidance policy for multiple fixed-wing UAVs. The core idea is to decompose the collision-avoiding flocking task into multiple subtasks and progressively increase the number of subtasks to be solved in a staged manner. Meanwhile, TSCAL iteratively alternates between the procedures of online learning and offline transfer. For online learning, we propose a hierarchical recurrent attention multiagent actor-critic (HRAMA) algorithm to learn the policies for the corresponding subtask(s) in each learning stage. For offline transfer, we develop two transfer mechanisms, i.e., model reload and buffer reuse, to transfer knowledge between two neighboring stages. A series of numerical simulations demonstrate the significant advantages of TSCAL in terms of policy optimality, sample efficiency, and learning stability. Finally, the high-fidelity hardware-in-the-loop (HITL) simulation is conducted to verify the adaptability of TSCAL. A video about the numerical and HITL simulations is available at https://youtu.be/R9yLJNYRIqY.

Task offloading and resource allocation in NOMA-VEC: A multi-agent deep graph reinforcement learning algorithm

Task offloading and resource allocation in NOMA-VEC: A multi-agent deep graph reinforcement learning algorithm

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

The AI circular hydrogen economist: Hydrogen supply chain design via hierarchical deep multi-agent reinforcement learning

Hydrogen supply chain (HSC) consists of various production, conditioning, transportation, storage, and distribution processes, all of which require extensive computational resources for precise modelling. In this context, artificial intelligence (AI) is emerging as a pivotal tool for addressing model-based decision-making challenges, courtesy of its rapid and efficient computational capabilities. This paper proposes a comprehensive HSC model consisting of an economic policy planner, a wholesale hydrogen market, a power supply system, a hydrogen distribution system (HDS), and hydrogen refueling stations (HRSs). It leverages an AI circular hydrogen economist approach based on a hierarchical deep multi-agent reinforcement learning (MARL) algorithm, offering a new alternative to traditional multi-objective bi-level optimization platforms. The model incorporates green, blue, and gray hydrogen production processes as viable hydrogen production pathways, with the hierarchical MARL’s agents representing the HDS and HRSs at two decision-making levels. Energy requirements are met through a combination of on-site renewable energy sources, the main power grid, or distributed power generation systems. Several HSC scenarios are examined with respect to various combinations of green hydrogen supply rates and carbon credits granted to them under optimum conditions. The results showed that the developed hierarchical MARL has the potential to replace mathematical programming (MP), uncovering a new economic-environmental trade-off between profit of HDS and operational costs of HRSs. Notably, green hydrogen transactions exponentially increase within the supply chain as the carbon credits exceed $1 per kilogram of hydrogen. While this research focuses on optimizing daily operations within the HSC, future efforts can aim to extend this optimization to forecasted annual operations.

A multi-agent deep reinforcement learning paradigm to improve the robustness and resilience of grid connected electric vehicle charging stations against the destructive effects of cyber-attacks

The rising deployment of electric vehicle charging stations (EVCS) in existing power grids and their integration with electric vehicles through communication systems are increasingly vulnerable to cyber-attacks, such as false data injection (FDI) and uncertainties of communication system. This research introduces an EVCS evaluation from a techno-economic standpoint, utilizing the developed and data-oriented TSKFS&MADRL technique to identify and rectify inaccuracies stemming from cyber-attacks like FDI and communication system delays. This method aims to enhance the resilience of EVCSs against sabotage attacks by ensuring fast dynamic responses. Initially, the study models the potential targets of hackers, such as communication systems and transducer sensors, and employs the Euclidean norm theory, weighted least square error method, and residual error technique to identify cyber-attacks resulting from FDI through the comparison of measured data with reference values based on probability distribution functions. Subsequently, the network control and recovery requirements are facilitated by the proposed controller. The proposed approach application has been demonstrated in the IEEE 33 bus, showcasing a 7.33 % lower experimental operation cost compared to the RL method and 12.15 % lower than the CNN method. Additionally, the FDI detection time is observed to be 40 % quicker than alternative methods based on the experimental outcomes.