Multi-agent Deep Reinforcement Learning Research Articles

Optimization of Edge–Cloud Collaborative Computing Resource Management for Internet of Vehicles Based on Multiagent Deep Reinforcement Learning

Optimization of Edge–Cloud Collaborative Computing Resource Management for Internet of Vehicles Based on Multiagent Deep Reinforcement Learning

Resource Allocation in UAV-D2D Networks: A Scalable Heterogeneous Multi-Agent Deep Reinforcement Learning Approach

In unmanned aerial vehicle (UAV)-assisted device-to-device (D2D) caching networks, the uncertainty from unpredictable content demands and variable user positions poses a significant challenge for traditional optimization methods, often making them impractical. Multi-agent deep reinforcement learning (MADRL) offers significant advantages in optimizing multi-agent system decisions and serves as an effective and practical alternative. However, its application in large-scale dynamic environments is severely limited by the curse of dimensionality and communication overhead. To resolve this problem, we develop a scalable heterogeneous multi-agent mean-field actor-critic (SH-MAMFAC) framework. The framework treats ground users (GUs) and UAVs as distinct agents and designs cooperative rewards to convert the resource allocation problem into a fully cooperative game, enhancing global network performance. We also implement a mixed-action mapping strategy to handle discrete and continuous action spaces. A mean-field MADRL framework is introduced to minimize individual agent training loads while enhancing total cache hit probability (CHP). The simulation results show that our algorithm improves CHP and reduces transmission delay. A comparative analysis with existing mainstream deep reinforcement learning (DRL) algorithms shows that SH-MAMFAC significantly reduces training time and maintains high CHP as GU count grows. Additionally, by comparing with SH-MAMFAC variants that do not include trajectory optimization or power control, the proposed joint design scheme significantly reduces transmission delay.

Multi-agent deep reinforcement learning for computation offloading in cooperative edge network

Multi-agent deep reinforcement learning for computation offloading in cooperative edge network

Autonomous ecologies of construction: Collaborative modular robotic material eco-systems with deep multi-agent reinforcement learning

To address the unprecedented challenges of construction pressurized by the global climate crisis, housing shortage, and growing shortage of skilled labor, this research presents a radical shift in the construction lifecycle of buildings, from linear processes that produce static continuous buildings to interrelational processes linking adaptative eco-systems of collaborative robots and reconfigurable building parts. Inspired by natural builders, the interdisciplinary field of collective robotic construction (CRC) offers the potential for scalable, adaptive, and resilient construction with simple robots. We establish a design framework for autonomous collaborative robotic construction (ACRC) through modular robotic material eco-systems (MRMES) trained with deep multi-agent reinforcement learning (DMARL). This involves the integration of three core aspects: (1) modular robotic material eco-systems (2) cyber-physical simulation and control with bidirectional feedback (3) adaptive intelligence through deep multi-agent reinforcement learning. The framework is implemented through three comparable case studies for collaborative modular robotic assembly of reconfigurable building parts.

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning

In recent years, reinforcement learning (RL) methods have shown powerful learning capabilities in single-vehicle autonomous driving. However, few studies have focused on multi-vehicle cooperative driving based on RL, particularly in the dynamically changing traffic environments of highway ramp merge zones. In this paper, a multi-agent deep reinforcement learning (MARL) framework for multi-vehicle cooperative decision-making is proposed based on actor–critic, which categorizes vehicles into two groups according to their origins in the merging area. At the same time, the complexity of the network is reduced and the training process of the model is accelerated by utilizing mechanisms such as partial parameter sharing and experience playback. Additionally, a combination of global and individual rewards is adopted to promote cooperation in connected autonomous vehicles (CAVs) and balance individual and group interests. The training performance of the model is compared under three traffic densities, and our method is also compared with state-of-the-art benchmark methods. The simulation results show that the proposed MARL framework can have stronger policy learning capability and stability under various traffic flow conditions. Moreover, it can also effectively improve the speed of vehicles in the merging zone and reduce traffic conflicts.

Dynamic Energy Management and Control of Networked Microgrids based on Load to Grid Services and Incentive-Based Demand Response Programs: A Multi-Agent Deep Reinforcement Learning Approach

This study has presented the energy management paradigm in a networked microgrid structure based on L2G services and considering incentive-based load response (IBDR) programs and energy market requirements to reduce operating costs, control and restore voltage and frequency index, providing the benefits of subscribers and distribution system operators. In this study, multi-objective functions such as optimal operation based on IBDR structure and energy market requirements, risk assessment, and L2G service approach are configured in the framework of central and local controllers. Optimal operation and risk assessment are analyzed by a multi-task learning algorithm based on multi-objective function and L2G service policies are evaluated based on multi-agent deep reinforcement learning. Control policies are sent by the communication system to the components affecting the optimal power distribution as well as the voltage and frequency controllers. L2G services have been evaluated in different scenarios such as plug-and-play operating conditions, load fluctuations, and operating in island mode. The results of optimal operation based on L2G services show that the IBDR program implementation reduces the total operation cost by 21%. Also, the total operating cost of the proposed framework is 13.97% less than the RL method and 27.8% less than the ANN method.

A Two-Stage Multi-Agent Deep Reinforcement Learning Method for Urban Distribution Network Reconfiguration Considering Switch Contribution

A Two-Stage Multi-Agent Deep Reinforcement Learning Method for Urban Distribution Network Reconfiguration Considering Switch Contribution

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

Multi-agent Deep Reinforcement Learning for cloud-based digital twins in power grid management

As the power industry becomes increasingly complex, Digital Twin (DT) technology has emerged as a crucial tool for enhancing grid resilience and operational efficiency by creating dynamic digital replicas of physical systems. These replicas enable accurate simulations and proactive management, but the vast amount of data generated by DT systems poses significant challenges for processing and analysis. Cloud computing offers a flexible solution by offloading these computational tasks to distributed resources, allowing for real-time analysis and scalable operations. However, this approach introduces complexities in task distribution and maintaining quality of service. Recent efforts have applied Deep Reinforcement Learning (DRL) to address these challenges, primarily using single-agent methods. However, these methods struggle with scalability and performance in increasingly complex cloud environments. To overcome these limitations, we propose an efficient task scheduling framework based on Multi-Agent Deep Q-Network (MADQN) principles, specifically designed to optimize both response times and operational costs. We provide a comprehensive design overview of our approach and conduct a thorough evaluation of its performance. The experimental results clearly indicate that our approach can considerably reduce response times amd lower operational costs compared to current methods, including state-of-the-art single-agent approaches.

Multi-Agent Deep Reinforcement Learning for Blockchain-Based Energy Trading in Decentralized Electric Vehicle Charger-Sharing Networks

With The integration of renewable energy sources into smart grids and electric vehicle (EV) charger-sharing networks is essential for achieving the goal of environmental sustainability. However, the uneven distribution of distributed energy trading among EVs, fixed charging stations (FCSs), and mobile charging stations (MCSs) introduces challenges such as inadequate supply at FCSs and prolonged latencies at MCSs. In this paper, we propose a multi-agent deep reinforcement learning (MADRL)-based auction algorithm for energy trading that effectively balances charger supply with energy demand in distributed EV charging markets, while also reducing total charging latency. Specifically, this involves a MADRL-based hierarchical auction that dynamically adapts to real-time conditions, optimizing the balance of supply and demand. During energy trading, each EV, acting as a learning agent, can refine its bidding strategy to participate in various local energy trading markets, thus enhancing both individual utility and global social welfare. Furthermore, we design a cross-chain scheme to securely record and verify transaction results of energy trading in decentralized EV charger-sharing networks to ensure integrity and transparency. Finally, experimental results show that the proposed algorithm significantly outperforms both the second-price and double auctions in increasing global social welfare and reducing total charging latency.

Short-Term Electricity Futures Investment Strategies for Power Producers Based on Multi-Agent Deep Reinforcement Learning

The global development and enhancement of electricity financial markets aim to mitigate price risk in the electricity spot market. Power producers utilize financial derivatives for both hedging and speculation, necessitating careful selection of portfolio strategies. Current research on investment strategies for power financial derivatives primarily emphasizes risk management, resulting in a lack of a comprehensive investment framework. This study analyzes six short-term electricity futures contracts: base day, base week, base weekend, peak day, peak week, and peak weekend. A multi-agent deep reinforcement learning algorithm, Dual-Q MADDPG, is employed to learn from interactions with both the spot and futures market environments, considering the hedging and speculative behaviors of power producers. Upon completion of model training, the algorithm enables power producers to derive optimal portfolio strategies. Numerical experiments conducted in the Nordic electricity spot and futures markets indicate that the proposed Dual-Q MADDPG algorithm effectively reduces price risk in the spot market while generating substantial speculative returns. This study contributes to lowering barriers for power generators in the power finance market, thereby facilitating the widespread adoption of financial instruments, which enhances market liquidity and stability.

Multi-Agent Deep Reinforcement Learning-Based Distributed Voltage Control of Flexible Distribution Networks with Soft Open Points

The increasing number of distributed generators (DGs) leads to the frequent occurrence of voltage violations in distribution networks. The soft open point (SOP) can adjust the transmission power between feeders, leading to the evolution of traditional distribution networks into flexible distribution networks (FDN). The problem of voltage violations can be effectively tackled with the flexible control of SOPs. However, the centralized control method for SOP may make it difficult to achieve real-time control due to the limitations of communication. In this paper, a distributed voltage control method is proposed for FDN with SOPs based on the multi-agent deep reinforcement learning (MADRL) method. Firstly, a distributed voltage control framework is proposed, in which the updating algorithm of the intelligent agent of MADRL is expounded considering experience sharing. Then, a Markov decision process for multi-area SOP coordinated voltage control is proposed, where the control areas are divided based on electrical distance. Finally, an IEEE 33-node test system and a practical system in Taiwan are used to verify the effectiveness of the proposed method. It shows that the proposed multi-area SOP coordinated control method can achieve real-time control while ensuring a better control effect.

The crowd cooperation approach for formation maintenance and collision avoidance using multi-agent deep reinforcement learning

The crowd cooperation approach for formation maintenance and collision avoidance using multi-agent deep reinforcement learning

Energy-Saving Multi-Agent Deep Reinforcement Learning Algorithm for Drone Routing Problem.

With the rapid advancement of drone technology, the efficient distribution of drones has garnered significant attention. Central to this discourse is the energy consumption of drones, a critical metric for assessing energy-efficient distribution strategies. Accordingly, this study delves into the energy consumption factors affecting drone distribution. A primary challenge in drone distribution lies in devising optimal, energy-efficient routes for drones. However, traditional routing algorithms, predominantly heuristic-based, exhibit certain limitations. These algorithms often rely on heuristic rules and expert knowledge, which can constrain their ability to escape local optima. Motivated by these shortcomings, we propose a novel multi-agent deep reinforcement learning algorithm that integrates a drone energy consumption model, namely EMADRL. The EMADRL algorithm first formulates the drone routing problem within a multi-agent reinforcement learning framework. It subsequently designs a strategy network model comprising multiple agent networks, tailored to address the node adjacency and masking complexities typical of multi-depot vehicle routing problem. Training utilizes strategy gradient algorithms and attention mechanisms. Furthermore, local and sampling search strategies are introduced to enhance solution quality. Extensive experimentation demonstrates that EMADRL consistently achieves high-quality solutions swiftly. A comparative analysis against contemporary algorithms reveals EMADRL's superior energy efficiency, with average energy savings of 5.96% and maximum savings reaching 12.45%. Thus, this approach offers a promising new avenue for optimizing energy consumption in last-mile distribution scenarios.

Multi-Agent DRL for Air-to-Ground Communication Planning in UAV-Enabled IoT Networks.

In this paper, we present a novel method to enhance the sum-rate effectiveness in full-duplex unmanned aerial vehicle (UAV)-assisted communication networks. Existing approaches often couple uplink and downlink associations, resulting in suboptimal performance, particularly in dynamic environments where user demands and network conditions are unpredictable. To overcome these limitations, we propose a decoupling of uplink and downlink associations for ground-based users (GBUs), significantly improving network efficiency. We formulate a comprehensive optimization problem that integrates UAV trajectory design and user association, aiming to maximize the overall sum-rate efficiency of the network. Due to the problem's non-convexity, we reformulate it as a Partially Observable Markov Decision Process (POMDP), enabling UAVs to make real-time decisions based on local observations without requiring complete global information. Our framework employs multi-agent deep reinforcement learning (MADRL), specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which balances centralized training with distributed execution. This allows UAVs to efficiently learn optimal user associations and trajectory controls while dynamically adapting to local conditions. The proposed solution is particularly suited for critical applications such as disaster response and search and rescue missions, highlighting the practical significance of utilizing UAVs for rapid network deployment in emergencies. By addressing the limitations of existing centralized and distributed solutions, our hybrid model combines the benefits of centralized training with the adaptability of distributed inference, ensuring optimal UAV operations in real-time scenarios.

A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States.

Multiagent deep reinforcement learning (DRL) makes optimal decisions dependent on system states observed by agents, but any uncertainty on the observations may mislead agents to take wrong actions. The mean-field actor-critic (MFAC) reinforcement learning is well-known in the multiagent field since it can effectively handle a scalability problem. However, it is sensitive to state perturbations that can significantly degrade the team rewards. This work proposes a Robust MFAC (RoMFAC) reinforcement learning that has two innovations: 1) a new objective function of training actors, composed of a policy gradient function that is related to the expected cumulative discount reward on sampled clean states and an action loss function that represents the difference between actions taken on clean and adversarial states and 2) a repetitive regularization of the action loss, ensuring the trained actors to obtain excellent performance. Furthermore, this work proposes a game model named a state-adversarial stochastic game (SASG). Despite the Nash equilibrium of SASG may not exist, adversarial perturbations to states in the RoMFAC are proven to be defensible based on SASG. Experimental results show that RoMFAC is robust against adversarial perturbations while maintaining its competitive performance in environments without perturbations.

Multi-Agent Deep Reinforcement Learning Based Scheduling Approach for Mobile Charging in Internet of Electric Vehicles

Multi-Agent Deep Reinforcement Learning Based Scheduling Approach for Mobile Charging in Internet of Electric Vehicles

Priority-Based Load Balancing With Multiagent Deep Reinforcement Learning for Space–Air–Ground Integrated Network Slicing

Priority-Based Load Balancing With Multiagent Deep Reinforcement Learning for Space–Air–Ground Integrated Network Slicing

Resource Allocation for Dynamic Platoon Digital Twin Networks: A Multi-Agent Deep Reinforcement Learning Method

Resource Allocation for Dynamic Platoon Digital Twin Networks: A Multi-Agent Deep Reinforcement Learning Method

Indoor Periodic Fingerprint Collections by Vehicular Crowdsensing via Primal-Dual Multi-Agent Deep Reinforcement Learning

Indoor Periodic Fingerprint Collections by Vehicular Crowdsensing via Primal-Dual Multi-Agent Deep Reinforcement Learning