Multi-Agent Deep Reinforcement Learning Approach Research Articles

Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management.

The utilization of large-scale distributed renewable energy (RE) promotes the development of the multimicrogrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy sufficiency. The multiagent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this article tackles this practical yet challenging issue by proposing a federated MADRL (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm, thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data are not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge National Laboratory distributed energy control communication laboratory MG (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Age of correlated information-optimal dynamic policy scheduling for sustainable Green IoT devices: A multi-agent deep reinforcement learning approach

The rapid progress in communication technologies and the widespread adoption of connected devices have given rise to various information-centric Internet-of-Things (IoT) systems, typically necessitating timely updates of information. Green IoT (G-IoT) strives to enhance the environment by reducing the power consumption of billions of devices engaged in extensive data exchange, addressing the substantial energy demand in the process. The Age of Correlated Information (AoCI) measures the freshness of information shared among two or more devices that contribute to the same decision-making process. Optimizing AoCI using Deep Reinforcement Learning (DRL) in IoT reduces energy consumption, optimizes resource utilization, and promotes environmentally conscious communication, contributing to the development of a sustainable G-IoT system. This paper focuses on scheduling the transmission of status update packets among interconnected G-IoT devices to minimize the application-specific long-term average AoCI. The problem is modeled as an NP-hard episodic Markov Decision Process (MDP), highlighting its computational complexity. To handle correlations and the curse of dimensionality, a multi-agent deep reinforcement learning algorithm, specifically the Multi-Agent Deep Deterministic Policy Grading (MADDPG) algorithm, is developed. The training progress displays episode rewards, with an environment designed to penalize multiple agents transmitting simultaneously or none at all, promoting cooperative behavior and minimizing the average age of correlated information. We provide comprehensive simulation results, including reward convergence, the learning process of actors and critics, and the resulting average AoCI with the number of episodes, demonstrating the effectiveness of the MADDPG algorithm.

Many-to-Many Task Offloading in Vehicular Fog Computing: A Multi-Agent Deep Reinforcement Learning Approach

Vehicular fog computing (VFC) has emerged as a promising solution to mitigate vehicular network computation load. In the hierarchical VFC, vehicles are employed as mobile fog nodes at the edge to provide reliable and low-latency services. Particularly, since privately-owned vehicles are rational nodes, their intentions for both computation provision and service demand should be considered instead of overestimating their willingness. To remunerate the participation intentions of vehicles as well as improve vehicular fog resource utilization in the large-scale VFC, the trading-based mechanism is a potential solution. In this paper, we propose a many-to-many task offloading framework based on the vehicular trading paradigm. This framework enables computational resource trading across different VFC subsystems and decides the multi-tier task offloading results based on the trading consensus. The trading process is viewed as a partially observable Markov decision process (POMDP) and a Multi-Agent Gated actor Attention Critic (MA-GAC) approach is designed to reach an effective and stable offload-and -serve cooperation among vehicles. Theoretical analyses and experiments verify the feasibility and efficiency of the proposed framework, and simulation results demonstrate that the coordinated MA-GAC approach not only benefits vehicles with higher long-term rewards but also optimizes the system social welfare in a distributed manner.

Distributed energy management of multi-area integrated energy system based on multi-agent deep reinforcement learning

This paper addresses the problem of distributed energy management in multi-area integrated energy systems (MA-IES) using a multi-agent deep reinforcement learning approach. The MA-IES consists of interconnected electric and thermal networks, incorporating renewable energy sources and heat conversion systems. The objective is to optimize the operation of the system while minimizing operational costs and maximizing renewable energy utilization. We propose a distributed energy management strategy that makes hierarchical decisions on intra-area heat energy and inter-area electric energy. The strategy is based on a multi-agent deep reinforcement learning framework, where each agent represents a component or unit in the MA-IES. We formulate the problem as a Markov Decision Process and employ Q-learning with experience replay and double networks to train the agents. The proposed strategy is evaluated using a simulation of a four-area MA-IES. The results demonstrate significant improvements in energy management compared to traditional methods, with higher renewable energy utilization and lower operational costs. Specifically, the strategy achieves 100% utilization of wind power, and decreases operational costs by 5.53%. Furthermore, it leverages the generalization capabilities of reinforcement learning to respond in real-time to uncertainties in demand and wind power output. The results highlight the advantages of the proposed strategy, making it a promising solution for optimizing the operation of multi-area integrated energy systems.

Weighted fair energy transfer in a UAV network: A multi-agent deep reinforcement learning approach

Flying Energy Sources (FESs) have been proven highly effective in transferring energy to battery-powered Unmanned Aerial Vehicles (UAVs). Such wireless transfer techniques in the literature were able to extend the flight duration of UAVs; however, they overlooked the fair distribution of energy among UAVs, which is of utmost importance for supporting diverse application demands. In this work, we quantify the urgency level of a UAV following its application responsibility, energy charging, and drainage rates and develop a framework for optimal energy transfer from FESs to UAVs in a weighted-fair way. Even though the developed framework can promote a highly balanced and fair energy distribution, its computation cannot always be done in polynomial time. Alternatively, to achieve a real-time solution, we further develop a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, namely AERIAL, to effectively model the complex interactions and dependencies among the UAVs and FESs and improve the wireless energy transfer process. The AERIAL system is implemented in the OpenAI Gym simulator platform and its performances have been compared with the state-of-the-art approaches. As high as improvements of 25.2% in fairness and 19.4% in average energy level demonstrated by the AERIAL system prove its effectiveness.

On-Chain and Off-Chain Data Management for Blockchain-Internet of Things: A Multi-Agent Deep Reinforcement Learning Approach

On-Chain and Off-Chain Data Management for Blockchain-Internet of Things: A Multi-Agent Deep Reinforcement Learning Approach

Peer-to-peer energy trading with energy trading consistency in interconnected multi-energy microgrids: A multi-agent deep reinforcement learning approach

Multi-energy microgrid technology is an essential for addressing the diversification of energy demand and local consumption of renewable energy sources. Peer-to-peer energy trading has emerged as a promising paradigm for the design of a decentralized trading framework. Therefore, this paper investigated the external peer-to-peer energy trading problem and internal energy conversion problem of interconnected multi-energy microgrids. The concept of energy trading consistency to avoid unreasonable energy trading behavior is first proposed and an off-design performance model of the energy conversion device is considered to more accurately reflect the operating status of the device. The complex decision-making problem with significantly large high-dimensional data is formulated as a partially observable Markov decision process and solved using the proposed multi-agent deep reinforcement learning approach combining the centralized training decentralized execution framework and soft actor-critic algorithm. Finally, the effectiveness of the proposed method was verified through a case simulation. The simulation results showed that the proposed method can reduce the total cost compared with the rule-based method.

Decoupling Patrolling Tasks for Water Quality Monitoring: A Multi-Agent Deep Reinforcement Learning Approach

Decoupling Patrolling Tasks for Water Quality Monitoring: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach

Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach

Enhancing Grid-Interactive Buildings Demand Response: Sequential Update-Based Multi-Agent Deep Reinforcement Learning Approach

Enhancing Grid-Interactive Buildings Demand Response: Sequential Update-Based Multi-Agent Deep Reinforcement Learning Approach

Multi-Level Objective Control of AVs at a Saturated Signalized Intersection with Multi-Agent Deep Reinforcement Learning Approach

Multi-Level Objective Control of AVs at a Saturated Signalized Intersection with Multi-Agent Deep Reinforcement Learning Approach

Resource Allocation in Uplink NOMA Systems: A Hybrid-Decision-Based Multi-Agent Deep Reinforcement Learning Approach

Resource Allocation in Uplink NOMA Systems: A Hybrid-Decision-Based Multi-Agent Deep Reinforcement Learning Approach

ATT-TA: A Cooperative Multiagent Deep Reinforcement Learning Approach for TSV Assignment in 3-D ICs

ATT-TA: A Cooperative Multiagent Deep Reinforcement Learning Approach for TSV Assignment in 3-D ICs

Adaptive Resource Scheduling in Permissionless Sharded-Blockchains: A Decentralized Multiagent Deep Reinforcement Learning Approach

Adaptive Resource Scheduling in Permissionless Sharded-Blockchains: A Decentralized Multiagent Deep Reinforcement Learning Approach

Energy-Aware MPTCP Scheduling in Heterogeneous Wireless Networks Using Multi-Agent Deep Reinforcement Learning Techniques

This paper proposes an energy-efficient scheduling scheme for multi-path TCP (MPTCP) in heterogeneous wireless networks, aiming to minimize energy consumption while ensuring low latency and high throughput. Each MPTCP sub-flow is controlled by an agent that cooperates with other agents using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. This approach enables the agents to learn decentralized policies through centralized training and decentralized execution. The scheduling problem is modeled as a multi-agent decision-making task. The proposed energy-efficient scheduling scheme, referred to as EE-MADDPG, demonstrates significant energy savings while maintaining lower latency and higher throughput compared to other state-of-the-art scheduling techniques. By adopting a multi-agent deep reinforcement learning approach, the agents can learn efficient scheduling policies that optimize various performance metrics in heterogeneous wireless networks.

Collective large-scale wind farm multivariate power output control based on hierarchical communication multi-agent proximal policy optimization

Wind power is becoming an increasingly vital source of renewable energy worldwide. However, controlling power generation in wind farms faces significant challenges due to the inherent complexity of these systems. To address this issue and maximize power output, we propose a novel communication-based multi-agent deep reinforcement learning approach for large-scale wind farm control. We introduce a multivariate power model for wind farms to analyze the impact of wake effects on power output. This model considers controllable variables such as axial induction factor, yaw angle, and tilt angle. To coordinate the continuous controls in large-scale wind farms, we propose the hierarchical communication multi-agent proximal policy optimization (HCMAPPO) algorithm. The wind farm is divided into multiple wind turbine aggregators (WTAs), and neighboring WTAs can exchange information through hierarchical communication to optimize power output. Our simulation results demonstrate that the multivariate HCMAPPO approach significantly increases wind farm power output compared to traditional PID control, coordinated model-based predictive control, and the multi-agent deep deterministic policy gradient algorithm. The HCMAPPO algorithm can be trained using a thirteen-turbine wind farm environment and effectively applied to large-scale wind farms. Importantly, as the wind farm scale increases, there is no significant increase in wind turbine blade fatigue damage from wake control. This multivariate HCMAPPO control strategy enables the collective maximization of power output in large-scale wind farms.

A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning

Multi-agent deep reinforcement learning (MADRL) approaches are at the forefront of contemporary research in optimum electric vehicle (EV) charging scheduling challenges. These techniques involve multiple agents that respond to a dynamic simulation environment to strategically integrate EV charging stations (EVCSs) on microgrids by incorporating the constraints posed by stochastic trip durations. In addition, recent research works have demonstrated that planning frameworks based on multi-objective optimization (MOO) techniques are suitable for the efficient functioning of microgrids comprising renewable energy sources (RESs) and battery energy storage systems (BESSs). Even though MADRL techniques have been used to solve the optimum EV charging scheduling challenges and MOO frameworks have been developed to determine the optimal RES-BESS allocation, the potential of merging MADRL and MOO is yet to be explored. Therefore, this research provides an opportunity to determine the effectiveness of combined MOO-MADRL dynamics and their computational efficacy. In this context, this work presents a novel Multi-objective Artificial Vultures Optimization Algorithm based on Multi-agent Deep Deterministic Policy Gradient (MOAVOA-MADDPG) planning framework for allocating RESs, BESSs, and EVCSs on microgrids. The objective function is formulated to optimize the network power losses, total installation and operational costs, greenhouse gas emissions, and system voltage stability. Moreover, the proposed framework incorporates the sporadic nature of RES systems and intends to improve the state of charge (SOC) of the EVs present in the network. The presented approach is validated using practical weather data and EV commuting behavior on the modified IEEE 33 bus network, two practical distribution feeders in Bangladesh, and the Turkish 141 bus network. According to the findings, the MOAVOA-MADDPG framework effectively accommodated the financial, technical, and environmental considerations with improved average SOC of the vehicles. Furthermore, statistical analysis, spacing, convergence, and hyper-volume metrics are employed to compare the suggested MOAVOA-MADDPG framework with five contemporary techniques. The findings indicate that, in every metric considered, the MOAVOA-MADDPG Pareto fronts provide superior solutions.

MW-MADDPG: a meta-learning based decision-making method for collaborative UAV swarm.

Unmanned Aerial Vehicles (UAVs) have gained popularity due to their low lifecycle cost and minimal human risk, resulting in their widespread use in recent years. In the UAV swarm cooperative decision domain, multi-agent deep reinforcement learning has significant potential. However, current approaches are challenged by the multivariate mission environment and mission time constraints. In light of this, the present study proposes a meta-learning based multi-agent deep reinforcement learning approach that provides a viable solution to this problem. This paper presents an improved MAML-based multi-agent deep deterministic policy gradient (MADDPG) algorithm that achieves an unbiased initialization network by automatically assigning weights to meta-learning trajectories. In addition, a Reward-TD prioritized experience replay technique is introduced, which takes into account immediate reward and TD-error to improve the resilience and sample utilization of the algorithm. Experiment results show that the proposed approach effectively accomplishes the task in the new scenario, with significantly improved task success rate, average reward, and robustness compared to existing methods.