Multi-agent Deep Reinforcement Learning Research Articles

Distributed multi-agent deep reinforcement learning for trajectory planning in UAVs-assisted edge offloading

Distributed multi-agent deep reinforcement learning for trajectory planning in UAVs-assisted edge offloading

A multi-UAV assisted task offloading and path optimization for mobile edge computing via multi-agent deep reinforcement learning

To tackle task offloading and path planning challenges in multi-UAV-assisted mobile edge computing, this paper proposes a task offloading and path optimization approach via multi-agent deep reinforcement learning. The primary goal is to minimize the overall energy consumption of the system and improve computational performance. Initially, we established a model for a multi-UAV-assisted mobile edge computing system that centrally manages the UAV network through software-defined networking technology. Subsequently, considering UAV load and fairness in user equipment-related services, we employ the multi-agent deep deterministic policy gradient algorithm to optimize task offloading and UAV path management, aiming at load balancing and reducing overall system energy consumption. Simulation results demonstrate our method’s effectiveness in reducing energy consumption and computation latency compared to benchmark algorithms. It ensures system efficiency, stability, and reliability, meeting mobile edge users’ service requests while utilizing computing resources efficiently.

Distribution network voltage control considering virtual power plants cooperative optimization with transactive energy

The integration of massive distributed energy resources (DERs) brings new challenges for the distribution networks (DNs) operation control. To control DERs effectively, virtual power plants (VPPs) are introduced into DNs. Conventionally, the control strategy of VPPs is formulated as an optimization model without considering voltage control and transactive energy (TE) in DNs. In this article, multiple VPPs cooperative optimization with TE is studied and cast as a nonlinear programming model. It aims to optimize the voltage control of DNs and the operation profits of VPPs. To dynamically capture the optimal operation state of VPPs in DNs, the distribution locational marginal pricing and DN partition are incorporated into the model. In general, this type of model is computationally challenging and mainly solved by approximate algorithms with low efficiency. Recently, multi-agent deep reinforcement learning (MADRL) is emerging as a scalable and promising data-driven approach. We propose an augmented MADRL solution to this problem. In the training stage, one soft actor-critic learning agent augmented by parameter sharing strategy is employed to simultaneously capture the state-action patterns of multiple VPPs. The prioritized experience replay technique is applied to improve learning efficiency and speed. Numerical test results demonstrate the superiority of the proposed method over benchmark methods.

Multi-agent cooperative area coverage: A two-stage planning approach based on reinforcement learning

Multi-agent area coverage aims to accomplish the complete traversal of the target area through cooperation between agents. Focusing on the problems of low coverage efficiency and weak practicability in the existing methods, we propose a two-stage area coverage method based on multi-agent deep reinforcement learning. In the first stage, we convert the coverage path planning problem into an optimal grid selection problem, and according to the equivalence of agents in cooperative tasks, we propose a distributed coverage path planning algorithm based on QMIX and a grid coverage map. The second stage is to realize the cooperative navigation control in a constrained environment with obstacles and non-ideal communication conditions. To implement the stage, we design a hybrid attention mechanism to adaptively aggregate important feature information of adjacent agents and obstacles, which efficiently exploits the limited local perception and communication capabilities of agents to perform cooperative control. The experimental results show that the proposed two-stage multi-agent area coverage method can accomplish the area coverage task in the environment with random obstacles, and the area coverage efficiency and robustness are significantly better than other reinforcement learning based or traditional coverage algorithms. In addition, the results also verify that the proposed method has the advantage of adapting to the dynamic changes in the number of agents and the communication range.

Multi-agent deep reinforcement learning based fully decentralized aggregation frequency regulation of electric vehicle

Electric vehicles (EVs) are high-quality flexible resources to provide frequency regulation services. However, the EV real-time participation in ancillary service market for frequency regulation (FRASM) under the existing aggregation framework faces data privacy and interest conflicts problems. To solve these issues, this paper proposes the fully-decentralized aggregator (FDA) as a trusted and non-profit agent to replace the traditional aggregator and help EVs interact with FRASM in a decentralized manner. Then a fully decentralized aggregation framework based on consortium blockchain is constructed to protect the interaction data security during frequency regulation process. Furthermore, the inference network is introduced so that EVs can learn better performance and optimize regulation mileage without sharing their private data. This paper proposes an inference network based multi-agent soft actor-critic (IN-MASAC) method that realizes the decentralized interaction between EVs and FRASM and considers multiple user preferences. The case studies demonstrate the efficiency and scalability of the fully decentralized aggregation framework and IN-MASAC method.

Multi-agent deep reinforcement learning-based multi-time scale energy management of urban rail traction networks with distributed photovoltaic–regenerative braking hybrid energy storage systems

The integration of photovoltaics (PVs), regenerative braking (RB) techniques, and energy storage devices has become crucial to promote energy conservation and emission reduction for a sustainable future of urban rail traction networks (URTNs). This paper proposes a tri-level multi-time scale energy management framework for the economic and low-carbon operation of URTNs with PV–RB hybrid energy storage systems (HESSs) based on multi-agent deep reinforcement learning (MADRL). A two-stage stochastic scheduling approach is developed to minimize daily operation and carbon trading costs at the upper level and correct day-ahead scheduling deviations against multi-source uncertainties at the middle level. A MADRL-based real-time energy management strategy is established to optimize the PV–RB power flow and promote its utilization by coordinating distributed HESSs at the lower level. The HESS control problem is formulated as a decentralized partially observable Markov decision process and solved by a multi-agent control algorithm based on monotonic value function factorization. A Copula-based spatio-temporal dependency model is devised to characterize the PV, passenger flow, and traction load uncertainties and generate daily URTN operation scenarios for enhancing day-ahead and intraday decisions. Comparative studies demonstrate the effectiveness of the proposed framework in terms of a cost reduction by 11.98% and a PV–RB energy utilization improvement by 13.94%.

Efficient multiple unmanned aerial vehicle-assisted data collection strategy in power infrastructure construction

Efficient data collection and sharing play a crucial role in power infrastructure construction. However, in an outdoor remote area, the data collection efficiency is reduced because of the sparse distribution of base stations (BSs). Unmanned aerial vehicles (UAVs) can perform as flying BSs for mobility and line-of-sight transmission features. In this paper, we propose a multiple temporary UAV-assisted data collection system in the power infrastructure scenario, where multiple temporary UAVs are employed to perform as relay or edge computing nodes. To improve the system performance, the task processing model selection, communication resource allocation, UAV selection, and task migration are jointly optimized. We designed a QMIX-based multi-agent deep reinforcement learning algorithm to find the final optimal solutions. The simulation results show that the proposed algorithm has better convergence and lower system costs than the current existing algorithms.

Multi-agent deep reinforcement learning for hyperspectral band selection with hybrid teacher guide

Due to the presence of noisy and highly redundant bands in hyperspectral images (HSIs), band selection serves as a key preprocessing for downstream classification tasks. Recently, deep reinforcement learning (DRL) has been developed as a new trend for band selection of HSIs. Existing DRL-based methods often adopt single-agent, which are prone to fall into local optima due to an excessive action space. The multi-agent methods provide a feasible solution, but often require too much computation. To address these problems, a novel multi-agent DRL method with hybrid teacher guide (MH-DRL) is proposed for band selection of HSIs. In MH-DRL, each agent corresponding to a spectral band decides whether this band is selected. Moreover, a presentation-evaluation network (PE-Net) is constructed to design the reward by evaluating the candidate band subsets without any fine-tuning and represent the state by extracting the spatial-spectral features of HSIs. Then, three kinds of experienced band selection models are regarded as the teachers and designed to participate in the band exploration of DRL, which can improve the learning effectiveness and efficiency by accumulating the external knowledge from diverse teacher models. Finally, deep Q-learning algorithm is designed to update the agents and improve their self-learning ability from continuous exploration. Experimental results on three widely-used HSI data verify the performance of the proposed method better than some advanced band selection algorithms of HSIs. Our source code is available at https://github.com/jiefeng0109/MH-DRL.

Multi-agent deep reinforcement learning with enhanced collaboration for distribution network voltage control

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses.

Blocklength Allocation and Power Control in UAV-Assisted URLLC System via Multi-agent Deep Reinforcement Learning

Integration of unmanned aerial vehicles (UAVs) with ultra-reliable and low-latency communication (URLLC) systems can improve the real-time communication performance for various industrial internet of things (IIoT) applications. Designing an intelligent resource allocation system is one of the challenges posed by an energy-constrained UAV communication system. Therefore, we formulate a sum rate maximization problem, subject to the UAVs’ energy by optimizing the blocklength allocation and the power control jointly in the uplink UAV-assisted URLLC systems, in which the probabilistic channel model between UAV and users is adopted. The problem is difficult to solve due to the non-convex objective function and the energy constraints, and also challenging to make fast decision in the complex communication environment. Thus, we propose a deep reinforcement learning (DRL)-based scheme to optimize the blocklength allocation and power control jointly. First, transform the original problem into the multi-agent reinforcement learning process, where each subcarrier is regarded as the agent that optimizes its individual blocklength allocation and power control. Then, each agent makes the intelligent decision to obtain the maximum reward value depending on the weighted segmented reward function, which is related to the UAV energy consumption and user rates to improve the rate performance. Finally, the simulation results show that the proposed scheme outperforms the benchmark schemes and has the stable convergence in different settings, such as the learning rate, the error probability, the subcarrier spacing, and the number of users.

Automatic depth matching method of well log based on deep reinforcement learning

In the traditional well log depth matching tasks, manual adjustments are required, which means significantly labor-intensive for multiple wells, leading to low work efficiency. This paper introduces a multi-agent deep reinforcement learning (MARL) method to automate the depth matching of multi-well logs. This method defines multiple top-down dual sliding windows based on the convolutional neural network (CNN) to extract and capture similar feature sequences on well logs, and it establishes an interaction mechanism between agents and the environment to control the depth matching process. Specifically, the agent selects an action to translate or scale the feature sequence based on the double deep Q-network (DDQN). Through the feedback of the reward signal, it evaluates the effectiveness of each action, aiming to obtain the optimal strategy and improve the accuracy of the matching task. Our experiments show that MARL can automatically perform depth matches for well-logs in multiple wells, and reduce manual intervention. In the application to the oil field, a comparative analysis of dynamic time warping (DTW), deep Q-learning network (DQN), and DDQN methods revealed that the DDQN algorithm, with its dual-network evaluation mechanism, significantly improves performance by identifying and aligning more details in the well log feature sequences, thus achieving higher depth matching accuracy.

An IOV Spectrum Sharing Approach based on Multi-Agent Deep Reinforcement Learning

Highly dynamic Internet of Vehicles spectrum sharing can share spectrum owned by vehicle-to-infrastructure links through multiple workshop links to achieve efficient resource allocation. Aiming at the problem that the rapid variations in channel states in highly dynamic vehicular environments can make it challenging for base stations to gather and manage information about instantaneous channel states, we present a multi-agent deep reinforcement learning-based V2X spectrum access algorithm. The algorithm is designed to optimize the throughput of V2I user under V2V user delay and reliability constraints, and uses the experience gained from interacting with the communication environment to update the Q network to improve spectrum and power allocation strategies. Implicit collaborative agents are trained through an improved DQN model combined with dueling network architecture and long short-term memory network layers and public rewards. With lagged Q-learning and concurrent experience replay trajectories, the training process was stabilized and the non-stationarity problem caused by concurrent learning of multiple agents was resolved. Simulation results demonstrate that our presented algorithm achieves a mean successful payload delivery rate of 95.89%, which is 16.48% greater than that of the randomized baseline algorithm. Our algorithm obtains approximately the optimal value and shows performance close to the centralized brute force algorithm, which provides a better strategy for further minimizing the signaling overhead of the Internet of Vehicles communication system.

Cooperative Optimization of Traffic Signals and Vehicle Speed Using a Novel Multi-Agent Deep Reinforcement Learning

Cooperative Optimization of Traffic Signals and Vehicle Speed Using a Novel Multi-Agent Deep Reinforcement Learning

Distributed Reactive Power Optimization for Energy Internet via Multiagent Deep Reinforcement Learning With Graph Attention Networks

Distributed Reactive Power Optimization for Energy Internet via Multiagent Deep Reinforcement Learning With Graph Attention Networks

Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach

Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach

Cooperative Sensing and Heterogeneous Information Fusion in VCPS: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Sensing and Heterogeneous Information Fusion in VCPS: A Multi-Agent Deep Reinforcement Learning Approach

Distributed Drive Autonomous Vehicle Trajectory Tracking Control Based on Multi-Agent Deep Reinforcement Learning

Distributed Drive Autonomous Vehicle Trajectory Tracking Control Based on Multi-Agent Deep Reinforcement Learning

Multi-agent DRL-based Multipath Scheduling for Video Streaming with QUIC

The popularization of video streaming brings challenges in satisfying diverse Quality of Service (QoS) requirements. The multipath extension of the Quick UDP Internet Connection (QUIC) protocol, also called MPQUIC, has the potential to improve video streaming performance with multiple simultaneously transmitting paths. The multipath scheduler of MPQUIC determines how to distribute the packets onto different paths. However, while applying current multipath schedulers into MPQUIC, our experimental results show that they fail to adapt to various receive buffer sizes of different devices and comprehensive QoS requirements of video streaming. These problems are especially severe under heterogeneous and dynamic network environments. To tackle these problems, we propose MARS, a Multi-agent deep Reinforcement learning (MADRL)-based Multipath QUIC Scheduler, which is able to promptly adapt to dynamic network environments. It exploits the MADRL method to learn a neural network for each path and generate scheduling policy. Besides, it introduces a novel multi-objective reward function that takes out-of-order queue size and different QoS metrics into consideration to realize adaptive scheduling optimization. We implement MARS in an MPQUIC prototype and deploy in Dynamic Adaptive Streaming over HTTP system. Then, we compare it with the state-of-the-art multipath schedulers in both emulated and real-world networks. Experimental results show that MARS outperforms the other schedulers with better adaptive capability regarding the receive buffer sizes and QoS.

Global optimization strategy of prosumer data center system operation based on multi-agent deep reinforcement learning

The escalating issues of high energy consumption and carbon emissions in data centers (DCs) necessitate the optimization of system operations. However, early optimization strategies were overly simplistic and lacked automated updating and iterative capabilities. With the evolution of artificial intelligence (AI), researchers have applied deep reinforcement learning (DRL) algorithms to system operations. However, the optimization focus has been limited to the internal systems, lacking global optimization. In this paper, a global optimization control strategy based on the Dueling double-deep Q network (D3QN) and value decomposition network (VDN) algorithms is proposed to make the DCs system operate more closely with the upstream, midstream, and downstream. By adjusting battery charging/discharging capacity, computational workload, and waste heat utilization heating temperature global synergistic optimization is achieved. Compared with without optimization, renewable energy waste, operation cost, total electricity consumption, and grid electricity consumption are reduced by 18.37%, 9.78%, 4.01%, and 29.74%, respectively. Additionally, a detailed comparison between non-algorithmic optimization and algorithmic optimization is provided, offering valuable insights for substantial energy savings and emissions reduction in DCs. The results demonstrate the importance of fully exploring the interactive potential between upstream energy supply, midstream computational workload, and downstream waste heat recovery to achieve synergistic global optimization of “computing power", “thermal energy" and “electrical energy" for the sustainable and green development of DCs or other prosumer buildings.

Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks

Due to the limitations caused by geographical conditions and economic requirements, it is difficult to provide computing services by terrestrial networks for mobile terminals in remote areas. To address this issue, mobile edge computing (MEC) servers can be deployed in the low earth orbit (LEO) satellites to act as a complement and accommodate the unserved terminals. However, offloading computing tasks to servers in satellites may increase the energy consumption of ground terminals. Considering the limited battery capacity of ground terminals, how to perform the computation offloading and resource allocation are key challenges in the LEO satellite edge computing networks. Therefore, in this paper, we investigate the energy minimization problem for LEO satellite edge computing networks, where a multi-agent deep reinforcement learning algorithm with global rewards is proposed to optimize the transmit power, CPU frequency, bit allocation, offloading decision and bandwidth allocation via a decentralized method. Simulation results show that our proposed algorithm can converge faster. Most importantly, compared with the random algorithm, the proximal policy optimization (PPO) algorithm, and the deep deterministic policy gradient (DDPG) algorithm, the ground terminals’ energy consumption can be effectively reduced by our proposed algorithm.