Multi-agent Deep Reinforcement Learning Algorithm Research Articles

Multi-agent reinforcement learning based computation offloading and resource allocation for LEO Satellite edge computing networks

Due to the limitations caused by geographical conditions and economic requirements, it is difficult to provide computing services by terrestrial networks for mobile terminals in remote areas. To address this issue, mobile edge computing (MEC) servers can be deployed in the low earth orbit (LEO) satellites to act as a complement and accommodate the unserved terminals. However, offloading computing tasks to servers in satellites may increase the energy consumption of ground terminals. Considering the limited battery capacity of ground terminals, how to perform the computation offloading and resource allocation are key challenges in the LEO satellite edge computing networks. Therefore, in this paper, we investigate the energy minimization problem for LEO satellite edge computing networks, where a multi-agent deep reinforcement learning algorithm with global rewards is proposed to optimize the transmit power, CPU frequency, bit allocation, offloading decision and bandwidth allocation via a decentralized method. Simulation results show that our proposed algorithm can converge faster. Most importantly, compared with the random algorithm, the proximal policy optimization (PPO) algorithm, and the deep deterministic policy gradient (DDPG) algorithm, the ground terminals’ energy consumption can be effectively reduced by our proposed algorithm.

Partially observable deep reinforcement learning for multi-agent strategy optimization of human-robot collaborative disassembly: A case of retired electric vehicle battery

The burgeoning electric vehicle (EV) industry has precipitated a commensurate surge in the consumption of EV batteries, which are currently labor-intensive and inefficient for the recycling and disassembly of EV batteries. However, it is a potential trend to enhance the efficacy and safety of the disassembly of EV batteries based on human-robot collaboration (HRC) method. Because of the uncertainty of retired EV battery disassembly and the inefficiency of the existing disassembling sequence, it is difficult to be fully accomplish through HRC disassembly. The collaborative disassembly of EV batteries by humans and robots can be conceptualized as agents engaging with and learning from the environment, and modeled as a multi-agent Markov game process. This paper aims to address the challenge of HRC in the disassembly of EV batteries by recognizing the dual attributes of partial observability and non-smoothness in the suitable disassembly scenario. A partially observable multi-agent reinforcement learning environment is constructed, incorporating the structural aspects of the EV battery and the disassembly task. The framework is extended to the QMIX-HRC algorithm on the QMIX architecture (as a value-based multi-agent deep reinforcement learning algorithm), specifically designed to tackle the sequence problem in human-robot collaborative disassembly of EV batteries. The optimization results would yield a task sequence to offer maximal global co-benefit during the exploration iteration, facilitating a reduction in labor costs and an enhancement of co-efficiency. The viability of the QMIX-HRC disassembly strategy would be verified through the eventual disassembly sequence of a simulated battery pack through a real human-robot collaborative disassembly station.

Multi-Agent Deep Reinforcement Learning-Based Fine-Grained Traffic Scheduling in Data Center Networks

In data center networks, when facing challenges such as traffic volatility, low resource utilization, and the difficulty of a single traffic scheduling strategy to meet demands, it is necessary to introduce intelligent traffic scheduling mechanisms to improve network resource utilization, optimize network performance, and adapt to the traffic scheduling requirements in a dynamic environment. This paper proposes a fine-grained traffic scheduling scheme based on multi-agent deep reinforcement learning (MAFS). This approach utilizes In-Band Network Telemetry to collect real-time network states on the programmable data plane, establishes the mapping relationship between real-time network state information and the forwarding efficiency on the control plane, and designs a multi-agent deep reinforcement learning algorithm to calculate the optimal routing strategy under the current network state. The experimental results demonstrate that compared to other traffic scheduling methods, MAFS can effectively enhance network throughput. It achieves a 1.2× better average throughput and achieves a 1.4–1.7× lower packet loss rate.

MADRLOM: A Computation offloading mechanism for software-defined cloud-edge computing power network

Cloud-edge computing power network often exhibits complex and heterogeneous structures, posing several challenges to computation offloading that significantly impact network performance and the efficient utilization of computation resources. In this paper, we propose a cloud-edge computing power network architecture that efficiently integrates cloud and edge computing resources into a single network system using software-defined networking technology to support upper-layer business applications. In this context, existing computation offloading methods struggle with issues like users' personalized requirements for latency and energy consumption, as well as the inability to adapt to dynamically complex environments. To overcome these challenges, we introduce a computation offloading mechanism, MADRLOM, focusing on the network's heterogeneity and modeling the computation offloading problem at the cloud-edge end. We formalize the offloading problem as a Markov process and employ a multi-agent deep reinforcement learning algorithm based on priority experience replay sampling to address the planning problem of computation offloading, allowing user equipment to continuously optimize offloading strategies in response to environmental changes and achieve rational resource allocation. Through dynamic offloading strategies, computation tasks can be intelligently allocated to appropriate computing nodes, thereby achieving optimal resource utilization. We utilized the Mininet simulation platform to construct the experimental environment for the software-defined cloud-edge computing power network and compared it with several representative computation offloading strategies. The experimental results demonstrate that the MADRLOM significantly reduces the total system overhead in the software-defined cloud-edge computing network and shows excellent adaptability to environmental changes.

Age of correlated information-optimal dynamic policy scheduling for sustainable Green IoT devices: A multi-agent deep reinforcement learning approach

The rapid progress in communication technologies and the widespread adoption of connected devices have given rise to various information-centric Internet-of-Things (IoT) systems, typically necessitating timely updates of information. Green IoT (G-IoT) strives to enhance the environment by reducing the power consumption of billions of devices engaged in extensive data exchange, addressing the substantial energy demand in the process. The Age of Correlated Information (AoCI) measures the freshness of information shared among two or more devices that contribute to the same decision-making process. Optimizing AoCI using Deep Reinforcement Learning (DRL) in IoT reduces energy consumption, optimizes resource utilization, and promotes environmentally conscious communication, contributing to the development of a sustainable G-IoT system. This paper focuses on scheduling the transmission of status update packets among interconnected G-IoT devices to minimize the application-specific long-term average AoCI. The problem is modeled as an NP-hard episodic Markov Decision Process (MDP), highlighting its computational complexity. To handle correlations and the curse of dimensionality, a multi-agent deep reinforcement learning algorithm, specifically the Multi-Agent Deep Deterministic Policy Grading (MADDPG) algorithm, is developed. The training progress displays episode rewards, with an environment designed to penalize multiple agents transmitting simultaneously or none at all, promoting cooperative behavior and minimizing the average age of correlated information. We provide comprehensive simulation results, including reward convergence, the learning process of actors and critics, and the resulting average AoCI with the number of episodes, demonstrating the effectiveness of the MADDPG algorithm.

Wind power investment equilibrium guided by reliability option considering electricity-carbon emission trading-trading green certificate joint market

Reliability option (RO) is a capacity adequacy mechanism and it is coupled with electricity market, carbon emission trading market and trading green certificate market. This article explores the impact of implementing the RO mechanism in a multi-market trading context on wind power investment equilibrium and user side cost, under different demand elasticities and the degrees of competition on the generation side. We introduce a novel multi-agent deep reinforcement learning algorithm, leveraging the twin delayed deep deterministic policy gradient, aimed at obtaining equilibrium. Initially, we propose a method for calculating the comprehensive income of power producers, integrating risk preferences using Conditional Value at Risk. Subsequently, we employ a decentralized partially observable Markov decision process to emulate multi-agent investment patterns. During each exploration phase, wind power output scenarios are meticulously crafted via Latin hypercube sampling. Evaluations of our algorithm underscore its superior convergence and discerning rationality. Numerical simulations reveal that in areas with high demand elasticity, the RO mechanism primarily reduces user side risk without much impact on equilibrium. Yet, in regions with low demand elasticity, its role in influencing wind power investment and user side risk hinges on the degree of market competition and the designated wind power RO sales ratio.

Multi-agent deep reinforcement learning based decision support model for resilient community post-hazard recovery

After a city-scale natural hazard, policymakers should plan sound decisions on the repair sequence to ensure the resilient recovery of the community, which consists of interdependent infrastructures. Stochastic scheduling for repairing interdependent infrastructure systems is a difficult control problem with huge decision spaces. This study proposes a novel decision support model to determine the optimal restoration policies for the purpose of maximizing disaster resilience. A simulation environment is first developed, consisting of hazard intensity assessment, components damage evaluation, system recovery simulation, and resilience quantification. The graph theory is utilized to represent the interdependencies among different systems, and the heterogeneous graph neural network is integrated into this framework to extract the topology and interdependency information of the whole community. The optimal repair policies approximated by neural networks are trained by a multi-agent deep reinforcement learning algorithm, considering uncertainties of the restoration process. The superiority and efficiency of the proposed method are demonstrated through a case study of the Tsinghua University campus, where different decision-making objectives are considered. The results show that the recovery trajectories determined by the proposed model have the highest performance compared to conventional methods. Besides, the proposed methodology based on transfer learning can achieve high computational efficiency for new damage scenarios. This model is promising to be a high-performance, robust decision-support tool for post-hazard repairing decisions.

Reliable adaptive edge-cloud collaborative DNN inference acceleration scheme combining computing and communication resources in optical networks

With the continuous development of the Artificial Intelligence of Things, deep neural network (DNN) models require a larger amount of computing capacity. The emerging edge-cloud collaboration architecture in optical networks is proposed as an effective solution, which combines edge computing with cloud computing to provide faster response and reduce the cloud load for compute-intensive tasks. The multi-layered DNN model can be divided into subtasks that are offloaded to edge and cloud servers for computation in this architecture. In addition, as bearer networks for computing capacity, once a server or link in optical networks fails, a large amount of data can be lost, so the robust reliability of the edge-cloud collaborative optical networks is very important. To solve the above problems, we design a reliable adaptive edge-cloud collaborative DNN inference acceleration scheme (RACAI) combining computing and communication resources. We formulate the RACAI into a mixed integer linear programming model and develop a multi-agent deep reinforcement learning algorithm (MADRL-RACIA) to jointly optimize DNN task partitioning, offloading, and protection. The simulation results show that compared with the benchmark schemes, the proposed MADRL-RACIA can provide a guarantee of reliability for more tasks under latency constraints and reduce the blocking probability.

Physical-assisted multi-agent graph reinforcement learning enabled fast voltage regulation for PV-rich active distribution network

Active distribution network is encountering serious voltage violations associated with the proliferation of distributed photovoltaic. Cutting-edge research has confirmed that voltage regulation techniques based on deep reinforcement learning manifest superior performance in addressing this issue. However, such techniques are typically applied to the specifically fixed network topologies and have insufficient learning efficiency. To address these challenges, a novel edge intelligence, featured by a multi-agent deep reinforcement learning algorithm with graph attention network and physical-assisted mechanism, is proposed. This novel method is unique in that it includes the graph attention network into reinforcement learning to capture spatial correlations and topological linkages among nodes, allowing agents to be “aware” of topology variations caused by reconfiguration real time. Furthermore, employing a relatively exact physical model to generate reference experiences and storing them in a replay buffer enables agents to identify effective actions faster during training and thus, greatly enhances the efficiency of learning voltage regulation laws. All agents are trained centralized to learn a coordinated voltage regulation strategy, which is then executed decentralized based solely on local observation for fast response. The proposed methodology is evaluated on the IEEE 33-node and 136-node systems, and it outperforms the previously implemented approaches in convergence and control performance.

Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay

The increasing penetration of distributed renewable energy resources brings a great challenge for real-time voltage security of distribution grids. The paper proposes a safe multi-agent deep reinforcement learning (MADRL) algorithm for real-time control of inverter-based Volt-Var control (VVC) in distribution grids considering communication delay to minimize the network power loss, while maintaining the nodal voltages in a safe range. The multi-agent VVC is modeled as a constrained Markov game, which is solved by the MADRL algorithm. In the training stage, the safety projection is added to the combined policy to analytically solve an action correction formulation to promote more efficient and safe exploration. In the real-time decision-making stage, a state synchronization block is designed to impute the data under the latest timestamp as the input of the agents deployed in a distributed manner, to avoid instability caused by communication delay. The simulation results show that the proposed algorithm performs well in safe exploration, and also achieves better performance under communication delay.

A traffic light control method based on multi-agent deep reinforcement learning algorithm

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration.

Addressing deadlock in large‐scale, complex rail networks via multi‐agent deep reinforcement learning

AbstractRail freight planning problems pose specific challenges that have attracted the attention of academics and industry professionals for many decades. They involve multiple types of assets (trains, stations, terminals, etc.) and are subjected to structural, operational and safety constraints. Even though various approaches have been proposed, few can address the complexity and size of real‐world scenarios, and decentralized techniques, like multi‐agent systems (MAS), have become more prevalent. The current state of the art in disciplines such as agent technology, reinforcement learning and discrete‐event simulation allows the implementation of complex architectures, with multiple actors interacting and learning simultaneously. Therefore, this study takes advantage of these current advances and proposes an innovative approach to real‐time traffic management problems in freight railway networks through multi‐agent deep reinforcement learning (MADRL). This study was motivated by the decision‐making scheduling problems arising in the Hunter Valley Coal Chain (HVCC), located in New South Wales, Australia. The MADRL algorithm uses as the training environment the simulation model currently utilized for capacity planning of the HVCC, allowing experiments with actual data. Thus, we enhanced the simulation model to accommodate a MAS with intelligent agents representing system elements, such as trains, dump stations, and load points. Furthermore, these agents act in a decentralized fashion based on local observations, constituting a partially‐observed Markov decision process (dec‐POMDP). Three variations of the MADRL approach are presented: a baseline model, an extended model, and one that directly addresses deadlocks. Finally, we present a transfer learning method that improves deadlock resolution and leverages performance. In the experiments, we explore specific, complex scenarios arising in the HVCC, where trains frequently face deadlock conditions. The baseline model outperforms a first‐come‐first‐serve (FCFS) based heuristic used by HVCC's simulation model and a genetic algorithm in instances with up to 60 trains – but fails in more complex scenarios. On the other hand, the most advanced model, which addresses deadlocks via transfer learning, always finds feasible solutions and produces policies that outperform the FCFS‐based heuristic in 94% of the instances.

Multi-agent deep reinforcement learning algorithm with self-adaption division strategy for VNF-SC deployment in SDN/NFV-Enabled Networks

Multi-agent deep reinforcement learning algorithm with self-adaption division strategy for VNF-SC deployment in SDN/NFV-Enabled Networks

Research on the Multiobjective and Efficient Ore-Blending Scheduling of Open-Pit Mines Based on Multiagent Deep Reinforcement Learning

In order to solve the problems of a slow solving speed and easily falling into the local optimization of an ore-blending process model (of polymetallic multiobjective open-pit mines), an efficient ore-blending scheduling optimization method based on multiagent deep reinforcement learning is proposed. Firstly, according to the actual production situation of the mine, the optimal control model for ore blending was established with the goal of minimizing deviations in ore grade and lithology. Secondly, the open-pit ore-matching problem was transformed into a partially observable Markov decision process, and the ore supply strategy was continuously optimized according to the feedback of the environmental indicators to obtain the optimal decision-making sequence. Thirdly, a multiagent deep reinforcement learning algorithm was introduced, which was trained continuously and modeled the environment to obtain the optimal strategy. Finally, taking a large open-pit metal mine as an example, the trained multiagent depth reinforcement learning algorithm model was verified via experiments, with the optimal training model displayed on the graphical interface. The experimental results show that the ore-blending optimization model constructed is more in line with the actual production requirements of a mine. When compared with the traditional multiobjective optimization algorithm, the efficiency and accuracy of the solution have been greatly improved, and the calculation results can be obtained in real-time.

A Multiagent Reinforcement Learning Approach for Wind Farm Frequency Control

As wind turbines (WTs) become more prevalent, there is an increasing interest in actively controlling their power output to participate in the frequency regulation for the power grid. Conventional frequency regulation controllers use fixed gains, making it difficult for the WT to adjust its kinetic energy uptake to its operating conditions and to collaborate effectively with other WTs in the wind farm. In addition, the design of conventional frequency controllers does not consider their impacts on the mechanical structure. To address these issues, in this article, we model the cooperative frequency control problem for all the WTs in a wind farm as a decentralized partially observable Markov decision process and use a multiagent deep reinforcement learning algorithm to solve it. We also develop a grid-connected wind farm simulation model based on MATLAB/Simulink and OpenFAST, which can reflect the detailed interactions between the electrical and mechanical components of WTs. Simulation results show that the proposed strategy is effective in reducing frequency drops and has less impact on mechanical structure deflections compared with traditional methods.

Adaptive Data Collection and Offloading in Multi-UAV-Assisted Maritime IoT Systems: A Deep Reinforcement Learning Approach

This paper studies the integration of data collection and offloading for maritime Internet of Things (IoT) systems with multiple unmanned aerial vehicles (UAVs). In the considered multi-UAV maritime IoT system, the UAVs act as the aerial base stations to complete the missions of data collection from buoys and data offloading to the offshore base station (OBS). In this case, the UAVs need to adaptively select the mission mode between data collection and data offloading according to the network resources and mission requirements. In this paper, we aimed to minimize the completion time of data collection and offloading missions for all UAVs by jointly optimizing the UAV trajectories, mission mode selection, transmit power of buoys, and association relationships between the UAVs and buoy/OBS. In order to solve the mixed-integer non-convex minimization problem, we first designed a multi-agent deep reinforcement learning algorithm based on a hybrid discrete and continuous action space to preliminarily obtain the UAV trajectories, mission mode selection, and the transmit power of buoys. Then, we propose an algorithm based on the stable marriage problem to determine the buoy–UAV and UAV–OBS association relationships. Finally, the simulation results show that the proposed algorithms can effectively shorten the total mission completion time of data collection and offloading for the multi-UAV-assisted maritime IoT system.

Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch

With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement learning algorithm is proposed to deal with distributed nonconvex economic dispatch problems. In the algorithm, agents run independent reinforcement learning algorithms and update their local Q-functions with a newly defined joint reward. The double network structure is adopted to approximate the Q-function so that the offline trained model can be used online to provide recommended power outputs for time-varying demands in real-time. By introducing the reward network, the competition mechanism between the reward network and the target network is established to determine a progressively stable target value, which achieves coordination among agents and pledges the losses of the Q-networks to converge well. Theoretical analysis is given and case studies are conducted to prove the advantages compared with existing approaches.

Curriculum Reinforcement Learning Based on K-Fold Cross Validation.

With the continuous development of deep reinforcement learning in intelligent control, combining automatic curriculum learning and deep reinforcement learning can improve the training performance and efficiency of algorithms from easy to difficult. Most existing automatic curriculum learning algorithms perform curriculum ranking through expert experience and a single network, which has the problems of difficult curriculum task ranking and slow convergence speed. In this paper, we propose a curriculum reinforcement learning method based on K-Fold Cross Validation that can estimate the relativity score of task curriculum difficulty. Drawing lessons from the human concept of curriculum learning from easy to difficult, this method divides automatic curriculum learning into a curriculum difficulty assessment stage and a curriculum sorting stage. Through parallel training of the teacher model and cross-evaluation of task sample difficulty, the method can better sequence curriculum learning tasks. Finally, simulation comparison experiments were carried out in two types of multi-agent experimental environments. The experimental results show that the automatic curriculum learning method based on K-Fold cross-validation can improve the training speed of the MADDPG algorithm, and at the same time has a certain generality for multi-agent deep reinforcement learning algorithm based on the replay buffer mechanism.

MACA: Multi-Agent with Credit Assignment for Computation Offloading in Smart Parks Monitoring

Video monitoring has a wide range of applications in a variety of scenarios, especially in smart parks. How to improve the efficiency of video data processing and reduce resource consumption have become of increasing concern. The high complexity of traditional computation offloading algorithms makes it difficult to apply them to real-time decision-making scenarios. Thus, we propose a multi-agent deep reinforcement learning algorithm with credit assignment (MACA) for computation offloading in smart park monitoring. By making online decisions after offline training, the agent can give consideration to both decision time and accuracy in effectively solving the problem of the curse of dimensionality. Via simulation, we compare the performance of MACA with traditional deep Q-network reinforcement learning algorithm and other methods. Our results show that MACA performs better in scenarios where there are a higher number of agents and can minimize request delay and reduce task energy consumption. In addition, we also provide results from a generalization capability verified experiment and ablation study, which demonstrate the contribution of MACA algorithm to each component.

Multi-agent deep reinforcement learning algorithm with trend consistency regularization for portfolio management

Multi-agent deep reinforcement learning algorithm with trend consistency regularization for portfolio management