Multi-agent Deep Reinforcement Learning Research Articles

Dynamic Pricing for Vehicle Dispatching in Mobility-as-a-Service Market via Multi-Agent Deep Reinforcement Learning

Dynamic Pricing for Vehicle Dispatching in Mobility-as-a-Service Market via Multi-Agent Deep Reinforcement Learning

Satellite-Terrestrial Coordinated Multi-Satellite Beam Hopping Scheduling Based on Multi-Agent Deep Reinforcement Learning

Satellite-Terrestrial Coordinated Multi-Satellite Beam Hopping Scheduling Based on Multi-Agent Deep Reinforcement Learning

Deep Reinforcement Learning-based Collaborative Multi-UAV Coverage Path Planning

Abstract The coverage path planning problem has gained significant attention in research due to its wide applicability and practical value in various fields such as logistics and distribution, smart homes, and unmanned vehicles. This paper focuses on studying the coverage path planning problem under multi-UAV collaboration to maximize the coverage of the mission area within a given time. To address this problem, we propose a multi-objective optimization model and reformulate it with the framework of Decentralized Partially Observable Markov Decision Process (Dec-POMDP). We then employ a multi-agent deep reinforcement learning (MADRL) method to solve the problem. Specifically, we introduce the ε—Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (ε—MADT3), which incorporates an exploration coefficient based on MATD3. This coefficient gradually decays with the number of iterations, allowing for a balance between exploration and exploitation. Numerous simulation results demonstrate that ε—MADT3 outperforms the baseline algorithm in terms of coverage rate and number of collisions.

Collision-Avoiding Flocking With Multiple Fixed-Wing UAVs in Obstacle-Cluttered Environments: A Task-Specific Curriculum- Based MADRL Approach.

Multiple unmanned aerial vehicles (UAVs) are able to efficiently accomplish a variety of tasks in complex scenarios. However, developing a collision-avoiding flocking policy for multiple fixed-wing UAVs is still challenging, especially in obstacle-cluttered environments. In this article, we propose a novel curriculum-based multiagent deep reinforcement learning (MADRL) approach called task-specific curriculum-based MADRL (TSCAL) to learn the decentralized flocking with obstacle avoidance policy for multiple fixed-wing UAVs. The core idea is to decompose the collision-avoiding flocking task into multiple subtasks and progressively increase the number of subtasks to be solved in a staged manner. Meanwhile, TSCAL iteratively alternates between the procedures of online learning and offline transfer. For online learning, we propose a hierarchical recurrent attention multiagent actor-critic (HRAMA) algorithm to learn the policies for the corresponding subtask(s) in each learning stage. For offline transfer, we develop two transfer mechanisms, i.e., model reload and buffer reuse, to transfer knowledge between two neighboring stages. A series of numerical simulations demonstrate the significant advantages of TSCAL in terms of policy optimality, sample efficiency, and learning stability. Finally, the high-fidelity hardware-in-the-loop (HITL) simulation is conducted to verify the adaptability of TSCAL. A video about the numerical and HITL simulations is available at https://youtu.be/R9yLJNYRIqY.

Task offloading and resource allocation in NOMA-VEC: A multi-agent deep graph reinforcement learning algorithm

Task offloading and resource allocation in NOMA-VEC: A multi-agent deep graph reinforcement learning algorithm

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Multi-UAV Positioning for Aerial Internet Service Management: A Multi-Agent Deep Reinforcement Learning Approach

The AI circular hydrogen economist: Hydrogen supply chain design via hierarchical deep multi-agent reinforcement learning

Hydrogen supply chain (HSC) consists of various production, conditioning, transportation, storage, and distribution processes, all of which require extensive computational resources for precise modelling. In this context, artificial intelligence (AI) is emerging as a pivotal tool for addressing model-based decision-making challenges, courtesy of its rapid and efficient computational capabilities. This paper proposes a comprehensive HSC model consisting of an economic policy planner, a wholesale hydrogen market, a power supply system, a hydrogen distribution system (HDS), and hydrogen refueling stations (HRSs). It leverages an AI circular hydrogen economist approach based on a hierarchical deep multi-agent reinforcement learning (MARL) algorithm, offering a new alternative to traditional multi-objective bi-level optimization platforms. The model incorporates green, blue, and gray hydrogen production processes as viable hydrogen production pathways, with the hierarchical MARL’s agents representing the HDS and HRSs at two decision-making levels. Energy requirements are met through a combination of on-site renewable energy sources, the main power grid, or distributed power generation systems. Several HSC scenarios are examined with respect to various combinations of green hydrogen supply rates and carbon credits granted to them under optimum conditions. The results showed that the developed hierarchical MARL has the potential to replace mathematical programming (MP), uncovering a new economic-environmental trade-off between profit of HDS and operational costs of HRSs. Notably, green hydrogen transactions exponentially increase within the supply chain as the carbon credits exceed $1 per kilogram of hydrogen. While this research focuses on optimizing daily operations within the HSC, future efforts can aim to extend this optimization to forecasted annual operations.

A multi-agent deep reinforcement learning paradigm to improve the robustness and resilience of grid connected electric vehicle charging stations against the destructive effects of cyber-attacks

The rising deployment of electric vehicle charging stations (EVCS) in existing power grids and their integration with electric vehicles through communication systems are increasingly vulnerable to cyber-attacks, such as false data injection (FDI) and uncertainties of communication system. This research introduces an EVCS evaluation from a techno-economic standpoint, utilizing the developed and data-oriented TSKFS&MADRL technique to identify and rectify inaccuracies stemming from cyber-attacks like FDI and communication system delays. This method aims to enhance the resilience of EVCSs against sabotage attacks by ensuring fast dynamic responses. Initially, the study models the potential targets of hackers, such as communication systems and transducer sensors, and employs the Euclidean norm theory, weighted least square error method, and residual error technique to identify cyber-attacks resulting from FDI through the comparison of measured data with reference values based on probability distribution functions. Subsequently, the network control and recovery requirements are facilitated by the proposed controller. The proposed approach application has been demonstrated in the IEEE 33 bus, showcasing a 7.33 % lower experimental operation cost compared to the RL method and 12.15 % lower than the CNN method. Additionally, the FDI detection time is observed to be 40 % quicker than alternative methods based on the experimental outcomes.

Advancing urban electric vehicle charging stations: AI-driven day-ahead optimization of pricing and Nudge strategies utilizing multi-agent deep reinforcement learning

Public charging stations (CSs) serve for electric vehicles (EVs) to charge during urban travel. Optimizing the charging time, location distribution, and power of EVs can increase the revenue of charging system operators (CSOs) and provide flexible regulation resources for the power grid. However, the optimization scheduling of CSs involves the charging choices of various users, which are influenced by their autonomy and bounded rationality. To guide users and encourage their participation in the charging schedule, we introduce the Nudge method from behavioral economics. To achieve collaborative optimization of non-economic Nudges and economic incentive strategies applying to multiple charging stations in a complex nonlinear environment involving users, CSO, and the transportation network, we leverage multi-agent deep reinforcement learning (MADRL). We construct a simulation environment using historical and survey data tailored to real users. This environment facilitates the training of agent groups to enhance decision-making processes. Case studies in a metropolis demonstrate that the agent group aimed at revenue improvement yields significant improvements in the CSO's revenue compared to fixed service fees and pricing strategies without Nudges. Moreover, the agent group aimed at power curve tracking achieves a lower average relative error in aligning the total charging power with the desired curve of the power system. This paper integrates sociological methods into the optimization of physical systems by MADRL, providing a new approach for the scheduling of EV charging considering user behavior.

Enhancing unmanned aerial vehicle communication through distributed ledger and multi-agent deep reinforcement learning for fairness and scalability

Enhancing unmanned aerial vehicle communication through distributed ledger and multi-agent deep reinforcement learning for fairness and scalability

Multi-agent DRL for edge computing: A real-time proportional compute offloading

In the Industrial Internet of Things, devices with limited computing power and energy storage often rely on offloading tasks to edge servers for processing. However, existing methods are plagued by the high cost of device communication and unstable training processes. Consequently, Deep reinforcement learning (DRL) has emerged as a promising solution to tackle the computation offloading problem. In this paper, we propose a framework called multi-agent twin delayed shared deep deterministic policy gradient algorithm (MASTD3) based on DRL. Firstly, we formulate the task offloading conundrum as a long-term optimization problem, which aids in mitigating the challenge of deciding between local or remote task execution by a device, leading to more effective task offloading management. Secondly, we enhance MASTD3 by introducing a priority experience replay buffer mechanism and a model sample replay buffer mechanism, thus improving sample utilization and overcoming the cold-start problem associated with long-term optimization. Moreover, we refine the actor-critic structure, enabling all agents to share the same critic network. This modification accelerates convergence speed during the training process and reduces computational costs during runtime. Finally, experimental results demonstrate that MASTD3 effectively addresses the proportional offloading problem, which is optimized by 44.32%, 29.26%, and 17.47% compared to DDPQN, MADDPG, and FLoadNet.

Multiobjective Optimization of Papermaking Wastewater Treatment Processes under Economic, Energy, and Environmental Goals.

Due to the heterogeneity of recycled paper materials and the production conditions, pollutants in papermaking wastewater fluctuate sharply over time. Quality control of the papermaking wastewater treatment process (PWTP) is challenging and costly. As regulations are also growing about the environmental effects of the PWTP on greenhouse gas (GHG) emission, energy consumption, etc., the PWTP formulates a complex multiobjective optimization problem. This research established a multiagent deep reinforcement learning framework to simultaneously optimize process cost, energy consumption, and GHG emission in the PWTP, subjected to the effluent quality, to realize economic, energy, and environmental (3E) goals. The biological treatment process of wastewater in paper mills was simulated using benchmark simulation model no. 1 (BSM1). The data generated based on the BSM manual was utilized for model training, and real data acquired from a local papermaking factory was used to estimate the model performance. The results show that the proposed method outperforms conventional techniques in identifying the best control strategies for multiple targets.

Low-carbon economic dispatch strategy for integrated electrical and gas system with GCCP based on multi-agent deep reinforcement learning

Low-carbon economic dispatch strategy for integrated electrical and gas system with GCCP based on multi-agent deep reinforcement learning

Multi-agent deep reinforcement learning for dynamic reconfigurable shop scheduling considering batch processing and worker cooperation

Reconfigurable manufacturing system is considered as a promising next-generation manufacturing paradigm. However, limited equipment and complex product processes add additional coupled scheduling problems, including resource allocation, batch processing and worker cooperation. Meanwhile, dynamic events bring uncertainty. Traditional scheduling methods are difficult to obtain good solutions quickly. To this end, this paper proposes a multi-agent deep reinforcement learning (DRL) based method for dynamic reconfigurable shop scheduling problem considering batch processing and worker cooperation to minimize the total tardiness cost. Specifically, a dual-agent DRL-based scheduling framework is first designed. Then, a multi-agent DRL-based training algorithm is developed, where two high-quality end-to-end action spaces are designed using rule adjustment, and an estimated tardiness cost driven reward function is proposed for order-level scheduling problem. Moreover, a multi-resource allocation heuristics is designed for the reasonable assignment of equipment and workers, and a batch processing rule is designed to determine the action of manufacturing cell based on workshop state. Finally, a strategy is proposed for handling new order arrivals, equipment breakdown and job reworks. Experimental results on 140 instances show that the proposed method is superior to scheduling rules, genetic programming, and two popular DRL-based methods, and can effectively deal with various disturbance events. Furthermore, a real-world assembly and debugging workshop case is studied to show that the proposed method is applicable to solve the complex reconfigurable shop scheduling problems.

Enhancing Grid-Interactive Buildings Demand Response: Sequential Update-Based Multiagent Deep Reinforcement Learning Approach

Enhancing Grid-Interactive Buildings Demand Response: Sequential Update-Based Multiagent Deep Reinforcement Learning Approach

Double-Timescale Multi-Agent Deep Reinforcement Learning for Flexible Payload in VHTS Systems

With the expansion of the very-high-throughput satellite (VHTS) system, the uneven distribution of traffic demands in time and space has become increasingly significant and cannot be ignored. It is a significant challenge to efficiently and dynamically allocate scarce on-board resources to ensure capacity and demand matching. The advancement of flexible payload technology provides the possibility to overcome this challenge. However, computational complexity is increasing due to the unsynchronized resource adjustment and the time-varying demands of the VHTS system. Therefore, we propose a double-timescale bandwidth and power allocation (DT-BPA) scheme to effectively manage the available resources in the flexible payload architecture. We use a multi-agent deep reinforcement learning (MADRL) algorithm aiming to meet the time-varying traffic demands of each beam and improve resource utilization. The simulation results demonstrate that the proposed DT-BPA algorithm enhanced the matching degree of capacity and demand as well as reduced the system’s power consumption. Additionally, it can be trained offline and implemented online, providing a more cost-effective solution for the VHTS system.

A coordinated active and reactive power optimization approach for multi-microgrids connected to distribution networks with multi-actor-attention-critic deep reinforcement learning

As a promising approach to managing distributed energy, the use of microgrids has attracted significant attention among those managing continuous connections to distribution networks. However, the barriers of the data sharing among different microgrids, the uncertainty of the distributed renewable sources and loads, and the nonlinear optimization of power flow make traditional model-based optimization methods difficult to be applied. In this paper, a data-driven coordinated active and reactive power optimization method is proposed for distribution networks with multi-microgrids. A multi-agent deep reinforcement learning (MADRL) method is used to protect the data privacy of each microgrids. Moreover, attention mechanism, which pays attention to crucial information, is presented to overcome the problem of slow convergence caused by the dimensionality explosion of the optimized variables. Two types of agents, controlling discrete action and continuous action devices, respectively, are formulated in coordinated optimization, which reduces voltage violations and improves the system operation efficiency. In addition, in order to improve the performance of the online agent model under variable operation conditions, the transfer learning is embedded in the training process of the MADRL. The proposed method is verified on a modified IEEE 33-bus distribution network with nine microgrids.

Joint Task and Computing Resource Allocation in Distributed Edge Computing Systems via Multi-Agent Deep Reinforcement Learning

Joint Task and Computing Resource Allocation in Distributed Edge Computing Systems via Multi-Agent Deep Reinforcement Learning

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain.

Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Multi-Agent deep reinforcement learning based task offloading and caching in computing power network

Abstract Computing power network (CPN) can alleviate the resource bottleneck in mobile edge computing (MEC). In this paper, we investigate the caching-assisted CPN system for task offloading. A combination of policies such as task offloading and caching is considered to reduce the energy consumption of the system under the condition of meeting a certain delay. Besides, we propose an advanced deep reinforcement learning (DRL) algorithm called Request Sensitive Multi Agent Soft Actor-Critic (RS-MASAC) algorithm to achieve minimal system energy consumption by obtaining the optimal policy. Simulation results show that our proposed algorithm has better performance than baseline algorithms.