Multi-agent Deep Reinforcement Learning Research Articles

Energy-Aware MPTCP Scheduling in Heterogeneous Wireless Networks Using Multi-Agent Deep Reinforcement Learning Techniques

This paper proposes an energy-efficient scheduling scheme for multi-path TCP (MPTCP) in heterogeneous wireless networks, aiming to minimize energy consumption while ensuring low latency and high throughput. Each MPTCP sub-flow is controlled by an agent that cooperates with other agents using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. This approach enables the agents to learn decentralized policies through centralized training and decentralized execution. The scheduling problem is modeled as a multi-agent decision-making task. The proposed energy-efficient scheduling scheme, referred to as EE-MADDPG, demonstrates significant energy savings while maintaining lower latency and higher throughput compared to other state-of-the-art scheduling techniques. By adopting a multi-agent deep reinforcement learning approach, the agents can learn efficient scheduling policies that optimize various performance metrics in heterogeneous wireless networks.

A Multi-Ship Collision Avoidance Algorithm Using Data-Driven Multi-Agent Deep Reinforcement Learning

Maritime Autonomous Surface Ships (MASS) are becoming of interest to the maritime sector and are also on the agenda of the International Maritime Organization (IMO). With the boom in global maritime traffic, the number of ships is increasing rapidly. The use of intelligent technology to achieve autonomous collision avoidance is a hot issue widely discussed in the industry. In the endeavor to solve this problem, multi-ship coordinated collision avoidance has become a crucial challenge. This paper proposes a multi-ship autonomous collision avoidance decision-making algorithm by a data-driven method and adopts the Multi-agent Deep Reinforcement Learning (MADRL) framework for its design. Firstly, the overall framework of this paper and its components follow the principle of “reality as primary and simulation as supplementary”, so a real data-driven AIS (Automatic Identification System) dominates the model construction. Secondly, the agent’s observation state is determined by quantifying the hazardous area. Then, based on a full understanding of the International Regulations for Preventing Collisions at Sea (COLREGs) and the preliminary data collection, this paper combines the statistical results of the real water traffic data to guide and design the algorithm framework and selects the representative influencing factors to be designed in the collision avoidance decision-making algorithm’s reward function. Next, we train the algorithmic model using both real data and simulation data. Meanwhile, Prioritized Experience Replay (PER) is adopted to accelerate the model’s learning efficiency. Finally, 40 encounter scenarios are designed and extended to verify the algorithm performance based on the idea of the Imazu problem. The experimental results show that this algorithm can efficiently make a ship collision avoidance decision in compliance with COLREGs. Multi-agent learning through shared network policies can ensure that the agents pass beyond the safe distance in unknown environments. We can apply the trained model to the system with different numbers of agents to provide a reference for the research of autonomous collision avoidance in ships.

Adaptive Resource Scheduling in Permissionless Sharded-Blockchains: A Decentralized Multiagent Deep Reinforcement Learning Approach

Adaptive Resource Scheduling in Permissionless Sharded-Blockchains: A Decentralized Multiagent Deep Reinforcement Learning Approach

Deep Multi-Agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

Optimization of multi-echelon spare parts inventory systems using multi-agent deep reinforcement learning

Multi-echelon inventory systems are commonly used in practice to satisfy widely distributed random demands of spare parts in an efficient and cost-effective manner. Optimization of a multi-echelon inventory system is a decision-making problem under uncertainties. Classic inventory policies (e.g. (s, S) and (R, Q)) that do not consider the inventory positions of other warehouses become suboptimal due to interrelationships among different warehouses caused by transshipment. The Markov decision process (MDP) is an effective tool for inventory optimization, which does not require a predetermined parameterized policy structure. Unfortunately, both the state and action spaces of MDP suffer from the curse of dimensionality when the number of warehouses increases. This paper optimizes the inventory of a large-scale multi-echelon inventory system using a new multi-agent deep reinforcement learning (MADRL) algorithm named EM-VDTD3 that is developed by introducing value decomposition and experience buffer modification into the twin delayed deep deterministic policy gradient (TD3) algorithm. Each agent in EM-VDTD3 manages a subsystem in the multi-echelon inventory system. Because different agents share the same network parameters, networks are customized to process subsystems with different parameters. Domain knowledge of inventory control is embedded in the learning process of EM-VDTD3 by adding expert experiences to the experience buffer. An efficient approximate method is developed to identify a teacher policy that generates expert experiences. Numerical studies about a spare part inventory system in the wind energy industry show that the proposed EM-VDTD3 outperforms benchmark methods.

Single-Agent Reinforcement Learning for Scalable Earth-Observing Satellite Constellation Operations

This work explores single-agent reinforcement learning for the multi-satellite agile Earth-observing scheduling problem. The objective of the problem is to maximize the weighted sum of imaging targets collected and downlinked while avoiding resource constraint violations on board the spacecraft. To avoid the computational complexity associated with multi-agent deep reinforcement learning while creating a robust and scalable solution, a policy is trained in a single satellite environment. This policy is then deployed on board each satellite in a Walker-delta constellation. A global set of targets is distributed to each satellite based on target access. The satellites communicate with one another to determine whether an imaging target is imaged or downlinked. Free communication, line-of-sight communication, and no communication are explored to determine how the communication assumptions and constellation design impact performance. Free communication is shown to produce the best performance, and no communication is shown to produce the worst performance. Line-of-sight communication performance is shown to depend heavily on the design of the constellation and how frequently the satellites can communicate with one another. To explore how higher-level coordination can impact performance, a centralized mixed-integer programming optimization approach to global target distribution is explored and compared to a decentralized approach. A genetic algorithm is also implemented for comparison purposes, and the proposed method is shown to achieve higher reward on average at a fraction of the computational cost.

Consistent epistemic planning for multiagent deep reinforcement learning

Consistent epistemic planning for multiagent deep reinforcement learning

Multi-agent deep reinforcement learning based decision support model for resilient community post-hazard recovery

After a city-scale natural hazard, policymakers should plan sound decisions on the repair sequence to ensure the resilient recovery of the community, which consists of interdependent infrastructures. Stochastic scheduling for repairing interdependent infrastructure systems is a difficult control problem with huge decision spaces. This study proposes a novel decision support model to determine the optimal restoration policies for the purpose of maximizing disaster resilience. A simulation environment is first developed, consisting of hazard intensity assessment, components damage evaluation, system recovery simulation, and resilience quantification. The graph theory is utilized to represent the interdependencies among different systems, and the heterogeneous graph neural network is integrated into this framework to extract the topology and interdependency information of the whole community. The optimal repair policies approximated by neural networks are trained by a multi-agent deep reinforcement learning algorithm, considering uncertainties of the restoration process. The superiority and efficiency of the proposed method are demonstrated through a case study of the Tsinghua University campus, where different decision-making objectives are considered. The results show that the recovery trajectories determined by the proposed model have the highest performance compared to conventional methods. Besides, the proposed methodology based on transfer learning can achieve high computational efficiency for new damage scenarios. This model is promising to be a high-performance, robust decision-support tool for post-hazard repairing decisions.

Collective large-scale wind farm multivariate power output control based on hierarchical communication multi-agent proximal policy optimization

Wind power is becoming an increasingly vital source of renewable energy worldwide. However, controlling power generation in wind farms faces significant challenges due to the inherent complexity of these systems. To address this issue and maximize power output, we propose a novel communication-based multi-agent deep reinforcement learning approach for large-scale wind farm control. We introduce a multivariate power model for wind farms to analyze the impact of wake effects on power output. This model considers controllable variables such as axial induction factor, yaw angle, and tilt angle. To coordinate the continuous controls in large-scale wind farms, we propose the hierarchical communication multi-agent proximal policy optimization (HCMAPPO) algorithm. The wind farm is divided into multiple wind turbine aggregators (WTAs), and neighboring WTAs can exchange information through hierarchical communication to optimize power output. Our simulation results demonstrate that the multivariate HCMAPPO approach significantly increases wind farm power output compared to traditional PID control, coordinated model-based predictive control, and the multi-agent deep deterministic policy gradient algorithm. The HCMAPPO algorithm can be trained using a thirteen-turbine wind farm environment and effectively applied to large-scale wind farms. Importantly, as the wind farm scale increases, there is no significant increase in wind turbine blade fatigue damage from wake control. This multivariate HCMAPPO control strategy enables the collective maximization of power output in large-scale wind farms.

Finding individual strategies for storage units in electricity market models using deep reinforcement learning

Modeling energy storage units realistically is challenging as their decision-making is not governed by a marginal cost pricing strategy but relies on expected electricity prices. Existing electricity market models often use centralized rule-based bidding or global optimization approaches, which may not accurately capture the competitive behavior of market participants. To address this issue, we present a novel method using multi-agent deep reinforcement learning to model individual strategies in electricity market models. We demonstrate the practical applicability of our approach using a detailed model of the German wholesale electricity market with a complete fleet of pumped hydro energy storage units represented as learning agents. We compare the results to widely used modeling approaches and demonstrate that the proposed method performs well and can accurately represent the competitive behavior of market participants. To understand the benefits of using reinforcement learning, we analyze overall profits, aggregated dispatch, and individual behavior of energy storage units. The proposed method can improve the accuracy and realism of electricity market modeling and help policymakers make informed decisions for future market designs and policies.

Learning in Cooperative Multiagent Systems Using Cognitive and Machine Models

Developing effective multi-agent systems (MASs) is critical for many applications requiring collaboration and coordination with humans. Despite the rapid advance of multi-agent deep reinforcement learning (MADRL) in cooperative MASs, one of the major challenges that remain is the simultaneous learning and interaction of independent agents in dynamic environments in the presence of stochastic rewards. State-of-the-art MADRL models struggle to perform well in Coordinated Multi-agent Object Transportation Problems (CMOTPs) wherein agents must coordinate with each other and learn from stochastic rewards. In contrast, humans often learn rapidly to adapt to non-stationary environments that require coordination among people. In this article, motivated by the demonstrated ability of cognitive models based on Instance-based Learning Theory (IBLT) to capture human decisions in many dynamic decision-making tasks, we propose three variants of multi-agent IBL models (MAIBLs). The idea of these MAIBL algorithms is to combine the cognitive mechanisms of IBLT and the techniques of MADRL models to deal with coordination MASs in stochastic environments from the perspective of independent learners. We demonstrate that the MAIBL models exhibit faster learning and achieve better coordination in a dynamic CMOTP task with various settings of stochastic rewards compared to current MADRL models. We discuss the benefits of integrating cognitive insights into MADRL models.

Physics-constrained deep reinforcement learning for flow field denoising

A multi-agent deep reinforcement learning (DRL)-based model is presented in this study to reconstruct flow fields from noisy data. A combination of reinforcement learning with pixel-wise rewards, physical constraints represented by the momentum equation and the pressure Poisson equation, and the known boundary conditions is used to build a physics-constrained deep reinforcement learning (PCDRL) model that can be trained without the target training data. In the PCDRL model, each agent corresponds to a point in the flow field and learns an optimal strategy for choosing pre-defined actions. The proposed model is efficient considering the visualisation of the action map and the interpretation of the model operation. The performance of the model is tested by using direct numerical simulation-based synthetic noisy data and experimental data obtained by particle image velocimetry. Qualitative and quantitative results show that the model can reconstruct the flow fields and reproduce the statistics and the spectral content with commendable accuracy. Furthermore, the dominant coherent structures of the flow fields can be recovered by the flow fields obtained from the model when they are analysed using proper orthogonal decomposition and dynamic mode decomposition. This study demonstrates that the combination of DRL-based models and the known physics of the flow fields can potentially help solve complex flow reconstruction problems, which can result in a remarkable reduction in the experimental and computational costs.

Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach

Collaborative vehicle routing occurs when carriers collaborate through sharing their transportation requests and performing transportation requests on behalf of each other. This achieves economies of scale, thus reducing cost, greenhouse gas emissions and road congestion. But which carrier should partner with whom, and how much should each carrier be compensated? Traditional game theoretic solution concepts are expensive to calculate as the characteristic function scales exponentially with the number of agents. This would require solving the vehicle routing problem (NP-hard) an exponential number of times. We therefore propose to model this problem as a coalitional bargaining game solved using deep multi-agent reinforcement learning, where – crucially – agents are not given access to the characteristic function. Instead, we implicitly reason about the characteristic function; thus, when deployed in production, we only need to evaluate the expensive post-collaboration vehicle routing problem once. Our contribution is that we are the first to consider both the route allocation problem and gain sharing problem simultaneously — without access to the expensive characteristic function. Through decentralised machine learning, our agents bargain with each other and agree to outcomes that correlate well with the Shapley value — a fair profit allocation mechanism. Importantly, we are able to achieve a reduction in run-time of 88%.

A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning

Multi-agent deep reinforcement learning (MADRL) approaches are at the forefront of contemporary research in optimum electric vehicle (EV) charging scheduling challenges. These techniques involve multiple agents that respond to a dynamic simulation environment to strategically integrate EV charging stations (EVCSs) on microgrids by incorporating the constraints posed by stochastic trip durations. In addition, recent research works have demonstrated that planning frameworks based on multi-objective optimization (MOO) techniques are suitable for the efficient functioning of microgrids comprising renewable energy sources (RESs) and battery energy storage systems (BESSs). Even though MADRL techniques have been used to solve the optimum EV charging scheduling challenges and MOO frameworks have been developed to determine the optimal RES-BESS allocation, the potential of merging MADRL and MOO is yet to be explored. Therefore, this research provides an opportunity to determine the effectiveness of combined MOO-MADRL dynamics and their computational efficacy. In this context, this work presents a novel Multi-objective Artificial Vultures Optimization Algorithm based on Multi-agent Deep Deterministic Policy Gradient (MOAVOA-MADDPG) planning framework for allocating RESs, BESSs, and EVCSs on microgrids. The objective function is formulated to optimize the network power losses, total installation and operational costs, greenhouse gas emissions, and system voltage stability. Moreover, the proposed framework incorporates the sporadic nature of RES systems and intends to improve the state of charge (SOC) of the EVs present in the network. The presented approach is validated using practical weather data and EV commuting behavior on the modified IEEE 33 bus network, two practical distribution feeders in Bangladesh, and the Turkish 141 bus network. According to the findings, the MOAVOA-MADDPG framework effectively accommodated the financial, technical, and environmental considerations with improved average SOC of the vehicles. Furthermore, statistical analysis, spacing, convergence, and hyper-volume metrics are employed to compare the suggested MOAVOA-MADDPG framework with five contemporary techniques. The findings indicate that, in every metric considered, the MOAVOA-MADDPG Pareto fronts provide superior solutions.

Data-driven load frequency cooperative control for multi-area power system integrated with VSCs and EV aggregators under cyber-attacks

This paper proposes a cooperative load frequency control (LFC) strategy based on a multi-agent deep reinforcement learning (MADRL) framework for the multi-area power system in the presence of voltage source converters (VSCs) and electric vehicle (EV) aggregators under cyber-attacks. Different from the existing LFC model, a novel transfer function of VSCs is first improved by the space-vector technique and integrated with EV aggregators to develop a multi-area training environment. By installing the agent in different control areas and interacting state transition information between agents and the new environment, the MADRL-based control strategy is achieved for centralized training and decentralized execution. Thus, the proposed MADRL method can coordinate thermal turbines, VSCs, as well as EV aggregators in the different control areas. Furthermore, a suitable cyber-attack model that can circumvent bad data detection (BDD) is reconstructed according to the perspective of adversaries for the LFC system. Then the double critic networks and parameter updating policy are designed to eliminate and mitigate the fluctuations caused by cyber-attacks. The comparative simulation with other control strategies on a three-area test power system demonstrates the superior performance of the proposed MADRL-based approach.

Consortium Blockchain-Based Spectrum Trading for Network Slicing in 5G RAN: A Multi-Agent Deep Reinforcement Learning Approach

Network slicing (NS) is envisioned as an emerging paradigm for accommodating different virtual networks on a common physical infrastructure. Considering the integration of blockchain and NS, a secure decentralized spectrum trading platform can be established for autonomous radio access network (RAN) slicing. Moreover, the realization of proper incentive mechanisms for fair spectrum trading is crucial for effective RAN slicing. This paper proposes a novel hierarchical framework for blockchain-empowered spectrum trading for NS in RAN. Specifically, we deploy a consortium blockchain platform for spectrum trading among spectrum providers and buyers for slice creation, and autonomous slice adjustment. For slice creation, the spectrum providers are infrastructure providers (InPs) and buyers are mobile virtual network operators (MVNOs). Then, underloaded MVNOs with extra spectrum to spare, trade with overloaded MVNOs, for slice spectrum adjustment. For proper incentive maximization, we propose a three-stage Stackelberg game framework among InPs, seller MVNOs, and buyer MVNOs, for joint optimal pricing and demand prediction strategies. Then, a multi-agent deep reinforcement learning (MADRL) method is designed to achieve a Stackelberg equilibrium (SE). Security assessment and extensive simulation results confirm the security and efficacy of our proposed method in terms of players’ utility maximization and fairness, compared with other baselines.

Digital Twin-Driven Collaborative Scheduling for Heterogeneous Task and Edge-End Resource via Multi-Agent Deep Reinforcement Learning

Digital Twin-Driven Collaborative Scheduling for Heterogeneous Task and Edge-End Resource via Multi-Agent Deep Reinforcement Learning

Learning-Based Real-Time Transmission Control for Multi-Path TCP Networks

The evolution of the Internet of Things and 5G/6G networks is ushered in the exponential proliferation of mobile devices with multiple wireless interfaces. Multi-Path TCP (MPTCP) is a widely used protocol that provides reliable connections with multiple parallel paths for these devices. To cope with the increasingly complex multi-path network environments, some learning-based methods have been proposed to achieve better control than existing heuristic methods in congestion control and multi-path scheduling. However, most learning-based efforts use Deep Reinforcement Learning (DRL) requiring the agent and the environment to evolve alternatively, which cannot adapt to the constantly active MPTCP networks. In this paper, we propose Real-time Multi-path Transmission Control (RMTC), a cognitive framework that achieves real-time congestion control and multi-path scheduling in MPTCP networks, allowing the agent and the environment to evolve simultaneously. It adopts the Multi-Agent DRL (MADRL) to improve the granularity of the control and handle the change in the number of sub-flows. Besides, RMTC develops a pacing mechanism in MPTCP connections for all congestion control algorithms to alleviate network fluctuations, which helps the agent make accurate decisions. Extensive experimental results show that RMTC obtains better network performance than existing learning-based or heuristic methods and considers TCP-friendliness and the Head-of-Line (HoL) blocking problems.

Multi-Agent Deep Reinforcement Learning Based Incentive Mechanism for Multi-Task Federated Edge Learning

Multi-Agent Deep Reinforcement Learning Based Incentive Mechanism for Multi-Task Federated Edge Learning

Multi-Agent Deep Reinforcement Learning for Joint Decoupled User Association and Trajectory Design in Full-Duplex Multi-UAV Networks

Multi-Agent Deep Reinforcement Learning for Joint Decoupled User Association and Trajectory Design in Full-Duplex Multi-UAV Networks