Multi-agent Reinforcement Learning Problem Research Articles

Communication-Efficient and Resilient Distributed Q-Learning.

This article investigates the problem of communication-efficient and resilient multiagent reinforcement learning (MARL). Specifically, we consider a setting where a set of agents are interconnected over a given network, and can only exchange information with their neighbors. Each agent observes a common Markov Decision Process and has a local cost which is a function of the current system state and the applied control action. The goal of MARL is for all agents to learn a policy that optimizes the infinite horizon discounted average of all their costs. Within this general setting, we consider two extensions to existing MARL algorithms. First, we provide an event-triggered learning rule where agents only exchange information with their neighbors if a certain triggering condition is satisfied. We show that this enables learning while reducing the amount of communication. Next, we consider the scenario where some of the agents can be adversarial (as captured by the Byzantine attack model), and arbitrarily deviate from the prescribed learning algorithm. We establish a fundamental trade-off between optimality and resilience when Byzantine agents are present. We then create a resilient algorithm and show almost sure convergence of all reliable agents' value functions to the neighborhood of the optimal value function of all reliable agents, under certain conditions on the network topology. When the optimal Q -values are sufficiently separated for different actions, we show that all reliable agents can learn the optimal policy under our algorithm.

Local Patch AutoAugment With Multi-Agent Collaboration

Data augmentation (DA) plays a critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA methods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose a more fine-grained automated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content together with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification, fine-grained image recognition and object detection ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</i> ., CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars, FGVC-Aircraft and Pascal VOC 2007). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources. Automatic Data Augmentation. Multi-Agent Reinforcement Learning.

Fully Decentralized Multiagent Communication via Causal Inference.

Many real-world tasks can be cast into multiagent (MA) reinforcement learning problems, and most algorithms in this field obey to the centralized learning and decentralized execution framework. However, enforcing centralized learning is impractical in many scenarios. Because it requires integrating the information from agents, while agents may not hope to share local information due to the issue of privacy. Thus, this article proposes a novel approach to achieve fully decentralized learning based on communication among multiple agents via reinforcement learning. Benefiting from causality analysis, an agent will choose the counterfactual that has the most significant influence on communication information of others. We find that this method can be applied in classic or complex MA scenarios and in federated learning domains, which are now attracting much attention.

MolOpt: Autonomous Molecular Geometry Optimization Using Multiagent Reinforcement Learning.

Most optimization problems require the user to select an algorithm and, to some extent, also tune it for better performance. Although intuition and knowledge about the problem can speed up these selection and fine-tuning processes, users often use trial-and-error methodologies, which can be time-consuming and inefficient. With all of that in mind and much more, the concept of "learned optimizers", "learning to learn", and "meta-learning" has been gathering attention in recent years. In this article, we propose MolOpt that uses multiagent reinforcement learning (MARL) for autonomous molecular geometry optimization (MGO). Typically MGO algorithms are hand-designed, but MolOpt uses MARL to learn a learned optimizer (policy) that can perform MGO without the need for other hand-designed optimizers. We cast MGO as a MARL problem, where each agent corresponds to a single atom in the molecule. MolOpt performs MGO by minimizing the forces on each atom of the molecule. Our experiments demonstrate the generalizing ability of MolOpt for the MGO of propane, pentane, heptane, hexane, and octane when trained on ethane, butane, and isobutane. In terms of performance, MolOpt outperforms the MDMin optimizer and demonstrates performance similar to that of the FIRE optimizer. However, it does not surpass the BFGS optimizer. The results demonstrate that MolOpt has the potential to introduce innovative advancements in MGO by providing a novel approach using reinforcement learning (RL), which may open up new research directions for MGO. Overall, this work serves as a proof-of-concept for the potential of MARL in MGO.

Deep Multi-Agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

Secrecy Rate Maximization in THz-Aided Heterogeneous Networks: A Deep Reinforcement Learning Approach

Densely deployed femtocells in heterogeneous networks (HetNets) improve the quality-of-service (QoS) and quality-of-experience (QoE) for both next-generation networks and end users, respectively. However, users near the femtocell edge, i.e., femto edge users (FEUs), which are far away from the Femto base station (FBS) may receive a low signal-to-interference-plus-noise ratio (SINR). Moreover, due to the distributed and dynamic nature of HetNets, femtocells may not resist various eavesdropping attacks from different attackers. Thus, it is essential to ensure the wireless channel's secrecy through which FEUs communicate with FBS since they may face eavesdropping attacks and may have low SINR and achievable data rates. Aiming to enhance the secrecy rate of femtocells, in this article, we formulate a joint optimization problem of beamforming vectors, artificial noise (AN), and power control for terahertz (THz)-enabled femtocells. Addressing this non-convex optimization problem is challenging due to the system's complexity and dynamic nature of FEUs. Thus, we translate the optimization problem into a multi-agent reinforcement learning (MARL) problem using the Markov decision process (MDP). Then, to solve the MDP with continuous action space, two deep reinforcement learning (DRL) based schemes, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Deterministic Policy Gradient-based Secrecy Maximization (DDPG-SM)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Asynchronous-Advantage-Actor-Critic based Secrecy Maximization (A3C-SM)</i> are proposed to maximize the secrecy rate of femtocells. The secrecy rate performance of the proposed schemes is compared and analyzed with Advantage-Actor-Critic (A2C) and state-of-the-art Perfect Secrecy Rate Maximization (PSRM) schemes. Simulation results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> schemes achieve 37.73% and 22.64% better average secrecy rate in comparison to the PSRM scheme. Also, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> scheme outperforms all other <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> , A2C, and PSRM schemes in terms of average secrecy rate, received SINR, and energy-efficiency of femtocells.

Cooperative bus holding and stop-skipping: A deep reinforcement learning framework

The bus control problem that combines holding and stop-skipping strategies is formulated as a multi-agent reinforcement learning (MARL) problem. Traditional MARL methods, designed for settings with joint action-taking, are incompatible with the asynchronous nature of at-stop control tasks. On the other hand, using a fully decentralized approach leads to environment non-stationarity, since the state transition of an individual agent may be distorted by the actions of other agents. To address it, we propose a design of the state and reward function that increases the observability of the impact of agents’ actions during training. An event-based mesoscopic simulation model is built to train the agents. We evaluate the proposed approach in a case study with a complex route from the Chicago transit network. The proposed method is compared to a standard headway-based control and a policy trained with MARL but with no cooperative learning. The results show that the proposed method not only improves level of service but it is also more robust towards uncertainties in operations such as travel times and operator compliance with the recommended action.

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks

Exploration under sparse rewards is a key challenge for multi-agent reinforcement learning problems. One possible solution to this issue is to exploit inherent task structures for an acceleration of exploration. In this paper, we present a novel exploration approach, which encodes a special structural prior on the reward function into exploration, for sparse-reward multi-agent tasks. Specifically, a novel entropic exploration objective which encodes the structural prior is proposed to accelerate the discovery of rewards. By maximizing the lower bound of this objective, we then propose an algorithm with moderate computational cost, which can be applied to practical tasks. Under the sparse-reward setting, we show that the proposed algorithm significantly outperforms the state-of-the-art algorithms in the multiple-particle environment, the Google Research Football and StarCraft II micromanagement tasks. To the best of our knowledge, on some hard tasks (such as 27m_vs_30m}) which have relatively larger number of agents and need non-trivial strategies to defeat enemies, our method is the first to learn winning strategies under the sparse-reward setting.

Fleet Rebalancing for Expanding Shared e-Mobility Systems: A Multi-Agent Deep Reinforcement Learning Approach

The electrification of shared mobility has become popular across the globe. Many cities have their new shared e-mobility systems deployed, with continuously expanding coverage from central areas to the city edges. A key challenge in the operation of these systems is fleet rebalancing, i.e., how EVs should be repositioned to better satisfy future demand. This is particularly challenging in the context of expanding systems, because i) the range of the EVs is limited while charging time is typically long, which constrain the viable rebalancing operations; and ii) the EV stations in the system are dynamically changing, i.e., the legitimate targets for rebalancing operations can vary over time. We tackle these challenges by first investigating rich sets of data collected from a real-world shared e-mobility system for one year, analyzing the operation model, usage patterns and expansion dynamics of this new mobility mode. With the learned knowledge we design a high-fidelity simulator, which is able to abstract key operation details of EV sharing at fine granularity. Then we model the rebalancing task for shared e-mobility systems under continuous expansion as a Multi-Agent Reinforcement Learning (MARL) problem, which directly takes the range and charging properties of the EVs into account. We further propose a novel policy optimization approach with action cascading, which is able to cope with the expansion dynamics and solve the formulated MARL. We evaluate the proposed approach extensively, and experimental results show that our approach outperforms the state-of-the-art, offering significant performance gain in both satisfied demand and net revenue.

Crowd-Sensing Enhanced Parking Patrol Using Sharing Bikes’ Trajectories

Illegal vehicle parking is a common urban problem faced by major cities in the world, as it incurs traffic jams, which lead to air pollution and traffic accidents. The government highly relies on active human efforts to detect illegal parking events. However, such an approach is extremely ineffective to cover a large city since the police have to patrol over the entire city roads. The massive and high-quality sharing bike trajectories from Mobike offer us a unique opportunity to design a ubiquitous illegal parking detection approach, as most of the illegal parking events happen at curbsides and have significant impact on the bike users. The detection result can guide the patrol schedule, i.e., send the patrol policemen to the region with higher illegal parking risks, and further improve the patrol efficiency. Inspired by this idea, three main components are employed in the proposed framework: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">trajectory pre-processing</i> , which filters outlier GPS points, performs map-matching, and builds trajectory indexes; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">illegal parking detection</i> , which models the normal trajectories, extracts features from the evaluation trajectories, and utilizes a distribution test-based method to discover the illegal parking events; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">patrol scheduling</i> , which leverages the detection result as reference context, and models the scheduling task as a multi-agent reinforcement learning problem to guide the patrol police. Finally, extensive experiments are presented to validate the effectiveness of illegal parking detection, as well as the improvement of patrol efficiency.

A Delay-Optimal Task Scheduling Strategy for Vehicle Edge Computing Based on the Multi-Agent Deep Reinforcement Learning Approach

Cloudlet-based vehicular networks are a promising paradigm to enhance computation services through a distributed computation method, where the vehicle edge computing (VEC) cloudlet are deployed in the vicinity of the vehicle. In order to further improve the computing efficiency and reduce the task processing delay, we present a parallel task scheduling strategy based on the multi-agent deep reinforcement learning (DRL) approach for delay-optimal VEC in vehicular networks, where multiple computation tasks select the target threads in a VEC server to execute the computing tasks. We model the target thread decision of computation tasks as a multi-agent reinforcement learning problem, which is further solved by using a task scheduling algorithm based on multi-agent DRL that is implemented in a distributed manner. The computation tasks, with each selection acting on the target thread acting as an agent, collectively interact with the VEC environment and receive observations with respect to a common reward and learn to reduce the task processing delay by updating the multi-agent deep Q network (MADQN) using the obtained experiences. The experimental results show that the proposed DRL-based scheduling algorithm can achieve significant performance improvement, reducing the task processing delay by 40% and increasing the processing probability of success for computation tasks by more than 30% compared with the traditional task scheduling algorithms.

Vehicular intelligent collaborative intersection driving decision algorithm in Internet of Vehicles

Connected and Automated Vehicles (CAV) are regarded as the developing trend of future transportation due to the advantages in terms of increasing traffic throughput, safety, and reducing energy consumption. One of the challenging CAV research problems is intelligent collaborative driving decision in dynamically changing traffic scenarios. The sixth-generation mobile communication technology (6G) plays an important role in attaining CAV intelligent driving in Internet of Vehicles (IoV) by provide significantly higher system performance, higher spectral efficiency, higher reliability, and higher security. This paper formulates the collaborative decision problem of CAVs at unsignalized intersections as a multi-agent reinforcement learning (MARL) problem, where CAVs entering an intersection safely cross the intersection and minimize their traffic time through collaborating to learn a strategy. An efficient and scalable MARL algorithm based on Proximal Policy Optimization (PPO) is applied to the dynamic intersection scenarios, where the parameter-sharing mechanism is used to improve the PPO algorithm to be a multi-agent version. Instead of the global reward, the local reward is used to promote cooperation between agents and achieve scalability. In addition, the action masking mechanism is adopted to improve the learning efficiency by filtering out invalid actions at each step. A joint simulation platform is built to verify the performance of the Parameter-Sharing PPO algorithm (PS-PPO). The simulation results show that the PS-PPO algorithm not only guarantees a low collision rate but also greatly reduces the vehicular travel time. This algorithm promotes cooperation among CAVs and performs well in the task of collaborative traffic flow at unsignalized intersections, especially in high-traffic volume scenarios. It is helpful to improve the safety and efficiency of traffic flow at unsignalized intersections.

Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning

We study a multi-agent reinforcement learning (MARL) problem where the agents interact over a given network. The goal of the agents is to cooperatively maximize the average of their entropy-regularized long-term rewards. To overcome the curse of dimensionality and to reduce communication, we propose a Localized Policy Iteration (LPI) algorithm that provably learns a near-globally-optimal policy using only local information. In particular, we show that, despite restricting each agent's attention to only its κ-hop neighborhood, the agents are able to learn a policy with an optimality gap that decays polynomially in κ. In addition, we show the finite-sample convergence of LPI to the global optimal policy, which explicitly captures the trade-off between optimality and computational complexity in choosing κ. Numerical simulations demonstrate the effectiveness of LPI.

Mitigating Jamming Attack in 5G Heterogeneous Networks: A Federated Deep Reinforcement Learning Approach

Jamming attack is one of the serious security breaches in the upcoming fifth-generation heterogeneous networks (5G HetNets). Most of the existing anti-jamming techniques, such as frequency hopping (FH) and direct sequence spread spectrum (DSSS) lack in self-adaptive capabilities to mitigate the security and privacy issues in highly dynamic 5G HetNet environment. In literature, although reinforcement learning (RL) has been explored a lot in designing various anti-jamming techniques to address the aforementioned problems, but these techniques suffer from the issue of large network resource consumption and slow convergence rate. To mitigate these issues, we propose a federated deep reinforcement learning (DRL) based anti-jamming technique for two-tier 5G HetNets. In the proposal, each femtocell of 5G HetNets is assumed to have multiple single antenna femto users (FUs) and a multi-antenna jammer used to jam the downlink signals from femto base station (FBS) to FUs. Aiming to improve the achievable rate at FUs in the presence of jammers, a joint optimization problem of beamforming and power allocation at FBSs is formulated by considering the quality-of-service (QoS) requirements of FUs. Due to the non-convex nature of the aforementioned optimization problem, we have used the Markov decision process (MDP) to transform the optimization problem into a multi-agent reinforcement learning (MARL) problem. Then, to solve this MDP with large number of states and action spaces, a federated deep reinforcement learning (DRL) scheme is proposed to maximize the achievable rate at FUs. The proposed scheme uses federated learning and dueling architecture of dueling double deep Q network (D3QN) to optimize the beamforming vectors and power allocation jointly at FBSs. The achievable rate performance of the proposed federated DRL scheme is compared with double deep Q network (DDQN) and deep Q network (DQN). Simulation results show that the proposed federated DRL scheme achieves 19.39% and 23.85% better achievable rate in comparison to DDQN and DQN schemes.

MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning.

Deep reinforcement learning (RL) has been applied extensively to solve complex decision-making problems. In many real-world scenarios, tasks often have several conflicting objectives and may require multiple agents to cooperate, which are the multi-objective multi-agent decision-making problems. However, only few works have been conducted on this intersection. Existing approaches are limited to separate fields and can only handle multi-agent decision-making with a single objective, or multi-objective decision-making with a single agent. In this paper, we propose MO-MIX to solve the multi-objective multi-agent reinforcement learning (MOMARL) problem. Our approach is based on the centralized training with decentralized execution (CTDE) framework. A weight vector representing preference over the objectives is fed into the decentralized agent network as a condition for local action-value function estimation, while a mixing network with parallel architecture is used to estimate the joint action-value function. In addition, an exploration guide approach is applied to improve the uniformity of the final non-dominated solutions. Experiments demonstrate that the proposed method can effectively solve the multi-objective multi-agent cooperative decision-making problem and generate an approximation of the Pareto set. Our approach not only significantly outperforms the baseline method in all four kinds of evaluation metrics, but also requires less computational cost.

A networked multi-agent reinforcement learning approach for cooperative FemtoCaching assisted wireless heterogeneous networks

To meet the explosive growth of mobile traffic requirement in the 5th generation (5 G) mobile system, FemtoCaching at the network edge has been regarded as a promising technique for the 5 G mobile system. In this paper, we focus on studying the cooperative FemtoCaching problem in wireless heterogeneous networks (HetNets), which is aimed to minimize the overall fetching delays of all users. Owing to the NP-hardness of the problem, we formulate the cooperative FemtoCaching problem as a Networked Multi-Agent Reinforcement Learning (NMARL) problem and accordingly propose a Soft Attentional Networked Multi-Agent Actor-Critic (SAN-AC) Reinforcement Learning algorithm. Simulation results demonstrate that the proposed algorithm can significantly increase the overall performance compared with existing work.

A review of cooperative multi-agent deep reinforcement learning

Deep Reinforcement Learning has made significant progress in multi-agent systems in recent years. The aim of this review article is to provide an overview of recent approaches on Multi-Agent Reinforcement Learning (MARL) algorithms. Our classification of MARL approaches includes five categories for modeling and solving cooperative multi-agent reinforcement learning problems: (I) independent learners, (II) fully observable critics, (III) value function factorization, (IV) consensus, and (IV) learn to communicate. We first discuss each of these methods, their potential challenges, and how these challenges were mitigated in the relevant papers. Additionally, we make connections among different papers in each category if applicable. Next, we cover some new emerging research areas in MARL along with the relevant recent papers. In light of MARL’s recent success in real-world applications, we have dedicated a section to reviewing these applications and articles. This survey also provides a list of available environments for MARL research. Finally, the paper is concluded with proposals on possible research directions.

Learning to Coordinate for a Worker-Station Multi-Robot System in Planar Coverage Tasks

For massive large-scale tasks, a multi-robot system (MRS) can effectively improve efficiency by utilizing each robot's different capabilities, mobility, and functionality. In this letter, we focus on the multi-robot coverage path planning (mCPP) problem in large-scale planar areas with random dynamic interferers in the environment, where the robots have limited resources. We introduce a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS consisting of multiple <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">workers</i> with limited resources for actual work, and one <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">station</i> with enough resources for resource replenishment. We aim to solve the mCPP problem for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS by formulating it as a fully cooperative multi-agent reinforcement learning problem. Then we propose an end-to-end decentralized online planning method, which simultaneously solves coverage planning for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">workers</i> and rendezvous planning for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">station</i> . Our method manages to reduce the influence of random dynamic interferers on planning, while the robots can avoid collisions with them. We conduct simulation and real robot experiments, and the comparison results show that our method has competitive performance in solving the mCPP problem for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS in metric of task finish time.

Multiagent Reinforcement Learning-Based Signal Planning for Resisting Congestion Attack in Green Transportation

Inefficient signal control will not only exaggerate traffic congestion, but also increase the fuel consumption and exhaust emissions. Thus, signal planning is highly important in green transportation. As the Connected vehicle (CV) technology has transformed today’s transportation systems by connecting vehicles and the transportation infrastructure through wireless communication, the CV-based signal control system has seen significant studies recently. Unfortunately, existing signal planning algorithms in use are developed for the signal-intersection, showing low traffic efficiency in the multi-intersection collaborative planning due to ignoring the traffic correlation among the neighboring intersections. In this work, we target the USDOT (U.S. Department of Transportation) sponsored CV-based traffic control system, and implement a multi-intersection traffic network. We model the multi-intersection collaborative signal planning problem as a multi-agent reinforcement learning problem, and present an actor-attention-critic algorithm to improve transportation efficiency and energy efficiency in green transportation, as well as resist congestion attack. Experiment results on the multi-intersection traffic network indicates that 1) compared to the baseline, our approach reduces the total delay by as high as 44.24%; 2) our method transports more vehicles passing the intersections meanwhile reduces the total CO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> emissions by 2.40%; 3) under the congestion attack, our approach shows robustness and reduces the total delay by as high as 64.33%.

Batch Active Learning with Graph Neural Networks via Multi-Agent Deep Reinforcement Learning

Graph neural networks (GNNs) have achieved tremendous success in many graph learning tasks such as node classification, graph classification and link prediction. For the classification task, GNNs' performance often highly depends on the number of labeled nodes and thus could be significantly hampered due to the expensive annotation cost. The sparse literature on active learning for GNNs has primarily focused on selecting only one sample each iteration, which becomes inefficient for large scale datasets. In this paper, we study the batch active learning setting for GNNs where the learning agent can acquire labels of multiple samples at each time. We formulate batch active learning as a cooperative multi-agent reinforcement learning problem and present a novel reinforced batch-mode active learning framework BiGeNe. To avoid the combinatorial explosion of the joint action space, we introduce a value decomposition method that factorizes the total Q-value into the average of individual Q-values. Moreover, we propose a novel multi-agent Q-network consisting of a graph convolutional network (GCN) component and a gated recurrent unit (GRU) component. The GCN component takes both the informativeness and inter-dependences between nodes into account and the GRU component enables the agent to consider interactions between selected nodes in the same batch. Experimental results on multiple public datasets demonstrate the effectiveness and efficiency of our proposed method.