Cooperative Multi-agent Reinforcement Learning Research Articles

Deep clustering of cooperative multi-agent reinforcement learning to optimize multi chiller HVAC systems for smart buildings energy management

Chillers are responsible for almost half of the total energy demand in buildings. Hence, the obligation of control systems of multi-chiller due to changes indoor environments is one of the most significant parts of a smart building. Such a controller is described as a nonlinear and multi-objective algorithm, and its fabrication is crucial to achieving the optimal balance between indoor thermal comfort and running a minimum number of chillers. This work proposes deep clustering of cooperative multi-agent reinforcement learning (DCCMARL) as well-suited to such system control, which supports centralized control by learning of agents. In MARL, since the learning of agents is based on discrete sets of actions and stats, this drawback significantly affects the model of agents for representing their actions with efficient performance. This drawback becomes considerably worse when increasing the number of agents, due to the increased complexity of solving MARL, which makes modeling policy very challenging. Therefore, the DCCMARL of multi-objective reinforcement learning is leveraging powerful frameworks of a hybrid clustering algorithm to deal with complexity and uncertainty, which is a critical factor that influences to the achievement of high levels of a performance action. The results showed that the ability of agents to manipulate the behavior of the smart building could improve indoor thermal conditions, as well as save energy up to 44.5% compared to conventional methods. It seems reasonable to conclude that agents' performance is influenced by what type of model structure.

Cooperative Learning for Smart Charging of Shared Autonomous Vehicle Fleets

We study the operational problem of shared autonomous electric vehicles that cooperate in providing on-demand mobility services while maximizing fleet profit and service quality. Therefore, we model the fleet operator and vehicles as interactive agents enriched with advanced decision-making aids. Our focus is on learning smart charging policies (when and where to charge vehicles) in anticipation of uncertain future demands to accommodate long charging times, restricted charging infrastructure, and time-varying electricity prices. We propose a distributed approach and formulate the problem as a semi-Markov decision process to capture its stochastic and dynamic nature. We use cooperative multiagent reinforcement learning with reshaped reward functions. The effectiveness and scalability of the proposed model are upgraded through deep learning. A mean-field approximation deals with environment instabilities, and hierarchical learning distinguishes high-level and low-level decisions. We evaluate our model using various numerical examples based on real data from ShareNow in Berlin, Germany. We show that the policies learned using our decentralized and dynamic approach outperform central static charging strategies. Finally, we conduct a sensitivity analysis for different fleet characteristics to demonstrate the proposed model’s robustness and provide managerial insights into the impacts of strategic decisions on fleet performance and derived charging policies. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.1187 .

Cross-Domain Service Function Chain Routing: Multiagent Reinforcement Learning Approaches

The service function chain (SFC) routing has been increasingly associated with multi-domain networks. However, several multi-domain SFC routing issues, linked to cross-domain interactions and emphasized by the heterogeneity of substrate topologies and administrative authorities, present a challenge. We propose the Cooperative Multi-agent Reinforcement Learning (CMRL) based algorithm for multi-domain SFC routing problem. Simulations on the real-world network topology demonstrate that our proposal can provide more than two and four times reduction in terms of the run time and network resource cost, respectively.

Neighborhood Cooperative Multiagent Reinforcement Learning for Adaptive Traffic Signal Control in Epidemic Regions

Nowadays, multiagent reinforcement learning (MARL) have shared significant advances in the adaptive traffic signal control (ATSC) problems. For most of the researches, agents are all isomorphic, which disregards the situation in which isomerous intersections cooperative together in a real ATSC scenario, especially in epidemic regions where different intersections have quite different levels of importance. To this end, this paper models the ATSC problem as a networked Markov game (NMG), in which agents take into account information, including traffic conditions of it and its connected neighbors. A cooperative MARL framework named neighborhood cooperative hysteretic DQN (NC-HDQN) is proposed. Specifically, for each NC-HDQN agent in the NMG, first, the framework analyses correlation degrees with their connected neighbors and weighs observations and rewards by these correlations. Second, NC-HDQN agents independently optimize their strategies on the weighted information using hysteretic DQN (HDQN), which is designed to learn optimal joint strategies in cooperative multiagent games. Third, a rule-based NC-HDQN method and a Pearson correlation coefficient based NC-HDQN method, i.e., empirical NC-HDQN (ENC-HDQN) and Pearson NC-HDQN (PNC-HDQN), respectively, are designed. The first method maps the correlation degree between two connected agents according to vehicle numbers on roads between the two agents. In contrast, the second method uses the Pearson correlation coefficient to calculate the correlation degree adaptively. Our methods are empirically evaluated in both a synthetic scenario and two real-world traffic scenarios and give better performances in almost every standard test metric for ATSC.

Graph attention mechanism based reinforcement learning for multi-agent flocking control in communication-restricted environment

To solve the poor performance of reinforcement learning (RL) in the multi-agent flocking cooperative control under the communication-restricted environments, we propose a multi-agent cooperative RL (MACRL) method based on the equivalent characteristics of the agents in the flocking task. A distance graph attention (GAT) mechanism is introduced into the policy network of the proposed MACRL to change the agent’s attention weights related to neighbors and reduce the influence of remote neighbors which have poor communication quality on the agent’s behavioral decision-making. Furthermore, a distance GAT-based MACRL (DGAT-MACRL) algorithm is proposed for multi-agent flocking control in the communication-restricted environment. The simulation results show that the proposed flocking algorithm has good adaptability to the communication delay and the communication distance constraint environments. The flocking control effect is significantly better than other RL-based flocking algorithms and traditional flocking algorithms. In addition, the good experimental performance has also confirmed that the proposed DGAT-MACRL is an effective solution to the problem of how to improve the adaptability of traditional RL to the flocking control system with a dynamic scale. Our algorithm provides a novel and practical algorithm offering an effective method for the cooperative task accomplished by multi-agent in non-ideal environments.

One4All: Manipulate one agent to poison the cooperative multi-agent reinforcement learning

Reinforcement Learning (RL) has achieved a plenty of breakthroughs in the past decade. Notably, existing studies have shown that RL is suffered from poisoning attack, which results in failure or even catastrophe in decision processes. However, these studies almost focus on single-agent RL setting, the cooperative Multi-Agent Reinforcement Learning (c-MARL) setting is less explored, which is a generalization of the single-agent RL setting and has achieved great success in many areas. As a sub-field of RL setting, c-MARL also faces some security issues, e.g., poisoning attack.In this paper, we introduce two novel poisoning attack techniques, i.e., State Noise Poisoning Attack (SNPA) and Target Action Poisoning Attack (TAPA), to attack the c-MARL setting stealthily and efficiently, which achieves the goal that poisoning the c-MARL setting while only manipulate one agent. The first attack technique, termed SNPA, a black-box attack method, modifies the state observation data of one agent, which results in the c-MARL setting performance degradation. The second attack technique, termed TAPA, a white-box attack method, injects a backdoor and triggers the target action by manipulating the action and reward function of one agent. The extensive experiments are conducted in two popular c-MARL games, i.e., MCSPT and MCHT. The experiment results show that the presented novel poisoning attacks, SNPA and TAPA, are effective on c-MARL scenarios. Specifically, the total reward is reduced by 1/3 and the winning rate of team drops down to 0 under the two proposed attacks. Furthermore, the TAPA experiments verify that the victim agent executes the target action once the backdoor is triggered. It is worth noting that the trigger rate of target action rises up to 99.31% and 99.01% in two c-MARL games.

A review of cooperative multi-agent deep reinforcement learning

Deep Reinforcement Learning has made significant progress in multi-agent systems in recent years. The aim of this review article is to provide an overview of recent approaches on Multi-Agent Reinforcement Learning (MARL) algorithms. Our classification of MARL approaches includes five categories for modeling and solving cooperative multi-agent reinforcement learning problems: (I) independent learners, (II) fully observable critics, (III) value function factorization, (IV) consensus, and (IV) learn to communicate. We first discuss each of these methods, their potential challenges, and how these challenges were mitigated in the relevant papers. Additionally, we make connections among different papers in each category if applicable. Next, we cover some new emerging research areas in MARL along with the relevant recent papers. In light of MARL’s recent success in real-world applications, we have dedicated a section to reviewing these applications and articles. This survey also provides a list of available environments for MARL research. Finally, the paper is concluded with proposals on possible research directions.

Transformer-Based Value Function Decomposition for Cooperative Multi-Agent Reinforcement Learning in StarCraft

The StarCraft II Multi-Agent Challenge (SMAC) was created to be a challenging benchmark problem for cooperative multi-agent reinforcement learning (MARL). SMAC focuses exclusively on the problem of StarCraft micromanagement and assumes that each unit is controlled individually by a learning agent that acts independently and only possesses local information; centralized training is assumed to occur with decentralized execution (CTDE). To perform well in SMAC, MARL algorithms must handle the dual problems of multi-agent credit assignment and joint action evaluation. This paper introduces a new architecture TransMix, a transformer-based joint action-value mixing network which we show to be efficient and scalable as compared to the other state-of-the-art cooperative MARL solutions. TransMix leverages the ability of transformers to learn a richer mixing function for combining the agents' individual value functions. It achieves comparable performance to previous work on easy SMAC scenarios and outperforms other techniques on hard scenarios, as well as scenarios that are corrupted with Gaussian noise to simulate fog of war.

Cooperative Multi-Agent Reinforcement-Learning-Based Distributed Dynamic Spectrum Access in Cognitive Radio Networks

With the development of wireless communication and Internet of Things (IoT), there are massive wireless devices that need to share the limited spectrum resources. Dynamic spectrum access (DSA) is a promising paradigm to remedy the problem of inefficient spectrum utilization brought upon by the historical command-and-control approach to spectrum allocation. In this article, we investigate the distributed DSA problem for multiusers in a typical multichannel cognitive radio network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP), and we propose a centralized off-line training and distributed online execution framework based on cooperative multi-agent reinforcement learning (MARL). We employ the deep recurrent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DRQN) to address the partial observability of the state for each cognitive user. The ultimate goal is to learn a cooperative strategy which maximizes the sum throughput of a cognitive radio network in a distributed fashion without information exchange between cognitive users. Finally, we validate the proposed algorithm in various settings through extensive experiments. The experimental results show that the proposed CoMARL-DSA algorithm outperforms the state-of-the-art deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning for spectrum access (DQSA) in terms of successful access rate and collision rate by at least 14% and 12%, respectively.

Learning to Coordinate for a Worker-Station Multi-Robot System in Planar Coverage Tasks

For massive large-scale tasks, a multi-robot system (MRS) can effectively improve efficiency by utilizing each robot's different capabilities, mobility, and functionality. In this letter, we focus on the multi-robot coverage path planning (mCPP) problem in large-scale planar areas with random dynamic interferers in the environment, where the robots have limited resources. We introduce a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS consisting of multiple <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">workers</i> with limited resources for actual work, and one <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">station</i> with enough resources for resource replenishment. We aim to solve the mCPP problem for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS by formulating it as a fully cooperative multi-agent reinforcement learning problem. Then we propose an end-to-end decentralized online planning method, which simultaneously solves coverage planning for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">workers</i> and rendezvous planning for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">station</i> . Our method manages to reduce the influence of random dynamic interferers on planning, while the robots can avoid collisions with them. We conduct simulation and real robot experiments, and the comparison results show that our method has competitive performance in solving the mCPP problem for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">worker-station</i> MRS in metric of task finish time.

Edge-Assisted Spectrum Sharing for Freshness-Aware Industrial Wireless Networks: A Learning-Based Approach

Information freshness is essential to industrial wireless networks (IWNs) and can be quantified by the age-of-information (AoI) metric. This paper addresses an AoI-aware spectrum sharing (AgeS) problem in IWNs, where multiple device-to-device (D2D) links opportunistically access the spectrum to satisfy their AoI constraints while maximizing primal links’ throughput. Particularly, we orchestrate the access of D2D links in a distributed manner. Since distributed scheduling results in incomplete observation, D2D links share the spectrum with uncertainty on the transmission environment. Therefore, we propose a distributed scheduling scheme, called D-age, to deal with the transmission uncertainty in the AgeS problem, where an adaptation of actor-critic method is adopted with AoI constraints tackled in the dual domain. To address the non-stationary environment and multi-agent credit assignment issue, cooperative multi-agent reinforcement learning (MARL) approach is developed, where multiple local actors are designed to guide D2D links to make real-time decisions via distributed scheduling policies, which are evaluated by an edge-assisted global critic with action-aware advantage functions. Integrated with graph attention networks (GATs), the critic selectively learns contextual information by assigning different importances to neighboring links, which enables the evaluation of scheduling policies in a scalable and computation-efficient manner. Theoretical guarantee of the time-averaged AoI constraints is provided and the effectiveness of D-age in terms of both AoI violation ratio and the capacity of primal links is demonstrated by simulation.

Multi-Level Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has become more and more popular over recent decades, and the need for high-level cooperation is increasing every day because of the complexity of the real-world environment. However, the multi-agent credit assignment problem that serves as the main obstacle to high-level coordination is still not addressed properly. Though lots of methods have been proposed, none of them have thought to perform credit assignments across multi-levels. In this paper, we aim to propose an approach to learning a better credit assignment scheme by credit assignment across multi-levels. First, we propose a hierarchical model that consists of the manager level and the worker level. The manager level incorporates the dilated Gated Recurrent Unit (GRU) to focus on high-level plans and the worker level uses GRU to execute primitive actions conditioned on high-level plans. Then, one centralized critic is designed for each level to learn each level’s credit assignment scheme. To this end, we construct a novel hierarchical MARL algorithm, named MLCA, which can achieve multi-level credit assignment. We also conduct experiments on three classical and challenging tasks to demonstrate the performance of the proposed algorithm against three baseline methods. The results show that our method gains great performance improvement across all maps that require high-level cooperation.

Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents

Multi-agent reinforcement learning is difficult to apply in practice, partially because of the gap between simulated and real-world scenarios. One reason for the gap is that simulated systems always assume that agents can work normally all the time, while in practice, one or more agents may unexpectedly “crash” during the coordination process due to inevitable hardware or software failures. Such crashes destroy the cooperation among agents and lead to performance degradation. In this work, we present a formal conceptualization of a cooperative multi-agent reinforcement learning system with unexpected crashes. To enhance the robustness of the system to crashes, we propose a coach-assisted multi-agent reinforcement learning framework that introduces a virtual coach agent to adjust the crash rate during training. We have designed three coaching strategies (fixed crash rate, curriculum learning, and adaptive crash rate) and a re-sampling strategy for our coach agent. To our knowledge, this work is the first to study unexpected crashes in a multi-agent system. Extensive experiments on grid-world and StarCraft II micromanagement tasks demonstrate the efficacy of the adaptive strategy compared with the fixed crash rate strategy and curriculum learning strategy. The ablation study further illustrates the effectiveness of our re-sampling strategy.

Qauxi: Cooperative multi-agent reinforcement learning with knowledge transferred from auxiliary task

Deep multi-agent reinforcement learning (MARL) can efficiently learn decentralized policies for real-world applications. However, current MARL methods suffer from the difficulty of transferring knowledge from already learned tasks to improve its exploration. In this paper, we propose a novel MARL method called Qauxi, which forms coordinated exploration scheme to improve the traditional MARL algorithms by reusing the meta-experience transferred from auxiliary task. We also use the weighting function to weight the importance of the joint action in monotonic loss function in order to focus on more important joint actions and thus avoid yielding suboptimal policies. Furthermore, we prove the convergence of Qauxi based on contraction mapping theorem. Qauxi is evaluated on the widely adopted StarCraft benchmarks (SMAC) across easy, hard, and super hard scenarios. Experimental results show that the proposed method outperforms the state-of-the-art MARL methods by a large margin in the most challenging super hard scenarios.

Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) faces significant scalability issues due to state and action spaces that are exponentially large in the number of agents. As environments grow in size, effective credit assignment becomes increasingly harder and often results in infeasible learning times. Still, in many real-world settings, there exist simplified underlying dynamics that can be leveraged for more scalable solutions. In this work, we exploit such locality structures effectively whilst maintaining global cooperation. We propose a novel, value-based multi-agent algorithm called LOMAQ, which incorporates local rewards in the Centralized Training Decentralized Execution paradigm. Additionally, we provide a direct reward decomposition method for finding these local rewards when only a global signal is provided. We test our method empirically, showing it scales well compared to other methods, significantly improving performance and convergence speed.

Multi-Agent Incentive Communication via Decentralized Teammate Modeling

Effective communication can improve coordination in cooperative multi-agent reinforcement learning (MARL). One popular communication scheme is exchanging agents' local observations or latent embeddings and using them to augment individual local policy input. Such a communication paradigm can reduce uncertainty for local decision-making and induce implicit coordination. However, it enlarges agents' local policy spaces and increases learning complexity, leading to poor coordination in complex settings. To handle this limitation, this paper proposes a novel framework named Multi-Agent Incentive Communication (MAIC) that allows each agent to learn to generate incentive messages and bias other agents' value functions directly, resulting in effective explicit coordination. Our method firstly learns targeted teammate models, with which each agent can anticipate the teammate's action selection and generate tailored messages to specific agents. We further introduce a novel regularization to leverage interaction sparsity and improve communication efficiency. MAIC is agnostic to specific MARL algorithms and can be flexibly integrated with different value function factorization methods. Empirical results demonstrate that our method significantly outperforms baselines and achieves excellent performance on multiple cooperative MARL tasks.

Multi-Agent Reinforcement Learning with General Utilities via Decentralized Shadow Reward Actor-Critic

We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a general utility. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. We derive the Decentralized Shadow Reward Actor-Critic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the ``shadow reward". DSAC converges to ϵ-stationarity in O(1/ϵ^2.5) or faster O(1/ϵ^2) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy. Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.

Batch Active Learning with Graph Neural Networks via Multi-Agent Deep Reinforcement Learning

Graph neural networks (GNNs) have achieved tremendous success in many graph learning tasks such as node classification, graph classification and link prediction. For the classification task, GNNs' performance often highly depends on the number of labeled nodes and thus could be significantly hampered due to the expensive annotation cost. The sparse literature on active learning for GNNs has primarily focused on selecting only one sample each iteration, which becomes inefficient for large scale datasets. In this paper, we study the batch active learning setting for GNNs where the learning agent can acquire labels of multiple samples at each time. We formulate batch active learning as a cooperative multi-agent reinforcement learning problem and present a novel reinforced batch-mode active learning framework BiGeNe. To avoid the combinatorial explosion of the joint action space, we introduce a value decomposition method that factorizes the total Q-value into the average of individual Q-values. Moreover, we propose a novel multi-agent Q-network consisting of a graph convolutional network (GCN) component and a gated recurrent unit (GRU) component. The GCN component takes both the informativeness and inter-dependences between nodes into account and the GRU component enables the agent to consider interactions between selected nodes in the same batch. Experimental results on multiple public datasets demonstrate the effectiveness and efficiency of our proposed method.

MAPDP: Cooperative Multi-Agent Reinforcement Learning to Solve Pickup and Delivery Problems

Cooperative Pickup and Delivery Problem (PDP), as a variant of the typical Vehicle Routing Problems (VRP), is an important formulation in many real-world applications, such as on-demand delivery, industrial warehousing, etc. It is of great importance to efficiently provide high-quality solutions of cooperative PDP. However, it is not trivial to provide effective solutions directly due to two major challenges: 1) the structural dependency between pickup and delivery pairs require explicit modeling and representation. 2) the cooperation between different vehicles is highly related to the solution exploration and difficult to model. In this paper, we propose a novel multi-agent reinforcement learning based framework to solve the cooperative PDP (MAPDP). First, we design a paired context embedding to well measure the dependency of different nodes considering their structural limits. Second, we utilize cooperative multi-agent decoders to leverage the decision dependence among different vehicle agents based on a special communication embedding. Third, we design a novel cooperative A2C algorithm to train the integrated model. We conduct extensive experiments on a randomly generated dataset and a real-world dataset. Experiments result shown that the proposed MAPDP outperform all other baselines by at least 1.64\% in all settings, and shows significant computation speed during solution inference.

Multi-agent cooperative structural vibration control of three coupled flexible beams based on value decomposition network

Multi-coupled flexible body (MCFB) structure has the characteristics of small damping, close modes and coupled vibration, and the structural vibration is difficult to freely attenuate, which will lead to rapid fatigue failure of the structure. In order to suppress the vibration of the MCFB system quickly, a system identification scheme and a multi-agent reinforcement learning (MARL) scheme are proposed to obtain the cooperative controller. The mathematical model of the three-coupled flexible beam (TCFB) system is identified by wavelet analysis method, particle swarm optimization (PSO) and sinusoidal excitation response method. Based on value decomposition network (VDN) and deterministic policy gradient (DPG), a cooperative MARL framework is constructed to obtain the controller (VDN-DPG controller). The cooperative VDN-DPG controller is applied to the vibration control simulation and experiments of the TCFB system, and compared with proportional–derivative (PD) control without cooperation. The results show that the performance of the VDN-DPG controller is better than that of the PD controller, which verifies the effectiveness of the multi-agent cooperative control scheme based on VDN-DPG.