Cooperative Reinforcement Learning Research Articles

Skill Matters: Dynamic skill learning for multi-agent cooperative reinforcement learning

Skill Matters: Dynamic skill learning for multi-agent cooperative reinforcement learning

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

Cooperative Reinforcement Learning for Military Drones over Large-Scale Battlefields

Cooperative Reinforcement Learning for Military Drones over Large-Scale Battlefields

LearnChain: Transparent and cooperative reinforcement learning on Blockchain

We consider multi-agent reinforcement learning (MARL) with the popular paradigm of centralized training and decentralized execution (CTDE). CTDE empowers sharing knowledge from agents in different environments for updating a shared model. A wide range of applications is supported through CTDE in MARL, such as self-driving vehicle coordination, traffic lights synchronization, or cooperation in various aspects of the Internet of Things (IoT), including resource management. Despite the drawbacks of relying on a central authority for handling model updates, incorporating multiple sources of data raises concerns about the trustworthiness of the process. For instance, participating agents could provide data in the favor of their experiences to shift the model towards certain behaviors. Similarly, sending falsified data for updates could lead to adversarial attacks. To overcome these challenges, it is essential to integrate the Ethereum Blockchain technology to handle model updates in the CTDE paradigm by achieving decentralized storage and consensus mechanism for model updates. In the literature, there exist multiple efforts that propose using reinforcement learning (RL) on Blockchain; however, none of them have considered updating MARL of CTDE on-chain, allowing transparent and auditable record of the training process. Therefore, we propose LearnChain, a framework that offers an integration between the CTDE mechanism and a Consortium Blockchain built between authorized participants, thus avoiding gas costs. At the core of LearnChain, RL is integrated with Quorum, offering separate smart contracts for deployment, data handling with incentive mechanisms, training, target update, and inference. Based on a real use-case entailing management of Vehicular Edge Computing tasks through multi-agent synchronization, we implement LearnChain and evaluate its performance and cost in different settings. Our results show the ability to improve learning from shared experiences and to adapt to environment changes on the Quorum BlockChain.

Distributed cooperative reinforcement learning for multi‐agent system with collision avoidance

AbstractIn this work, we present an optimal cooperative control scheme for a multi‐agent system in an unknown dynamic obstacle environment, based on an improved distributed cooperative reinforcement learning (RL) strategy with a three‐layer collaborative mechanism. The three collaborative layers are collaborative perception layer, collaborative control layer, and collaborative evaluation layer. The incorporation of collaborative perception expands the perception range of a single agent, and improves the early warning ability of the agents for the obstacles. Neural networks (NNs) are employed to approximate the cost function and the optimal controller of each agent, where the NN weight matrices are collaboratively optimized to achieve global optimal performance. The distinction of the proposed control strategy is that cooperation of the agents is embodied not only in the input of NNs (in a collaborative perception layer) but also in their weight updating procedure (in the collaborative evaluation and collaborative control layers). Comparative simulations are carried out to demonstrate the effectiveness and performance of the proposed RL‐based cooperative control scheme.

Task Computation Offloading for Multi-Access Edge Computing via Attention Communication Deep Reinforcement Learning

This paper investigates how to enhance the Multi-access Edge Computing (MEC) systems performance with the aid of device-to-device (D2D) communication computation offloading. By adequately exploiting a novel computation offloading mechanism based on D2D collaboration, users can efficiently share computational resources with each other. However, it is challenging to distinguish valuable information that truly promotes a collaborative decision, as worthless information can hinder collaboration among users. In addition, the transmission of large volumes of information requires high bandwidth and incurs significant latency and computational complexity, resulting in unacceptable costs. In this paper, we propose an efficient D2D-assisted MEC computation offloading framework based on Attention Communication Deep Reinforcement Learning (ACDRL), which simulates the interactions between related entities, including device-to-device collaboration in the horizontal and device-to-edge offloading in the vertical. Secondly, we developed a distributed cooperative reinforcement learning algorithm that includes an attention mechanism that skews computational resources towards active users to avoid unnecessary resource wastage in large-scale MEC systems. Finally, to improve the effectiveness and rationality of cooperation among users, we introduce a communication channel to integrate information from all users in a communication group, thus facilitating cooperative decision-making. The proposed framework is benchmarked, and the experimental results show that the proposed framework can effectively reduce latency and provide valuable insights for practical design compared to other baseline approaches.

A graph neural network based deep reinforcement learning algorithm for multi-agent leader-follower flocking

Flocking control is one of the important topics in multi-agent system (MAS), and has great value in both military and civilian applications. At present, the slow cluster consensus and poor control strategy in the stochastic environment are still serious problems faced by the flocking control algorithms. Focusing on the problems, we propose a leader-follower flocking algorithm based on a novel reinforcement learning (RL) model. We construct a homogeneous graph neural network (GNN) based multi-agent deep deterministic policy gradient (herein HGNN-MADDPG) algorithm model for multi-agent flocking control system. In HGNN-MADDPG, we design a GNN with loss weight to complete the feature extraction of the structural information and improve the perception ability of agents to the system structure. Furthermore, in order to improve training efficiency, we also propose a new cooperative reinforcement learning architecture in which the network and experience between agents are shared. Simulation results show that the HGNN-MADDPG has a better learning effect and faster convergence than the traditional RL algorithms. In addition, the comparative experiments demonstrate the leader-follower flocking algorithm based on HGNN-MADDPG has a faster cluster consensus and more stable control than the traditional and other RL-based leader-follower algorithms in the stochastic environment.

Fuzzy Q-learning-based approach for real-time energy management of home microgrids using cooperative multi-agent system

The development of plug-in electrical vehicles (PEVs) and distributed energy resources in home microgrids (H-MGs) is gaining attention in recent years. Accordingly, it is vital to apply a suitable energy management system to provide the required energy of the PEV and ensure the health of the energy storage systems in the H-MG. This paper proposes a continuous real-time cooperative multi-agent system (MAS) for H-MG. Real-time MAS interacts with H-MG agents to perform as independent learners applying a distributed and cooperative reinforcement learning method, while state variables are shared to coordinate their real-time behavior. Therefore, the fuzzy Q-learning (FQL) method is investigated to control the agents in the continuous state space and achieve an efficient solution. The experimental results highlight the effectiveness of the suggested control strategy to guarantee the energy supply and reduce the electricity market price. The results of the simulation demonstrate that using the proposed cooperative MAS with the real-time FQL method in the energy management of the H-MG will reduce the electricity market price by 10%, increase the health of the storage system by 35.67%, and enhance the utilization of the PV system to charge PEV for 8% at the peak load demand.

BRGR: Multi-agent cooperative reinforcement learning with bidirectional real-time gain representation

BRGR: Multi-agent cooperative reinforcement learning with bidirectional real-time gain representation

Learnchain: Transparent and Cooperative Reinforcement Learning on Blockchain

Learnchain: Transparent and Cooperative Reinforcement Learning on Blockchain

Federated reinforcement learning approach for detecting uncertain deceptive target using autonomous dual UAV system

This paper develops a cooperative federated reinforcement learning (RL) strategy that enables two unmanned aerial vehicles (UAVs) to cooperate in learning and predicting the movements of an intelligent deceptive target in a given search area. The proposed strategy allows the UAVs to autonomously cooperate, through information exchange of the gained experience to maximize the target detection performance and accelerate the learning speed while maintaining privacy. Specifically, we consider a monitoring model that includes a search area, a charging station, two cooperative UAVs, an intelligent deceptive uncertain moving target, and a fake (false) target. Each UAV is equipped with a limited-capacity rechargeable battery and a communication unit for exchanging the gained experience. The problem of maximizing the detection probability of the uncertain deceptive target using cooperative UAVs is mathematically modeled as a search-benefit maximization problem, which is then reformulated as a Markov decision process (MDP) due to the uncertainty nature of the problem. Because there is no prior information on the targets’ movement, a cooperative RL, is utilized to tackle the problem. The proposed cooperative RL-based algorithm is a distributed collaborative mechanism that enables the two UAVs, i.e., agents, to individually interact with the operating environment and maximize their cumulative rewards by converging to a shared policy while achieving privacy. Simulation results indicate that a cooperative RL-based dual UAV system can noticeably improve the target detection probability, reduce the detection performance, and accelerate the learning speed.

Structured Cooperative Reinforcement Learning With Time-Varying Composite Action Space.

In recent years, reinforcement learning has achieved excellent results in low-dimensional static action spaces such as games and simple robotics. However, the action space is usually composite, composed of multiple sub-action with different functions, and time-varying for practical tasks. The existing sub-actions might be temporarily invalid due to the external environment, while unseen sub-actions can be added to the current system. To solve the robustness and transferability problems in time-varying composite action spaces, we propose a structured cooperative reinforcement learning algorithm based on the centralized critic and decentralized actor framework, called SCORE. We model the single-agent problem with composite action space as a fully cooperative partially observable stochastic game and further employ a graph attention network to capture the dependencies between heterogeneous sub-actions. To promote tighter cooperation between the decomposed heterogeneous agents, SCORE introduces a hierarchical variational autoencoder, which maps the heterogeneous sub-action space into a common latent action space. We also incorporate an implicit credit assignment structure into the SCORE to overcome the multi-agent credit assignment problem in the fully cooperative partially observable stochastic game. Performance experiments on the proof-of-concept task and precision agriculture task show that SCORE has significant advantages in robustness and transferability for time-varying composite action space.

FusionSum: Abstractive summarization with sentence fusion and cooperative reinforcement learning

When summarizing an article, humans are habituated to fuse multiple related sentences to make the summary more concise and coherent. But most of the previous work focuses on the grammaticality of the fusion process and neglects the mechanism behind which sentences should be fused together. And there also lacks an effective training method for bridging the modules in the model to approach a global optimization. In this paper we propose FusionSum, a novel framework that imitates the behaviors of humans in summarization by explicitly modeling the sentence grouping and fusion process. It consists of an entity-aware sentence grouping module to identify salient sentences and combine them into groups, after which a unified sentence fusion module rewrites each group into a summary sentence. Furthermore, we also study the collaboration problem between these two modules and propose cooperative reinforcement learning method, which plays a maximax two-player game to approach global optimization. Automatic evaluation shows that our model significantly outperforms the strong baselines on three prevalent corpora. Human evaluation further demonstrates the summaries generated by our model are more concise and coherent.

Generating individual intrinsic reward for cooperative multiagent reinforcement learning

Multiagent reinforcement learning holds considerable promise to deal with cooperative multiagent tasks. Unfortunately, the only global reward shared by all agents in the cooperative tasks may lead to the lazy agent problem. To cope with such a problem, we propose a generating individual intrinsic reward algorithm, which introduces an intrinsic reward encoder to generate an individual intrinsic reward for each agent and utilizes the hypernetworks as the decoder to help to estimate the individual action values of the decomposition methods based on the generated individual intrinsic reward. Experimental results in the StarCraft II micromanagement benchmark prove that the proposed algorithm can increase learning efficiency and improve policy performance.

Analysing factorizations of action-value networks for cooperative multi-agent reinforcement learning

Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. However, given the lack of theoretical insight, it remains unclear what the employed neural networks are learning, or how we should enhance their learning power to address the problems on which they fail. In this work, we empirically investigate the learning power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results extend those in Castellini et al. (Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS’19.International Foundation for Autonomous Agents and Multiagent Systems, pp 1862–1864, 2019) and quantify how well various approaches can represent the requisite value functions, and help us identify the reasons that can impede good performance, like sparsity of the values or too tight coordination requirements.

Hierarchically and Cooperatively Learning Traffic Signal Control

Deep reinforcement learning (RL) has been applied to traffic signal control recently and demonstrated superior performance to conventional control methods. However, there are still several challenges we have to address before fully applying deep RL to traffic signal control. Firstly, the objective of traffic signal control is to optimize average travel time, which is a delayed reward in a long time horizon in the context of RL. However, existing work simplifies the optimization by using queue length, waiting time, delay, etc., as immediate reward and presumes these short-term targets are always aligned with the objective. Nevertheless, these targets may deviate from the objective in different road networks with various traffic patterns. Secondly, it remains unsolved how to cooperatively control traffic signals to directly optimize average travel time. To address these challenges, we propose a hierarchical and cooperative reinforcement learning method-HiLight. HiLight enables each agent to learn a high-level policy that optimizes the objective locally by selecting among the sub-policies that respectively optimize short-term targets. Moreover, the high-level policy additionally considers the objective in the neighborhood with adaptive weighting to encourage agents to cooperate on the objective in the road network. Empirically, we demonstrate that HiLight outperforms state-of-the-art RL methods for traffic signal control in real road networks with real traffic.

Maximizing network throughput by cooperative reinforcement learning in clustered solar-powered wireless sensor networks

Power management in wireless sensor networks is very important due to the limited energy of batteries. Sensor nodes with harvesters can extract energy from environmental sources as supplemental energy to break this limitation. In a clustered solar-powered sensor network where nodes in the network are grouped into clusters, data collected by cluster members are sent to their cluster head and finally transmitted to the base station. The goal of the whole network is to maintain an energy neutrality state and to maximize the effective data throughput of the network. This article proposes an adaptive power manager based on cooperative reinforcement learning methods for the solar-powered wireless sensor networks to keep harvested energy more balanced among the whole clustered network. The cooperative strategy of Q-learning and SARSA( λ) is applied in this multi-agent environment based on the node residual energy, the predicted harvested energy for the next time slot, and cluster head energy information. The node takes action accordingly to adjust its operating duty cycle. Experiments show that cooperative reinforcement learning methods can achieve the overall goal of maximizing network throughput and cooperative approaches outperform tuned static and non-cooperative approaches in clustered wireless sensor network applications. Experiments also show that the approach is effective in response to changes in the environment, changes in its parameters, and application-level quality of service requirements.

Wildfire Front Monitoring with Multiple UAVs using Deep Q-Learning

Wildfires destroy thousands of hectares every summer all over the globe. To provide an effective response and to mitigate wildfires impact, firefighters require a real-time monitoring of the fire front. This paper proposes a cooperative reinforcement learning (RL) framework that allows a team of autonomous unmanned aerial vehicles (UAVs) to learn how to monitor a fire front. In the literature, independent Q-learners were proposed to solve a wildfire monitoring task with two UAVs. Here we propose a framework that can be easily extended to a larger number of UAVs. Our framework builds on two methods: multiple single trained Q-learning agents (MSTA) and value decomposition networks (VDN). MSTA trains a single UAV controller, which is then "copied" to each of the UAVs in the team. In contrast, VDN trains agents to learn how to cooperate. We benchmarked in simulations our two considered methods – MSTA and VDN – against two state-of-the-art approaches: independent Q-learners and a joint Q-learner. Simulation results show that our considered methods outperform state-of-the-art approaches in a wildfire front monitoring task with up to 9 fixed-wing and multi-copter UAVs.

A Robust Channel Access Using Cooperative Reinforcement Learning for Congested Vehicular Networks

Vehicular Ad-hoc Network (VANET) is an emerging technique dedicated to wireless vehicular communication to improve transportation safety by exchanging driving information between vehicles. For safety purposes, vehicles periodically broadcast a safety packet via Vehicle-to-Vehicle (V2V) communication. Accordingly, VANET safety applications demand a reliable exchange of the safety packet with high Packet Delivery Ratio (PDR), acceptable latency, and communication fairness. However, the communication performance significantly degrades due to numerous packet collisions when a large number of vehicles simultaneously access limited channel resources for the safety broadcast. In particular, the problem grows more severe in congested VANETs absent infrastructures since vehicles must control channel access using a self-adaptive scheme without external assistance. Thus, a robust and decentralized channel access protocol for VANETs is required to achieve road safety. In this paper, we propose an intelligent channel access algorithm empowered by cooperative Reinforcement Learning (RL), in which vehicles coordinate the channel access in a fully-decentralized manner. We also consider a proper interaction scheme between vehicles for enhancing the V2V safety broadcast in infrastructure-less congested VANETs. We provide evaluation results with extensive simulations according to various levels of traffic congestion. Simulations confirm the superior performance of the algorithm: the algorithm has a 20% increase in PDR compared to the latest RL-based channel access scheme. Furthermore, the algorithm satisfies the low latency requirement of VANET safety applications as well as both short-term and long-term communication fairness.

Performance comparison of multiagent cooperative reinforcement learning algorithms for dynamic decision making in retail shop application

A novel approach by expertise based on multiagent cooperative reinforcement learning algorithms (EMCRLA) for dynamic decision making in the retail application is proposed in this paper. Performance comparison between cooperative reinforcement learning algorithms and expertise-based multiagent cooperative reinforcement learning algorithms is demonstrated. Different cooperation schemes for multi-agent cooperative reinforcement learning, i.e., EGroup scheme, EDynamic scheme is proposed here. This approach is developed for a three retailer stores in the retail marketplace. Retailers will be able to help each other and obtain profit from cooperation knowledge through learning their own strategies that exactly stand for their aims and benefit. Dynamic consumer performance is noticeably learned using the proposed algorithms. The paper illustrates results of cooperative reinforcement learning algorithms of three shop agents for the period of one year sale duration and then demonstrated the results using proposed approach for three shop agents for one year sale duration.