Multi-agent Reinforcement Learning Method Research Articles

Value function factorization with dynamic weighting for deep multi-agent reinforcement learning

In many real-world scenarios, multiple agents necessitate coordination with each other because of their limited observation and communication capability. Deep multi-agent reinforcement learning has demonstrated significant success in such challenging settings making use of value decomposition. One of the representative methods is QMIX, which factorizes the multi-agent global Q-value into individual Q-values and limits the joint action Q-value to a monotonic assumption leveraging an implicit mixing method. However, this assumption restricts it to representing certain value functions in which the ordering of an agent’s actions is based on the actions of others. WQMIX presents two weighting schemes to tackle this restriction but the weighting function is simple that limits the performance of methods, more appropriate weighting scheme is required to be considered. To tackle this issue, we present a more complex and accurate weighting scheme, which we call Dynamic Weighting (DW), as opposed to the fixed weighting in WQMIX. Our proposed method DW-QMIX guarantees a more general decomposition than QMIX or WQMIX and places accurate importance on the better joint actions thus leading to obtaining the optimal policy. Extensive experiments on the simulation environments and real-life systems demonstrate that our proposed method outperforms the existing multi-agent reinforcement learning methods.

Performance Evaluation of Multiagent Reinforcement Learning Based Training Methods for Swarm Fighting

In this paper, we conducted a performance evaluation of two multiagent reinforcement learning based training methods for swarm fighting, namely, the multiagent reinforcement learning (MARL) training method, and the combined multiagent reinforcement learning and behavior cloning (MARL-BC) training method. The behavior cloning expert is taken from some well-trained model in the final steady phase by the MARL training method. From the perspective of winning rate, the performances of these two different training methods can be divided into three phases. In the first phase, learning progresses slowly for both these two training methods. As the model trained by the MARL training method grows stronger, the experience of the behavior cloning expert gradually becomes useful, and the second phase kicks off where the MARL-BC training method takes obvious advantage. Surprisingly, the advantage of the MARL-BC training method will disappear as the learning progress goes on because in this final phase the expert of the behavior cloning training method can no longer offer the right strategy in presence of the ever changing environment and opponent.

Method of Multi-Agent Reinforcement Learning in Systems with a Variable Number of Agents

Multi-agent reinforcement learning methods are one of the newest and actively developing areas of machine learning. Among the methods of multi-agent reinforcement learning, one of the most promising is the MADDPG method, the advantage of which is the high convergence of the learning process. The disadvantage of the MADDPG method is the need to ensure the equality of the number of agents N at the training stage and the number of agents K at the functioning stage. At the same time, target multi-agent systems (MAS), such as groups of UAVs or mobile ground robots, are systems with a variable number of agents, which does not allow the use of the MADDPG method in them. To solve this problem, the article proposes an improved MADDPG method for multi-agent reinforcement learning in systems with a variable number of agents. The improved MADDPG method is based on the hypothesis that to perform its functions, an agent needs information about the state of not all other MAS agents, but only a few nearest neighbors. Based on this hypothesis, a method of hybrid joint / independent learning of MAS with a variable number of agents is proposed, which involves training a small number of agents N to ensure the functioning of an arbitrary number of agents K, K> N. The experiments have shown that the improved MADDPG method provides an efficiency of MAS functioning com-parable to the original method with varying the number of K agents at the stage of functioning within wide limits.

CRMRL: Collaborative Relationship Meta Reinforcement Learning for Effectively Adapting to Type Changes in Multi-Robotic System

Multi-agent reinforcement learning methods have been widely used for multi-robotic systems, and meta-learning methods are also applied to help robots reuse prior experiences to guide new tasks learning. But in some multi-robotic tasks, the robot types cannot be determined in advance or may dynamically change in practical application, bringing urgent challenges. When the robot types change, the prior experiences may be outdated and useless for new robots, even leading to negative transfer. It significantly limits the performance of most existing meta reinforcement learning methods. We get inspiration from the structure of bees, ants, and neurons. These colonies can overcome the interference caused by individual replacement, and always maintain dynamic stability. We find the relationship among individuals is more important than individuals themselves in these colonies, and propose a collaborative relationship meta reinforcement learning method (CRMRL). It concentrates on the relationship among robots, and reuses the collective knowledge to alleviate the interference caused by robot changes, dealing with the challenge of different combinations and dynamic changes in multi-robotic systems. The experiments are carried out on the StarCraft II platform and Webots simulator. The extensive results indicate that our method has noticeable improvement on many indicators compared with the traditional meta reinforcement learning methods.

Oracle-Guided Deep Reinforcement Learning for Large-Scale Multi-UAVs Flocking and Navigation

The flocking and navigation control of large-scale Unmanned Aerial Vehicle (UAV) swarms have received a lot of research interest due to the wide applications of UAVs in many fields. Compared to traditional non-learning-based flocking and navigation control methods, reinforcement learning-based methods have advantages in model-free, flexibility, and adaptability. In this paper, we formulate the flocking and navigation control of the UAV swarm as a Markov Decision Process (MDP) and use multi-agent reinforcement learning methods to solve the problem. There are two significant challenges introduced by reinforcement learning: the scalability issue and the partial observations of each UAV. We adopt the independent learning and parameter sharing scheme to tackle the scalability issue, which extends the single-agent reinforcement learning algorithms to the multi-agent scenario. For the partial observations, we propose an oracle-guided two-stage training and execution scheme, which utilizes the flock center during the training phase but avoids the dependence on the flock center during the execution phase. We design the oracle-guided observations and rewards and build a highly efficient simulation environment to conduct experiments. Simulation results show that the policy trained with our method performs well with up to thirty-two UAVs and outperforms the policy trained with local observations.

Common belief multi-agent reinforcement learning based on variational recurrent models

The tacit cooperation among human teams benefits from the fact that consensus can be reached on a task through common belief. Similar to human social groups, agents in distributed learning systems can also rely on common belief to achieve cooperation under the condition of limited communication. In this paper, we show the role of common belief among agents in completing cooperative tasks, by proposing the Common Belief Multi-Agent (CBMA) reinforcement learning method. CBMA is a novel value-based method that infers the belief between agents with a variational model and models the environment with a variational recurrent neural network. We validate CBMA on two grid-world games as well as the StarCraft II micromanagement benchmark. Experimental results show that the learned common belief by CBMA can improve performance in both discrete and continuous state settings.

A Multi-Agent Reinforcement Learning Method With Route Recorders for Vehicle Routing in Supply Chain Management

In the modern supply chain system, large-scale transportation tasks require the collaborative work of multiple vehicles to be completed on time. Over the past few decades, multi-vehicle route planning was mainly implemented by heuristic algorithms. However, these algorithms face the dilemma of long computation time. In recent years, some machine learning-based methods are also proposed for vehicle route planning, but the existing algorithms can hardly solve multi-vehicle time-sensitive problems. To overcome this problem, we propose a novel multi-agent reinforcement learning model, which optimizes the route length and the vehicle’s arrival time simultaneously. The model is based on the encoder-decoder framework. The encoder mines the relationship between the customer nodes in the problem, and the decoder generates the route of each vehicle iteratively. Specially, we design multiple route recorders to extract the route history information of vehicles and realize the communication between them. In the inferring phase, the model could immediately generate routes for all vehicles in a new instance. To further improve the performance of the model, we devise a multi-sampling strategy and obtain the balance boundary between computation time and performance improvement. In addition, we propose a simulation-based vehicle configuration method to select the optimal number of vehicles in real applications. For validation, we conduct a series of experiments on problems with different customer amounts and various vehicle numbers. The results show that the proposed model outperforms other typical algorithms in both performance and calculation time.

Option-Based Multi-Agent Reinforcement Learning for Painting With Multiple Large-Sized Robots

To ensure an efficient and conflict-free task allocation, this paper presents an option-based multi-agent reinforcement learning (OMARL) method for a cooperative multi-station multi-robot system in aircraft painting application. The problem can be described as a fully-connected graph, whose nodes and edges represent stations and trajectories between stations, respectively. The aim is to minimize the maximum execution time of conflict-free aircraft painting with multiple robots. Traditional methods cannot handle mutual collision avoidance well when robots are in large size. Therefore, this work solved the planning problem with multi-agent reinforcement learning algorithm based on option framework. The moving trajectories were pre-planned by a simplified 2D A* algorithm, which is fast and reduces rotations. The agents would finish the whole task with less time, and potential collisions were punished at every time step. Hierarchical option-based model with shared network parameters greatly reduced the complexity of training for painting planning scenarios. A fuzzy clustering method was introduced, and the clustering scores of stations would contribute to greedy agents. Comparisons were made on static and dynamic situations among several planning methods. OMARL can give optimal schedules as well as, or better than the most efficient traditional method does. Furthermore, OMARL can provide satisfactory adjustments in real time when practical process is different from plan or there is one broken robot. In contrast, traditional methods take lots of time to recalculate the plan. The results indicate that OMARL can achieve better performance in multi-station multi-robot task allocation when the collision between robots cannot be ignored.

MIRA: Model-Based Imagined Rollouts Augmentation for Non-Stationarity in Multi-Agent Systems

One of the challenges in multi-agent systems comes from the environmental non-stationarity that policies of all agents are evolving individually over time. Many existing multi-agent reinforcement learning (MARL) methods have been proposed to address this problem. However, these methods rely on a large amount of training data and some of them require agents to intensely communicate, which is often impractical in real-world applications. To better tackle the non-stationarity problem, this article combines model-based reinforcement learning (MBRL) and meta-learning and proposes a method called Model-based Imagined Rollouts Augmentation (MIRA). Based on an environment dynamics model, distributed agents can independently perform multi-agent rollouts with opponent models during exploitation and learn to infer the environmental non-stationarity as a latent variable using the rollouts. Based on the world model and latent-variable inference module, we perform multi-agent soft actor-critic implementation for centralized training and decentralized decision making. Empirical results on the Multi-agent Particle Environment (MPE) have proved that the algorithm has a very considerable improvement in sample efficiency as well as better convergent rewards than state-of-the-art MARL methods, including COMA, MAAC, MADDPG, and VDN.

Cost optimization of UAV-MEC network calculation offloading: A multi-agent reinforcement learning method

In the Unmanned Aerial Vehicle (UAV)-assisted mobile edge computing (MEC) system, most existing studies do not effectively consider the task offloading of user equipment (UEs) and the economic issues related to MEC service providers. Firstly, this paper considers the UE offloading cost and the pricing of MEC server, and establishes the UE, UAV cost and UAV revenue model. For the established non-convex optimization problem, it is further described as a Markov decision process (MDP), and a multi-agent reinforcement deep learning algorithm (MADRL) is proposed to minimize the system energy consumption by jointly optimizing power control, resource allocation and UE association, so as to effectively improve the overall revenue of UAV under the premise of ensuring the system performance. The simulation results show that our scheme can effectively improve the UAV revenue, and is significantly better than the random scheme without energy consumption optimization. In addition, the revenue obtained by our design scheme is 10.7% higher than that of the comparison scheme.

Explaining deep reinforcement learning decisions in complex multiagent settings: towards enabling automation in air traffic flow management.

With the objective to enhance human performance and maximize engagement during the performance of tasks, we aim to advance automation for decision making in complex and large-scale multi-agent settings. Towards these goals, this paper presents a deep multi agent reinforcement learning method for resolving demand - capacity imbalances in real-world Air Traffic Management settings with thousands of agents. Agents comprising the system are able to jointly decide on the measures to be applied to resolve imbalances, while they provide explanations on their decisions: This information is rendered and explored via appropriate visual analytics tools. The paper presents how major challenges of scalability and complexity are addressed, and provides results from evaluation tests that show the abilities of models to provide high-quality solutions and high-fidelity explanations.

A Graph Attention Mechanism-Based Multiagent Reinforcement-Learning Method for Task Scheduling in Edge Computing

Multi-access edge computing (MEC) enables end devices with limited computing power to provide effective solutions while dealing with tasks that are computationally challenging. When each end device in an MEC scenario generates multiple tasks, how to reasonably and effectively schedule these tasks is a large-scale discrete action space problem. In addition, how to exploit the objectively existing spatial structure relationships in the given scenario is also an important factor to be considered in task-scheduling algorithms. In this work, we consider indivisible, time-sensitive tasks under this scenario and formalize the task-scheduling problem to minimize the long-term losses. We propose a multiagent collaborative deep reinforcement learning (DRL)-based distributed scheduling algorithm based on graph attention neural networks (GATs) to solve task-scheduling problems in the MEC scenario. Each end device creates a graph representation agent to extract potential spatial features in the scenario and a scheduling agent to extract the timing-related features of the tasks and make scheduling decisions using a gated recurrent unit (GRU). The simulation results show that, compared with several baseline algorithms, our proposed algorithm can take advantage of the spatial positional relationship of devices in the environment, significantly reduce the average delay and drop rate, and improve link utilization.

Centralized and Accelerated Multiagent Reinforcement Learning Method with Automatic Reward Setting

Centralized and Accelerated Multiagent Reinforcement Learning Method with Automatic Reward Setting

Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach

In order to achieve the target of carbon peak and carbon neutrality, electric vehicles (EVs) have increasingly received a prominent interest to electrify the transportation sector due to their advantages of mobility and flexibility on handling complicated transportation and power networks. However, it is still challenging to realize the significant potential of EVs towards an emerging low-carbon transition. Previous works have focused on vehicle-to-grid (V2G) technology that allows for an increased utilization of EVs to make arbitrage by the temporal differentials of electricity prices. Nevertheless, the economic potential of EVs flexibility may not be fully exploited lacking an appropriate business model. This paper addresses this challenge by developing a coupled power-transportation network for cooperative EVs to optimize the provision of multiple inter-dependent services, including charging service, demand management service, carbon intensity service, and balancing service. In order to unlock this value, the EVs operation problem has already been tackled using model-based optimization approaches, which may raise privacy issues since the requirement for global information and also can be time consuming due to the high variability of transportation and power networks. In this paper, we propose a model-free hierarchical and hybrid multi-agent reinforcement learning method to learn the routing and scheduling decisions of EVs in a coupled power-transportation network with the objective of optimizing multi-service provisions. To this end, EVs do not reply on any knowledge of the simulated environment and are capable of handling system uncertainties via the learning process. Extensive case studies based on a 15-bus radial power distribution network and a 9-node 12-edge transportation network are developed to show that the proposed method outperforms the conventional learning algorithms in terms of policy quality and convergence speed. Finally, the generalizability and scalability are also investigated for different environment circumstances and EV numbers.

Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures

The increasing penetration of renewable energy sources has the potential to contribute towards the decarbonisation of the building energy sector. However, this transition brings its own challenges including that of energy integration and potential grid instability issues arising due the stochastic nature of variable renewable energy sources. One potential approach to address these issues is demand side management, which is increasingly seen as a promising solution to improve grid stability. This is achieved by exploiting demand flexibility and shifting peak demand towards periods of peak renewable energy generation. However, the energy flexibility of a single building needs to be coordinated with other buildings to be used in a flexibility market. In this context, multi-agent systems represent a promising tool for improving the energy management of buildings at the district and grid scale. The present research formulates the energy management of four buildings equipped with thermal energy storage and PV systems as a multi-agent problem. Two multi-agent reinforcement learning methods are explored: a centralised (coordinated) controller and a decentralised (cooperative) controller, which are benchmarked against a rule-based controller. The two controllers were tested for three different climates, outperforming the rule-based controller by 3% and 7% respectively for cost, and 10% and 14% respectively for peak demand. The study shows that the multi-agent cooperative approach may be more suitable for districts with heterogeneous objectives within the individual buildings.

SCC-rFMQ: a multiagent reinforcement learning method in cooperative Markov games with continuous actions

SCC-rFMQ: a multiagent reinforcement learning method in cooperative Markov games with continuous actions

Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning

AbstractIn this paper, we expolore Multi-Agent Reinforcement Learning (MARL) methods for unmanned aerial vehicle (UAV) cluster. Considering that the current UAV cluster is still in the program control stage, the fully autonomous and intelligent cooperative combat has not been realised. In order to realise the autonomous planning of the UAV cluster according to the changing environment and cooperate with each other to complete the combat goal, we propose a new MARL framework. It adopts the policy of centralised training with decentralised execution, and uses Actor-Critic network to select the execution action and then to make the corresponding evaluation. The new algorithm makes three key improvements on the basis of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. The first is to improve learning framework; it makes the calculated Q value more accurate. The second is to add collision avoidance setting, which can increase the operational safety factor. And the third is to adjust reward mechanism; it can effectively improve the cluster’s cooperative ability. Then the improved MADDPG algorithm is tested by performing two conventional combat missions. The simulation results show that the learning efficiency is obviously improved, and the operational safety factor is further increased compared with the previous algorithm.

Reward Design for Intelligent Intersection Control to Reduce Emission

The transportation industry is one of the main contributors to global warming since it is responsible for a quarter of greenhouse gas emissions. Due to society’s crucial dependence on fossil fuels and the rapid increase in mobility demands, the reduction of global vehicle emissions evolved into a significant challenge. In the urban transportation areas, signalized intersections can be considered the main bottlenecks in mitigating congestion and, therefore, vehicle emission. Our research focuses on the Traffic Signal Control problem since the efficient control of these intersections can significantly impact the productive hours and, through emission, the health of the citizens along with the depressing challenge of climate change. The Traffic Signal Control problem is well-studied and solved via several different techniques. However, most recently, Single and Multi-Agent Reinforcement Learning methods have arisen thanks to their performance and real-time applicability. Although rewarding schemes, which are the most crucial aspects of this method, do not seem to evolve at the same pace as the utilized techniques. In this paper, we propose a novel rewarding concept to compare its performance with the most common rewarding strategies in the literature. The results indicate that our approach outperforms its contenders from the literature in both classic and sustainability measures.

Unmanned Aerial Vehicle Swarm Cooperative Decision-Making for SEAD Mission: A Hierarchical Multiagent Reinforcement Learning Approach

Unmanned aerial vehicle (UAV) swarm cooperative decision-making has attracted increasing attentions because of its low-cost, reusable, and distributed characteristics. However, existing non-learning-based methods rely on small-scale, known scenarios, and cannot solve complex multi-agent cooperation problem in large-scale, uncertain scenarios. This paper proposes a hierarchical multi-agent reinforcement learning (HMARL) method to solve the heterogeneous UAV swarm cooperative decision-making problem for the typical suppression of enemy air defense (SEAD) mission, which is decoupled into two sub-problems, i.e., the higher-level target allocation (TA) sub-problem and the lower-level cooperative attacking (CA) sub-problem. We establish a HMARL agent model, consisting of a multi-agent deep Q network (MADQN) based TA agent and multiple independent asynchronous proximal policy optimization (IAPPO) based CA agents. MADQN-TA agent can dynamically adjust the TA schemes according to the relative position. To encourage exploration and promote learning efficiency, the Metropolis criterion and inter-agent information exchange techniques are introduced. IAPPO-CA agent adopts independent learning paradigm, which can easily scale with the number of agents. Comparative simulation results validate the effectiveness, robustness, and scalability of the proposed method.

A Multi-Agent Collaborative Environment Learning Method for UAV Deployment and Resource Allocation

The dynamic position deployment and resource allocation of the unmanned aerial vehicle (UAV) communication networks has great significance in terms of interference management, coverage enhancement, and capacity improvement. Since the transmission power and energy resources of the UAVs are limited and the actual communication environment is complex and time-varying, it is challenging for the multiple UAVs to dynamically make decisions to ensure the communication performance of the system. Meanwhile, the centralized architecture may generate a certain degree of communication delay and affect communication efficiency. Facing this challenge, a resource allocation algorithm for the UAV networks based on multi-agent collaborative environment learning is proposed. This method is based on a distributed architecture. Each UAV is modeled as an independent agent, which improves the utility of the UAV networks through the dynamic selection decisions of its deployment position, transmission power, and occupied sub-channels. Each UAV learns the mapping of the network information to the position deployment and resource selection decisions based on the reinforcement learning algorithm according to partial of the state information it can observe. For the overall network, a multi-agent reinforcement learning method based on federated learning is designed on the purpose of realizing information interaction and combined dispatching of the UAVs. In the multi-agent system, the framework of federated learning is introduced to realize the sharing of non-privacy data among the UAVs. Simulation results indicate that the proposed method can effectively improve the network utility compared with the multi-agent deep reinforcement learning algorithm without information interaction.