Multi-agent Reinforcement Learning Method Research Articles

Cooperative Multi-Agent Reinforcement Learning With Approximate Model Learning

In multi-agent reinforcement learning, it is essential for agents to learn communication protocol to optimize collaboration policies and to solve unstable learning problems. Existing methods based on actor-critic networks solve the communication problem among agents. However, these methods have difficulty in improving sample efficiency and learning robust policies because it is not easy to understand the dynamics and nonstationary of the environment as the policies of other agents change. We propose a method for learning cooperative policies in multi-agent environments by considering the communications among agents. The proposed method consists of recurrent neural network-based actor-critic networks and deterministic policy gradients to centrally train decentralized policies. The actor networks cause the agents to communicate using forward and backward paths and to determine subsequent actions. The critic network helps to train the actor networks by sending gradient signals to the actors according to their contribution to the global reward. To address issues with partial observability and unstable learning, we propose using auxiliary prediction networks to approximate state transitions and the reward function. We used multi-agent environments to demonstrate the usefulness and superiority of the proposed method by comparing it with existing multi-agent reinforcement learning methods, in terms of both learning efficiency and goal achievements in the test phase. The results demonstrate that the proposed method outperformed other alternatives.

WRFMR: A Multi-Agent Reinforcement Learning Method for Cooperative Tasks

Multi-agent reinforcement learning (MARL) for cooperative tasks has been extensively studied in recent years. The balance of exploration and exploitation is crucial to MARL algorithms’ performance in terms of the learning speed and the quality of the obtained strategy. To this end, we propose an algorithm known as the weighted relative frequency of obtaining the maximal reward (WRFMR), which uses a weight parameter and the action probability to balance exploration and exploitation and accelerate convergence to the optimal joint action. For the WRFMR algorithm, each agent needs to share the state and the immediate reward and does not need to observe the actions of the other agents. Theoretical analysis on the model of WRFMR in cooperative repeated games shows that each optimal joint action is an asymptotically stable critical point if the component action of every optimal joint action is unique. The box-pushing task, the distributed sensor network (DSN) task, and a strategy game known as blood battlefield are used for empirical studies. Both the DSN task and the box-pushing task involve full cooperation, while blood battle comprises both cooperation and competition. The simulation results show that the WRFMR algorithm outperforms the other algorithms regarding the success rate and the learning speed.

Utilizing Observed Information for No-Communication Multi-Agent Reinforcement Learning toward Cooperation in Dynamic Environment

This paper proposes a multi-agent reinforcement learning method without communication toward dynamic environments, called profit minimizing reinforcement learning with oblivion of memory (PMRL-OM)....

SA-IGA: a multiagent reinforcement learning method towards socially optimal outcomes

In multiagent environments, the capability of learning is important for an agent to behave appropriately in face of unknown opponents and dynamic environment. From the system designer's perspective, it is desirable if the agents can learn to coordinate towards socially optimal outcomes, while also avoiding being exploited by selfish opponents. To this end, we propose a novel gradient ascent based algorithm (SA-IGA) which augments the basic gradient-ascent algorithm by incorporating social awareness into the policy update process. We theoretically analyze the learning dynamics of SA-IGA using dynamical system theory and SA-IGA is shown to have linear dynamics for a wide range of games including symmetric games. The learning dynamics of two representative games (the prisoner's dilemma game and the coordination game) are analyzed in details. Based on the idea of SA-IGA, we further propose a practical multiagent learning algorithm, called SA-PGA, based on Q-learning update rule. Simulation results show that SA-PGA agent can achieve higher social welfare than previous social-optimality oriented Conditional Joint Action Learner (CJAL) and also is robust against individually rational opponents by reaching Nash equilibrium solutions.

Efficient Training Techniques for Multi-Agent Reinforcement Learning in Combat Tasks

Multi-agent combat scenarios often appear in many real-time strategy games. Efficient learning for such scenarios is an indispensable step towards general artificial intelligence. Multi-agent reinforcement learning (MARL) algorithms have attracted much interests, but few of them have been shown effective for such scenarios. Most of previous research is focused on revising the learning mechanism of MARL algorithms, for example, trying different types of neural networks. The study of training techniques for improving the performance of MARL algorithms has not been paid much attention. In this paper we propose three efficient training techniques for a multi-agent combat problem which is originated from an unmanned aerial vehicle (UAV) combat scenario. The first one is the scenario-transfer training, which utilizes the experience obtained in simpler combat tasks to assist the training for complex tasks. The next one is the self-play training, which can continuously improve the performance by iteratively training agents and their counterparts. Finally, we consider using combat rules to assist the training, which is named as the rule-coupled training. We combine the three training techniques with two popular multi-agent reinforcement learning methods, multi-agent deep q-learning and multi-agent deep deterministic policy gradient (proposed by Open AI in 2017), respectively. The results show that both the converging speed and the performance of the two methods are significantly improved through the three training techniques.

Improved Multi-Agent Reinforcement Learning for Path Planning-Based Crowd Simulation

The combination of multi-agent technology and reinforcement learning methods has been recognized as an effective way which is used in path planning-based crowd simulation. However, the existing solution is still not satisfactory due to the problem in the mutual influence of agents. Therefore, an improved multi-agent reinforcement learning method (IMARL algorithm) is introduced. In this method, the intersection of the pedestrian trajectory extracted from the real video is first used as the state space for reinforcement learning. The crowd is grouped and the leader is selected. A bulletin board is added to the reinforcement learning algorithm of multi-agent to store the empirical knowledge of the learning process, and the navigation agent passes information between the leader and the bulletin board. The original social force model was improved, and the cohesive force of visual factors was added to the force formula. The IMARL algorithm is combined with the improved social force model for crowd evacuation simulation. Using a two-layer control mechanism, the leader in the upper layer uses the decision process based on the IMARL algorithm to select the path, and the individuals in the bottom group use the improved social force model to evacuate. The method of this paper not only solves the dimensionality disaster problem of reinforcement learning but also improves the convergence speed. The evacuation efficiency is effectively improved in crowd evacuation simulation experiments. In addition, it can also provide specific guidance scheme for crowd evacuation improvement and assistant decision support for the prevention and management of large-scale group trampling incidents.

A New Multi-Agent Reinforcement Learning Method Based on Evolving Dynamic Correlation Matrix

Multi-agent reinforcement learning approaches can be roughly classified into two categories. One is the agent-based approach which can be implemented in real distributed systems, though most approaches of this type cannot provide meaningful theoretical verifications. The other can be seen as the more formalized approach, which can provide theoretical results. However, most of current algorithms usually require unrealistic global communication, which makes them impractical for real distributed systems. In this article, we propose a dynamic correlation matrix based multi-agent reinforcement learning approach where the meta-parameters are evolved using an evolutionary algorithm. We believe that our approach is able to fill the gap between the two kinds of traditional multi-agent reinforcement learning approaches by providing both agent-level implementation and system-level convergence verification. The basic idea of this approach is that agents learn not only from local environmental feedback, i.e., their own experiences and rewards, but also from other agents’ experiences. In this way, the agents’ learning speed can be increased significantly. The performance of the proposed algorithm is demonstrated on a number of application scenarios, including blackjack games, urban traffic control systems and multi-robot foraging.

Energy Optimization of Solar Micro-Grid Using Multi Agent Reinforcement Learning

In this paper, two solar Photovoltaic (PV) systems are considered; one in the department with capacity of 100 kW and the other in the hostel with capacity of 200 kW. Each one has battery and load. The capital cost and energy savings by conventional methods are compared and it is proved that the energy dependency from grid is reduced in solar micro-grid element, operating in distributed environment. In the smart grid frame work, the grid energy consumption is further reduced by optimal scheduling of the battery, using Reinforcement Learning. Individual unit optimization is done by a model free reinforcement learning method, called Q-Learning and it is compared with distributed operations of solar micro-grid using a Multi Agent Reinforcement Learning method, called Joint Q-Learning. The energy planning is designed according to the prediction of solar PV energy production and observed load pattern of department and the hostel. A simulation model was developed using Python programming.

あるクラスのジレンマ問題に対するマルチエージェント強化学習法

あるクラスのジレンマ問題に対するマルチエージェント強化学習法

A novel modular Q-learning architecture to improve performance under incomplete learning in a grid soccer game

Multi-agent reinforcement learning methods suffer from several deficiencies that are rooted in the large state space of multi-agent environments. This paper tackles two deficiencies of multi-agent reinforcement learning methods: their slow learning rate, and low quality decision-making in early stages of learning. The proposed methods are applied in a grid-world soccer game. In the proposed approach, modular reinforcement learning is applied to reduce the state space of the learning agents from exponential to linear in terms of the number of agents. The modular model proposed here includes two new modules, a partial-module and a single-module. These two new modules are effective for increasing the speed of learning in a soccer game. We also apply the instance-based learning concepts, to choose proper actions in states that are not experienced adequately during learning. The key idea is to use neighbouring states that have been explored sufficiently during the learning phase. The results of experiments in a grid-soccer game environment show that our proposed methods produce a higher average reward compared to the situation where the proposed method is not applied to the modular structure.

A multi-agent reinforcement learning approach to robot soccer

In this paper, a multi-agent reinforcement learning method based on action prediction of other agent is proposed. In a multi-agent system, action selection of the learning agent is unavoidably impacted by other agents' actions. Therefore, joint-state and joint-action are involved in the multi-agent reinforcement learning system. A novel agent action prediction method based on the probabilistic neural network (PNN) is proposed. PNN is used to predict the actions of other agents. Furthermore, the sharing policy mechanism is used to exchange the learning policy of multiple agents, the aim of which is to speed up the learning. Finally, the application of presented method to robot soccer is studied. Through learning, robot players can master the mapping policy from the state information to the action space. Moreover, multiple robots coordination and cooperation are well realized.

Research on Path-Planning of Manipulator Based on Multi-Agent Reinforcement Learning

Because of the dynamic characteristic of high nonlinear,strong coupling and variable structure,it is difficult to perform effective controlling on the robot manipulator by conventional controlling theory.In this paper,a new approach of multi-agent reinforcement learning method based on Kohonen net is proposed which is used in the multi-agent environment of robot manipulator path-planning and the simulation experiment shows the validity of this method.

A novel multi-agent reinforcement learning approach for job scheduling in Grid computing

Grid computing utilizes distributed heterogeneous resources to support large-scale or complicated computing tasks, and an appropriate resource scheduling algorithm is fundamentally important for the success of Grid applications. Due to the complex and dynamic properties of Grid environments, traditional model-based methods may result in poor scheduling performance in practice. Scalability and adaptability are among the key objectives of Grid job scheduling. In this paper, a novel multi-agent reinforcement learning method, called ordinal sharing learning (OSL) method, is proposed for job scheduling problems, especially, for realizing load balancing in Grids. The approach circumvents the scalability problem by using an ordinal distributed learning strategy, and realizes multi-agent coordination based on an information-sharing mechanism with limited communication. Simulation results show that the OSL method can achieve the goal of load balancing effectively, and its performance is even comparable to some centralized scheduling algorithm in most cases. The convergence property and adaptability of the proposed method are also illustrated.

Adaptive Division-of-Labor Control Algorithm for Multi-Robot Systems

An advanced function for multi-robot systems is the division of labor. There are some studies proposing a multi-agent reinforcement learning method for a division of labor. However, it often requires much time to converge. Many studies focusing on division-of-labor control inspired biological phenomenon have been reported. In those methods, whether heterogeneous or homogeneous state is determined by self-organization, however, group performance improvement is not guaranteed because decentralized control is typically complicated. In this study, we propose adaptive division-of-labor control, enabling adaptive selection of homogeneous or heterogeneous group state. We demonstrate the adaptability of proposal method versus working conditions and address the performance improvement by mathematical analysis. To evaluate the effectiveness of the proposed method, we treat foraging by multi-robot systems and confirm that the robot group inevitably organizes the division of labor with group performance improvement in computer simulations.

Cooperative Multiagent Congestion Control for High-Speed Networks

An adaptive multiagent reinforcement learning method for solving congestion control problems on dynamic high-speed networks is presented. Traditional reactive congestion control selects a source rate in terms of the queue length restricted to a predefined threshold. However, the determination of congestion threshold and sending rate is difficult and inaccurate due to the propagation delay and the dynamic nature of the networks. A simple and robust cooperative multiagent congestion controller (CMCC), which consists of two subsystems: a long-term policy evaluator, expectation-return predictor and a short-term rate selector composed of action-value evaluator and stochastic action selector elements has been proposed to solve the problem. After receiving cooperative reinforcement signals generated by a cooperative fuzzy reward evaluator using game theory, CMCC takes the best action to regulate source flow with the features of high throughput and low packet loss rate. By means of learning procedures, CMCC can learn to take correct actions adaptively under time-varying environments. Simulation results showed that the proposed approach can promote the system utilization and decrease packet losses simultaneously.

A Multi-Agent Reinforcement Learning Method Based on Role Tracking

A Multi-Agent Reinforcement Learning Method Based on Role Tracking

A multiagent reinforcement learning method based on the model inference of the other agents

AbstractThis paper proposes a reinforcement learning method based on an actor‐critic architecture, which can be applied to multiagent competitive games. As an example, we deal with the card game “Hearts.” We introduce a method combining an actor‐critic learning and model inference of the other agents. A single Hearts game is divided into three stages, and three actors are prepared so that one of them determines actions separately in each stage. In the middle stage, in particular, actions are determined based on expected temporal‐difference error. Computer simulation results suggest the importance of the model inference of the other agents. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(12): 67–76, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10110