Multi-agent Reinforcement Learning Technique Research Articles

Research on Multi cabin Collaborative Assembly Method Based on Multi Agent Reinforcement Learning

Abstract In the domain of multi-cabin assembly, the prevailing method entails step-by-step docking assembly, which often leads to the accumulation of errors throughout the assembly process. This paper employs multi-agent reinforcement learning techniques to facilitate collaborative assembly of the lateral force device assembly, thereby enhancing assembly quality. Initially, the assembly process of the final assembly is scrutinized to establish a conducive learning environment. Subsequently, a novel agent configuration is proposed, wherein two agents are deployed to represent a single motion unit. Following this, a diverse range of algorithms is employed to train the model, enabling the selection of the most appropriate algorithm and optimization of pertinent parameters. The resultant multi-agent assembly paths are then generated for comprehensive analysis and comparison. Finally, simulation verification of the paths is conducted, accompanied by an analysis of the assembly-induced stress during the assembly process.

A review of research on reinforcement learning algorithms for multi-agents

In recent years, multi-agent reinforcement learning techniques have been widely used and evolved in the field of artificial intelligence. However, traditional reinforcement learning methods have limitations such as long training time, large sample data requirements, and highly delayed rewards. Therefore, this paper systematically and specifically studies the MARL algorithm. Firstly, this paper uses Citespace software to visually analyze the existing literature on multi-agent reinforcement learning and briefly indicates the research hotspots and key research directions in this field. Secondly, the applications of traditional reinforcement learning algorithms under two task objects, namely single-agent and multi-agent systems, are described in detail. Then, the paper highlights the diverse applications, challenges, and corresponding solutions of MARL algorithmic techniques in the field of MAS. Finally, the paper points out future research directions based on the existing limitations of the algorithm. Through this paper, readers will gain a systematic and in-depth understanding of MARL algorithms and how they can be utilized to better address the various challenges posed by MAS.

Multi-Agent Reinforcement Learning for Traffic Flow Management of Autonomous Vehicles

Intelligent traffic management systems have become one of the main applications of Intelligent Transportation Systems (ITS). There is a growing interest in Reinforcement Learning (RL) based control methods in ITS applications such as autonomous driving and traffic management solutions. Deep learning helps in approximating substantially complex nonlinear functions from complicated data sets and tackling complex control issues. In this paper, we propose an approach based on Multi-Agent Reinforcement Learning (MARL) and smart routing to improve the flow of autonomous vehicles on road networks. We evaluate Multi-Agent Advantage Actor-Critic (MA2C) and Independent Advantage Actor-Critical (IA2C), recently suggested Multi-Agent Reinforcement Learning techniques with smart routing for traffic signal optimization to determine its potential. We investigate the framework offered by non-Markov decision processes, enabling a more in-depth understanding of the algorithms. We conduct a critical analysis to observe the robustness and effectiveness of the method. The method's efficacy and reliability are demonstrated by simulations using SUMO, a software modeling tool for traffic simulations. We used a road network that contains seven intersections. Our findings show that MA2C, when trained on pseudo-random vehicle flows, is a viable methodology that outperforms competing techniques.

COLREGs-Compliant Multi-Ship Collision Avoidance Based on Multi-Agent Reinforcement Learning Technique

The congestion of waterways can easily lead to traffic hazards. Moreover, according to the data, the majority of sea collisions are caused by human error and the failure to comply with the Convention on the International Regulation for the preventing Collision at Sea (COLREGs). To avoid this situation, ship automatic collision avoidance has become one of the most important research issues in the field of marine engineering. In this study, an efficient method is proposed to solve multi-ship collision avoidance problems based on the multi-agent reinforcement learning (MARL) algorithm. Firstly, the COLREGs and ship maneuverability are considered for achieving multi-ship collision avoidance. Subsequently, the Optimal Reciprocal Collision Avoidance (ORCA) algorithm is utilized to detect and reduce the risk of collision. Ships can operate at the safe velocity computed by the ORCA algorithm to avoid collisions. Finally, the Nomoto three-degrees-of-freedom (3-DOF) model is used to simulate the maneuvers of ships. According to the above information and algorithms, this study designs and improves the state space, action space and reward function. For validating the effectiveness of the method, this study designs various simulation scenarios with thorough performance evaluations. The simulation results indicate that the proposed method is flexible and scalable in solving multi-ship collision avoidance, complying with COLREGs in various scenarios.

Creating Interactive Crowds with Reinforcement Learning

The entertainment industry, as well as the field of Computer Graphics, frequently faces the issue of creating large virtual crowds that would populate a scene. One of the ways to achieve that, particularly with modern rendering techniques, is by using simulation -- this, however, is nontrivial to design and control. The main goal of my PhD work is working towards the creation of a tool enabling the creation of virtual crowds that one can interact with, and we believe the best way to that is through Multiagent Reinforcement Learning techniques. These animated crowds can then be used both in movies and video games. Especially for the latter, it is highly desirable that both the crowd as a whole, as well as the individual characters, can react to the user's input in real time.

Multi-Agent Reinforcement Learning Based Distributed Transmission in Collaborative Cloud-Edge Systems

Millimeter-wave (mmWave) cloud-edge collaboration has emerged as an effective solution for low-latency transmission by harnessing the potentials of cloud and edge nodes (eNodes) as well as abundant bandwidth. In a mmWave dense network, centralized processing and coordinated multi-points processing require large amounts of signaling exchange overhead, which in turn increase the latency. As such, distributed mechanism is demanded. In this paper, we investigate joint user scheduling and beam selection strategies for eNodes and formulate the problem as a constrained Markov decision process. The objective and constraints are long-term average network delay cost minimization and instantaneous quality of service (QoS) guarantee for each user equipment. To update the distributed strategies, we apply the multi-agent reinforcement learning technique and propose a hybrid learning framework by extending the actor-critic model, where centralized training and decentralized execution are implemented. We utilize the Lagrange technique to design the reward functions. Additionally, a novel actor network architecture and an exploration scheme are proposed. Simulation results validate the effectiveness of the proposed intelligent distributed algorithm with a high degree of scalability, and show superior performance in terms of long-term average network delay cost and QoS satisfaction rates compared with other methods.

PSO-Based: MARL Approach for Frequency Regulation of Multi-area Power System

In this study, a particle swarm optimization (PSO) based Multi-Agent Reinforcement Learning (MARL) approach for load frequency control (LFC) of multi-area power system over communication network considering communication delay and packet loss is presented. A networked control system (NCS) is framed to model a constant delay and packet loss using Markovian approach. The controller using PSO based MARL technique having two levels of controlled action in each area of power system act as an estimator agent and controller agent respectively. The proposed method for frequency regulation of multi-area power system is studied in detail and verified for different types of load disturbances. In addition, to check the system dynamic performance of electric power system a mean square errors (MSEs) of states with different value of communication delay are computed. The simulation results of this study is validated and compared with some pre-design methods available from literatures, along with capacity of multi agent system (MAS) for application of LFC in power system over communication network.

Distributed Multiagent Coordinated Learning for Autonomous Driving in Highways Based on Dynamic Coordination Graphs

Autonomous driving is one of the most important AI applications and has attracted extensive interest in recent years. A large number of studies have successfully applied reinforcement learning techniques in various aspects of autonomous driving, ranging from low-level control of driving maneuvers to higher level of strategic decision-making. However, comparatively less progress has been made in investigating how co-existing autonomous vehicles would interact with each other in a common environment and how reinforcement learning can be helpful in such situations by applying multiagent reinforcement learning techniques in the high-level strategic decision-making of the following or overtaking for a group of autonomous vehicles in highway scenarios. Learning to achieve coordination among vehicles in such situations is challenging due to the unique feature of vehicular mobility, which renders it infeasible to directly apply the existing coordinated learning approaches. To solve this problem, we propose using dynamic coordination graph to model the continuously changing topology during vehicles’ interactions and come up with two basic learning approaches to coordinate the driving maneuvers for a group of vehicles. Several extension mechanisms are then presented to make these approaches workable in a more complex and realistic setting with any number of vehicles. The experimental evaluation has verified the benefits of the proposed coordinated learning approaches, compared with other approaches that learn without coordination or rely on some traditional mobility models based on some expert driving rules.

Energy efficient data collection in sink-centric wireless sensor networks: A cluster-ring approach

Sink-centric traffic pattern is common in wireless sensor networks (WSN), which typically causes higher energy consumption of the sensor nodes near the sink node (called ‘hot spot’ problem). Clustering combined with careful traffic flow control can alleviate a hot spot by dispersing the energy burden concentration. Existing clustering schemes treat each clusters as an entity for energy efficiency optimization. We propose to group a set of clusters into ‘cluster-rings’, which is a chain of clusters that are equal distance away from the sink, and conduct energy efficiency optimization at the cluster-ring level. More specifically, we first present a novel method to compose a cluster structure. Next, we present an algorithm that gradually optimizes the traffic flow control at the cluster-ring level by using a multi-agent reinforcement learning technique. Then, we present an algorithm that makes cluster-level traffic routing decision on the basis of cluster-ring level traffic flow control results. Via simulations, it is shown that the proposed scheme result near-optimal performance and can adapt to dynamic changes of network-wide traffic generation.

A reinforcement learning-based routing for delay tolerant networks

Delay Tolerant Reinforcement-Based (DTRB) is a delay tolerant routing solution for IEEE 802.11 wireless networks which enables device to device data exchange without the support of any pre-existing network infrastructure. The solution utilizes Multi-Agent Reinforcement Learning techniques to learn about routes in the network and forward/replicate the messages that produce the best reward. The rewarding process is executed by a learning algorithm based on the distances between the nodes, which are calculated as a function of time from the last meetings. DTRB is a flooding-based delay tolerant routing solution. The simulation results show that DTRB can deliver more messages than a traditional delay tolerant routing solution does in densely populated areas, with similar end-to-end delay and lower network overhead.

From cognition to docition: The teaching radio paradigm for distributed & autonomous deployments

We advocate for a novel communication paradigm of docition which facilitates distributed and autonomous networking at minimal control overhead and maximal performance. We consider that the nodes in foreseen networks are intelligent radios able to learn and thus self-adapt to prior set performance targets within a given surrounding environment. We briefly review the state-of-the-art of purely distributed learning algorithms, and we identify the most appropriate approaches allowing for self-adaptation to particular system dynamics. In such distributed settings, however, the learning is typically complex, imprecise and slow due to mutually-impacting decisions resulting in non-stationarities. The docitive paradigm proposes a timely solution which encourages more knowledgeable nodes to teach surrounding nodes to speed up the development of their cognitive state. We advocate for different degrees of docition, such as teaching at start-up or run-time, and demonstrate that this improves the convergence speed and precision of known cognitive algorithms. We evaluate the docitive paradigm in the context of a femtocell network modeled as a multi-agent system, where the agents are the femto access points, implementing a realtime multi-agent reinforcement learning technique known as decentralized Q-learning. We propose different docitive algorithms and we show their superiority to the well know paradigm of independent learning.

A Comprehensive Survey of Multiagent Reinforcement Learning

Multiagent systems are rapidly finding applications in a variety of domains, including robotics, distributed control, telecommunications, and economics. The complexity of many tasks arising in these domains makes them difficult to solve with preprogrammed agent behaviors. The agents must, instead, discover a solution on their own, using learning. A significant part of the research on multiagent learning concerns reinforcement learning techniques. This paper provides a comprehensive survey of multiagent reinforcement learning (MARL). A central issue in the field is the formal statement of the multiagent learning goal. Different viewpoints on this issue have led to the proposal of many different goals, among which two focal points can be distinguished: stability of the agents' learning dynamics, and adaptation to the changing behavior of the other agents. The MARL algorithms described in the literature aim---either explicitly or implicitly---at one of these two goals or at a combination of both, in a fully cooperative, fully competitive, or more general setting. A representative selection of these algorithms is discussed in detail in this paper, together with the specific issues that arise in each category. Additionally, the benefits and challenges of MARL are described along with some of the problem domains where the MARL techniques have been applied. Finally, an outlook for the field is provided.