Multi-agent Reinforcement Learning Model Research Articles

Enhancing urban metro system resilience under disruptive events through multi-agent reinforcement learning

With urban rail transit (URT) systems playing an increasingly crucial role in megacity mobility, the rising frequency of service disruptions may pose substantial passenger delays and safety concerns. This emphasizes the need to prioritize resilience enhancements for the URT system. Bus bridging, an effective resilience-enhancing measure, faces challenges that effectively capture the unique features of bridging buses and stranded passengers, and rely on predetermined routes and fixed frequencies, diminishing service flexibility and efficiency in dynamic bridging environments. To overcome these challenges, we propose a novel approach using a multi-agent reinforcement learning model. This model treats bus bridging as a collaborative multi-agent task with the competition. Specifically, it enables bridging buses to adaptively select the next visiting station based on environmental characteristics. Furthermore, a competitive reward function is introduced to facilitate effective collaboration among all agents while preventing malicious competition. The case study results demonstrate that the proposed enhancing resilience method outperforms traditional reinforcement learning method and classic bus bridging strategy and can reduce the average waiting time and average travel time, especially in resource-limited and complex scenarios.

Multi-mode filter target tracking method for mobile robot using multi-agent reinforcement learning

Multi-mode filtering target tracking for mobile robot has important research significance for robot path planning, motion control and tracking robot targets. To address the problem that it is difficult for mobile robot to track targets in unknown environments, a multi-mode filtering target tracking method for mobile robot using multi-agent reinforcement learning is proposed. First, based on the extended kalman filter and combined with probability data association, the observation values in different motion modes are calculated and the system state is updated in real time, to achieve concurrent mapping, location and state estimation for each robot. Second, state communication among multi-mobile robot are established, and the multi-agent reinforcement learning model is built. Finally, based on the agent reinforcement learning model, based on the communication and communication among multi-tracking agents and the location of the target agent, the target tracking task is allocated to shorten the total path of the tracking agent, and the multi-target tracking of multi-mobile robot is completed by constantly learning and updating the strategy. The results show that the maximum tracking error of the target is 7 mm, and the calculation time of each step is only 51.1ms. The tracking success rate, observation rate and coverage rate are all high. It has important research value in robot location and navigation, service robot and other fields.

A multi-agent reinforcement learning model for inventory transshipments under supply chain disruption

The COVID-19 pandemic has significantly disrupted global Supply Chains (SCs), emphasizing the importance of SC resilience, which refers to the ability of SCs to return to their original or more desirable state following disruptions. This study focuses on collaboration, a key component of SC resilience, and proposes a novel collaborative structure that incorporates a fictitious agent to manage inventory transshipment decisions between retailers in a centralized manner while maintaining the retailers’ autonomy in ordering. The proposed collaborative structure offers the following advantages from SC resilience and operational perspectives: (i) it facilitates decision synchronization for enhanced collaboration among retailers, and (ii) it allows retailers to collaborate without the need for information sharing, addressing the potential issue of information sharing reluctance. Additionally, this study employs non-stationary probability to capture the deeply uncertain nature of the ripple effect and the highly volatile customer demand caused by the pandemic. A new Reinforcement Learning (RL) algorithm is developed to handle non-stationary environments and to implement the proposed collaborative structure. Experimental results demonstrate that the proposed collaborative structure using the new RL algorithm achieves superior SC resilience compared with centralized inventory management systems with transshipment and decentralized inventory management systems without transshipment using traditional RL algorithms.

Real-Time Coordinated Operation of Power and Autonomous Electric Ride-Hailing Systems

Integration of self-driving functions in electric vehicles is radically changing the transportation systems, and representing an opportunity for power utilities to develop innovative solutions for harnessing the spatio-temporal charging flexibility of autonomous electric vehicles (AEVs). This paper develops a multi-agent reinforcement learning model for intelligent real-time coordinated operation of interdependent autonomous electric ride-hailing system (AERS) and power distribution system (PDS), which enables the coordinated routing and charging of AEVs while ensuring the quality of service by meeting spatio-temporal passenger demand and regularly charging the batteries. The proposed model adopts a modified deep Q-network method with multi-agent rollout to determine the near-optimal solution for the coordinated operation problem that determines the routing and charging of numerous AEVs in real-time. The proposed model is tested on a 13-node transportation network, adopted from the Salt Lake City transportation system, and the IEEE 33-bus test power distribution system to showcase the efficiency of the proposed model in determining the real-time routing and charging decisions for AEVs while meeting the operational constraints of both AERS and PDS.

Multi-agent reinforcement learning for traffic congestion on one-way multi-lane highways

ABSTRACT In the last decade, agent-based modelling and simulation has emerged as a potential approach to study complex systems in the real world, such as traffic congestion. Complex systems could be modelled as a collection of autonomous agents, who observe the external environment, interact with each other and perform suitable actions. In addition, reinforcement learning, a branch of Machine Learning, that models the learning process of a single agent as a Markov decision process, has recently achieved remarkable results in several domains (e.g. Atari games, Dota 2, Go, Self-driving cars, Protein folding, etc.), especially with the invention of deep reinforcement learning. Multi-agent reinforcement learning, by taking advantage of these two approaches, is a new technique that can be used to further study complex systems. In this article, we present a multi-agent reinforcement learning model for traffic congestion on one-way multi-lane highways and experiment with six reinforcement learning algorithms in this setting.

Energy efficient behavior modeling for demand side recommender system in solar microgrid applications using multi-agent reinforcement learning model

Electricity consumers are often faced with challenges relating to the choice of an optimal energy saving plan. Increasing integration of transient renewable energy sources promises tantalizing solutions but also poses emerging stability challenges for the electricity grid. Demand side management using battery energy storage systems (BESSs) is crucial towards extending the physical limits of existing electricity grid. However, problems related to consumer behavior towards adoption of energy/battery efficiency measures and consumer comfort feedback exist. In this study, we present BESS technologies that are embedded into the grid and further enhanced with the use of reinforcement learning control and recommendation system technologies for improving the grid reliability, attaining self-consumption and demand response goals. The novelty of the proposed work is highlighted by using a separate class of active controller for BESS technologies, thereby separating it from loads which determine user comfort. Similarly, an adaptive demand side recommender scheme was used to provide recommendations targeting various microgrid entities. The result of the study shows that operating BESS using the multi-agent reinforcement learning control strategy achieved a maximum peak load reduction of about 24.5% alongside 94% comfort improvements in certain loads. The linear reduction in peak load was further enhanced by the BESS efficiency-related recommendations when compared to the baseline scenario.

Multi-Agent DRL-Based Lane Change With Right-of-Way Collaboration Awareness

Lane change is a common-yet-challenging driving behavior for automated vehicles. To improve the safety and efficiency of automated vehicles, researchers have proposed various lane-change decision models. However, most of the existing models consider lane-change behavior as a one-player decision-making problem, ignoring the essential multi-agent properties when vehicles are driving in traffic. Such models lead to deficiencies in interaction and collaboration between vehicles, which results in hazardous driving behaviors and overall traffic inefficiency. In this paper, we revisit the lane-change problem and propose a bi-level lane-change behavior planning strategy, where the upper level is a novel multi-agent deep reinforcement learning (DRL) based lane-change decision model and the lower level is a negotiation based right-of-way assignment model. We promote the collaboration performance of the upper-level lane-change decision model from three crucial aspects. First, we formulate the lane-change decision problem with a novel multi-agent reinforcement learning model, which provides a more appropriate paradigm for collaboration than the single-agent model. Second, we encode the driving intentions of surrounding vehicles into the observation space, which can empower multiple vehicles to implicitly negotiate the right-of-way in decision-making and enable the model to determine the right-of-way in a collaborative manner. Third, an ingenious reward function is designed to allow the vehicles to consider not only ego benefits but also the impact of changing lanes on traffic, which will guide the multi-agent system to learn excellent coordination performance. With the upper-level lane-change decisions, the lower-level right-of-way assignment model is used to guarantee the safety of lane-change behaviors. The experiments show that the proposed approaches can lead to safe, efficient, and harmonious lane-change behaviors, which boosts the collaboration between vehicles and in turn improves the safety and efficiency of the overall traffic. Moreover, the proposed approaches promote the microscopic synchronization of vehicles, which can lead to the macroscopic synchronization of traffic flow.

A Multi-Agent Reinforcement Learning Method With Route Recorders for Vehicle Routing in Supply Chain Management

In the modern supply chain system, large-scale transportation tasks require the collaborative work of multiple vehicles to be completed on time. Over the past few decades, multi-vehicle route planning was mainly implemented by heuristic algorithms. However, these algorithms face the dilemma of long computation time. In recent years, some machine learning-based methods are also proposed for vehicle route planning, but the existing algorithms can hardly solve multi-vehicle time-sensitive problems. To overcome this problem, we propose a novel multi-agent reinforcement learning model, which optimizes the route length and the vehicle’s arrival time simultaneously. The model is based on the encoder-decoder framework. The encoder mines the relationship between the customer nodes in the problem, and the decoder generates the route of each vehicle iteratively. Specially, we design multiple route recorders to extract the route history information of vehicles and realize the communication between them. In the inferring phase, the model could immediately generate routes for all vehicles in a new instance. To further improve the performance of the model, we devise a multi-sampling strategy and obtain the balance boundary between computation time and performance improvement. In addition, we propose a simulation-based vehicle configuration method to select the optimal number of vehicles in real applications. For validation, we conduct a series of experiments on problems with different customer amounts and various vehicle numbers. The results show that the proposed model outperforms other typical algorithms in both performance and calculation time.

Dynamic energy scheduling and routing of a large fleet of electric vehicles using multi-agent reinforcement learning

As the world’s population and economy grow, demand for energy increases as well. Smart grids can be a cost-effective solution to overcome increases in energy demand and ensure power security. Current applications of smart grids involve a large numbers of agents (e.g., electric vehicles). Since each agent must interact with other agents when taking decisions (e.g., movement and scheduling), the computational complexity of smart grid systems increases exponentially with the number of agents. Computational tractability of planning is a significant barrier to implementation of large-scale smart grids of electric vehicles.Existing solution approaches such as mixed-integer programming and dynamic programming are not computationally efficient for high-dimensional problems. This paper proposes a reformulation of a Mixed-Integer Programming model into a Decentralized Markov Decision Process model and solves it using a Multi-Agent Reinforcement Learning algorithm to address the scalability issues of large-scale smart grid systems. The Decentralized Markov Decision Process model uses centralized training and distributed execution: agents are trained using a unique actor network for each agent and a shared critic network, and then agent execute actions independently from other agents to reduce computation time. The performance of the Multi-Agent Reinforcement Learning model is assessed under different configurations of customers and electric vehicles, and compared to the results from deep reinforcement learning and three heuristic algorithms. The simulation results demonstrate that the Multi-Agent Reinforcement Learning algorithm can reduce simulation time significantly compared to deep reinforcement learning, genetic algorithm, particle swarm optimization, and the artificial fish swarm algorithm. The superior performance of the proposed method indicates that it may be a realistic solution for large-scale implementation.

Stock Price Formation: Precepts from a Multi-Agent Reinforcement Learning Model

Stock Price Formation: Precepts from a Multi-Agent Reinforcement Learning Model

Intelligent multi-agent reinforcement learning model for resources allocation in cloud computing

Now more than ever, optimizing resource allocation in cloud computing is becoming more critical due to the growth of cloud computing consumers and meeting the computing demands of modern technology. Cloud infrastructures typically consist of heterogeneous servers, hosting multiple virtual machines with potentially different specifications, and volatile resource usage. This makes the resource allocation face many issues such as energy conservation, fault tolerance, workload balancing, etc. Finding a comprehensive solution that considers all these issues is one of the essential concerns of cloud service providers. This paper presents a new resource allocation model based on an intelligent multi-agent system and reinforcement learning method (IMARM). It combines the multi-agent characteristics and the Q-learning process to improve the performance of cloud resource allocation. IMARM uses the properties of multi-agent systems to dynamically allocate and release resources, thus responding well to changing consumer demands. Meanwhile, the reinforcement learning policy makes virtual machines move to the best state according to the current state environment. Also, we study the impact of IMARM on execution time. The experimental results showed that our proposed solution performs better than other comparable algorithms regarding energy consumption and fault tolerance, with reasonable load balancing and respectful execution time.

Evaluating the learning and performance characteristics of self-organizing systems with different task features

AbstractSelf-organizing systems (SOS) are developed to perform complex tasks in unforeseen situations with adaptability. Predefining rules for self-organizing agents can be challenging, especially in tasks with high complexity and changing environments. Our previous work has introduced a multiagent reinforcement learning (RL) model as a design approach to solving the rule generation problem of SOS. A deep multiagent RL algorithm was devised to train agents to acquire the task and self-organizing knowledge. However, the simulation was based on one specific task environment. Sensitivity of SOS to reward functions and systematic evaluation of SOS designed with multiagent RL remain an issue. In this paper, we introduced a rotation reward function to regulate agent behaviors during training and tested different weights of such reward on SOS performance in two case studies: box-pushing and T-shape assembly. Additionally, we proposed three metrics to evaluate the SOS: learning stability, quality of learned knowledge, and scalability. Results show that depending on the type of tasks; designers may choose appropriate weights of rotation reward to obtain the full potential of agents’ learning capability. Good learning stability and quality of knowledge can be achieved with an optimal range of team sizes. Scaling up to larger team sizes has better performance than scaling downwards.

Searching collaborative agents for multi-plane localization in 3D ultrasound.

3D ultrasound (US) has become prevalent due to its rich spatial and diagnostic information not contained in 2D US. Moreover, 3D US can contain multiple standard planes (SPs) in one shot. Thus, automatically localizing SPs in 3D US has the potential to improve user-independence and scanning-efficiency. However, manual SP localization in 3D US is challenging because of the low image quality, huge search space and large anatomical variability. In this work, we propose a novel multi-agent reinforcement learning (MARL) framework to simultaneously localize multiple SPs in 3D US. Our contribution is four-fold. First, our proposed method is general and it can accurately localize multiple SPs in different challenging US datasets. Second, we equip the MARL system with a recurrent neural network (RNN) based collaborative module, which can strengthen the communication among agents and learn the spatial relationship among planes effectively. Third, we explore to adopt the neural architecture search (NAS) to automatically design the network architecture of both the agents and the collaborative module. Last, we believe we are the first to realize automatic SP localization in pelvic US volumes, and note that our approach can handle both normal and abnormal uterus cases. Extensively validated on two challenging datasets of the uterus and fetal brain, our proposed method achieves the average localization accuracy of 7.03∘/1.59mm and 9.75∘/1.19mm. Experimental results show that our light-weight MARL model has higher accuracy than state-of-the-art methods.

Distributed Learning for Vehicle Routing Decision in Software Defined Internet of Vehicles

With the increasing number of vehicles, the traffic congestion is becoming more and more serious. In order to alleviate such a problem, this article considers transmission and inference delay of cloud centralized computing in the software defined Internet of Vehicles (SDIoV), and builds a new SDIoV architecture based on edge intelligence, for supporting real-time vehicle routing decision through distributed multi-agent reinforcement learning model. Then, a software defined device collaboration optimization method is designed to improve the efficiency of distributed training. Combined with multi-agent reinforcement learning, a distributed-learning-based vehicle routing decision algorithm (DLRD) is proposed to adaptively adjust vehicle routing online. The performed simulations show that the DLRD can successfully realize real-time routing decision for vehicles and alleviate traffic congestion with the dynamic changes of the road environment.

Stock Market Forecasting from Multi-Source Data using Tolerance Based Multi-Agent Deep Reinforcement Learning

Analyzing and forecasting the future trends in stock market is challenging due to the ever increasing size of stock data. Modern techniques extract the stock indicators from the web data to forecast the stock movements. However, most previous studies were based on single source of data for extracting these indicators. This might not be effective in obtaining all the possible diverse factors that influence the market movements. Multi-source data has been rarely applied for stock prediction and even those techniques have limitations in handling larger data. In an attempt to utilize multi-source data more effectively for extracting stock indicators and improve the forecasting accuracy of stock movements, this paper developed a stock market forecasting model using Tolerance based Multi-Agent Deep Reinforcement Learning (TMA-DRL) model. The TMA-DRL model effectively combines the quantitative stock data with the indicators i.e. the events extracted from news data and sentiments extracted from tweets. This forecasting model utilizes Random forests to extract the twitter opinions and Restricted Boltzmann Machine (RBM) for event extraction from news data. Combining these indicators, the TMA-DRL model leads to improved data learning and provides highly accurate prediction of future stock trends. Datasets for evaluation were collected from three sources namely Twitter, Market News and Stock exchange, for 12 months period. Evaluation results illustrate the effectiveness of the proposed TMA-DRL stock market forecasting model which makes predictions with high accuracy and less time complexity.

Emergence of linguistic conventions in multi-agent reinforcement learning.

Recently, emergence of signaling conventions, among which language is a prime example, draws a considerable interdisciplinary interest ranging from game theory, to robotics to evolutionary linguistics. Such a wide spectrum of research is based on much different assumptions and methodologies, but complexity of the problem precludes formulation of a unifying and commonly accepted explanation. We examine formation of signaling conventions in a framework of a multi-agent reinforcement learning model. When the network of interactions between agents is a complete graph or a sufficiently dense random graph, a global consensus is typically reached with the emerging language being a nearly unique object-word mapping or containing some synonyms and homonyms. On finite-dimensional lattices, the model gets trapped in disordered configurations with a local consensus only. Such a trapping can be avoided by introducing a population renewal, which in the presence of superlinear reinforcement restores an ordinary surface-tension driven coarsening and considerably enhances formation of efficient signaling.

Emergent behaviors and scalability for multi-agent reinforcement learning-based pedestrian models

This paper analyzes the emergent behaviors of pedestrian groups that learn through the multiagent reinforcement learning model developed in our group. Five scenarios studied in the pedestrian model literature, and with different levels of complexity, were simulated in order to analyze the robustness and the scalability of the model. Firstly, a reduced group of agents must learn by interaction with the environment in each scenario. In this phase, each agent learns its own kinematic controller, that will drive it at a simulation time. Secondly, the number of simulated agents is increased, in each scenario where agents have previously learnt, to test the appearance of emergent macroscopic behaviors without additional learning. This strategy allows us to evaluate the robustness and the consistency and quality of the learned behaviors. For this purpose several tools from pedestrian dynamics, such as fundamental diagrams and density maps, are used. The results reveal that the developed model is capable of simulating human-like micro and macro pedestrian behaviors for the simulation scenarios studied, including those where the number of pedestrians has been scaled by one order of magnitude with respect to the situation learned.

A multi-agent reinforcement learning approach to dynamic service composition

As a promising implementation of software systems, service composition has attracted significant attention and research, which constructs complex and value-added applications by composing existing single services to reduce the deployment time and cost. However, as the services on the Internet and the external environment are frequently changeable, these demand the service composition must be adaptive and dynamic to address these changes automatically. Therefore, this paper describes a multi-agent reinforcement learning model for the dynamic optimization of web service composition. In this model, agent can utilize reinforcement learning algorithms to interact with environment in real time to compute optimal composition strategy dynamically, and multi-agents mechanism can keep higher effectiveness in contrast to single-agent reinforcement learning. We propose a distributed Q-learning algorithm, which decompose the task into many sub-tasks and make every agent focus on own sub-task, to accelerate the convergence rate. In addition, we also introduce experience sharing strategy to improve the efficiency. As a result, these methods allow composite service to dynamically adjust itself to fit a varying environment, where the properties of the component services continue changing. Finally, a series of comparable experiments with traditional Q-learning algorithm demonstrate that our algorithms have certain validity, higher efficiency and obvious advantages.

Effective service composition using multi-agent reinforcement learning

As online services may keep evolving, service composition should maintain certain adaptivity especially for a dynamic composition environment. Meanwhile, the large number of potential candidate services poses scalability concerns, which demand efficient composition solutions. This paper presents a multi-agent reinforcement learning model for Web service composition that effectively addresses the above challenges. In particular, we model a service composition as a Markov Decision Process. Based on the model, agents in a team would benefit from one another. In contrast to single-agent reinforcement-learning, our method can speed up the convergence to an optimal policy. We develop two multi-agent reinforcement learning algorithms. The first one introduces the concept of articulate state and distributed Q-learning to speed up the convergence time. The second one proposes the experience sharing strategy to improve the efficiency of learning. As the learning process continues throughout the life-cycle of a service composition, our algorithms can automatically adapt to the change of environment and the evolving component services. We conduct a simulation study to compare our algorithm with other similar reinforcement learning algorithms, including the traditional Q-learning algorithm, a multi-agent Sarsa algorithm, a Q-learning algorithm based on gaussian process, and a multi-agent Q-learning algorithm, to justify the effectiveness of our model and algorithm.

Multi-agent Reinforcement Learning for Route Guidance System

Nowadays, multi-agent systems are used to create applications in a variety of areas, including economics, management, transportation, telecommunications, etc. Importantly, in many domains, the reinforcement learning agents try to learn a task by directly interacting with its environment. The main challenge in route guidance system is to direct vehicles to their destination in a dynamic traffic situation, with the aim of reducing travel times and ensuring efficient use of available road network capacity. This paper proposes a multi-agent reinforcement learning algorithm to find the best and shortest path between the origin and destination nodes. The shortest path such as the lowest cost is calculated using multi-agent reinforcement learning model and it will be suggested to the vehicle drivers in a route guidance system. The proposed algorithm has been evaluated based on Dijkstra's algorithm to find the optimal solution using Kuala Lumpur (KL) road network map. A number of route cases have been used to evaluate the proposed approach based on the road network problems. Finally, the experiment results demonstrate that the proposed approach is feasible and efficient.