Multi-agent Reinforcement Learning Research Articles

CESDQL: Communicative experience-sharing deep Q-learning for scalability in multi-robot collaboration with sparse reward

Owing to the massive transformation in industrial processes and logistics, warehouses are also undergoing advanced automation. The application of Autonomous Mobile Robots (a.k.a. multi-robots) is one of the important elements of overall warehousing automation. The autonomous collaborative behaviour of the multi-robots can be considered as employment on a control task and, thus, can be optimised using multi-agent reinforcement learning (MARL). Consequently, an autonomous warehouse is to be represented by an MARL environment. An MARL environment replicating an autonomous warehouse poses the challenge of exploration due to sparse reward leading to inefficient collaboration. This challenge aggravates further with an increase in the number of robots and the grid size, i.e., scalability. This research proposes Communicative Experience-Sharing Deep Q-Learning (CESDQL) based on Q-learning, a novel hybrid multi-robot communicative framework for scalability for MARL collaboration with sparse rewards, where exploration is challenging and makes collaboration difficult. CESDQL makes use of experience-sharing through collective sampling from the Experience (Replay) buffer and communication through Communicative Deep recurrent Q-network (CommDRQN), a Q-function approximator. Through empirical evaluation of CESDQL in a variety of collaborative scenarios, it is established that CESDQL outperforms the baselines in terms of convergence and stable learning. Overall, CESDQL achieves 5%, 69%, 60%, 211%, 171%, 3.8% & 10% more final accumulative training returns than the closest performing baseline by scenario, and, 27%, 10.33% & 573% more final average training returns than the closest performing baseline by the big-scale scenario.

Feature selection integrating Shapley values and mutual information in reinforcement learning: An application in the prediction of post-operative outcomes in patients with end-stage renal disease

Background:In predicting post-operative outcomes for patients with end-stage renal disease, our study faced challenges related to class imbalance and a high-dimensional feature space. Therefore, with a focus on overcoming class imbalance and improving interpretability, we propose a novel feature selection approach using multi-agent reinforcement learning. Methods:We proposed a multi-agent feature selection model based on a comprehensive reward function that combines classification model performance, Shapley additive explanations values, and the mutual information. The definition of rewards in reinforcement learning is crucial for model convergence and performance improvement. Initially, we set a deterministic reward based on the mutual information between variables and the target class, selecting variables that are highly dependent on the class, thus accelerating convergence. We then prioritised variables that influence the minority class on a sample basis and introduced a dynamic reward distribution strategy using shapley additive explanations values to improve interpretability and solve the class imbalance problem. Results:Involving the integration of electronic medical records, anesthesia records, operating room vital signs, and pre-operative anesthesia evaluations, our approach effectively mitigated class imbalance and demonstrated superior performance in ablation analysis. Our model achieved a 16% increase in the minority class F1 score and an 8.2% increase in the overall F1 score compared to the baseline model without feature selection. Conclusion:This study contributes important research findings that show that the multi-agent-based feature selection method can be a promising approach for solving the class imbalance problem.

Cooperative price-based demand response program for multiple aggregators based on multi-agent reinforcement learning and shapley-value

Demand response (DR) plays an essential role in power system management. To facilitate the implementation of these techniques, many aggregators have appeared in response as new mediating entities in the electricity market. These actors exploit the technologies to engage customers in DR programs, offering grid services like load scheduling. However, the growing number of aggregators has become a new challenge, making it difficult for utilities to manage the load scheduling problem. This paper presents a multi-agent reinforcement Learning (MARL) approach to a price-based DR program for multiple aggregators. A dynamic pricing scheme based on discounts is proposed to encourage residential customers to change their consumption patterns. This strategy is based on a cooperative framework for a set of DR Aggregators (DRAs). The DRAs take advantage of a reward offered by a Distribution System Operator (DSO) for performing a peak-shaving over the total system aggregated demand. Furthermore, a Shapley-Value-based reward sharing mechanism is implemented to fairly determine the individual contribution and calculate the individual reward for each DRA. Simulation results verify the merits of the proposed model for a multi-aggregator system, improving DRAs’ pricing strategies considering the overall objectives of the system. Consumption peaks were managed by reducing the Peak-to-Average Ratio (PAR) by 15%, and the MARL mechanism’s performance was improved in terms of reward function maximization and convergence time, the latter being reduced by 29%.

Efficient multi-agent reinforcement learning HVAC power consumption optimization

Efficient multi-agent reinforcement learning HVAC power consumption optimization

A collaborative-learning multi-agent reinforcement learning method for distributed hybrid flow shop scheduling problem

As the increasing level of implementation of artificial intelligence technology in solving complex engineering optimization problems, various learning mechanisms, including deep learning (DL) and reinforcement learning (RL), have been developed for manufacturing scheduling. In this paper, a collaborative-learning multi-agent RL method (CL-MARL) is proposed for solving distributed hybrid flow-shop scheduling problem (DHFSP), minimizing both makespan and total energy consumption. First, the DHFSP is formulated as the Markov decision process, the features of machines and jobs are represented as state and observation matrixes according to their characteristics, the candidate operation set is used as action space, and a reward mechanism is designed based on the machine utilization. Next, a set of critic networks and actor networks, consist of recurrent neural networks and fully connected networks, are employed to map the states and observations into the output values. Then, a novel distance matching strategy is designed for each agent to select the most appropriate action at each scheduling step. Finally, the proposed CL-MARL model is trained through multi-agent deep deterministic policy gradient algorithm in collaborative-learning manner. The numerical results prove the effectiveness of the proposed multi-agent system, and the comparisons with existing algorithms demonstrate the high-potential of CL-MARL in solving DHFSP.

Real-time bidding with multi-agent reinforcement learning in multi-channel display advertising

Real-time bidding with multi-agent reinforcement learning in multi-channel display advertising

Robust dynamic real-time control strategies for high-frequency bus service: a multi-agent reinforcement learning framework

Robust dynamic real-time control strategies for high-frequency bus service: a multi-agent reinforcement learning framework

High-Sample-Efficient Multiagent Reinforcement Learning for Navigation and Collision Avoidance of UAV Swarms in Multitask Environments

High-Sample-Efficient Multiagent Reinforcement Learning for Navigation and Collision Avoidance of UAV Swarms in Multitask Environments

Adaptive Multi-layer Attention Double Dueling Deep Q-Network for Muti-agent Reinforcement Learning

Abstract. In multi-agent fields, traditional muti-agent DQN methods often suffer from overestimation bias and overestimation of unimportant actions, especially when state-action Q-value differences are slight. To deal with such issue, we present an adaptive Multi-layer Attention Double Dueling Deep Q-Network (MAD-D3QN) model, aiming to improve decision-making accuracy in complex multi-agent environments. The proposed model utilizes two attention layers that dynamically calculate state value and action advantage weights, facilitating more precise Q-value estimation and reducing the common overestimation bias. Related experiments carried out in StarWar II scenarios show that the MAD-D3QN model obviously outperforms traditional methods (IQL,DQN), achieving higher decision efficiency and robustness. Our findings demonstrates that the MAD-D3QN framework not only promotes the state-of-the-art in multi-agent reinforcement learning but also provides potential applications in real-world cooperative tasks. Future research will delve into the integration of advanced multi-agent communication structures to further enhance model adaptability.

Autonomous ecologies of construction: Collaborative modular robotic material eco-systems with deep multi-agent reinforcement learning

To address the unprecedented challenges of construction pressurized by the global climate crisis, housing shortage, and growing shortage of skilled labor, this research presents a radical shift in the construction lifecycle of buildings, from linear processes that produce static continuous buildings to interrelational processes linking adaptative eco-systems of collaborative robots and reconfigurable building parts. Inspired by natural builders, the interdisciplinary field of collective robotic construction (CRC) offers the potential for scalable, adaptive, and resilient construction with simple robots. We establish a design framework for autonomous collaborative robotic construction (ACRC) through modular robotic material eco-systems (MRMES) trained with deep multi-agent reinforcement learning (DMARL). This involves the integration of three core aspects: (1) modular robotic material eco-systems (2) cyber-physical simulation and control with bidirectional feedback (3) adaptive intelligence through deep multi-agent reinforcement learning. The framework is implemented through three comparable case studies for collaborative modular robotic assembly of reconfigurable building parts.

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning

In recent years, reinforcement learning (RL) methods have shown powerful learning capabilities in single-vehicle autonomous driving. However, few studies have focused on multi-vehicle cooperative driving based on RL, particularly in the dynamically changing traffic environments of highway ramp merge zones. In this paper, a multi-agent deep reinforcement learning (MARL) framework for multi-vehicle cooperative decision-making is proposed based on actor–critic, which categorizes vehicles into two groups according to their origins in the merging area. At the same time, the complexity of the network is reduced and the training process of the model is accelerated by utilizing mechanisms such as partial parameter sharing and experience playback. Additionally, a combination of global and individual rewards is adopted to promote cooperation in connected autonomous vehicles (CAVs) and balance individual and group interests. The training performance of the model is compared under three traffic densities, and our method is also compared with state-of-the-art benchmark methods. The simulation results show that the proposed MARL framework can have stronger policy learning capability and stability under various traffic flow conditions. Moreover, it can also effectively improve the speed of vehicles in the merging zone and reduce traffic conflicts.

Message Action Adapter Framework in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has demonstrated significant potential in enabling cooperative agents. The communication protocol, which is responsible for message exchange between agents, is crucial in cooperation. However, communicative MARL systems still face challenges due to the noisy messages in complex multi-agent decision processes. This issue often stems from the entangled representation of observations and messages in policy networks. To address this, we propose the Message Action Adapter Framework (MAAF), which first trains individual agents without message inputs and then adapts a residual action based on message components. This separation isolates the effect of messages on action inference. We explore how training the MAAF framework with model-agnostic message types and varying optimization strategies influences adaptation performance. The experimental results indicate that MAAF achieves competitive performance across multiple baselines despite utilizing only half of the available communication, and shows an average improvement of 7.6% over the full attention-based communication approach. Additional findings suggest that different message types result in significant performance variations, emphasizing the importance of environment-specific message types. We demonstrate how the proposed architecture separates communication channels, effectively isolating message contributions.

Integrated Early Flood Prediction using Sentinel-2 Imagery with VANET-MARL-based Deep Neural RNN

<p style="line-height:200%;"><span style="font-family:"Times New Roman","serif";font-size:12.0pt;line-height:200%;" lang="EN-US">Floods are a major contributor to the destruction of infrastructure and the overall economy of afflicted countries, leading to loss of life and significant damage. Remote sensing, satellite imagery photography, global positioning system, and geographic information system (GIS) are commonly used to identify floods and analyze the associated damages. The research presented here integrates Sentinel-2 satellite images, VANET with Multi-Agent Reinforcement Learning (MARL), and a deep neural RNN for early flood prediction. Sentinel-2 imaging delivers extensive geographical and temporal data regarding land cover and water bodies, while VANET-MARL provides real-time ground-truth information and distributed decision-making capabilities. The Deep Neural RNN efficiently acquires intricate patterns from the combined data to forecast the likelihood, intensity, and scope of floods. The experimental findings clearly show that the suggested system outperforms standard methods in terms of accuracy, precision, recall, and lead time. VANET-MARL integration improves the system's ability to adapt and remain strong in changing circumstances. The results showed that the method was 94.8% accurate in early predicting floods.</span></p>

Skill matters: Dynamic skill learning for multi-agent cooperative reinforcement learning

With the popularization of intelligence, the necessity of cooperation between intelligent machines makes the research of collaborative multi-agent reinforcement learning (MARL) more extensive. Existing approaches typically address this challenge through task decomposition of the environment or role classification of agents. However, these studies may rely on the sharing of parameters between agents, resulting in the homogeneity of agent behavior, which is not effective for complex tasks. Or training that relies on external rewards is difficult to adapt to scenarios with sparse rewards. Based on the above challenges, in this paper we propose a novel dynamic skill learning (DSL) framework for agents to learn more diverse abilities motivated by internal rewards. Specifically, the DSL has two components: (i) Dynamic skill discovery, which encourages the production of meaningful skills by exploring the environment in an unsupervised manner, using the inner product between a skill vector and a trajectory representation to generate intrinsic rewards. Meanwhile, the Lipschitz constraint of the state representation function is used to ensure the proper trajectory of the learned skills. (ii) Dynamic skill assignment, which utilizes a policy controller to assign skills to each agent based on its different trajectory latent variables. In addition, in order to avoid training instability caused by frequent changes in skill selection, we introduce a regularization term to limit skill switching between adjacent time steps. We thoroughly tested the DSL approach on two challenging benchmarks, StarCraft II and Google Research Football. Experimental results show that compared with strong benchmarks such as QMIX and RODE, DSL effectively improves performance and is more adaptable to difficult collaborative scenarios.

Dynamic Integrated Scheduling of Production Equipment and Automated Guided Vehicles in a Flexible Job Shop Based on Deep Reinforcement Learning

The high-quality development of the manufacturing industry necessitates accelerating its transformation towards high-end, intelligent, and green development. Considering logistics resource constraints, the impact of dynamic disturbance events on production, and the need for energy-efficient production, the integrated scheduling of production equipment and automated guided vehicles (AGVs) in a flexible job shop environment is investigated in this study. Firstly, a static model for the integrated scheduling of production equipment and AGVs (ISPEA) is developed based on mixed-integer programming, which aims to optimize the maximum completion time and total production energy consumption (EC). In recent years, reinforcement learning, including deep reinforcement learning (DRL), has demonstrated significant advantages in handling workshop scheduling issues with sequential decision-making characteristics, which can fully utilize the vast quantity of historical data accumulated in the workshop and adjust production plans in a timely manner based on changes in production conditions and demand. Accordingly, a DRL-based approach is introduced to address the common production disturbances in emergency order insertions. Combined with the characteristics of the ISPEA problem and an event-driven strategy for handling dynamic events, four types of agents, namely workpiece selection, machine selection, AGV selection, and target selection agents, are set up, which refine workshop production status characteristics as observation inputs and generate rules for selecting workpieces, machines, AGVs, and targets. These agents are trained offline using the QMIX multi-agent reinforcement learning framework, and the trained agents are utilized to solve the dynamic ISPEA problem. Finally, the effectiveness of the proposed model and method is validated through a comparison of the solution performance with other typical optimization algorithms for various cases.

Multi-agent reinforcement learning with synchronized and decomposed reward automaton synthesized from reactive temporal logic

Multi-agent systems (MAS) consist of multiple autonomous agents interacting to achieve collective objectives. Multi-agent reinforcement learning (MARL) enhances these systems by enabling agents to learn optimal behaviors through interaction, thus improving their coordination in dynamic environments. However, MARL faces significant challenges in adapting to complex dependencies on past states and actions, which are not adequately represented by the current state alone in reactive systems. This paper addresses these challenges by considering MAS operating under task specifications formulated as Generalized Reactivity of rank 1 (GR(1)). These synthesized strategies are used as a priori knowledge to guide the learning. To tackle the difficulties of handling non-Markovian tasks in reactive systems, we propose a novel synchronized decentralized training paradigm that guides agents to learn within the MARL framework using a reward structure constructed from decomposed synthesized strategies of GR(1). We initially formalize the synthesis of GR(1) strategies as a reachability problem of winning states of the system. Subsequently, we develop a decomposition mechanism that constructs individual reward structures for decentralized MARL, incorporating potential values calculated through value iteration. Theoretical proofs are provided to verify that the safety and liveness properties are preserved. We evaluate our approach against other state-of-the-art methods under various GR(1) specifications and scenario maps, demonstrating superior learning efficacy and optimal rewards per episode. Additionally, we show that the decentralized training paradigm outperforms the centralized training paradigm. The value iteration strategy used to calculate potential values for the reward structure is compared against two other strategies, showcasing its advantages.

Adaptive multi-agent reinforcement learning for dynamic pricing and distributed energy management in virtual power plant networks

This paper presents a novel approach to dynamic pricing and distributed energy management in virtual power plant (VPP) networks using multi-agent reinforcement learning (MARL). As the energy landscape evolves towards greater decentralization and renewable integration, traditional optimization methods struggle to address the inherent complexities and uncertainties. Our proposed MARL framework enables adaptive, decentralized decision-making for both the distribution system operator (DSO) and individual VPPs, optimizing economic efficiency while maintaining grid stability. We formulate the problem as a Markov decision process and develop a custom MARL algorithm that leverages actor-critic architectures and experience replay. Extensive simulations across diverse scenarios demonstrate that our approach consistently outperforms baseline methods, including Stackelberg game models and model predictive control, achieving an 18.73% reduction in costs and a 22.46% increase in VPP profits. The MARL framework shows particular strength in scenarios with high renewable energy penetration, where it improves system performance by 11.95% compared with traditional methods. Furthermore, our approach demonstrates superior adaptability to unexpected events and mis-predictions, highlighting its potential for real-world implementation.

Selective Information Communication Considering the Influence of Each Other’s Messages in Multi-Agent Reinforcement Learning

Selective Information Communication Considering the Influence of Each Other’s Messages in Multi-Agent Reinforcement Learning

Coordinating multi-agent reinforcement learning via dual collaborative constraints

Many real-world multi-agent tasks exhibit a nearly decomposable structure, where interactions among agents within the same interaction set are strong while interactions between different sets are relatively weak. Efficiently modeling the nearly decomposable structure and leveraging it to coordinate agents can enhance the learning efficiency of multi-agent reinforcement learning algorithms for cooperative tasks, while existing works typically fail. To overcome this limitation, this paper proposes a novel algorithm named Dual Collaborative Constraints (DCC) that identifies the interaction sets as subtasks and achieves both intra-subtask and inter-subtask coordination. Specifically, DCC employs a bi-level structure to periodically distribute agents into multiple subtasks, and proposes both local and global collaborative constraints based on mutual information to facilitate both intra-subtask and inter-subtask coordination among agents. These two constraints ensure that agents within the same subtask reach a consensus on their local action selections and all of them select superior joint actions that maximize the overall task performance. Experimentally, we evaluate DCC on various cooperative multi-agent tasks, and its superior performance against multiple state-of-the-art baselines demonstrates its effectiveness.

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning