Multi-agent Reinforcement Learning Approach Research Articles

Beyond the matrix: Experimental approaches to studying cognitive agents in social-ecological systems

Social-ecological systems, in which agents interact with each other and their environment are important both for sustainability applications and for under- standing how human cognition functions in context. In such systems, the en- vironment shapes the agents' experience and actions, and in turn collective action of agents changes social and physical aspects of the environment. Here we review current investigation approaches, which rely on a lean design, with discrete actions and outcomes and little scope for varying environmental pa- rameters and cognitive demands. We then introduce multiagent reinforcement learning (MARL) approach, which builds on modern artificial intelligence tech- niques, which provides new avenues to model complex social worlds, while pre- serving more of their characteristics, and allowing them to capture a variety of social phenomena. These techniques can be fed back to the laboratory where they make it easier to design experiments in complex social situations without compromising their tractability for computational modeling. We showcase the potential MARL by discussing several recent studies that have used it, detail- ing the way environmental settings and cognitive constraints can lead to the emergence of complex cooperation strategies. This novel approach can help re- searchers bring together insights from human cognition, sustainability, and AI, to tackle real world problems of social-ecological systems.

Effective Patient Scheduling For Public Health Events Based On Deep Reinforcement Learning

Amidst the current worldwide pandemic of the novel coronavirus, there are still issues about the efficient allocation of emergency supplies and shortcomings in recovery protocols. To enhance individuals' health and overall welfare, it is imperative to enhance and refine the emergency patient scheduling framework for health emergencies while upholding the idea of a society with an inherent future for humanity. The paper proposed an Efficient Patient Scheduling for Public Health Emergencies (EPS-PHE) using Multiagent Reinforcement Learning (MRL). Originally intended to solve the PHE and improve EPS, the technology was conceived as a distributed multiagent system. MRL has proposed a decentralized incentive system evaluation technique for the EPS-PHE input. An incentive type called a Direct Incentive (DI) is used to show the relationships between individuals or details about two closely connected events. The quality of data transmission in EPS-PHE or the efficacy of the emergency network is measured using a Collective Incentive (CI). Simulation research has shown the value of the Deep Learning (DL) - MRL approach in EPS-PHE framework optimization.

Decentralized Counterfactual Value with Threat Detection for Multi-Agent Reinforcement Learning in mixed cooperative and competitive environments

This paper proposes a fully decentralized approach to address the challenge of general mixed cooperation and competition within the domain of Multi-Agent Reinforcement Learning (MARL). Conventional MARL approaches do not achieve full decentralization as they necessitate either the communication of implicit information or the retention of a centralized critic, rendering them impractical in mixed cooperative and competitive environments. To address these challenges, this paper proposes a Decentralized Counterfactual Value (DCV) to model the behaviors of other agents and mitigate the non-stationary problem, accompanied by a Threat Detection (TD) mechanism to discern latent competitive or cooperative relationships. In addition, DCVTD is incorporated into both value-based and policy-based RL paradigms with theoretical convergence guarantee. Finally, empirical validation across four representative environments demonstrates the superior performance of DCVTD in terms of collective returns, computational efficiency, and agent scalability over other fully decentralized approaches, centralized training with decentralized execution approaches, and alternative approaches involving agent modeling or reward shaping in comprehensive experiments.

XRL-FlexSBR: Multi-agent reinforcement learning-driven flexible SBR control with explainable performance guarantee under diverse influent conditions

The sequencing batch reactor (SBR) process stands out for its small footprint and operational flexibility. However, the SBR process is highly nonlinear and subject to influent disturbances. In this study, we suggested an explainable multi-agent reinforcement learning (XRL) approach coupled with multi-agent reinforcement learning (MARL) and explainable AI (XAI); then, an XRL-driven flexible SBR control (XRL-FlexSBR) system was developed to conduct multivariate control the SBR process autonomously. Influent big datasets including biochemical oxygen demand (BOD) and total nitrogen (TN) were collected from the wastewater treatment plants (WWTPs) of South Korea. Then, the Gaussian mixture model was utilized to cluster the diverse influent conditions and the SBR mechanistic model was developed. A game abstraction method based on a two-stage attention network (G2ANET), one of MARL algorithms, was employed to manipulate dissolved oxygen (DO) and extra carbon injection (EC) controllers in the SBR process; furthermore, layer-wise relevance propagation (LRP) of explainable AI (XAI) technique was utilized to evaluate a control performance guarantee by G2ANET. The results verified that XRL-FlexSBR can control the DO and EC controllers in the SBR process while reducing the energy consumption by 4.93 % on average and maintaining effluent quality criteria across the diverse influent conditions. Furthermore, XAI explained that the improved control performance of XRL-FlexSBR agents is attributed to their understanding of the mechanism underlying SBR operations without human intervention. Hence the proposed XRL-FlexSBR can flexibly control the SBR to improve sustainability and profitability under varying influent conditions.

Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach

In the context of the expanding diversity of energy demands, an increasing number of heterogeneous Multi-energy Microgrids (MEMGs) are engaging in the collaborative framework of the Multi-energy Multi-microgrid System (MEMMG). However, following this trend, the existing centralized Integrated Energy Management System (IEMS) control strategy is unreliable for future energy systems, characterized by more complex optimization control and a flexible system structure. This paper introduces a hierarchical Multi-agent Deep Reinforcement Learning (HMADRL) approach for distributed IEMS in MEMMG. Firstly, by employing a hierarchical approach, this method simplifies control complexity by segmenting the overarching control challenge into tasks of collaborative planning and action control, which are distributed across varied multi-agent scenes. Considering both macro and microeconomic factors, alongside carbon emissions, the optimal operation of MEMMG is realized through a bottom-up edge multi-agent control approach, in contrast to traditional top-down centralized methods. Secondly, in the phase of the inter-MEMG collaborative strategy, the Centralized Training Decentralized Execution (CTDE) framework is adopted to overcome the problems of unstable training environments and large-scale agent training, and each heterogeneous microgrid can develop local strategies independently with the assurance that their internal data will not be overly exposed. Thirdly, within each MEMG, the Trust-Region (TR) model is introduced for multi-agent action control, adeptly addressing the effects of mutual exclusion in decision-making time series. Simultaneously, an initialized Hot Experience Pool (HEP) is implemented, efficiently reducing exploration in complex, high-dimensional spaces. Finally, the off-time agent model is integrated into the HMADRL environment and undergoes secondary training based on real interactions, thereby deriving the optimal energy management policy. The proposed method markedly reduces reliance on exact physical modeling systems. Comparative simulations validate the proposed control scheme’s efficacy.

Computation Rate Maximization for SCMA-Aided Edge Computing in IoT Networks: A Multi-Agent Reinforcement Learning Approach

Computation Rate Maximization for SCMA-Aided Edge Computing in IoT Networks: A Multi-Agent Reinforcement Learning Approach

Multi-task scheduling in vehicular edge computing: a multi-agent reinforcement learning approach

Multi-task scheduling in vehicular edge computing: a multi-agent reinforcement learning approach

AMARL: An Attention-Based Multiagent Reinforcement Learning Approach to the Min-Max Multiple Traveling Salesmen Problem.

In recent years, the multiple traveling salesmen problem (MTSP or multiple TSP) has received increasing research interest and one of its main applications is coordinated multirobot mission planning, such as cooperative search and rescue tasks. However, it is still challenging to solve MTSP with improved inference efficiency as well as solution quality in varying situations, e.g., different city positions, different numbers of cities, or agents. In this article, we propose an attention-based multiagent reinforcement learning (AMARL) approach, which is based on the gated transformer feature representations for min-max multiple TSPs. The state feature extraction network in our proposed approach adopts the gated transformer architecture with reordering layer normalization (LN) and a new gate mechanism. It aggregates fixed-dimensional attention-based state features irrespective of the number of agents and cities. The action space of our proposed approach is designed to decouple the interaction of agents' simultaneous decision-making. At each time step, only one agent is assigned to a non-zero action so that the action selection strategy can be transferred across tasks with different numbers of agents and cities. Extensive experiments on min-max multiple TSPs were conducted to illustrate the effectiveness and advantages of the proposed approach. Compared with six representative algorithms, our proposed approach achieves state-of-the-art performance in solution quality and inference efficiency. In particular, the proposed approach is suitable for tasks with different numbers of agents or cities without extra learning, and experimental results demonstrate that the proposed approach realizes powerful transfer capability across tasks.

Competitive pricing for ride-sourcing platforms with MARL

With the advancement of autonomous driving, the competition between shared autonomous vehicles (SAVs) and the existing human-driven vehicles (HVs) ride-sourcing platforms has emerged as a prominent research area within the fields of transportation and management. Solving the pricing problem, considering the competition between HV and SAV platforms, is critical for the platform’s operation. The competitive pricing problem is hard to solve since it is a partially observable Markov decision process (POMDP), that is, only the strategies of other platforms (such as price and wage) are observable, while their states are unobservable. It is also a multivariable sequential problem pricing since spatial-temporal pricing varies with region and time. To tackle this complex problem, this paper implements the multi-agent reinforcement learning (MARL) approach of multi-agent deep deterministic policy gradient (MADDPG) to an mesoscopic simulation system. The simulation replicates the intricate dynamics within a competitive environment where SAV and HV platforms coexist. MADDPG effectively resolves the competitive pricing issue using data derived from simulation. The findings indicate that the SAV platform secures greater profits with a smaller fleet, as it eliminates the need to compensate drivers. Meanwhile, for the same reason, even with fewer vehicles, the SAV platform can set lower prices to attract more passengers and thereby generate higher profits. We further evaluate the efficacy of dynamic pricing versus spatial-temporal pricing and discover that spatial-temporal pricing yields greater profits.

Age of information minimization in UAV-assisted data harvesting networks by multi-agent deep reinforcement curriculum learning

Unmanned Aerial Vehicles (UAVs) are increasingly being used for data harvesting from Wireless Sensor Nodes (SNs). This study aims to minimize the Age of Information (AoI) during data collection, while also considering the energy sustainability of the UAVs. Addressing the challenge of optimizing performance in Multi-Agent Reinforcement Learning (MARL) systems as the number of agents increases, our study introduces a MARL approach combined with curriculum learning and evolutionary strategy. This innovation specifically targets the performance issue in traditional MARL setups when scaling up the number of agents. By applying curriculum learning and evolutionary strategies, our method not only enhances MARL scalability but also integrates energy-efficient charging mechanisms, effectively enhancing system performance in large-scale deployments. Numerical results show that our proposed algorithm outperforms baselines in terms of AoI and charging, proving its effectiveness in managing the complexities of large-scale MARL systems.

Multi-agent deep reinforcement learning with enhanced collaboration for distribution network voltage control

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses.

Integrated Velocity Optimization and Energy Management Strategy for Hybrid Electric Vehicle Platoon: A Multiagent Reinforcement Learning Approach

Integrated Velocity Optimization and Energy Management Strategy for Hybrid Electric Vehicle Platoon: A Multiagent Reinforcement Learning Approach

A multi-agent reinforcement learning approach for ART adaptive control in automated container terminals

Intelligent transportation equipment represented by Artificial Intelligence Robot of Transportation (ART) has been gradually applied to automated container terminals (ACT). ARTs have higher intelligence and distributed autonomous decision-making capability compared to AGVs. The production and operation of ACT require strict loading plans based on container types and destination ports, which puts strict requirements on the ART loading task sequence. ART travel time from the yard to the quay is highly uncertain due to special scenarios such as traffic flow convergence and turning in narrow areas, which will lead to long waits for ART due to out-of-sequence arrivals at the quay. Therefore, it is necessary to regulate the speed of ART to adjust the timing of ART arriving at the quay. However, existing scheduling systems tend to ignore the impact of ART speed and loading sequence constraints on terminal operations. To optimize the multi-ART control problem of ACT, the waiting time of ART at the quay is taken as the optimization goal, and the speed regulation of ART during each task is defined as a Markov decision process. Then, by incorporating reinforcement learning and a multi-agent approach, an adaptive decision-making method is developed to generate the optimal speed regulation strategy. This method interacts with the multi-agent simulation environment to obtain real-time status information and learns strategies online to improve the intelligence of individual agents and the collaborative capabilities of the agent group. Finally, a multi-agent simulation model embedded with a reinforcement learning agent is established on AnyLogic software. Simulation results show that this method can effectively reduce the waiting time of ART, improve the coherence and efficiency of loading operations, and exhibit good performance in online scenarios that consider emergency tasks. The results indicate that the speed of ART in ACTs with parallel layout of the yard can impact the operational efficiency of the system. Furthermore, by appropriately defining the reward function and training method, collaborative behaviour among equipment can be enhanced, ultimately optimizing the overall performance of the system.

Synergizing Transfer Learning and Multi-Agent Systems for Thermal Parametrization in Induction Traction Motors

Maintaining optimal temperatures in the critical parts of an induction traction motor is crucial for railway propulsion systems. A reduced-order lumped-parameter thermal network (LPTN) model enables computably inexpensive, accurate temperature estimation; however, it requires empirically based parameter estimation exercises. The calibration process is typically performed in labs in a controlled experimental setting, which is associated with a lot of supervised human efforts. However, the exploration of machine learning (ML) techniques in varied domains has enabled the model parameterization in the drive system outside the laboratory settings. This paper presents an innovative use of a multi-agent reinforcement learning (MARL) approach for the parametrization of an LPTN model. First, a set of reinforcement learning agents are trained to estimate the optimized thermal parameters using the simulated data in several driving cycles (DCs). The selection of a reinforcement learning agent and the level of neurons in the RL model is made based on variability of the driving cycle data. Furthermore, transfer learning is performed on a new driving cycle data collected on the measurement setup. Statistical analysis and clustering techniques are proposed for the selection of an RL agent that has been pre-trained on the historical data. It is established that by synergizing within reinforcement learning techniques, it is possible to refine and adjust the RL learning models to effectively capture the complexities of thermal dynamics. The proposed MARL framework shows its capability to accurately reflect the motor’s thermal behavior under various driving conditions. The transfer learning usage in the proposed approach could yield significant improvement in the accuracy of temperature prediction in the new driving cycles data. This approach is proposed with the aim of developing more adaptive and efficient thermal management strategies for railway propulsion systems.

Competitive Pricing for Resource Trading in Sliced Mobile Networks: A Multi-Agent Reinforcement Learning Approach

The emergence of network slicing as a flagship technology in 5G networks has not only enhanced network expansion and flexibility in resource management for service continuity, but also provided an avenue for establishing a viable market for resource sharing. To optimize the network's resource usage, stakeholders are encouraged to take pragmatic steps toward dynamic resource sharing. This paper designs a techno-economic model for the strategic interactions among multiple competing mobile virtual network operators (MVNOs) and their users in a trading marketplace. We formulate the dynamic pricing problem as a two-stage Stackelberg game, where the MVNOs are leaders, and the users are followers. In the first stage, the MVNOs compete to set their differentiated unit prices using a negotiation mechanism while considering system-level network load. Then, the users decide their purchasing volumes to match the prices of the MVNOs. We transform the game-based optimization problem into a stochastic Markov decision process (MDP) problem and propose a multi-agent deep Q-network (MADQN) method that obtains an optimal solution for the formulated game. Simulation results and analysis reveal that the proposed algorithm achieves convergence under the competitive pricing scheme (CPS) and independent pricing scheme (IPS) while enhancing MVNOs and users' utilities at acceptable levels.

Adaptive Incentive for Cross-Silo Federated Learning in IIoT: A Multiagent Reinforcement Learning Approach

Adaptive Incentive for Cross-Silo Federated Learning in IIoT: A Multiagent Reinforcement Learning Approach

A Multi-Agent Reinforcement Learning Approach for Safe and Efficient Behavior Planning of Connected Autonomous Vehicles

A Multi-Agent Reinforcement Learning Approach for Safe and Efficient Behavior Planning of Connected Autonomous Vehicles

Computation Offloading and Trajectory Planning of Multi-UAV-Enabled MEC: A Knowledge-Assisted Multiagent Reinforcement Learning Approach

Computation Offloading and Trajectory Planning of Multi-UAV-Enabled MEC: A Knowledge-Assisted Multiagent Reinforcement Learning Approach

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

A Multi-Agent Reinforcement Learning-Based Grant-Free Random Access Protocol for mMTC Massive MIMO Networks

The expected huge number of connected devices in Internet of Things (IoT) applications characterizes the massive machine-type communication (mMTC) scenario, one prominent use case of beyond fifth-generation (B5G) systems. To meet mMTC connectivity requirements, grant-free (GF) random access (RA) protocols are seen as a promising solution due to the small amount of data that MTC devices usually transmit. In this paper, we propose a GF RA protocol based on a multi-agent reinforcement learning approach, applied to aid IoT devices in selecting the least congested RA pilots. The rewards obtained by the devices in collision cases resemble the congestion level of the chosen pilot. To enable the operation of the proposed method in a realistic B5G network scenario and aiming to reduce signaling overheads and centralized processing, the rewards in our proposed method are computed by the devices taking advantage of a large number of base station antennas. Numerical results demonstrate the superior performance of the proposed method in terms of latency, network throughput, and per-device throughput compared with other protocols.