Multi-agent Reinforcement Learning Approach Research Articles

Synergizing Transfer Learning and Multi-Agent Systems for Thermal Parametrization in Induction Traction Motors

Maintaining optimal temperatures in the critical parts of an induction traction motor is crucial for railway propulsion systems. A reduced-order lumped-parameter thermal network (LPTN) model enables computably inexpensive, accurate temperature estimation; however, it requires empirically based parameter estimation exercises. The calibration process is typically performed in labs in a controlled experimental setting, which is associated with a lot of supervised human efforts. However, the exploration of machine learning (ML) techniques in varied domains has enabled the model parameterization in the drive system outside the laboratory settings. This paper presents an innovative use of a multi-agent reinforcement learning (MARL) approach for the parametrization of an LPTN model. First, a set of reinforcement learning agents are trained to estimate the optimized thermal parameters using the simulated data in several driving cycles (DCs). The selection of a reinforcement learning agent and the level of neurons in the RL model is made based on variability of the driving cycle data. Furthermore, transfer learning is performed on a new driving cycle data collected on the measurement setup. Statistical analysis and clustering techniques are proposed for the selection of an RL agent that has been pre-trained on the historical data. It is established that by synergizing within reinforcement learning techniques, it is possible to refine and adjust the RL learning models to effectively capture the complexities of thermal dynamics. The proposed MARL framework shows its capability to accurately reflect the motor’s thermal behavior under various driving conditions. The transfer learning usage in the proposed approach could yield significant improvement in the accuracy of temperature prediction in the new driving cycles data. This approach is proposed with the aim of developing more adaptive and efficient thermal management strategies for railway propulsion systems.

Competitive Pricing for Resource Trading in Sliced Mobile Networks: A Multi-Agent Reinforcement Learning Approach

The emergence of network slicing as a flagship technology in 5G networks has not only enhanced network expansion and flexibility in resource management for service continuity, but also provided an avenue for establishing a viable market for resource sharing. To optimize the network's resource usage, stakeholders are encouraged to take pragmatic steps toward dynamic resource sharing. This paper designs a techno-economic model for the strategic interactions among multiple competing mobile virtual network operators (MVNOs) and their users in a trading marketplace. We formulate the dynamic pricing problem as a two-stage Stackelberg game, where the MVNOs are leaders, and the users are followers. In the first stage, the MVNOs compete to set their differentiated unit prices using a negotiation mechanism while considering system-level network load. Then, the users decide their purchasing volumes to match the prices of the MVNOs. We transform the game-based optimization problem into a stochastic Markov decision process (MDP) problem and propose a multi-agent deep Q-network (MADQN) method that obtains an optimal solution for the formulated game. Simulation results and analysis reveal that the proposed algorithm achieves convergence under the competitive pricing scheme (CPS) and independent pricing scheme (IPS) while enhancing MVNOs and users' utilities at acceptable levels.

Adaptive Incentive for Cross-Silo Federated Learning in IIoT: A Multiagent Reinforcement Learning Approach

Adaptive Incentive for Cross-Silo Federated Learning in IIoT: A Multiagent Reinforcement Learning Approach

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

A Multi-Agent Reinforcement Learning-Based Grant-Free Random Access Protocol for mMTC Massive MIMO Networks

The expected huge number of connected devices in Internet of Things (IoT) applications characterizes the massive machine-type communication (mMTC) scenario, one prominent use case of beyond fifth-generation (B5G) systems. To meet mMTC connectivity requirements, grant-free (GF) random access (RA) protocols are seen as a promising solution due to the small amount of data that MTC devices usually transmit. In this paper, we propose a GF RA protocol based on a multi-agent reinforcement learning approach, applied to aid IoT devices in selecting the least congested RA pilots. The rewards obtained by the devices in collision cases resemble the congestion level of the chosen pilot. To enable the operation of the proposed method in a realistic B5G network scenario and aiming to reduce signaling overheads and centralized processing, the rewards in our proposed method are computed by the devices taking advantage of a large number of base station antennas. Numerical results demonstrate the superior performance of the proposed method in terms of latency, network throughput, and per-device throughput compared with other protocols.

A Hybrid Multi-Agent Reinforcement Learning Approach for Spectrum Sharing in Vehicular Networks

Efficient spectrum sharing is essential for maximizing data communication performance in Vehicular Networks (VNs). In this article, we propose a novel hybrid framework that leverages Multi-Agent Reinforcement Learning (MARL), thereby combining both centralized and decentralized learning approaches. This framework addresses scenarios where multiple vehicle-to-vehicle (V2V) links reuse the frequency spectrum preoccupied by vehicle-to-infrastructure (V2I) links. We introduce the QMIX technique with the Deep Q Networks (DQNs) algorithm to facilitate collaborative learning and efficient spectrum management. The DQN technique uses a neural network to approximate the Q value function in high-dimensional state spaces, thus mapping input states to (action, Q value) tables that facilitate self-learning across diverse scenarios. Similarly, the QMIX is a value-based technique for multi-agent environments. In the proposed model, each V2V agent having its own DQN observes the environment, receives observation, and obtains a common reward. The QMIX network receives Q values from all agents considering individual benefits and collective objectives. This mechanism leads to collective learning while V2V agents dynamically adapt to real-time conditions, thus improving VNs performance. Our research finding highlights the potential of hybrid MARL models for dynamic spectrum sharing in VNs and paves the way for advanced cooperative learning strategies in vehicular communication environments. Furthermore, we conducted an in-depth exploration of the simulation environment and performance evaluation criteria, thus concluding in a comprehensive comparative analysis with cutting-edge solutions in the field. Simulation results show that the proposed framework efficiently performs against the benchmark architecture in terms of V2V transmission probability and V2I peak data transfer.

A smart inventory management system with medication demand dependencies in a hospital supply chain: A multi-agent reinforcement learning approach

In light of the intense need for quality health and well-being, the healthcare industry must improve its operations by strengthening its service delivery and reducing overall costs. A smart inventory management system for managing medicines in a hospital supply chain (HSC) is one of the best solutions as its advantages ensure availability of affordable medicines as well as inventory cost reduction. Significant effort is made on inventory management from literature; however, smart inventory systems for HSC considering the various complexities and uniqueness of healthcare systems are limited. This paper attempts to fill this gap by developing a stochastic semi-Markov decision process model which is solved by a multi-agent reinforcement learning method. Data collected from a multispecialty hospital in India’s eastern region serves to validate the proposed model. The results present the optimal order quantities and optimal inventory control policy for the HSC considering the uncertain medication demand at the hospital pharmacy and multiple point-of-care units, and demand dependencies among medicines prescribed to the patients admitted in the hospital. The study also has significant managerial implications for enhancing HSC’s inventory performance, improving the healthcare service delivery, reducing healthcare cost, thereby ensuring the affordable, accessible, and quality healthcare to the society.

Optimizing UAV-UGV coalition operations: A hybrid clustering and multi-agent reinforcement learning approach for path planning in obstructed environment

One of the most critical applications undertaken by Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) is reaching predefined targets by following the most time-efficient routes while avoiding collisions. Unfortunately, UAVs are hampered by limited battery life, and UGVs face challenges in reachability due to obstacles and elevation variations, which is why a coalition of UAVs and UGVs can be highly effective. Existing literature primarily focuses on one-to-one coalitions, which constrains the efficiency of reaching targets. In this work, we introduce a novel approach for a UAV-UGV coalition with a variable number of vehicles, employing a modified mean-shift clustering algorithm (MEANCRFT) to segment targets into multiple zones. This approach of assigning targets to various circular zones based on density and range significantly reduces the time required to reach these targets. Moreover, introducing variability in the number of UAVs and UGVs in a coalition enhances task efficiency by enabling simultaneous multi-target engagement. In our approach, each vehicle of the coalitions is trained using two advanced deep reinforcement learning algorithms in two separate experiments, namely Multi-agent Deep Deterministic Policy Gradient (MADDPG) and Multi-agent Proximal Policy Optimization (MAPPO). The results of our experimental evaluation demonstrate that the proposed MEANCRFT-MADDPG method substantially surpasses current state-of-the-art techniques, nearly doubling efficiency in terms of target navigation time and task completion rate.

Distributed Edge Caching for Zero Trust-Enabled Connected and Automated Vehicles: A Multi-Agent Reinforcement Learning Approach

Distributed Edge Caching for Zero Trust-Enabled Connected and Automated Vehicles: A Multi-Agent Reinforcement Learning Approach

Towards Efficient Coordination in Multi-Agent Reinforcement Learning through Hybrid Information-Driven Approaches

Abstract: This paper introduces a novel framework to enhance coordination in multi-agent reinforcement learning (MARL) systems by integrating mutual information (MMI) principles with information-driven strategies. Firstly, we propose a variational approach leveraging MMI to promote coordinated behaviors among agents by regulating the cumulative return alongside the simultaneous mutual information between multi-agent actions. Through the introduction of a latent variable inducing nonzero mutual information and the application of a variational bound, a tractable lower bound is derived for the MMI-regularized objective function. This bound combines maximum entropy reinforcement learning with reducing uncertainty in other agents' actions. Subsequently, we present a practical algorithm, Variational Maximum Mutual Information Multi-Agent Actor-Critic (VM3-AC), utilizing policy iteration to maximize the derived lower bound, following a centralized learning with decentralized execution (CTDE) paradigm. Secondly, we explore the challenges of large state spaces and limited computational resources in distributed multi-agent systems, proposing a hybrid information-driven MARL approach. This approach integrates informationtheoretic models as heuristics to aid navigation in sparse state spaces, complemented by information-based rewards within an RL framework to learn higher-level policies efficiently. Our preliminary findings suggest that this hybrid approach could enhance exploration efficiency significantly, demonstrating approximately three orders of magnitude improvement over naive baseline metrics. Although still in its early stages, this work presents a promising direction for future research in achieving efficient coordination in MARL systems

ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

Value function factorization has achieved great success in multi-agent reinforcement learning by optimizing joint action-value functions through the maximization of factorized per-agent utilities. To ensure Individual-Global-Maximum property, existing works often focus on value factorization using monotonic functions, which are known to result in restricted representation expressiveness. In this paper, we analyze the limitations of monotonic factorization and present ConcaveQ, a novel non-monotonic value function factorization approach that goes beyond monotonic mixing functions and employs neural network representations of concave mixing functions. Leveraging the concave property in factorization, an iterative action selection scheme is developed to obtain optimal joint actions during training. It is used to update agents’ local policy networks, enabling fully decentralized execution. The effectiveness of the proposed ConcaveQ is validated across scenarios involving multi-agent predator-prey environment and StarCraft II micromanagement tasks. Empirical results exhibit significant improvement of ConcaveQ over state-of-the-art multi-agent reinforcement learning approaches.

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

Cooperative multiagent reinforcement learning approaches are increasingly being used to make decisions in contested and dynamic environments, which tend to be wildly different from the environments used to train them. As such, there is a need for a more in-depth understanding of their resilience and robustness in conditions such as network partitions, node failures, or attacks. In this article, we propose a modeling and simulation framework that explores the resilience of four c-MARL models when faced with different types of attacks, and the impact that training with different perturbations has on the effectiveness of these attacks. We show that c-MARL approaches are highly vulnerable to perturbations of observation, action reward, and communication, showing more than 80% drop in the performance from the baseline. We also show that appropriate training with perturbations can dramatically improve performance in some cases, however, can also result in overfitting, making the models less resilient against other attacks. This is a first step toward a more in-depth understanding of the resilience c-MARL models and the effect that contested environments can have on their behavior and toward resilience of complex systems in general.

Demand response model: A cooperative-competitive multi-agent reinforcement learning approach

This study introduces a novel Demand Response (DR) Model based on Multi-agent Reinforcement Learning (MARL-DR), comprised of a pricing and incentives scheme aimed at improving the accuracy of existing demand response strategies. Furthermore, this new approach represents a flexibility solution to prevent sharp price variations caused by the high penetration of unconventional renewable energy, which are directly passed on to end-users. The model employs a cooperative-competitive MARL-DR technique based on Q-learning, with the goal of determining optimal prices and incentives that maximize benefits for both customers and electric Service Provider (SP). In this regard, the model has the capability to offer customers pricing options in both real-time and time-of-use, to actively adjust each user's consumption. Additionally, through demand characterization factors, such as the coincidence factor (CF), the of typical user behavior is improved, and the influence of individual user demand on system peak demand is more accurately detected. It is also demonstrated that the cooperative-competitive approach offers better performance compared to other approaches. Finally, a sensitivity analysis is presented at various stages of the model to verify its accuracy and efficiency in pricing and incentives formulation.

Big-IDS: a decentralized multi agent reinforcement learning approach for distributed intrusion detection in big data networks

Big-IDS: a decentralized multi agent reinforcement learning approach for distributed intrusion detection in big data networks

Collaborative Hyperspectral Image Processing Using Satellite Edge Computing

The advancement of nanosatellite techniques has boosted the growth of satellite-originated data and applications. Satellite edge computing (SEC) is envisioned to provide in-orbit processing of the sensed data to save the scarce terrestrial-satellite communication resources and support mission-critical services. While most of the existing SEC studies mainly focus on general computing tasks, we present a two-tier collaborative processing framework for the important and unique hyperspectral image (HSI) processing task. Our framework carefully selects bands out of the collected HSIs and sends them back for further analysis. We first conduct a comprehensive data analysis to reveal the non-trivial relationship between the band selection and the eventual analytic performance. We then formulate the band selection problem in this collaborative setting as a utility maximization problem that jointly considers the analytic, energy, and communication factors. A novel multi-agent reinforcement learning approach, named MaHSI, is proposed to solve it in the dynamic SEC environment. Our multi-agent design judiciously embeds the complex correlations among bands as collaborations among agents and significantly reduces the exploration space. Extensive experiments on real-world HSI datasets prove that our approach not only outperforms the existing classical band selection algorithms in accuracy and inference speed but also brings the highest utility to the satellites.

Optimizing Electric Vehicle Charging Recommendation in Smart Cities: A Multi-Agent Reinforcement Learning Approach

As global awareness for preserving natural energy sustainability rises, electric vehicles (EVs) are increasingly becoming a preferred choice for transportation because of their ability to emit zero emissions, conserve energy, and reduce pollution, especially in smart cities with sustainable development. Nonetheless, the lack of adequate EV charging infrastructure remains a significant problem that has resulted in varying charging demands at different locations and times, particularly in developing countries. As a consequence, this inadequacy has posed a challenge for EV drivers, particularly those in smart cities, as they face difficulty in locating suitable charging stations. Nevertheless, the recent development of deep reinforcement learning is a promising technology that has the potential to improve the charging experience in several ways over the long term. This paper proposes a novel approach for recommending EV charging stations using multi-agent reinforcement learning (MARL) algorithms by comparing several popular algorithms, including the deep deterministic policy gradient, deep Q-network, multi-agent DDPG (MADDPG), Real, and Random, in optimizing the placement and allocation of the EV charging stations. The results demonstrated that MADDPG outperformed other algorithms in terms of the Mean Charge Waiting Time, CFT, and Total Saving Fee, thus indicating its superiority in addressing the EV charging station problem in a multi-agent setting. The collaborative and communicative nature of the MADDPG algorithm played a key role in achieving these results. Hence, this approach could provide a better user experience, increase the adoption of EVs, and be extended to other transportation-related problems. Overall, this study highlighted the potential of MARL as a powerful approach for solving complex optimization problems in transportation and beyond. This would also contribute to the development of more efficient and sustainable transportation systems in smart cities for sustainable development.

Intelligent Caching for Seamless High-Quality Streaming in Vehicular Networks: A Multi-Agent Reinforcement Learning Approach

Intelligent Caching for Seamless High-Quality Streaming in Vehicular Networks: A Multi-Agent Reinforcement Learning Approach

Coexistence Between Task- and Data-Oriented Communications: A Whittle’s Index Guided Multiagent Reinforcement Learning Approach

Coexistence Between Task- and Data-Oriented Communications: A Whittle’s Index Guided Multiagent Reinforcement Learning Approach

A reinforcement learning approach to vehicle coordination for structured advanced air mobility

Advanced Air Mobility (AAM) has emerged as a pioneering concept designed to optimize the efficacy and ecological sustainability of air transportation. Its core objective is to provide highly automated air transportation services for passengers or cargo, operating at low altitudes within urban, suburban, and rural regions. AAM seeks to enhance the efficiency and environmental viability of the aviation sector by revolutionizing the way air travel is conducted. In a complex aviation environment, traffic management and control are essential technologies for safe and effective AAM operations. One of the most difficult obstacles in the envisioned AAM systems is vehicle coordination at merging points and intersections. The escalating demand for air mobility services, particularly within urban areas, poses significant complexities to the execution of such missions. In this study, we propose a novel multi-agent reinforcement learning (MARL) approach to efficiently manage high-density AAM operations in structured airspace. Our approach provides effective guidance to AAM vehicles, ensuring conflict avoidance, mitigating traffic congestion, reducing travel time, and maintaining safe separation. Specifically, intelligent learning-based algorithms are developed to provide speed guidance for each AAM vehicle, ensuring secure merging into air corridors and safe passage through intersections. To validate the effectiveness of our proposed model, we conduct training and evaluation using BlueSky, an open-source air traffic control simulation environment. Through the simulation of thousands of aircraft and the integration of real-world data, our study demonstrates the promising potential of MARL in enabling safe and efficient AAM operations. The simulation results validate the efficacy of our approach and its ability to achieve the desired outcomes.

AMARL: An Attention-Based Multiagent Reinforcement Learning Approach to the Min-Max Multiple Traveling Salesmen Problem.

In recent years, the multiple traveling salesmen problem (MTSP or multiple TSP) has received increasing research interest and one of its main applications is coordinated multirobot mission planning, such as cooperative search and rescue tasks. However, it is still challenging to solve MTSP with improved inference efficiency as well as solution quality in varying situations, e.g., different city positions, different numbers of cities, or agents. In this article, we propose an attention-based multiagent reinforcement learning (AMARL) approach, which is based on the gated transformer feature representations for min-max multiple TSPs. The state feature extraction network in our proposed approach adopts the gated transformer architecture with reordering layer normalization (LN) and a new gate mechanism. It aggregates fixed-dimensional attention-based state features irrespective of the number of agents and cities. The action space of our proposed approach is designed to decouple the interaction of agents' simultaneous decision-making. At each time step, only one agent is assigned to a non-zero action so that the action selection strategy can be transferred across tasks with different numbers of agents and cities. Extensive experiments on min-max multiple TSPs were conducted to illustrate the effectiveness and advantages of the proposed approach. Compared with six representative algorithms, our proposed approach achieves state-of-the-art performance in solution quality and inference efficiency. In particular, the proposed approach is suitable for tasks with different numbers of agents or cities without extra learning, and experimental results demonstrate that the proposed approach realizes powerful transfer capability across tasks.