Multi-agent Approach Research Articles

Optimizing UAV-UGV coalition operations: A hybrid clustering and multi-agent reinforcement learning approach for path planning in obstructed environment

One of the most critical applications undertaken by Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) is reaching predefined targets by following the most time-efficient routes while avoiding collisions. Unfortunately, UAVs are hampered by limited battery life, and UGVs face challenges in reachability due to obstacles and elevation variations, which is why a coalition of UAVs and UGVs can be highly effective. Existing literature primarily focuses on one-to-one coalitions, which constrains the efficiency of reaching targets. In this work, we introduce a novel approach for a UAV-UGV coalition with a variable number of vehicles, employing a modified mean-shift clustering algorithm (MEANCRFT) to segment targets into multiple zones. This approach of assigning targets to various circular zones based on density and range significantly reduces the time required to reach these targets. Moreover, introducing variability in the number of UAVs and UGVs in a coalition enhances task efficiency by enabling simultaneous multi-target engagement. In our approach, each vehicle of the coalitions is trained using two advanced deep reinforcement learning algorithms in two separate experiments, namely Multi-agent Deep Deterministic Policy Gradient (MADDPG) and Multi-agent Proximal Policy Optimization (MAPPO). The results of our experimental evaluation demonstrate that the proposed MEANCRFT-MADDPG method substantially surpasses current state-of-the-art techniques, nearly doubling efficiency in terms of target navigation time and task completion rate.

Dynamic, fair, and efficient routing for cooperative autonomous vehicle fleets

This paper addresses challenges in agricultural cooperative autonomous fleet routing through the proposition, modeling, and resolution of the Dynamic Vehicle Routing Problem with Fair Profits and Time Windows (DVRP-FPTW). The aim is to dynamically optimize routes for a vehicle fleet serving tasks within assigned time windows, emphasizing fair and efficient solutions. Our DVRP-FPTW accommodates unforeseen events like task modifications or vehicle breakdowns, ensuring adherence to task demand, vehicle capacities, and autonomies. The proposed model incorporates mandatory and optional tasks, including optional ones in operational vehicle routes if not compromising the vehicles’ profits. Including asynchronous and distributed column generation heuristics, the proposed Multi-Agent-based architecture DIMASA for the DVRP-FPTW dynamically adapts to unforeseen events. Systematic Egalitarian social welfare optimization is used to iteratively maximize the profit of the least profitable vehicle, prioritizing fairness across the fleet in light of unforeseen events. This improves upon existing dynamic and multi-period VRP models that rely on prior knowledge of demand changes. Our approach allows vehicle agents to maintain privacy while sharing minimal local data with a fleet coordinator agent. We propose publicly available benchmark instances for both static and dynamic VRP-FPTW. Simulation results demonstrate the effectiveness of our DVRP-FPTW model and our multi-agent system solution approach in coordinating large, dynamically evolving cooperative autonomous fleets fairly and efficiently in close to real-time.

Multiagent Q-Learning Approach for the Recharging Scheduling of Electric Automated Guided Vehicles in Container Terminals

In recent years, advancements in battery technology have led to increased adoption of electric automated guided vehicles in container terminals. Given how critical these vehicles are to terminal operations, this trend requires efficient recharging scheduling for automated guided vehicles, and the main challenges arise from limited charging station capacity and tight vehicle schedules. Motivated by the dynamic nature of the problem, the recharging scheduling problem for an entire vehicle fleet given capacitated stations is formulated as a Markov decision process model. Then, it is solved using a multiagent Q-learning (MAQL) approach to produce a recharging schedule that minimizes the delay of jobs. Numerical experiments show that under a stochastic environment in terms of vehicle travel time, MAQL enables the exploration of better scheduling by coordinating across the entire vehicle fleet and charging facilities and outperforms various benchmark approaches, with an additional improvement of 18.8% on average over the best rule-based heuristic and 5.4% over the predetermined approach. Funding: This work was supported by the National Natural Science Foundation of China [Grant 72101203], the Shaanxi Provincial Key R&D Program, China [Grant 2022KW-02], and the Singapore Maritime Institute [Grant SMI-2017-SP-002]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.0113 .

Distributed Edge Caching for Zero Trust-Enabled Connected and Automated Vehicles: A Multi-Agent Reinforcement Learning Approach

Distributed Edge Caching for Zero Trust-Enabled Connected and Automated Vehicles: A Multi-Agent Reinforcement Learning Approach

Dynamic Scheduling of IoV Edge Cloud Service Functions Under NFV: A Multi-Agent Reinforcement Learning Approach

Dynamic Scheduling of IoV Edge Cloud Service Functions Under NFV: A Multi-Agent Reinforcement Learning Approach

Deep-silicon photon-counting x-ray projection denoising through reinforcement learning.

In recent years, deep reinforcement learning (RL) has been applied to various medical tasks and produced encouraging results. In this paper, we demonstrate the feasibility of deep RL for denoising simulated deep-silicon photon-counting CT (PCCT) data in both full and interior scan modes. PCCT offers higher spatial and spectral resolution than conventional CT, requiring advanced denoising methods to suppress noise increase. In this work, we apply a dueling double deep Q network (DDDQN) to denoise PCCT data for maximum contrast-to-noise ratio (CNR) and a multi-agent approach to handle data non-stationarity. Using our method, we obtained significant image quality improvement for single-channel scans and consistent improvement for all three channels of multichannel scans. For the single-channel interior scans, the PSNR (dB) and SSIM increased from 33.4078 and 0.9165 to 37.4167 and 0.9790 respectively. For the multichannel interior scans, the channel-wise PSNR (dB) increased from 31.2348, 30.7114, and 30.4667 to 31.6182, 30.9783, and 30.8427 respectively. Similarly, the SSIM improved from 0.9415, 0.9445, and 0.9336 to 0.9504, 0.9493, and 0.0326 respectively. Our results show that the RL approach improves image quality effectively, efficiently, and consistently across multiple spectral channels and has great potential in clinical applications.

ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

Value function factorization has achieved great success in multi-agent reinforcement learning by optimizing joint action-value functions through the maximization of factorized per-agent utilities. To ensure Individual-Global-Maximum property, existing works often focus on value factorization using monotonic functions, which are known to result in restricted representation expressiveness. In this paper, we analyze the limitations of monotonic factorization and present ConcaveQ, a novel non-monotonic value function factorization approach that goes beyond monotonic mixing functions and employs neural network representations of concave mixing functions. Leveraging the concave property in factorization, an iterative action selection scheme is developed to obtain optimal joint actions during training. It is used to update agents’ local policy networks, enabling fully decentralized execution. The effectiveness of the proposed ConcaveQ is validated across scenarios involving multi-agent predator-prey environment and StarCraft II micromanagement tasks. Empirical results exhibit significant improvement of ConcaveQ over state-of-the-art multi-agent reinforcement learning approaches.

A simulation and experimentation architecture for resilient cooperative multiagent reinforcement learning models operating in contested and dynamic environments

Cooperative multiagent reinforcement learning approaches are increasingly being used to make decisions in contested and dynamic environments, which tend to be wildly different from the environments used to train them. As such, there is a need for a more in-depth understanding of their resilience and robustness in conditions such as network partitions, node failures, or attacks. In this article, we propose a modeling and simulation framework that explores the resilience of four c-MARL models when faced with different types of attacks, and the impact that training with different perturbations has on the effectiveness of these attacks. We show that c-MARL approaches are highly vulnerable to perturbations of observation, action reward, and communication, showing more than 80% drop in the performance from the baseline. We also show that appropriate training with perturbations can dramatically improve performance in some cases, however, can also result in overfitting, making the models less resilient against other attacks. This is a first step toward a more in-depth understanding of the resilience c-MARL models and the effect that contested environments can have on their behavior and toward resilience of complex systems in general.

Demand response model: A cooperative-competitive multi-agent reinforcement learning approach

This study introduces a novel Demand Response (DR) Model based on Multi-agent Reinforcement Learning (MARL-DR), comprised of a pricing and incentives scheme aimed at improving the accuracy of existing demand response strategies. Furthermore, this new approach represents a flexibility solution to prevent sharp price variations caused by the high penetration of unconventional renewable energy, which are directly passed on to end-users. The model employs a cooperative-competitive MARL-DR technique based on Q-learning, with the goal of determining optimal prices and incentives that maximize benefits for both customers and electric Service Provider (SP). In this regard, the model has the capability to offer customers pricing options in both real-time and time-of-use, to actively adjust each user's consumption. Additionally, through demand characterization factors, such as the coincidence factor (CF), the of typical user behavior is improved, and the influence of individual user demand on system peak demand is more accurately detected. It is also demonstrated that the cooperative-competitive approach offers better performance compared to other approaches. Finally, a sensitivity analysis is presented at various stages of the model to verify its accuracy and efficiency in pricing and incentives formulation.

Age of correlated information-optimal dynamic policy scheduling for sustainable Green IoT devices: A multi-agent deep reinforcement learning approach

The rapid progress in communication technologies and the widespread adoption of connected devices have given rise to various information-centric Internet-of-Things (IoT) systems, typically necessitating timely updates of information. Green IoT (G-IoT) strives to enhance the environment by reducing the power consumption of billions of devices engaged in extensive data exchange, addressing the substantial energy demand in the process. The Age of Correlated Information (AoCI) measures the freshness of information shared among two or more devices that contribute to the same decision-making process. Optimizing AoCI using Deep Reinforcement Learning (DRL) in IoT reduces energy consumption, optimizes resource utilization, and promotes environmentally conscious communication, contributing to the development of a sustainable G-IoT system. This paper focuses on scheduling the transmission of status update packets among interconnected G-IoT devices to minimize the application-specific long-term average AoCI. The problem is modeled as an NP-hard episodic Markov Decision Process (MDP), highlighting its computational complexity. To handle correlations and the curse of dimensionality, a multi-agent deep reinforcement learning algorithm, specifically the Multi-Agent Deep Deterministic Policy Grading (MADDPG) algorithm, is developed. The training progress displays episode rewards, with an environment designed to penalize multiple agents transmitting simultaneously or none at all, promoting cooperative behavior and minimizing the average age of correlated information. We provide comprehensive simulation results, including reward convergence, the learning process of actors and critics, and the resulting average AoCI with the number of episodes, demonstrating the effectiveness of the MADDPG algorithm.

Collaborative Hyperspectral Image Processing Using Satellite Edge Computing

The advancement of nanosatellite techniques has boosted the growth of satellite-originated data and applications. Satellite edge computing (SEC) is envisioned to provide in-orbit processing of the sensed data to save the scarce terrestrial-satellite communication resources and support mission-critical services. While most of the existing SEC studies mainly focus on general computing tasks, we present a two-tier collaborative processing framework for the important and unique hyperspectral image (HSI) processing task. Our framework carefully selects bands out of the collected HSIs and sends them back for further analysis. We first conduct a comprehensive data analysis to reveal the non-trivial relationship between the band selection and the eventual analytic performance. We then formulate the band selection problem in this collaborative setting as a utility maximization problem that jointly considers the analytic, energy, and communication factors. A novel multi-agent reinforcement learning approach, named MaHSI, is proposed to solve it in the dynamic SEC environment. Our multi-agent design judiciously embeds the complex correlations among bands as collaborations among agents and significantly reduces the exploration space. Extensive experiments on real-world HSI datasets prove that our approach not only outperforms the existing classical band selection algorithms in accuracy and inference speed but also brings the highest utility to the satellites.

Many-to-Many Task Offloading in Vehicular Fog Computing: A Multi-Agent Deep Reinforcement Learning Approach

Vehicular fog computing (VFC) has emerged as a promising solution to mitigate vehicular network computation load. In the hierarchical VFC, vehicles are employed as mobile fog nodes at the edge to provide reliable and low-latency services. Particularly, since privately-owned vehicles are rational nodes, their intentions for both computation provision and service demand should be considered instead of overestimating their willingness. To remunerate the participation intentions of vehicles as well as improve vehicular fog resource utilization in the large-scale VFC, the trading-based mechanism is a potential solution. In this paper, we propose a many-to-many task offloading framework based on the vehicular trading paradigm. This framework enables computational resource trading across different VFC subsystems and decides the multi-tier task offloading results based on the trading consensus. The trading process is viewed as a partially observable Markov decision process (POMDP) and a Multi-Agent Gated actor Attention Critic (MA-GAC) approach is designed to reach an effective and stable offload-and -serve cooperation among vehicles. Theoretical analyses and experiments verify the feasibility and efficiency of the proposed framework, and simulation results demonstrate that the coordinated MA-GAC approach not only benefits vehicles with higher long-term rewards but also optimizes the system social welfare in a distributed manner.

Imbalance Cost-Aware Energy Scheduling for Prosumers Towards UAM Charging: A Matching and Multi-Agent DRL Approach

Imbalance Cost-Aware Energy Scheduling for Prosumers Towards UAM Charging: A Matching and Multi-Agent DRL Approach

DEVELOPMENT OF AN ONTOLOGICAL MODEL OF DEEP LEARNING NEURAL NETWORKS

This research paper examines the challenges and prospects associated with the integration of artificial neural networks and knowledge bases. The focus is on leveraging this integration to address practical problems. The paper explores the development, training, and integration of artificial neural net- works, emphasizing their adaptation to knowledge bases. This adaptation involves processes such as in- tegration, communication, representation of ontological structures, and interpretation by the knowledge base of the artificial neural network's representation through input and output. The paper also delves into the direction of establishing an intellectual environment conducive to the development, training, and integration of adapted artificial neural networks with knowledge bases. The knowledge base embedded in an artificial neural network is constructed using a homogeneous semantic network, and knowledge processing employs a multi-agent approach. The representation of artificial neural networks and their specifications within a unified semantic model of knowledge representation is detailed, encompassing text-based specifications in the language of knowledge representation with theoretical semantics. The models shared with the knowledge base include dynamic and other types that vary in their capabilities for knowledge representation. Furthermore, the paper conducts an analysis of approaches to creating artificial neural networks across various libraries of the high-level programming language Python. It explores techniques for developing arti- ficial neural networks within the Python development environment, investigating the key features and func- tions of these libraries. A comparative analysis of neural networks created in object-oriented programming languages is provided, along with the development of an ontological model for deep learning neural net- works.

Distributed energy management of multi-area integrated energy system based on multi-agent deep reinforcement learning

This paper addresses the problem of distributed energy management in multi-area integrated energy systems (MA-IES) using a multi-agent deep reinforcement learning approach. The MA-IES consists of interconnected electric and thermal networks, incorporating renewable energy sources and heat conversion systems. The objective is to optimize the operation of the system while minimizing operational costs and maximizing renewable energy utilization. We propose a distributed energy management strategy that makes hierarchical decisions on intra-area heat energy and inter-area electric energy. The strategy is based on a multi-agent deep reinforcement learning framework, where each agent represents a component or unit in the MA-IES. We formulate the problem as a Markov Decision Process and employ Q-learning with experience replay and double networks to train the agents. The proposed strategy is evaluated using a simulation of a four-area MA-IES. The results demonstrate significant improvements in energy management compared to traditional methods, with higher renewable energy utilization and lower operational costs. Specifically, the strategy achieves 100% utilization of wind power, and decreases operational costs by 5.53%. Furthermore, it leverages the generalization capabilities of reinforcement learning to respond in real-time to uncertainties in demand and wind power output. The results highlight the advantages of the proposed strategy, making it a promising solution for optimizing the operation of multi-area integrated energy systems.

Cooperative traffic signal control through a counterfactual multi-agent deep actor critic approach

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput.

Modelling motorized and non-motorized vehicle conflicts using multiagent inverse reinforcement learning approach

Microsimulation models are effective for analysing road users’ interaction behaviour and assessing different facilities’ performance. However, only a few studies have developed simulation models for studying motorized and non-motorized vehicles conflicts. This is likely due to mixed traffic’s complexity and heterogeneity and the difficulty in accurately capturing road users’ avoidance maneuver. This study aims to adopt a multiagent simulation model to replicate road users’ microscopic behaviour and collision avoidance mechanisms in traffic conflict scenarios. Road users’ reward functions are recovered by the multiagent inverse reinforcement learning approach. The multiagent Actor-Critic deep learning algorithm is used to predict road users’ evasive action and assess their optimal policies. The findings demonstrate that the multiagent simulation model provides highly accurate predictions of road users’ trajectories and collision avoidance strategies. Furthermore, the results demonstrate a strong correlation between the predicted traffic conflict indicator from the simulated trajectories and that from the actual trajectories.

Optimizing Electric Vehicle Charging Recommendation in Smart Cities: A Multi-Agent Reinforcement Learning Approach

As global awareness for preserving natural energy sustainability rises, electric vehicles (EVs) are increasingly becoming a preferred choice for transportation because of their ability to emit zero emissions, conserve energy, and reduce pollution, especially in smart cities with sustainable development. Nonetheless, the lack of adequate EV charging infrastructure remains a significant problem that has resulted in varying charging demands at different locations and times, particularly in developing countries. As a consequence, this inadequacy has posed a challenge for EV drivers, particularly those in smart cities, as they face difficulty in locating suitable charging stations. Nevertheless, the recent development of deep reinforcement learning is a promising technology that has the potential to improve the charging experience in several ways over the long term. This paper proposes a novel approach for recommending EV charging stations using multi-agent reinforcement learning (MARL) algorithms by comparing several popular algorithms, including the deep deterministic policy gradient, deep Q-network, multi-agent DDPG (MADDPG), Real, and Random, in optimizing the placement and allocation of the EV charging stations. The results demonstrated that MADDPG outperformed other algorithms in terms of the Mean Charge Waiting Time, CFT, and Total Saving Fee, thus indicating its superiority in addressing the EV charging station problem in a multi-agent setting. The collaborative and communicative nature of the MADDPG algorithm played a key role in achieving these results. Hence, this approach could provide a better user experience, increase the adoption of EVs, and be extended to other transportation-related problems. Overall, this study highlighted the potential of MARL as a powerful approach for solving complex optimization problems in transportation and beyond. This would also contribute to the development of more efficient and sustainable transportation systems in smart cities for sustainable development.

Active Fault-Tolerant Control of a Four-Wheel Independent Steering System Based on the Multi-Agent Approach

In this paper, we present an active fault-tolerant, vehicle motion control, based on the multi-agent approach for a four-wheel independent steering system. This improves handling stability and active safety of the vehicle in the case of actuator faults. Firstly, an eight-degree-of-freedom four-input vehicle model is established by using the vector transformation method, which takes the four-wheel angle as the input and the yaw rate and the sideslip angle, obtained by the lateral force of a single wheel, as the outputs. Then, we established the topological structure of the four-wheel independent steering subsystems based on graph theory. Aiming at the known fault conditions, an adaptive sliding mode active fault-tolerant control based on the multi-agent four-wheel independent steering system is proposed, in which the unmodeled part of the system is estimated using adaptive methods. For unknown fault conditions, an active fault-tolerant control of the four-wheel independent steering system based on state observer is proposed. An extended state observer with sliding mode surface function as a state variable is designed to estimate actuator faults. Finally, it was demonstrated that the designed state observer and distributed active fault-tolerant control method reduced the impact of actuator failures on vehicle steering and improved the stability of the steering process based on the Lyapunov theory. The hardware in the loop experimental results validate the effectiveness of the proposed novel method.

Sum Rate Optimization for Multi-IRS-Aided Multi-BS Communication System Based on Multi-Agent

Intelligent reflecting surface (IRS) is a revolutionizing technology for improving the spectral and energy efficiency of future wireless networks. In this paper, we consider a downlink large-scale system empowered by multi-IRS to aid communication between the multiple base stations (BSs) and multiple user equipment (UEs). We target maximizing the sum rate by jointly optimizing the UE association, the transmit powers of BSs, and the configurations of the IRS beamforming. Due to the applicability restrictions of conventional optimization methods and their high complexity with large-scale networks in dynamic environments, deep reinforcement (DRL) learning is adopted as an alternative approach to finding optimal solutions. First, we model the optimization problem as a multi-agent Markov decision problem (MAMDP). Then, because large-scale wireless networks are naturally complex and changeable, and because many entities interact and affect how the whole system works, it is important to use a multi-agent approach to understand the complex dependencies and relationships between the different parts. In order to solve the problem, we propose a cooperative multi-agent deep reinforcement learning (MADRL)-based algorithm that works for both continuous and discrete IRS phase shifts. Simulation results validate that the proposed algorithm surpasses iterative optimization benchmarks regarding both sum rate performance and convergence.