Multi-agent Reinforcement Learning Research Articles

Wall Modeling of Turbulent Flows with Varying Pressure Gradients Using Multi-Agent Reinforcement Learning

We propose a framework for developing wall models for large-eddy simulation that is able to capture pressure-gradient effects using multi-agent reinforcement learning. Within this framework, the distributed reinforcement learning agents receive off-wall environmental states, including pressure gradient and turbulence strain rate, ensuring adaptability to a wide range of flows characterized by pressure-gradient effects and separations. Based on these states, the agents determine an action to adjust the wall eddy viscosity and, consequently, the wall-shear stress. The model training is in situ with wall-modeled large-eddy simulation grid resolutions and does not rely on the instantaneous velocity fields from high-fidelity simulations. Throughout the training, the agents compute rewards from the relative error in the estimated wall-shear stress, which allows them to refine an optimal control policy that minimizes prediction errors. Employing this framework, wall models are trained for two distinct subgrid-scale models using low-Reynolds-number flow over periodic hills. These models are validated through simulations of flows over periodic hills at higher Reynolds numbers and flows over the Boeing Gaussian bump. The developed wall models successfully capture the acceleration and deceleration of wall-bounded turbulent flows under pressure gradients and outperform the equilibrium wall model in predicting skin friction.

Graph-based multi-agent reinforcement learning for collaborative search and tracking of multiple UAVs

This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.

UAV Confrontation and Evolutionary Upgrade Based on Multi-Agent Reinforcement Learning

UAV Confrontation and Evolutionary Upgrade Based on Multi-Agent Reinforcement Learning

Distributed Routing Optimization Algorithm for FANET Based on Multiagent Reinforcement Learning

Distributed Routing Optimization Algorithm for FANET Based on Multiagent Reinforcement Learning

WToE: Learning When to Explore in Multiagent Reinforcement Learning.

Existing multiagent exploration works focus on how to explore in the fully cooperative task, which is insufficient in the environment with nonstationarity induced by agent interactions. To tackle this issue, we propose When to Explore (WToE), a simple yet effective variational exploration method to learn WToE under nonstationary environments. WToE employs an interaction-oriented adaptive exploration mechanism to adapt to environmental changes. We first propose a novel graphical model that uses a latent random variable to model the step-level environmental change resulting from interaction effects. Leveraging this graphical model, we employ the supervised variational auto-encoder (VAE) framework to derive a short-term inferred policy from historical trajectories to deal with the nonstationarity. Finally, agents engage in exploration when the short-term inferred policy diverges from the current actor policy. The proposed approach theoretically guarantees the convergence of the Q -value function. In our experiments, we validate our exploration mechanism in grid examples, multiagent particle environments and the battle game of MAgent environments. The results demonstrate the superiority of WToE over multiple baselines and existing exploration methods, such as MAEXQ, NoisyNets, EITI, and PR2.

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

QFuture: Learning Future Expectation Cognition in Multiagent Reinforcement Learning

QFuture: Learning Future Expectation Cognition in Multiagent Reinforcement Learning

A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learning.

State-of-the-art (SOTA) multiagent reinforcement algorithms distinguish themselves in many ways from their single-agent equivalences. However, most of them still totally inherit the single-agent exploration-exploitation strategy. Naively inheriting this strategy from single-agent algorithms causes potential collaboration failures, in which the agents blindly follow mainstream behaviors and reject taking minority responsibility. We name this problem the responsibility diffusion (RD) as it shares similarities with the same-name social psychology effect. In this work, we start by theoretically analyzing the cause of this RD problem, which can be traced back to the exploration-exploitation dilemma of multiagent systems (especially large-scale multiagent systems). We address this RD problem by proposing a policy resonance (PR) approach which modifies the collaborative exploration strategy of agents by refactoring the joint agent policy while keeping individual policies approximately invariant. Next, we show that SOTA algorithms can equip this approach to promote the collaborative performance of agents in complex cooperative tasks. Experiments are performed in multiple test benchmark tasks to illustrate the effectiveness of this approach.

Multi-Agent Reinforcement Learning for Coordinating Communication and Control

Multi-Agent Reinforcement Learning for Coordinating Communication and Control

An Intelligent Strategy Decision Method for Collaborative Jamming Based on Hierarchical Multi-Agent Reinforcement Learning

An Intelligent Strategy Decision Method for Collaborative Jamming Based on Hierarchical Multi-Agent Reinforcement Learning

Computation Rate Maximization for SCMA-Aided Edge Computing in IoT Networks: A Multi-Agent Reinforcement Learning Approach

Computation Rate Maximization for SCMA-Aided Edge Computing in IoT Networks: A Multi-Agent Reinforcement Learning Approach

Multi-Agent Reinforcement Learning Based Uplink OFDMA for IEEE 802.11ax Networks

Multi-Agent Reinforcement Learning Based Uplink OFDMA for IEEE 802.11ax Networks

Multi-Agent Reinforcement Learning for Cooperative Trajectory Design of UAV-BS Fleets in Terrestrial/Non-Terrestrial Integrated Networks

Multi-Agent Reinforcement Learning for Cooperative Trajectory Design of UAV-BS Fleets in Terrestrial/Non-Terrestrial Integrated Networks

Mitigating Bus Bunching via Hierarchical Multi-Agent Reinforcement Learning

Mitigating Bus Bunching via Hierarchical Multi-Agent Reinforcement Learning

The AI circular hydrogen economist: Hydrogen supply chain design via hierarchical deep multi-agent reinforcement learning

Hydrogen supply chain (HSC) consists of various production, conditioning, transportation, storage, and distribution processes, all of which require extensive computational resources for precise modelling. In this context, artificial intelligence (AI) is emerging as a pivotal tool for addressing model-based decision-making challenges, courtesy of its rapid and efficient computational capabilities. This paper proposes a comprehensive HSC model consisting of an economic policy planner, a wholesale hydrogen market, a power supply system, a hydrogen distribution system (HDS), and hydrogen refueling stations (HRSs). It leverages an AI circular hydrogen economist approach based on a hierarchical deep multi-agent reinforcement learning (MARL) algorithm, offering a new alternative to traditional multi-objective bi-level optimization platforms. The model incorporates green, blue, and gray hydrogen production processes as viable hydrogen production pathways, with the hierarchical MARL’s agents representing the HDS and HRSs at two decision-making levels. Energy requirements are met through a combination of on-site renewable energy sources, the main power grid, or distributed power generation systems. Several HSC scenarios are examined with respect to various combinations of green hydrogen supply rates and carbon credits granted to them under optimum conditions. The results showed that the developed hierarchical MARL has the potential to replace mathematical programming (MP), uncovering a new economic-environmental trade-off between profit of HDS and operational costs of HRSs. Notably, green hydrogen transactions exponentially increase within the supply chain as the carbon credits exceed $1 per kilogram of hydrogen. While this research focuses on optimizing daily operations within the HSC, future efforts can aim to extend this optimization to forecasted annual operations.

Adaptive task migration strategy with delay risk control and reinforcement learning for emergency monitoring

The timely and reliable handling of post-disaster emergency monitoring tasks is crucial for effective rescue operations. UAV-assisted edge computing plays a pivotal role in the rapid deployment of such systems. However, challenges persist due to communication and computation resource bottlenecks when dealing with delay-sensitive monitoring tasks. In dynamic post-disaster environments, effective task scheduling and resource allocation decisions directly impact the system’s ability to process tasks. Therefore, this paper proposes an adaptive task migration decision-making system for emergency monitoring tasks in UAV-assisted edge computing. Firstly, We decomposed the optimization objectives based on the task processing workflow, then devised a stepwise delay risk control and resource recovery mechanism based on early discarding. Secondly, by integrating multi-agent reinforcement learning (MARL), optimal strategies for task offloading, UAV queue scheduling, and communication resource allocation are learned to enhance the decision system’s environmental awareness and maximize the successful completion of emergency monitoring tasks. Simulation experiments demonstrate that the algorithm significantly improves the success rate of migration tasks and data processing capacity, thereby validating its convergence and effectiveness.

Distributed and Multi-Agent Reinforcement Learning Framework for Optimal Electric Vehicle Charging Scheduling

The increasing number of electric vehicles (EVs) necessitates the installation of more charging stations. The challenge of managing these grid-connected charging stations leads to a multi-objective optimal control problem where station profitability, user preferences, grid requirements and stability should be optimized. However, it is challenging to determine the optimal charging/discharging EV schedule, since the controller should exploit fluctuations in the electricity prices, available renewable resources and available stored energy of other vehicles and cope with the uncertainty of EV arrival/departure scheduling. In addition, the growing number of connected vehicles results in a complex state and action vectors, making it difficult for centralized and single-agent controllers to handle the problem. In this paper, we propose a novel Multi-Agent and distributed Reinforcement Learning (MARL) framework that tackles the challenges mentioned above, producing controllers that achieve high performance levels under diverse conditions. In the proposed distributed framework, each charging spot makes its own charging/discharging decisions toward a cumulative cost reduction without sharing any type of private information, such as the arrival/departure time of a vehicle and its state of charge, addressing the problem of cost minimization and user satisfaction. The framework significantly improves the scalability and sample efficiency of the underlying Deep Deterministic Policy Gradient (DDPG) algorithm. Extensive numerical studies and simulations demonstrate the efficacy of the proposed approach compared with Rule-Based Controllers (RBCs) and well-established, state-of-the-art centralized RL (Reinforcement Learning) algorithms, offering performance improvements of up to 25% and 20% in reducing the energy cost and increasing user satisfaction, respectively.

UAV swarm air combat maneuver decision-making method based on multi-agent reinforcement learning and transferring

UAV swarm air combat maneuver decision-making method based on multi-agent reinforcement learning and transferring

Joint optimization of communication and mission performance for multi-UAV collaboration network: A multi-agent reinforcement learning method

In emergency rescue, target search and other mission scenarios with Unmanned Aerial Vehicles (UAVs), the Relay UAVs (RUs) and Mission UAVs (MUs) can collaborate to accomplish tasks in unknown environments. In this paper, we investigate the problem of trajectory planning and power control for the MU and RU collaboration. Firstly, considering the characteristics of multi-hop data transmission between the MU and Ground Control Station, a multi-UAV collaborative coverage model is designed. Meanwhile, a UAV control algorithm named MUTTO is proposed based on multi-agent reinforcement learning. In order to solve the problem of the unknown information about the number and locations of targets, the geographic coverage rate is used to replace the target coverage rate for decision making. Then, the reward functions of two types of UAVs are designed separately for the purpose of better cooperation. By simultaneously planning the trajectory and transmission power of the RU and MU, the mission target coverage rate and network transmission rate are maximized while the energy consumption of the UAV is minimized. Finally, numerical simulations results show that MUTTO can solve the UAV network control problem in an efficient way and has better performance than the benchmark method.

Environmental-Impact-Based Multi-Agent Reinforcement Learning

To promote cooperation and strengthen the individual impact on the collective outcome in social dilemmas, we propose the Environmental-impact Multi-Agent Reinforcement Learning (EMuReL) method where each agent estimates the “environmental impact” of every other agent, that is, the difference in the current environment state compared to the hypothetical environment in the absence of that other agent. Inspired by the inequity aversion model, the agent then compares its own reward with that of its fellows multiplied by their environmental impacts. If its reward exceeds the scaled reward of one of its fellows, the agent takes “social responsibility” toward that fellow by reducing its own reward. Therefore, the less influential an agent is in reaching the current state, the more social responsibility is taken by other agents. Experiments in the Cleanup (resp. Harvest) test environment demonstrated that agents trained based on EMuReL learned to cooperate more effectively and obtained 54% (39%) and 20% (44%) more total rewards while preserving the same cooperation levels compared to when they were trained based on the two state-of-the-art reward reshaping methods: inequity aversion and social influence.