Multi-agent Reinforcement Learning Algorithm Research Articles

Multi-Agent Reinforcement Learning for Extended Flexible Job Shop Scheduling

An extended flexible job scheduling problem is presented with characteristics of technology and path flexibility (dual flexibility), varied transportation time, and an uncertain environment. The scheduling can greatly increase efficiency and security in complex scenarios, e.g., distributed vehicle manufacturing, and multiple aircraft maintenance. However, optimizing the scheduling puts forward higher requirements on accuracy, real time, and generalization, while subject to the curse of dimension and usually incomplete information. Various coupling relations among operations, stations, and resources aggravate the problem. To deal with the above challenges, we propose a multi-agent reinforcement learning algorithm where the scheduling environment is modeled as a decentralized partially observable Markov decision process. Each job is regarded as an agent that decides the next triplet, i.e., operation, station, and employed resource. This paper is novel in addressing the flexible job shop scheduling problem with dual flexibility and varied transportation time in consideration and proposing a double Q-value mixing (DQMIX) optimization algorithm under a multi-agent reinforcement learning framework. The experiments of our case study show that the DQMIX algorithm outperforms existing multi-agent reinforcement learning algorithms in terms of solution accuracy, stability, and generalization. In addition, it achieves better solution quality for larger-scale cases than traditional intelligent optimization algorithms.

Retracted: Nursing Value Analysis and Risk Assessment of Acute Gastrointestinal Bleeding Using Multiagent Reinforcement Learning Algorithm.

[This retracts the article DOI: 10.1155/2022/7874751.].

A Multi-Agent Reinforcement Learning Method for Omnidirectional Walking of Bipedal Robots.

Achieving omnidirectional walking for bipedal robots is considered one of the most challenging tasks in robotics technology. Reinforcement learning (RL) methods have proved effective in bipedal walking tasks. However, most existing methods use state machines to switch between multiple policies and achieve omnidirectional gait, which results in shaking during the policy switching process for bipedal robots. To achieve a seamless transition between omnidirectional gait and transient motion for full-size bipedal robots, we propose a novel multi-agent RL method. Firstly, a multi-agent RL algorithm based on the actor-critic framework is designed, and policy entropy is introduced to improve exploration efficiency. By learning agents with parallel initial state distributions, we minimize reliance on gait planner effectiveness in the Robot Operating System (ROS). Additionally, we design a novel heterogeneous policy experience replay mechanism based on Euclidean distance. Secondly, considering the periodicity of bipedal robot walking, we develop a new periodic gait function. Including periodic objectives in the policy can accelerate the convergence speed of training periodic gait functions. Finally, to enhance the robustness of the policy, we construct a novel curriculum learning method by discretizing Gaussian distribution and incorporate it into the robot's training task. Our method is validated in a simulation environment, and the results show that our method can achieve multiple gaits through a policy network and achieve smooth transitions between different gaits.

Mean-field reinforcement learning for decentralized task offloading in vehicular edge computing

Vehicular Edge Computing (VEC) is a promising paradigm for providing low-latency and high-reliability services in the Internet of Vehicles (IoV). The increasing number of mobile devices and the diverse resource requirements of the growing IoV have resulted in a shift from centralized resource management to a decentralized approach. This shift offers improved fault tolerance, scalability, and privacy preservation. However, constructing collaborative awareness and coordination mechanisms between multiple vehicles and edge nodes in a decentralized manner is a challenge. To address this issue, we propose a decentralized many-to-many task offloading method that aims to minimize the average task completion latency of vehicles. In this study, we propose a data-sharing mechanism between vehicles and edge servers using the digital twin service, which provides global environmental perceptions to the vehicles by a low-cost approach. Additionally, we develop a mean-field multi-agent reinforcement learning algorithm to generate coordinated task offloading schemes. Instead of interacting with multiple agents, the vehicle only needs to respond to the mean action of the environment. Based on this transition, the agent generates coordinated task offloading decisions in complex scenarios. We evaluate the performance of our method using real urban traffic data. Experiment results verify the efficiency of our proposed method.

An off-policy multi-agent stochastic policy gradient algorithm for cooperative continuous control

Multi-agent reinforcement learning (MARL) algorithms based on trust regions (TR) have achieved significant success in numerous cooperative multi-agent tasks. These algorithms restrain the Kullback–Leibler (KL) divergence (i.e., TR constraint) between the current and new policies to avoid aggressive update steps and improve learning performance. However, the majority of existing TR-based MARL algorithms are on-policy, meaning that they require new data sampled by current policies for training and cannot utilize off-policy (or historical) data, leading to low sample efficiency. This study aims to enhance the data efficiency of TR-based learning methods. To achieve this, an approximation of the original objective function is designed. In addition, it is proven that as long as the update size of the policy (measured by the KL divergence) is restricted, optimizing the designed objective function using historical data can guarantee the monotonic improvement of the original target. Building on the designed objective, a practical off-policy multi-agent stochastic policy gradient algorithm is proposed within the framework of centralized training with decentralized execution (CTDE). Additionally, policy entropy is integrated into the reward to promote exploration, and consequently, improve stability. Comprehensive experiments are conducted on a representative benchmark for multi-agent MuJoCo (MAMuJoCo), which offers a range of challenging tasks in cooperative continuous multi-agent control. The results demonstrate that the proposed algorithm outperforms all other existing algorithms by a significant margin.

Robust Multiagent Reinforcement Learning for UAV Systems: Countering Byzantine Attacks

Multiple unmanned aerial vehicle (multi-UAV) systems have gained significant attention in applications, such as aerial surveillance and search and rescue missions. With the recent development of state-of-the-art multiagent reinforcement learning (MARL) algorithms, it is possible to train multi-UAV systems in collaborative and competitive environments. However, the inherent vulnerabilities of multiagent systems pose significant privacy and security risks when deploying general and conventional MARL algorithms. The presence of even a single Byzantine adversary within the system can severely degrade the learning performance of UAV agents. This work proposes a Byzantine-resilient MARL algorithm that leverages a combination of geometric median consensus and a robust state update model to mitigate, or even eliminate, the influence of Byzantine attacks. To validate its effectiveness and feasibility, the authors include a multi-UAV threat model, provide a guarantee of robustness, and investigate key attack parameters for multiple UAV navigation scenarios. Results from the experiments show that the average rewards during a Byzantine attack increased by up to 60% for the cooperative navigation scenario compared with conventional MARL techniques. The learning rewards generated by the baseline algorithms could not converge during training under these attacks, while the proposed method effectively converged to an optimal solution, proving its viability and correctness.

V-Learning—A Simple, Efficient, Decentralized Algorithm for Multiagent Reinforcement Learning

A major challenge of multiagent reinforcement learning (MARL) is the curse of multiagents, where the size of the joint action space scales exponentially with the number of agents. This remains to be a bottleneck for designing efficient MARL algorithms, even in a basic scenario with finitely many states and actions. This paper resolves this challenge for the model of episodic Markov games. We design a new class of fully decentralized algorithms—V-learning, which provably learns Nash equilibria (in the two-player zero-sum setting), correlated equilibria, and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with [Formula: see text], where Ai is the number of actions for the ith player. This is in sharp contrast to the size of the joint action space, which is [Formula: see text]. V-learning (in its basic form) is a new class of single-agent reinforcement learning (RL) algorithms that convert any adversarial bandit algorithm with suitable regret guarantees into an RL algorithm. Similar to the classical Q-learning algorithm, it performs incremental updates to the value functions. Different from Q-learning, it only maintains the estimates of V-values instead of Q-values. This key difference allows V-learning to achieve the claimed guarantees in the MARL setting by simply letting all agents run V-learning independently. Funding: This work was partially supported by Office of Naval Research Grant N00014-22-1-2253.

Multi-Agent Multi-Target Pursuit with Dynamic Target Allocation and Actor Network Optimization

In this paper, we consider the cooperative decision-making problem for multi-target tracking in multi-agent systems using multi-agent deep reinforcement learning algorithms. Multi-agent multi-target pursuit has faced new challenges in practical applications, where pursuers need to plan collision-free paths and appropriate multi-target allocation strategies to determine which target to track at the current time for each pursuer. We design three feasible multi-target allocation strategies from different perspectives. We compare our allocation strategies in the multi-agent multi-target pursuit environment that models collision risk and verify the superiority of the allocation strategy marked as POLICY3, considering the overall perspective of agents and targets. We also find that there is a significant gap in the tracking policies learned by agents when using the multi-agent reinforcement learning algorithm MATD3. We propose an improved algorithm, DAO-MATD3, based on dynamic actor network optimization. The simulation results show that the proposed POLICY3-DAO-MATD3 method effectively improves the efficiency of completing multi-agent multi-target pursuit tasks.

A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning

Target search is widely applied in military reconnaissance, geological exploration and personnel search and rescue. Most target search algorithms perform well in single target search but are inefficient or even ineffective in multi-target search. To solve the multi-target search problem in an uncertain environment, this paper constructs a multi-agent massive target cooperative search mission planning model and proposes an improved reinforcement learning algorithm using the action preference selection strategy. Based on the Reinforce algorithm, this algorithm solves the problem of invalid searches within the stochastic strategy by changing the preferred action selection method. The proposed method improves the efficiency of multiple agents in the search for targets without collision using a cooperative mechanism and reward rules based on the odor effect. Simulation experiments are conducted in three aspects to verify the effectiveness and robustness of the improved algorithm and compare it with other reinforcement learning algorithms in the field of multi-agent learning. The results demonstrate that the improved algorithm has obvious advantages in terms of mission success rate, target search rate and average search time, and the movement trajectory of multiple agents is more concise.

Multi-Agent Reinforcement Learning in non-zero-sum games: Algorithms and applications

Multi-Agent Reinforcement Learning (MARL) algorithms have been utilized in resource allocation modelling, game theory analysis, formation of alliances and so on. The potential for future research and applications of MARL in non-zero-sum games is vast, including integrating MARL with game theory and mechanism design, developing online learning algorithms for dynamic environments, and exploring applications in various domains. Currently, there is a relative lack of review literature on this area of research. This paper, therefore, aims to fill this gap. This work will first introduce the role of MARL in non-zero-sum games. Then it will discuss in detail the practical applications of MARL in economics, social sciences, and political science, which are typical of non-zero-sum games while presenting possibilities and challenges for future research. Finally, a summary is given at the end of the paper. This article provides insights into the current and future possibilities of using MARL in non-zero-sum games.

Multi-Agent Reinforcement Learning Control of a Hydrostatic Wind Turbine-Based Farm

This paper leverages multi-agent reinforcement learning (MARL) to develop an efficient control system for a wind farm comprising a new type of wind turbines with hydrostatic transmission. The primary motivation for hydrostatic wind turbines (HWT) is increased reliability, and reduced manufacturing, operating, and maintaining costs by removing troublesome components and reducing nacelle weight. Nevertheless, the high system complexity of HWT and the wake effect pose significant challenges for the control of HWT-based wind farms. We therefore propose a MARL algorithm named multi-agent policy optimization (MAPO), which allows agents (turbines) to gradually improve their control policies by repeatedly interacting with the environment to learn an optimal operation curve for wind farms. Simulation results based on a wind farm simulator, FAST.Farm, show that MAPO outperforms the greedy policy and a popular learning-based method, multi-agent deep deterministic policy gradient (MADDPG), in terms of power generation.

Joint Resource Allocation and Drones Relay Selection for Large-Scale D2D Communication Underlaying Hybrid VLC/RF IoT Systems

Relay-aided Device-to-Device (D2D) communication combining visible light communication (VLC) with radio frequency (RF) is a promising paradigm in the internet of things (IoT). Static relay limits the flexibility and maintaining connectivity of relays in Hybrid VLC/RF IoT systems. By using a drone as a relay station, it is possible to avoid obstacles such as buildings and to communicate in a line-of-sight (LoS) environment, which naturally aligns with the requirement of VLC Systems. To further support the application of VLC in the IoT, subject to the challenges imposed by the constrained coverage, the lack of flexibility, poor reliability, and connectivity, drone relay-aided D2D communication appears on the horizon and can be cost-effectively deployed for the large-scale IoT. This paper proposes a joint resource allocation and drones relay selection scheme, aiming to maximize the D2D system sum rate while ensuring the quality of service (QoS) requirements for cellular users (CUs) and D2D users (DUs). First, we construct a two-phase coalitional game to tackle the resource allocation problem, which exploits the combination of VLC and RF, as well as incorporates a greedy strategy. After that, a distributed cooperative multi-agent reinforcement learning (MARL) algorithm, called WoLF policy hill-climbing (WoLF-PHC), is proposed to address the drones relay selection problem. Moreover, to further reduce the computational complexity, we propose a lightweight neighbor–agent-based WoLF-PHC algorithm, which only utilizes historical information of neighboring DUs. Finally, we provide an in-depth theoretical analysis of the proposed schemes in terms of complexity and signaling overhead. Simulation results illustrate that the proposed schemes can effectively improve the system performance in terms of the sum rate and outage probability with respect to other outstanding algorithms.

Reinforcement Learning for Multiaircraft Autonomous Air Combat in Multisensor UCAV Platform

Autonomous air combat has received significant attention from researchers working on artificial intelligence (AI) applications. Most previous research on autonomous air combat has focused on one-on-one air combat scenarios in which air combat situational information is considered to be precisely observable. However, most modern air combats are conducted in formations, where air combat situational information is obtained from multiple sensors. Therefore, we introduce a novel automated maneuver decision architecture for close-range multi-aircraft air combat scenarios under the multi-sensor UCAV platform that can handle air combat scenarios with variable-sized formations. Then, a multi-agent reinforcement learning algorithm is proposed to obtain the strategy. The training performance of the training algorithm is evaluated and the obtained strategy is analyzed in different air combat scenarios it is found that these formations exhibit effective cooperative behavior in symmetric and asymmetric situations. Finally, we give ideas for the engineering implementation of a maneuver control architecture. This study provides a solution for future multi-aircraft autonomous air combat.

Graphon mean-field control for cooperative multi-agent reinforcement learning

The marriage between mean-field theory and reinforcement learning has shown a great capacity to solve large-scale control problems with homogeneous agents. To break the homogeneity restriction of mean-field theory, a recent interest is to introduce graphon theory to the mean-field paradigm. In this paper, we propose a graphon mean-field control (GMFC) framework to approximate cooperative heterogeneous multi-agent reinforcement learning (MARL) with nonuniform interactions and heterogeneous reward functions and state transition functions among agents and show that the approximate order is of O(1N), with N the number of agents. By discretizing the graphon index of GMFC, we further introduce a smaller class of GMFC called block GMFC, which is shown to well approximate cooperative MARL in terms of the value function and the policy. Finally, we design a Proximal Policy Optimization based algorithm for block GMFC that converges to the optimal policy of cooperative MARL. Our empirical studies on several examples demonstrate that our GMFC approach is comparable with the state-of-art MARL algorithms while enjoying better scalability.

Heuristically Assisted Multiagent RL-Based Framework for Computation Offloading and Resource Allocation of Mobile-Edge Computing

Mobile edge computing (MEC) as a promising technology enables it to satisfy ever-increasing demands for low-latency and ultra-reliable services. However, due to the limitations of computing capability and the dynamic network environment, it is challenging to process massive data with low latency. In this paper, we consider a dynamic MEC network with a high-performance edge server, multiple time-varying channels, and multiple mobile devices. We aim to find a policy that can maximize the processing success rate of computational tasks and the fairness index of the system while minimizing the process delays. To this end, we propose a heuristic-assisted multiagent reinforcement learning (RL) based framework to realize the joint optimization of computation offloading and resource allocation. On one hand, heuristic search is introduced in this framework to find a better resource allocation policy in edge servers and further assist the multiagent RL algorithm to determine offloading policy in mobile devices. On the other hand, a novel parametrized multiagent RL algorithm based on soft actor-critic (SAC) is also proposed to broaden the effectiveness and availability of the proposed framework. Simulation results of the average cumulative reward, success rate, processing delay, and fairness index fully verify the superiority of the proposed framework and algorithm for supporting this problem.

Intelligent Hierarchical Network Slicing Based on Dynamic Multi-Connectivity in Cell-Free Distributed Massive MIMO Systems

In order to provide customized services for the future sixth-generation (6 G) mass business, we propose a two-timescale intelligent radio access network (RAN) slicing scheme under the architecture of cell-free distributed massive multiple-input multiple-output (MIMO) systems. Cell-free distributed massive MIMO systems have powerful macro diversity gain and multi-user interference suppression capabilities to improve the performance of different slices, and are more flexible in terms of service types based on network slicing. In the proposed scheme, we utilize the long-term and short-term trends of network to achieve adaptive resource allocation at different timescales, so as to utilize resources more effectively and meet performance requirements in parallel. Moreover, multi-connectivity is utilized to improve link reliability and further improve system performance, and user clustering reduces the impact of pilot contamination on system performance. In order to implement the RAN control strategy effectively, an efficient two-level deep reinforcement learning framework is proposed and the multi-agent reinforcement learning algorithm is used to realize efficient network resource interaction in multi-device scenarios. Simulation results further verify the effectiveness of the proposed intelligent RAN slicing scheme.

RIS Aided NR-U and Wi-Fi Coexistence in Single Cell and Multiple Cell Networks on Unlicensed Bands

To address the scarcity of spectrum resource incurred by explosively increasing traffic load in existing fifth generation (5G) networks, we consider exploiting Reconfigurable intelligent surface (RIS) to harmonize the coexistence of cellular networks and WiFi system on the unlicensed bands. In particular, RIS is capable of not only improving the performance of new radio on unlicensed bands (NR-U) by increasing the signal to interference and noise ratio (SINR) of cellular users, but also decreasing the interference to the existing WiFi system caused by NR-U. Thus, we investigate the performance of RIS aided coexisting networks both in single cell and multiple cell scenarios. For the single cell scenario, we aim to improve the worst transmission rate of UEs in NR-U network. The problem is solved by a proposed alternating optimization-based solution. For the multiple cell scenario, cell cooperation is exploited jointly with multiple RISs to maximize the sum-rate of all UEs. Although the problem is more challenging than that of single cell case, we propose a multi-agent reinforcement learning (MARL) algorithm to solve it. Numeral results verify the effectiveness of our proposed schemes and show that deploying RISs is a promising solution for the enhancement of NR-U networks.

Heterogeneous optimal formation control of nonlinear multi-agent systems with unknown dynamics by safe reinforcement learning

This article presents the problem of distributed training with a decentralized execution policy as a safe, optimal formation control for a heterogeneous nonlinear multi-agent system. The control objective is to guarantee safety while achieving optimal performance. This objective is achieved by introducing novel distributed optimization problems with cost and local control barrier functions (CBFs). Designing an optimal formation controller is defined as optimal performance and modeled by a cost function. A local CBF trains a safe controller to ensure multi-agent systems operate within the safe regions. Instead of optimizing constrained optimization problems, this method generates safe, optimal controllers from unconstrained optimization problems by utilizing local CBFs. As a result, the presented approach has a lower computational cost than constrained optimization problems. It is proven that the proposed controller's optimality and stability are not affected by adding the local CBF to the cost function. A safe, optimal policy is iteratively derived using a new off-policy multi-agent reinforcement learning (MARL) algorithm that does not need knowledge of the agents' dynamics. Finally, the effectiveness of the proposed algorithm is evaluated through simulation of the collision-free problem of the multi-quadrotor formation control.

LJIR: Learning Joint-Action Intrinsic Reward in cooperative multi-agent reinforcement learning

Effective exploration is the key to achieving high returns for reinforcement learning. Agents must explore jointly in multi-agent systems to find the optimal joint policy. Due to the exploration problem and the shared reward, the policy-based multi-agent reinforcement learning algorithms face policy overfitting, which may lead to the joint policy falling into a local optimum. This paper introduces a novel general framework called Learning Joint-Action Intrinsic Reward (LJIR) for improving multi-agent reinforcement learners’ joint exploration ability and performance. LJIR observes agents’ state and joint actions to learn to construct an intrinsic reward online that can guide effective joint exploration. With the novel combination of Transformer and random network distillation, LJIR selects the novel states to give more intrinsic rewards, which help agents find the best joint actions. LJIR can dynamically adjust the weight of exploration and exploitation during training and keep the policy invariance finally. To ensure LJIR seamlessly adopts existing MARL algorithms, we also provide a flexible combination method for intrinsic and external rewards. Empirical results on the SMAC benchmark show that the proposed method achieves state-of-the-art performance in challenging tasks.

Improving Sample Efficiency of Multiagent Reinforcement Learning With Nonexpert Policy for Flocking Control

Control algorithms of a multi-agent system (MAS) have been applied to many Internet-of-Things devices, such as unmanned aerial vehicles and autonomous underwater vehicles. Flocking control is a crucial problem in MAS to enhance the safety and cooperativity of agents, which requires the agents to maintain the flock when navigating to a target position and avoiding collisions. In comparison with traditional algorithms, methods based on multi-agent reinforcement learning (MARL) can solve the problem of flocking control more flexibly and adapt to more complex environments. However, MARL-based methods demand a huge number of interactions between agents and the environment, resulting in the problem of sample inefficiency. In this paper, we propose Non-expert Policy Aided MARL (NPA-MARL) to improve sample efficiency, which utilizes a fundamental MARL algorithm and a prior policy whose performance can be non-expert. Before online MARL training, NPA-MARL generates demonstrations by the non-expert policy to pretrain agents, while preventing overfitting demonstrations. During online training, NPA-MARL instructs agents to imitate the non-expert policy if the non-expert policy is better in agents’ recognition. We leverage NPA-MARL to solve the problem of flocking control. Experimental results show that NPA-MARL improves sample efficiency and policy performance in flocking control. Besides, NPA-MARL has the scalability of more agents and the flexibility of choice of the non-expert policy and fundamental MARL algorithm.