Multi-agent Reinforcement Learning Framework Research Articles

Markov game based on reinforcement learning solution against cyber–physical attacks in smart grid

The rapid expansion of information networks has amplified vulnerabilities in cyberspace, particularly within the power grid and communication layers of smart grid systems. Safeguarding smart grid security has become imperative for users concerned about both online and offline privacy. Detecting attacks in smart grids is crucial for ensuring the safety and continuity of the electrical network. This paper introduces a novel stealth attack-defense game that examines the impact of these invasions on the availability, confidentiality, and integrity of phasor measurement units. The proposed system considers both attack and defense perspectives. Attackers focus on key parameters to covertly infiltrate system layers while defending the smart grid system requires advanced knowledge of data flow and critical insights to establish robust protections. To address this cybersecurity challenge, we propose a game approach based on a partially observed Markov game with an improved Shapley Q-value and a multi-agent reinforcement learning framework, implemented in cooperative, communicative, and dynamic environments. The evaluation results demonstrate the robustness of the proposed approach against coordinated attacks in smart grid environments. Additionally, the results emphasize the significance of attack distribution as a crucial trace for identifying the attack source. Our approach shows promise for addressing various types of attacks against smart grid systems.

Learning Advanced Locomotion for Quadrupedal Robots: A Distributed Multi-Agent Reinforcement Learning Framework with Riemannian Motion Policies

Recent advancements in quadrupedal robotics have explored the motor potential of these machines beyond simple walking, enabling highly dynamic skills such as jumping, backflips, and even bipedal locomotion. While reinforcement learning has demonstrated excellent performance in this domain, it often relies on complex reward function tuning and prolonged training times, and the interpretability is not satisfactory. Riemannian motion policies, a reactive control method, excel in handling highly dynamic systems but are generally limited to fully actuated systems, making their application to underactuated quadrupedal robots challenging. To address these limitations, we propose a novel framework that treats each leg of a quadrupedal robot as an intelligent agent and employs multi-agent reinforcement learning to coordinate the motion of all four legs. This decomposition satisfies the conditions for utilizing Riemannian motion policies and eliminates the need for complex reward functions, simplifying the learning process for high-level motion modalities. Our simulation experiments demonstrate that the proposed method enables quadrupedal robots to learn stable locomotion using three, two, or even a single leg, offering advantages in training speed, success rate, and stability compared to traditional approaches, and better interpretability. This research explores the possibility of developing more efficient and adaptable control policies for quadrupedal robots.

Cooperative traffic optimization with multi-agent reinforcement learning and evolutionary strategy: Bridging the gap between micro and macro traffic control

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM).

Multi-agent reinforcement learning framework based on information fusion biometric ticketing data in different public transport modes

In smart cities, biometric technologies have become extensively used for ticket authentication on public transport. Information fusion plays a key role in biometric ticketing, allowing ticket validation with more data source validation in different public transport modes. This paper proposes a novel biometric technology-based mobile ticket application-based system. We formulate the problem as a multi-agent reinforcement learning framework for biometric ticketing in multi-transport environments. Specifically, we propose the Asynchronous Advantage Critic Biometric Ticketing Framework (A3CBTF) algorithm, which consists of different schemes based on the proposed system. The proposed algorithm framework operates in hybrid transport modes using a parallel reinforcement learning scheme. A key advantage of A3CBTF is that it enables passengers to use a single ticket for various public transport modes. Additionally, even when a passenger’s mobile device is stolen, lost, or has a dead battery, they can still validate their tickets through different information fusion sources, such as fingerprint and face recognition. A3CBTF is a multi-agent system that integrates mobile, transport, edge, and cloud servers to facilitate ticket validation in a distributed environment. By optimizing both convex and concave optimizations, A3CBTF ensures efficient ticket validation with minimal processing time and maximizes validation rewards across different biometric technologies. Experimental results demonstrate that A3CBTF outperforms mobile off with other options such as fingerprint and face recognition in public transport as compared to other ticketing systems.

A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning

Faced with the challenges of security strategy design brought about by the complexity of attack behavior and the dynamism of network structure, dynamic hierarchical intelligent defense methods have shown their effectiveness. However, in complex network environments, their application requires a higher level of coordination mechanisms. Therefore, this paper proposes a hierarchical multi-agent reinforcement learning network attack and defense game and cooperative defense decision-making method, which autonomously and efficiently completes the formulation of defense strategies and defense behavior responses. Firstly, we construct a Stackelberg hypergame model of cyberspace conflicts, and under the condition of information loss, characterize the multi-layer dynamic defense coordination response mechanism. Secondly, By utilizing a hierarchical multi-agent reinforcement learning method as the driving force for game evolution, we sequentially solve the Nash equilibrium of the game, and form a dynamic autonomous defense strategy. Finally, we construct a hierarchical multi-agent reinforcement learning framework, which decouples the defense decision problem, reduces the dimension of the defense action space and the exploration difficulty of the strategy space, and learns coordinated defense strategies more efficiently. We used the CybORG (Cyber Operations Research Gym) environment for simulation. We compared and analyzed the autonomously generated cyber defense strategies with other related works, verifying the superior coordination performance of our method in defense strategy generation and control.

Carbon Footprint Reduction for Sustainable Data Centers in Real-Time

As machine learning workloads are significantly increasing energy consumption, sustainable data centers with low carbon emissions are becoming a top priority for governments and corporations worldwide. This requires a paradigm shift in optimizing power consumption in cooling and IT loads, shifting flexible loads based on the availability of renewable energy in the power grid, and leveraging battery storage from the uninterrupted power supply in data centers, using collaborative agents. The complex association between these optimization strategies and their dependencies on variable external factors like weather and the power grid carbon intensity makes this a hard problem. Currently, a real-time controller to optimize all these goals simultaneously in a dynamic real-world setting is lacking. We propose a Data Center Carbon Footprint Reduction (DC-CFR) multi-agent Reinforcement Learning (MARL) framework that optimizes data centers for the multiple objectives of carbon footprint reduction, energy consumption, and energy cost. The results show that the DC-CFR MARL agents effectively resolved the complex interdependencies in optimizing cooling, load shifting, and energy storage in real-time for various locations under real-world dynamic weather and grid carbon intensity conditions. DC-CFR significantly outperformed the industry-standard ASHRAE controller with a considerable reduction in carbon emissions (14.5%), energy usage (14.4%), and energy cost (13.7%) when evaluated over one year across multiple geographical regions.

UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution

In recent years, there has been a growing interest in utilizing reinforcement learning (RL) to optimize long-term rewards in recommender systems. Since industrial recommender systems are typically designed as multi-stage systems, RL methods with a single agent face challenges when optimizing multiple stages simultaneously. The reason is that different stages have different observation spaces, and thus cannot be modeled by a single agent. To address this issue, we propose a novel UNidirectional-EXecution-based multi-agent Reinforcement Learning (UNEX-RL) framework to reinforce the long-term rewards in multi-stage recommender systems. We show that the unidirectional execution is a key feature of multi-stage recommender systems, bringing new challenges to the applications of multi-agent reinforcement learning (MARL), namely the observation dependency and the cascading effect. To tackle these challenges, we provide a cascading information chain (CIC) method to separate the independent observations from action-dependent observations and use CIC to train UNEX-RL effectively. We also discuss practical variance reduction techniques for UNEX-RL. Finally, we show the effectiveness of UNEX-RL on both public datasets and an online recommender system with over 100 million users. Specifically, UNEX-RL reveals a 0.558% increase in users' usage time compared with single-agent RL algorithms in online A/B experiments, highlighting the effectiveness of UNEX-RL in industrial recommender systems.

Adaptive overlay selection at the SD-WAN edges: A reinforcement learning approach with networked agents

Software-Defined Wide Area Networks (SD-WANs) have emerged as a promising solution to address the connectivity demands of modern distributed enterprises. However, the effective application of the Software Defined Networking (SDN) paradigm in such broad and dynamic environments remains a significant challenge. In this paper, we present two novel contributions. First, we design a decentralized control plane for SD-WANs that leverages edge-based network monitoring and overlays’ configuration. Then we present a Reinforcement Learning-based orchestration plane that leverages local information for the enforcement of SD-WAN policies. Since traditional approaches suffer either a lack of scalability due to the problem’s complexity or suboptimal performance due to isolated decision-making, the proposed approach leverages a cooperative Multi-Agent Reinforcement Learning framework. Our novel cooperative approach is based on per-site agents that exchange a small amount of information to enhance performance while preserving scalability. To validate the efficacy of our proposed approach, we conducted an extensive experimental evaluation considering diverse SD-WAN scenarios. Results show that our framework is able to satisfy global network policies for a multi-site SD-WAN with different QoS requirements and cost constraints.

Multi-Agent Reinforcement Learning for Extended Flexible Job Shop Scheduling

An extended flexible job scheduling problem is presented with characteristics of technology and path flexibility (dual flexibility), varied transportation time, and an uncertain environment. The scheduling can greatly increase efficiency and security in complex scenarios, e.g., distributed vehicle manufacturing, and multiple aircraft maintenance. However, optimizing the scheduling puts forward higher requirements on accuracy, real time, and generalization, while subject to the curse of dimension and usually incomplete information. Various coupling relations among operations, stations, and resources aggravate the problem. To deal with the above challenges, we propose a multi-agent reinforcement learning algorithm where the scheduling environment is modeled as a decentralized partially observable Markov decision process. Each job is regarded as an agent that decides the next triplet, i.e., operation, station, and employed resource. This paper is novel in addressing the flexible job shop scheduling problem with dual flexibility and varied transportation time in consideration and proposing a double Q-value mixing (DQMIX) optimization algorithm under a multi-agent reinforcement learning framework. The experiments of our case study show that the DQMIX algorithm outperforms existing multi-agent reinforcement learning algorithms in terms of solution accuracy, stability, and generalization. In addition, it achieves better solution quality for larger-scale cases than traditional intelligent optimization algorithms.

Harnessing Online Knowledge Transfer for Enhanced Search and Rescue Decisions via Multi-Agent Reinforcement Learning

In the rapidly evolving domain of the Internet of Things (IoT), devices play an instrumental role in high-stakes scenarios like search and rescue (SAR) operations. Traditional decision-making processes within SAR missions often struggle to cope with the dynamic and unpredictable nature of such environments, leading to inefficiencies and delayed responses. This paper aims to explore the potential of multi-agent reinforcement learning (MARL) to improve the decision-making process within SAR operations underpinned by IoT. Functional, current methods are limited by their static decision frameworks and inability to adapt in real time to the chaotic variables present in SAR situations. We introduced a novel MARL framework and compared its performance against benchmark strategies, specifically the multi-agent deep deterministic policy gradient (MADDPG) approach. Uniquely enhanced by online knowledge transfer, the framework leverages the capabilities of the deep deterministic policy gradient (DDPG) method. The preliminary findings underscore the proposed framework’s superior efficiency and speed in SAR contexts. Our research highlights MARL’s transformative potential, positing it as a groundbreaking strategy for IoT-based decision making in high-pressure SAR environments with suggestions for further studies in varied real-world scenarios.

Multiagent Reinforcement Learning-Based Orbital Edge Offloading in SAGIN Supporting Internet of Remote Things

We investigate a computing task scheduling problem in space-air-ground integrated network (SAGIN) for Internet of remote Things (IoRT). In the considered scenario, the unmanned aerial vehicles (UAVs) collect computing tasks from IoRT devices and make offloading decisions, in which the tasks can be computed at the UAVs or offloaded to the low earth orbital (LEO) satellite. The optimization objective is to design offloading strategies for each UAV that maximum the number of tasks satisfies delay constraint and reduces the energy consumption of UAVs. We formulate this problem as a nonlinear integer optimization problem, and remodel it as a stochastic game to solve it. To copy with the dynamic and complexity environment, we propose a learning-based orbital edge offloading (LOEF) approach which can coordinate all UAVs to learn the optimal offloading strategies. This method is based on the Actor-Critic framework of multi-agent reinforcement learning, the Actor observes the environment and output the current offloading decision, the Critic evaluates the action of the Actor and coordinate the actions of all UAV to increase the reward of system. The simulation results show that the proposed offloading strategy converges fast, increases the number of satisfying the delay constraints tasks and the energy utilization of UAVs compared with the baseline strategies.

A multiagent reinforcement learning framework for off-policy evaluation in two-sided markets

A multiagent reinforcement learning framework for off-policy evaluation in two-sided markets

Generalized multi-agent competitive reinforcement learning with differential augmentation

Observation generalization in multi-agent reinforcement learning is essential for real-world applications as it enables agents to formulate effective policies to deal with environmental uncertainty. However, most existing studies seldom consider the generalization ability of agents in competitive scenarios, which is not in line with the actual process of intelligent groups in real applications. We propose a generalized multi-agent reinforcement learning framework (GMARL) by incorporating the data augmentation mechanism of differential privacy, which enhances the agent’s adaptability and improves the model’s stability. Specifically, we introduce a Laplacian mechanism with temporary adjacency sensitivity into the training and add an observation supplement to the agent’s input. Under this mechanism, the agent intelligently considers the environment fluctuation, thus guarding against the adverse effects of the uncertain changes in the environment states. We show the rigorous guarantee of the difference balance between the observation supplement and the original data, and propose a series of variants upon the GMARL framework. Extensive experimental results on five benchmark multi-agent competitive environments against with five state-of-the-art algorithms validate the effectiveness and robustness of our proposed method.

A multi-agent reinforcement learning framework for optimizing financial trading strategies based on TimesNet

An increasing number of studies have shown the effectiveness of using deep reinforcement learning to learn profitable trading strategies from financial market data. However, a single-agent model is not sufficient to handle complex financial scenarios. To address this problem, a novel approach called Multi-Agent Double Deep Q-Network (Later called MADDQN) is proposed in this study, which reasonably balances the pursuit of maximum revenue and the avoidance of risk under the multi-agent reinforcement learning framework by innovatively employing two different agents represented respectively by two time-series feature extraction networks, TimesNet, and the Multi-Scale Convolutional Neural Network. Furthermore, to achieve a more generalized model suitable for different underlying assets, a mixed dataset containing three major U.S. stock indexes is collected. And the proposed model has been pre-trained in this dataset and subsequently refined for the specified asset. The results from experiments on five different stock indices show that the proposed MADDQN has an average cumulative return of 23.08%, outperforming the other baseline methods. Besides, the multi-agent model demonstrates its advantage in balancing the risk and revenue, in comparison with the single-agent models. Additionally, The generalization experiments confirm that the proposed MADDQN method after pre-training in the proposed mixed dataset could be stably transferred to the other underlying assets with a refinement. These findings indicate that the proposed framework not only achieves good performance in complex financial market environments but also is able to generalize robustly across different scenarios in various markets.

MAFSIDS: a reinforcement learning-based intrusion detection model for multi-agent feature selection networks

Large unbalanced datasets pose challenges for machine learning models, as redundant and irrelevant features can hinder their effectiveness. Furthermore, the performance of intrusion detection systems (IDS) can be further degraded by the emergence of new network attack types. To address these issues, we propose MAFSIDS (Multi-Agent Feature Selection Intrusion Detection System), a DQL (Deep Q-Learning) based IDS.MAFSIDS comprises a feature self-selection algorithm and a DRL (Deep Reinforcement Learning) attack detection module. The feature self-selection algorithm leverages a multi-agent reinforcement learning framework, which redefines the feature selection problem by converting the traditional {2}^{N} feature selection space into N agent representations. This approach reduces model complexity and enhances the search strategy for feature selection. To ensure accurate feature representation and expedite the feature selection process, we have also developed a GCN (Graph Convolutional Network) method that extracts deeper features from the data. The DRL attack detection module utilizes the Mini-Batchs technique to encode the data, allowing reinforcement learning to be applied in a supervised learning context. This integration improves accuracy. Additionally, the policy network in this module is designed to be minimalist, enhancing model efficiency. To evaluate the performance of our model, we conducted comprehensive simulation experiments using Python. We tested the model using the CSE-CIC-IDS2018 and NSL-KDD datasets, achieving impressive accuracy rates of 96.8% and 99.1%, as well as F1-Scores of 96.3% and 99.1%, respectively. The selected feature subset successfully eliminates approximately 80% of redundant features compared to the original feature set. Furthermore, we compared our proposed model with other popular machine-learning models.

Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos.

Adversarial robustness assessment for video recognition models has raised concerns owing to their wide applications on safety-critical tasks. Compared with images, videos have much high dimension, which brings huge computational costs when generating adversarial videos. This is especially serious for the query-based black-box attacks where gradient estimation for the threat models is usually utilized, and high dimensions will lead to a large number of queries. To mitigate this issue, we propose to simultaneously eliminate the temporal and spatial redundancy within the video to achieve an effective and efficient gradient estimation on the reduced searching space, and thus query number could decrease. To implement this idea, we design the novel Adversarial spatial-temporal Focus (AstFocus) attack on videos, which performs attacks on the simultaneously focused key frames and key regions from the inter-frames and intra-frames in the video. AstFocus attack is based on the cooperative Multi-Agent Reinforcement Learning (MARL) framework. One agent is responsible for selecting key frames, and another agent is responsible for selecting key regions. These two agents are jointly trained by the common rewards received from the black-box threat models to perform a cooperative prediction. By continuously querying, the reduced searching space composed of key frames and key regions is becoming precise, and the whole query number becomes less than that on the original video. Extensive experiments on four mainstream video recognition models and three widely used action recognition datasets demonstrate that the proposed AstFocus attack outperforms the SOTA methods, which is prevenient in fooling rate, query number, time, and perturbation magnitude at the same time.

A Cooperative Multiagent Reinforcement Learning Framework for Droplet Routing in Digital Microfluidic Biochips

A Cooperative Multiagent Reinforcement Learning Framework for Droplet Routing in Digital Microfluidic Biochips

Multi-objective reinforcement learning for designing ethical multi-agent environments

AbstractThis paper tackles the open problem of value alignment in multi-agent systems. In particular, we propose an approach to build an ethical environment that guarantees that agents in the system learn a joint ethically-aligned behaviour while pursuing their respective individual objectives. Our contributions are founded in the framework of Multi-Objective Multi-Agent Reinforcement Learning. Firstly, we characterise a family of Multi-Objective Markov Games (MOMGs), the so-called ethical MOMGs, for which we can formally guarantee the learning of ethical behaviours. Secondly, based on our characterisation we specify the process for building single-objective ethical environments that simplify the learning in the multi-agent system. We illustrate our process with an ethical variation of the Gathering Game, where agents manage to compensate social inequalities by learning to behave in alignment with the moral value of beneficence.

Decentralized signal control for multi-modal traffic network: A deep reinforcement learning approach

Managing traffic flow at intersections in a large-scale network remains challenging. Multi-modal signalized intersections integrate various objectives, including minimizing the queue length and maintaining constant bus headway. Inefficient traffic signals and bus headway control strategies may cause severe traffic jams, high delays for bus passengers, and bus bunching that harms bus line operations. To simultaneously improve the level of service for car traffic and the bus system in a multi-modal network, this paper integrates bus priority and holding with traffic signal control via decentralized controllers based on Reinforcement Learning (RL). The controller agents act and learn from a synthetic traffic environment built with the microscopic traffic simulator SUMO. Action information is shared among agents to achieve cooperation, forming a Multi-Agent Reinforcement Learning (MARL) framework. The agents simultaneously aim to minimize vehicles' total stopping time and homogenize the forward and backward space headways for buses approaching intersections at each decision step. The Deep Q-Network (DQN) algorithm is applied to manage the continuity of the state space. The tradeoff between the bus transit and car traffic objectives is discussed using various numerical experiments. The introduced method is tested in scenarios with distinct bus lane layouts and bus line deployments. The proposed controller outperforms model-based adaptive control methods and the centralized RL method regarding global traffic efficiency and bus transit stability. Furthermore, the remarkable scalability and transferability of trained models are demonstrated by applying them to several different test networks without retraining.

A deep multi-agent reinforcement learning framework for autonomous aerial navigation to grasping points on loads

Deep reinforcement learning, by taking advantage of neural networks, has made great strides in the continuous control of robots. However, in scenarios where multiple robots are required to collaborate with each other to accomplish a task, it is still challenging to build an efficient and scalable multi-agent control system due to increasing complexity. In this paper, we regard each unmanned aerial vehicle (UAV) with its manipulator as one agent, and leverage the power of multi-agent deep deterministic policy gradient (MADDPG) for the cooperative navigation and manipulation of a load. We propose solutions for addressing navigation to grasping point problem in targeted and flexible scenarios, and mainly focus on how to develop model-free policies for the UAVs without relying on a trajectory planner. To overcome the challenges of learning in scenarios with an increasing number of grasping points, we incorporate the demonstrations from an Optimal Reciprocal Collision Avoidance (ORCA) algorithm into our framework to guide the policy training and adapt two novel techniques into the architecture of MADDPG. Furthermore, curriculum learning with the attention mechanism is utilized by reusing knowledge from fewer grasping points to facilitate the training of a load with more points. Our experiments were validated by a load with three, four and six grasping points respectively in Coppeliasim simulator and then transferred into the real world with Crazyflie quadrotors. Our results show that the average tracking deviations from the desirable grasping point to the final position of the UAV can be less than 10 cm in some real-world experiments. Compared with state-of-the-art model-free reinforcement learning and swarm optimization algorithms, results show that our proposed methods outperform other baselines with a reasonable success rate especially in the scenarios with more grasping points. Furthermore, the learned optimal policies enable UAVs to reach and hover over all the grasping points before manipulation without any collision. We conducted a comprehensive analysis of both targeted and flexible navigation, highlighting their respective advantages and disadvantages.