Multi-agent Deep Deterministic Policy Gradient Research Articles

Intelligent Decision-Making and Human Language Communication Based on Deep Reinforcement Learning in a Wargame Environment

The application of artificial intelligence (AI) in games has been significantly developed and attracted much attention over the past few years. This article not only leverages the reinforcement learning multiagent deep deterministic policy gradient algorithm to realize the dynamic decision-making of game AI but also creatively incorporates deep learning and natural language processing technologies in the wargame field to transform game context situation maps into textual suggestions in wargame confrontation. In this article, we effectively integrate reinforcement learning technologies, deep learning technologies, and natural language processing technologies to generalize the semantic text output at state-of-the-art accuracy, which plays an important role in human understanding of game AI behavior. The experimental results are promising and can be used to verify the feasibility, accuracy, and performance of our proposed model in extensive simulations against benchmarking methods.

Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: A cloud-edge collaboration architecture

With the construction of the power Internet of Things (IoT), communication between smart devices in urban distribution networks has been gradually moving towards high speed, high compatibility, and low latency, which provides reliable support for reconfiguration optimization in urban distribution networks. Thus, this study proposed a deep reinforcement learning based multi-level dynamic reconfiguration method for urban distribution networks in a cloud-edge collaboration architecture to obtain a real-time optimal multi-level dynamic reconfiguration solution. First, the multi-level dynamic reconfiguration method was discussed, which included feeder-, transformer-, and substation-levels. Subsequently, the multi-agent system was combined with the cloud-edge collaboration architecture to build a deep reinforcement learning model for multi-level dynamic reconfiguration in an urban distribution network. The cloud-edge collaboration architecture can effectively support the multi-agent system to conduct “centralized training and decentralized execution” operation modes and improve the learning efficiency of the model. Thereafter, for a multi-agent system, this study adopted a combination of offline and online learning to endow the model with the ability to realize automatic optimization and updation of the strategy. In the offline learning phase, a Q-learning-based multi-agent conservative Q-learning (MACQL) algorithm was proposed to stabilize the learning results and reduce the risk of the next online learning phase. In the online learning phase, a multi- agent deep deterministic policy gradient (MADDPG) algorithm based on policy gradients was proposed to explore the action space and update the experience pool. Finally, the effectiveness of the proposed method was verified through a simulation analysis of a real-world 445-node system.

AutoMan: Resource-efficient provisioning with tail latency guarantees for microservices

Modern user-facing services are progressively evolving from large monolithic applications to complex graphs of loosely-coupled microservices. While microservice architecture greatly improves the efficiency of development and operation, it also complicates resource allocation and performance guarantee due to complex dependencies across different microservices. To prevent resource wastage and ensure user satisfaction, we present AutoMan, a learning-driven resource manager for microservices that enables much more efficient resource provisioning while guaranteeing the end-to-end tail latency Service Level Objective (SLO). AutoMan leverages a multi-agent deep deterministic policy gradient (MADDPG)-based method to capture the dependencies across different microservices and to allocate a proper amount of resources to each microservice subject to the target end-to-end tail latency SLO. During runtime, it further proactively identifies the critical microservices responsible for performance anomaly by deriving partial SLOs mathematically, and performs dynamic reprovisioning to mitigate the potential SLO violations. Testbed experiments show that AutoMan can save CPU and memory resources by up to 49.6% and 29.1% on average, while guaranteeing the same end-to-end tail latency objective.

Autonomous collaborative combat strategy of unmanned system group in continuous dynamic environment based on PD-MADDPG

In this study, we studied the Unmanned System Group (USG) Autonomous Collaborative Combat Strategy (ACCS) and the Parallel Decoupling-Multi-agent Deep Deterministic Policy Gradient (PD-MADDPG) algorithm was proposed. This was conducted on the background of USG confrontation game problem between ourselves and the enemy in the continuous dynamic environment of future aerial combat. An independent Parallel Benchmark Critic (PB-Critic) network and Parallel Decoupling Critic (PD-Critic) network for each member of USG was developed to maximize the whole reward and member reward of USG under time parallelism. By integrating Forerunner Mechanism (FM) into the parallel decoupling reward function, the reward sparse problem confronted by parallel decoupling reward function at the beginning of the algorithmic training was controlled as well as the convergence efficiency of USG in the continuous dynamic environment of future aerial combat. Introduction of Symmetric Attention Mechanism (SAM) to the Critic-network and Actor-network shortened the screening radius of helpful information related to the confrontation game of USG of both sides of ourselves and enemy in the continuous dynamic environment. Assuming the multiple typical confrontation game control strategies of USG of the enemy in the continuous dynamic environment, the effectiveness and feasibility of the strategies were simulated and verified. The PD-MADDPG improved the multiple inadequate natural endowments of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) which showed that the policy training convergence and execution stability were greatly enhanced, and USG’s behavioral autonomy for collaborative combat in the continuous dynamic environment was further improved.

The Study of Crash-Tolerant, Multi-Agent Offensive and Defensive Games Using Deep Reinforcement Learning

In the multi-agent offensive and defensive game (ODG), each agent achieves its goal by cooperating or competing with other agents. The multi-agent deep reinforcement learning (MADRL) method is applied in similar scenarios to help agents make decisions. In various situations, the agents of both sides may crash due to collisions. However, the existing algorithms cannot deal with the situation where the number of agents reduces. Based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm, we study a method to deal with a reduction in the number of agents in the training process without changing the structure of the neural network (NN), which is called the frozen agent method for the MADDPG (FA-MADDPG) algorithm. In addition, we design a distance–collision reward function to help agents learn strategies better. Through the experiments in four scenarios with different numbers of agents, it is verified that the algorithm we proposed can not only successfully deal with the problem of agent number reduction in the training stage but also show better performance and higher efficiency than the MADDPG algorithm in simulation.

Anti-conflict AGV path planning in automated container terminals based on multi-agent reinforcement learning

AGV conflict prevention path planning is a key factor to improve transportation cost and operation efficiency of the container terminal. This paper studies the anti-conflict path planning problem of Automated Guided Vehicle (AGV) in the horizontal transportation area of the Automated Container Terminals (ACTs). According to the characteristics of magnetic nail guided AGVs, a node network is constructed. Through the analysis of two conflict situations, namely the opposite conflict situation and same point occupation conflict situation, an integer programming model is established to obtain the shortest path. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is proposed to solve the problem, and the Gumbel-Softmax strategy is applied to discretize the scenario created by the node network. A series of numerical experiments are conducted to verify the effectiveness and the efficiency of the model and the algorithm.

PASCAL: PopulAtion-Specific Curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms

Flocking with a swarm of unmanned aerial vehicles (UAVs) has been playing an important role in various applications. However, the complexity of developing a collision-free flocking policy for a UAV swarm increases tremendously with the swarm scale. In this paper, a novel curriculum-based multi-agent deep reinforcement learning (MADRL) approach is proposed for flocking with collision avoidance of large-scale fixed-wing UAV swarms by progressively increasing the population of UAVs, which we call PopulAtion-Specific Curriculum-based MADRL (PASCAL). Specifically, PASCAL has two core components of policy learning and knowledge transfer. For policy learning, an improved multi-agent deep deterministic policy gradient algorithm is proposed to accelerate the learning process. For knowledge transfer, an attention-based population-invariant network is designed to handle dynamic dimensional inputs, enabling the reloading and fine-tuning of the previously learned models. Finally, both numerical and semi-physical simulations demonstrate the advantages of PASCAL in terms of learning efficiency and generalization capability for different swarm scales.

Game-theoretic physical layer authentication for spoofing detection in internet of things

The Internet of Things (IoT) has permeated various fields relevant to our lives. In these applications, countless IoT devices transmit vast amounts of data, which often carry important and private information. To prevent malicious users from spoofing these information, the first critical step is effective authentication. Physical Layer Authentication (PLA) employs unique characteristics inherent to wireless signals and physical devices and is promising in the IoT due to its flexibility, low complexity, and transparency to higher layer protocols. In this paper, the focus is on the interaction between multiple malicious spoofers and legitimate receivers in the PLA process. First, the interaction is formulated as a static spoof detection game by including the spoofers and receivers as players. The best authentication threshold of the receiver and the attack rate of the spoofers are consideblack as Nash Equilibrium (NE). Then, closed-form expressions are derived for all NEs in the static environment in three cases: multiplayer games, zero-sum games with collisions, and zero-sum games without collisions. Considering the dynamic environment, a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is propsed to analyse the interactions of recevier and spoofers. Last, comprehensive simulation experiments are conducted and demonstrate the impact of environmental parameters on the NEs, which provides guidance to design effective PLA schemes.

Auto-Tuning with Reinforcement Learning for Permissioned Blockchain Systems

In a permissioned blockchain, performance dictates its development, which is substantially influenced by its parameters. However, research on auto-tuning for better performance has somewhat stagnated because of the difficulty posed by distributed parameters; thus, it is possible only with difficulty to propose an effective auto-tuning optimization scheme. To alleviate this issue, we lay a solid basis for our research by first exploring the relationship between parameters and performance in Hyperledger Fabric, a permissioned blockchain, and we propose Athena, a Fabric-based auto-tuning system that can automatically provide parameter configurations for optimal performance. The key of Athena is designing a new Permissioned Blockchain Multi-Agent Deep Deterministic Policy Gradient (PB-MADDPG) to realize heterogeneous parameter-tuning optimization of different types of nodes in Fabric. Moreover, we select parameters with the most significant impact on accelerating recommendation. In its application to Fabric, a typical permissioned blockchain system, with 12 peers and 7 orderers, Athena achieves a throughput improvement of 470.45% and a latency reduction of 75.66% over the default configuration. Compared with the most advanced tuning schemes (CDBTune, Qtune, and ResTune), our method is competitive in terms of throughput and latency.

One-Layer Real-Time Optimization Using Reinforcement Learning: A Review with Guidelines

This paper reviews real-time optimization from a reinforcement learning point of view. The typical control and optimization system hierarchy depend on the layers of real-time optimization, supervisory control, and regulatory control. The literature about each mentioned layer is reviewed, supporting the proposal of a benchmark study of reinforcement learning using a one-layer approach. The multi-agent deep deterministic policy gradient algorithm was applied for economic optimization and control of the isothermal Van de Vusse reactor. The cooperative control agents allowed obtaining sufficiently robust control policies for the case study against the hybrid real-time optimization approach.

Reputation-Aware Multi-Agent DRL for Secure Hierarchical Federated Learning in IoT

Aiming at protecting device data privacy, Federated Learning (FL) is a framework of distributed machine learning in which devices’ local model parameters are exchanged with a centralized server without revealing the actual data. Hierarchical Federated Learning (HFL) framework was introduced to improve FL communication efficiency where devices are clustered and seek model consensus with the support of edge servers (e.g., base stations). Devices in a cluster submit their local model updates to their assigned local edge server for aggregation at each iteration. The edge servers transmit the aggregated models to a centralized server and establish a global consensus. However, similar to FL, adversaries may threaten the security and privacy of HFL. The client devices within a cluster may deliberately provide unreliable local model updates through poisoning attacks or poor-quality model updates due to inconsistent communication channels, increased device mobility, or inadequate device resources. To address the above challenges, this paper investigates the client selection problem in the HFL framework to eliminate the impact of unreliable clients while maximizing the global model accuracy of HFL. Each FL edge server is equipped with a Deep Reinforcement Learning (DRL)-based reputation model to optimally measure the reliability and trustworthiness of FL workers within its cluster. A Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is utilized to enhance the accuracy and stability of the HFL global model, given the workers’ dynamic behaviors in the HFL environment. The experimental results indicate that our proposed MADDPG improves the accuracy and stability of HFL compared with the conventional reputation model and single-agent DDPG-based reputation model.

Fairness-Aware Task Loss Rate Minimization for Multi-UAV Enabled Mobile Edge Computing

In practical systems, a computing task generated by an Internet of Things device (IoTD) is usually given a valid period ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">vap</monospace> ). The tasks that cannot be executed within the <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">vap</monospace> will be dropped. The main goal of this letter is to minimize the task loss rate ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TLR</monospace> ) in unmanned-air-vehicle (UAV) assisted mobile edge computing (MEC) due to the timeout. Furthermore, to ensure an equal service opportunity for the IoTDs and an equal energy consumption ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EC</monospace> ) level for the UAVs, the issues of the <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TLR</monospace> -fairness between IoTDs and the <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EC</monospace> -fairness between UAVs are also considered. This is formulated as the mixed integer nonlinear programming (MINLP), which is difficult to be addressed by traditional methods, especially when the fast decision-making process is required. To address this problem, we present a new solution based on the multi-agent deep deterministic policy gradient (MA-DDPG) to optimize the flight trajectory, the association between the UAVs and IoTDs and the task scheduling of the IoTDs. Simulation results verify the effectiveness of the proposed MA-DDPG based algorithm.

Multi-Agent Deep Reinforcement Learning Based Transmission Latency Minimization for Delay-Sensitive Cognitive Satellite-UAV Networks

With the ubiquitous deployment of a massive number of Internet-of-Things (IoT) devices, the satellite-aerial networks are becoming a promising candidate to provide flexible and seamless service for IoT applications. Concerning about the spectrum scarcity issue, we present a cognitive satellite-aerial network where the multiple unmanned aerial vehicles (UAVs) can share spectrum with the satellite without interfering with satellite communications. To further improve spectral efficiency, non-orthogonal multiple access (NOMA) technique is adopted in this network. Considering the delay-sensitive quality-of-service (QoS) requirement, a joint trajectory and power optimization problem is formulated to minimize the total transmission latency over a long-term task period. In order to reduce the computational complexity and ease the burden of information exchange by using the centralized DRL methods, we propose a multi-agent deep deterministic policy gradient (MADDPG) based algorithm which adopts the framework of centralized training with decentralized execution to solve the sophisticated problem. The simulation results show the proposed algorithm can achieve satisfactory performance through joint trajectory control and power allocation for the UAVs compared with other methods.

Do Deep Reinforcement Learning Agents Model Intentions?

Inferring other agents’ mental states, such as their knowledge, beliefs and intentions, is thought to be essential for effective interactions with other agents. Recently, multi-agent systems trained via deep reinforcement learning have been shown to succeed in solving various tasks. Still, how each agent models or represents other agents in their environment remains unclear. In this work, we test whether deep reinforcement learning agents trained with the multi-agent deep deterministic policy gradient (MADDPG) algorithm explicitly represent other agents’ intentions (their specific aims or plans) during a task in which the agents have to coordinate the covering of different spots in a 2D environment. In particular, we tracked over time the performance of a linear decoder trained to predict the final targets of all agents from the hidden-layer activations of each agent’s neural network controller. We observed that the hidden layers of agents represented explicit information about other agents’ intentions, i.e., the target landmark the other agent ended up covering. We also performed a series of experiments in which some agents were replaced by others with fixed targets to test the levels of generalization of the trained agents. We noticed that during the training phase, the agents developed a preference for each landmark, which hindered generalization. To alleviate the above problem, we evaluated simple changes to the MADDPG training algorithm which lead to better generalization against unseen agents. Our method for confirming intention modeling in deep learning agents is simple to implement and can be used to improve the generalization of multi-agent systems in fields such as robotics, autonomous vehicles and smart cities.

Deep reinforcement learning based topology-aware voltage regulation of distribution networks with distributed energy storage

Both the high penetration of clean energy with strong fluctuation and the complicated variable operation condition bring great challenges to the voltage regulation of the distribution network. To deal with the problem, a topology-aware voltage regulation multi-agent deep reinforcement learning (MADRL) algorithm is proposed. The distributed energy storages (DESs) are modeled as agents to regulate voltage autonomously in real-time, which could fast adapt to dynamic topological scenarios. Firstly, taking the minimization of voltage fluctuation and maximization of reserve capacity as the target, the optimal voltage regulation model is established. Secondly, a topology extraction method considering voltage sensitivity is proposed for dynamic topology clustering, and the obtained typical topology is added to the observation set of agents. Then, the optimal voltage regulation model is formulated to the decentralized partially observable Markov decision process (Dec-POMDP) framework, in which only local information is required for the agent during the test process to decision-making to realize the hierarchical and partitioned control of voltage. Finally, the multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to solve the Dec-POMDP model. The feasibility and superiority of the proposed algorithm are verified and analyzed in the simulation under different scenarios.

Deep Reinforcement Learning Car-Following Model Considering Longitudinal and Lateral Control

The lateral control of the vehicle is significant for reducing the rollover risk of high-speed cars and improving the stability of the following vehicle. However, the existing car-following (CF) models rarely consider lateral control. Therefore, a CF model with combined longitudinal and lateral control is constructed based on the three degrees of freedom vehicle dynamics model and reinforcement learning method. First, 100 CF segments were selected from the OpenACC database, including 50 straight and 50 curved road trajectories. Afterward, the deep deterministic policy gradient (DDPG) car-following model and multi-agent deep deterministic policy gradient (MADDPG) car-following model were constructed based on the deterministic policy gradient theory. Finally, the models are trained with the extracted trajectory data and verified by comparison with the observed data. The results indicate that the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model are both safer and more comfortable than the human-driven vehicle (HDV) on straight roads and curved roads. Under the premise of safety, the vehicle under the control of the MADDPG model has the highest road traffic flow efficiency. The maximum lateral offset of the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model in straight road conditions is respectively reduced by 80.86% and 71.92%, compared with the HDV, and the maximum lateral offset in the curved road conditions is lessened by 83.67% and 78.95%. The proposed car following model can provide a reference for developing an adaptive cruise control system considering lateral stability.

Extensible Hierarchical Multi-Agent Reinforcement-Learning Algorithm in Traffic Signal Control

Reinforcement-learning (RL) algorithms have made great achievements in many scenarios. However, in large-scale traffic signal control (TSC) scenarios, RL still falls into local optima when controlling multiple signal lights. To solve this problem, we propose a novel goal-based multi-agent hierarchical model (GMHM). Specifically, we divide the traffic environment into several regions. The region contains a virtual manager and several workers who control the traffic lights. The manager assigns goals to each worker by observing the environment, and the worker makes decisions according to the environment state and the goal. For the worker, we adapted the goal-based multi-agent deep deterministic policy gradient (MADDPG) algorithm combined with hierarchical reinforcement learning. In this way, we simplify tasks and allow agents to cooperate more efficiently. We carried out experiments on both grid traffic scenarios and real-world scenarios in the SUMO simulator. The experimental results show the performance advantages of our algorithm compared with state-of-the-art algorithms.

Multi-timescale voltage control for distribution system based on multi-agent deep reinforcement learning

Nowadays a large number of distributed generators and controllable elements connecting to the distribution network have resulted in higher requirements for voltage control. Existing voltage control methods require a specific physical model and are time-consuming with the complexity increasing. To overcome these challenges, this paper proposes a multi-timescale voltage control scheme using multi-agent deep reinforcement learning (MADRL). Firstly, considering the control requirements of different timescale regulators, a multi-timescale voltage control model is formulated. Then the control variables are assigned to multiple agents, and a MADRL-based voltage control scheme is proposed, which is based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm and applies Gumbel-Softmax distribution to solve discrete actions. Through centralized learning and decentralized execution, this scheme can obtain the optimal coordinated control strategy of multiple regulators adaptively and achieve the effect of distributed control. Finally, case studies are conducted on the modified IEEE-123 bus system, and the performance of the proposed method is compared with the conventional model-based optimal voltage control scheme and other DRL-based methods. The results show that the MADRL-based method possesses better performance than other methods.

Hybrid attention-oriented experience replay for deep reinforcement learning and its application to a multi-robot cooperative hunting problem

Multiple robots complete a cooperative hunting task by obtaining environmental information and autonomously learning hunting decision-making strategies. However, with the increase in the number of environment participants, it becomes difficult for robots to process a large amount of environmental information. Thus, a multi-robot cooperative hunting decision-making method called hybrid attention-oriented experience replay in multi-agent deep deterministic policy gradient (HAER-MADDPG) is proposed. First, a hybrid attention module is designed to pay greater attention to key information in a large amount of environmental information by integrating it with the multi-agent deep deterministic policy gradient (MADDPG). The method then combines hybrid attention and prioritized experience replay to improve the utilization of experience samples. Finally, the proposed algorithm is tested through a predator–prey game. The results show that the effectiveness, convergence speed, and scalability of the proposed algorithm are better than those of the baseline algorithms. In addition, HAER-MADDPG is effectively applied to a hunting task with real robots.

Multi-agent reinforcement mechanism design for dynamic pricing-based demand response in charging network

Dynamic pricing for the charging network needs to consider the interdependency of charging stations’ decision-making, and acquire user’s demand information and their price sensitivity on demands. Most existing pricing schemes assume users consistently follow stable and inferrable patterns, while it remains crucial to explore how they may be influenced by historic prices in a charging market, their private utility functions, and the best demand in response. To integrate the strategic user-station interactions and uncertainties into the network dynamic pricing, this paper proposes a multi-agent reinforcement mechanism design framework to simultaneously determine the optimal charging prices for multiple charging stations over a period considering power output limits, unexpected arriving requests, and undetermined charging demands of self-interested users who aim to maximize their own utility. Specifically, the station–user interaction is modelled as a mechanism design problem, and station–station cooperation is captured by the Markov game and solved by multi-agent deep deterministic policy gradient. The objective is to maximize the long-term network revenue considering the social welfare of all users. We evaluate our framework through an experimental study, and the results demonstrate that our approach outperforms the non-cooperative deep deterministic policy gradient algorithm and time-of-use pricing scheme.