Prioritized Experience Replay Research Articles

Green Computation Offloading With DRL in Multi‐Access Edge Computing

ABSTRACTIn multi‐access edge computing (MEC), computational task offloading of mobile terminals (MT) is expected to provide the green applications with the restriction of energy consumption and service latency. Nevertheless, the diverse statuses of a range of edge servers and mobile terminals, along with the fluctuating offloading routes, present a challenge in the realm of computational task offloading. In order to bolster green applications, we present an innovative computational task offloading model as our initial approach. In particular, the nascent model is constrained by energy consumption and service latency considerations: (1) Smart mobile terminals with computational capabilities could serve as carriers; (2) The diverse computational and communication capacities of edge servers have the potential to enhance the offloading process; (3) The unpredictable routing paths of mobile terminals and edge servers could result in varied information transmissions. We then propose an improved deep reinforcement learning (DRL) algorithm named PS‐DDPG with the prioritized experience replay (PER) and the stochastic weight averaging (SWA) mechanisms based on deep deterministic policy gradients (DDPG) to seek an optimal offloading mode, saving energy consumption. Next, we introduce an enhanced deep reinforcement learning (DRL) algorithm named PS‐DDPG, incorporating the prioritized experience replay (PER) and stochastic weight averaging (SWA) techniques rooted in deep deterministic policy gradients (DDPG). This approach aims to identify an efficient offloading strategy, thereby reducing energy consumption. Fortunately, algorithm is proposed for each MT, which is responsible for making decisions regarding task partition, channel allocation, and power transmission control. Our developed approach achieves the ultimate estimation of observed values and enhances memory via write operations. The replay buffer holds data from previous time slots to upgrade both the actor and critic networks, followed by a buffer reset. Comprehensive experiments validate the superior performance, including stability and convergence, of our algorithm when juxtaposed with prior studies.

Path Planning in Complex Environments Using Attention-Based Deep Deterministic Policy Gradient

The traditional Deep Deterministic Policy Gradient (DDPG) algorithm frequently exhibits a notable reduction in success rate when transferred to new environments after being trained in complex simulation settings. To address these issues, this paper adopts a Multi-Environment (Multi-Env) parallel training approach and integrates Multi-Head Attention (MHA) and Prioritized Experience Replay (PER) into the DDPG framework, optimizing the reward function to form the MAP-DDPG algorithm. This approach enhances the algorithm’s generalization capability and execution efficiency. Through comparative training and testing of the DDPG and MAP-DDPG algorithms in both simulation and real-world environments, the experimental results demonstrate that MAP-DDPG significantly improves generalization and execution efficiency over the DDPG algorithm. Specifically, in simulation environment tests, the MAP-DDPG algorithm achieved an average 30% increase in success rate and reduced the average time to reach the target point by 23.7 s compared to the DDPG algorithm. These results indicate that the MAP-DDPG algorithm significantly enhances path planning generalization and execution efficiency, providing a more effective solution for path planning in complex environments.

Adaptive dynamic programming–based fault-tolerant control of multi-UAV formation system

With the increasing utilization of unmanned aerial vehicle (UAV) swarms in civilian and military field applications, it is crucial to eliminate the potential impacts of UAV faults on task completion and efficiency. To address this issue, the present paper investigates a fault-tolerant control scheme based on the adaptive dynamic programming (ADP) method and fault observer. The whole closed-loop system is composed of a position loop and an attitude loop. An ADP-based position-tracking controller is established with an improved cost function that can simultaneously represent actuator faults, control inputs, and tracking errors. In addition, a single-layer critic neural network is constructed to estimate the cost function and approximate control inputs. In the process of adjusting the weights of the neural network, a unique adjustment method combining policy gradient and prioritized experience replay is adopted to improve the data usage efficiency and speed up the weight convergence. Furthermore, a sliding mode–based attitude controller is utilized for the inner-loop attitude subsystem. The significant feature of this scheme is that it has the ability to self-learn and self-adapt. A Lyapunov stability analysis is performed to verify whether the signals are uniformly ultimately bounded. Finally, simulation experiments are conducted to demonstrate the effectiveness of the proposed scheme.

Analyzing deep reinforcement learning model decisions with Shapley additive explanations for counter drone operations

As the use of drones continues to increase, their capabilities pose a threat to airspace safety when they are misused. Deploying AI models for intercepting these unwanted drones becomes crucial. However, these AI models, such as deep learning models, often operate as “black boxes”, making it hard to trust their decision-making system. This also affects end-users’ confidence in these AI systems. In this paper, the explainability of deep reinforcement learning is investigated and a deep reinforcement learning (DRL) method, double deep Q-network with dueling architecture and prioritized experience replay is applied to train the AI models. To make the AI model decisions more transparent and to understand the reasoning behind the AI decisions for counter-drone systems, Shapley Additive Explanations (SHAP) method is implemented. After training the DRL agent, experience replay is visualized, and the absolute SHAP values are calculated to explain the key factors that influence the deep reinforcement learning agent’s choices. The integration of DRL with explainable AI methods such as SHAP demonstrates significant potential for the advancement of robust and efficient counter-drone systems.

A deep residual reinforcement learning algorithm based on Soft Actor-Critic for autonomous navigation

The problem of autonomous navigation has attracted significant attention from robotics research community in the last few decades. In this paper, we address the problem of low data utilization due to the large amount of episode experience value data. A maximum entropy algorithm based on prioritized experience replay (Learning Good Experience based on Soft Actor-Criti, LGE-SAC) is proposed to quickly reproduce past good experience episodes. As the deep reinforcement learning method is susceptible to failure to plan ahead and explore the target position in a long sequence environment, a deep Residual Soft Actor-Critic (RSAC) is proposed to alleviate this problem. The reinforcement learning policy is fused with the Artificial Potential Field method to improve the generalization ability of the proposed algorithm, thus improving robot adaptation in new test environments. In order to validate the effectiveness of the proposed algorithm, we conducted simulation experiments in Gazebo simulator environment and real experiments on a Turtlebot3 robot equipped with LiDAR sensor. Simulation and experiment results show that the proposed algorithm effectively avoids obstacles and succeeds in reaching the goal compared to other obstacle avoidance algorithms. In comparison with the Artificial Potential Field method, the planning success rate of the proposed RSAC algorithm in the test environment is increased by 30%, and at the same time, the number of planning steps is reduced by half, and the generalization ability is improved.

Traffic navigation via reinforcement learning with episodic-guided prioritized experience replay

Deep Reinforcement Learning (DRL) models play a fundamental role in autonomous driving applications; however, they typically suffer from sample inefficiency because they often require many interactions with the environment to learn effective policies. This makes the training process time-consuming. To address this shortcoming, Prioritized Experience Replay (PER) has proven to be effective by prioritizing samples with high Temporal-Difference (TD) error for learning. In this context, this study contributes to artificial intelligence by proposing a sample-efficient DRL algorithm called Episodic-Guided Prioritized Experience Replay (EPER). The core innovation of EPER lies in the utilization of an episodic memory, dedicated to storing successful training episodes. Within this memory, expected returns for each state–action pair are extracted. These returns, combined with TD error-based prioritization, form a novel objective function for deep Q-network training. To prevent excessive determinism, EPER introduces exploration into the learning process by incorporating a regularization term into the objective function that allows exploration of state-space regions with diverse Q-values. The proposed EPER algorithm is suitable to train a DRL agent for handling episodic tasks, and it can be integrated into off-policy DRL models. EPER is employed for traffic navigation through scenarios such as highway driving, merging, roundabout, and intersection to showcase its application in engineering. The attained results denote that, compared with the PER and an additional state-of-the-art training technique, EPER is superior in expediting the training of the agent and learning a more optimal policy that leads to lower collision rates within the constructed navigation scenarios.

Learning Multi-Agent Cooperation via Considering Actions of Teammates.

Recently value-based centralized training with decentralized execution (CTDE) multi-agent reinforcement learning (MARL) methods have achieved excellent performance in cooperative tasks. However, the most representative method among these methods, Q-network MIXing (QMIX), restricts the joint action Q values to be a monotonic mixing of each agent's utilities. Furthermore, current methods cannot generalize to unseen environments or different agent configurations, which is known as ad hoc team play situation. In this work, we propose a novel Q values decomposition that considers both the return of an agent acting on its own and cooperating with other observable agents to address the nonmonotonic problem. Based on the decomposition, we propose a greedy action searching method that can improve exploration and is not affected by changes in observable agents or changes in the order of agents' actions. In this way, our method can adapt to ad hoc team play situation. Furthermore, we utilize an auxiliary loss related to environmental cognition consistency and a modified prioritized experience replay (PER) buffer to assist training. Our extensive experimental results show that our method achieves significant performance improvements in both challenging monotonic and nonmonotonic domains, and can handle the ad hoc team play situation perfectly.

Joint 3D trajectory and phase shift optimization via deep reinforcement learning for RIS-assisted UAV communication systems

Unmanned aerial vehicle (UAV) can be deployed as aerial base station to provide communication services for the user equipments (UEs). However, in urban environments, the links between UAV and UEs might be frequently blocked by obstacles, leading to severely adverse effects on the quality of service (QoS) of UEs. Additionally, due to the limited energy of the UAV, it might not always be feasible to re-establish the line-of-sight (LoS) links by frequently adjusting the positions of the UAV. In this context, the reconfigurable intelligent surface (RIS) is utilized to enhance the transmission range of UAV-UE links by reflecting incident signals to UEs. In this paper, we investigate the RIS-assisted UAV communication systems with the goal of maximizing the energy efficiency of the UAV through a joint optimization of the UAV’s trajectory and the RIS’s phase shift. The formulated optimization problem is non-convex, and challenging to solve in a polynomial time. Therefore, an effective deep reinforcement learning (DRL)-based solution, named Dueling DQN-PER is proposed, which combines the Dueling DQN algorithm with the prioritized experience replay (PER) technique. To ensure the fairness among all UEs, we design a service fairness index, and integrate it into the reward function when designing the proposed algorithm. Numerical results demonstrate that: 1) the proposed Dueling DQN-PER algorithm is capable of improving the system energy efficiency and has a better training performance than benchmark schemes; 2) by devising the service fairness index, the fairness among all UEs is ensured while enhancing the system performance in energy efficiency; 3) the RIS-assisted UAV communication systems benefit from significant energy efficiency gain over the systems without RIS.

Discrete Space Deep Reinforcement Learning Algorithm Based on Support Vector Machine Recursive Feature Elimination

Algorithms for training agents with experience replay have advanced in several domains, primarily because prioritized experience replay (PER) developed from the double deep Q-network (DDQN) in deep reinforcement learning (DRL) has become a standard. PER-based algorithms have achieved significant success in the image and video domains. However, the exceptional results observed in images and videos are not as effective in many domains with simple action spaces and relatively small states, particularly in discrete action spaces with sparse rewards. Moreover, most advanced techniques may improve sampling efficiency using deep learning algorithms rather than reinforcement learning. However, there is growing evidence that deep learning algorithms cannot generalize during training. Therefore, this study proposes an algorithm suitable for discrete action space environments that uses the sample efficiency of PER based on DDQN but incorporates support vector machine recursive feature elimination (SVM-RFE) without enhancing the sampling efficiency through deep learning algorithms. The proposed algorithm exhibited considerable performance improvements in classical OpenAI Gym environments that did not use images or videos as inputs. In particular, simple discrete space environments with reflection symmetry, such as Cart–Pole, exhibited a faster and more stable learning process. These results suggest that the application of SVM-RFE, which leverages the orthogonality of support vector machines (SVMs) across learning patterns, can be appropriate when the data in the reinforcement learning environment demonstrate symmetry.

Computation Offloading with Privacy-Preserving in Multi-Access Edge Computing: A Multi-Agent Deep Reinforcement Learning Approach

The rapid development of mobile communication technologies and Internet of Things (IoT) devices has introduced new challenges for multi-access edge computing (MEC). A key issue is how to efficiently manage MEC resources and determine the optimal offloading strategy between edge servers and user devices, while also protecting user privacy and thereby improving the Quality of Service (QoS). To address this issue, this paper investigates a privacy-preserving computation offloading scheme, designed to maximize QoS by comprehensively considering privacy protection, delay, energy consumption, and the task discard rate of user devices. We first formalize the privacy issue by introducing the concept of privacy entropy. Then, based on quantified indicators, a multi-objective optimization problem is established. To find an optimal solution to this problem, this paper proposes a computation offloading algorithm based on the Twin delayed deep deterministic policy gradient (TD3-SN-PER), which integrates clipped double-Q learning, prioritized experience replay, and state normalization techniques. Finally, the proposed method is evaluated through simulation analysis. The experimental results demonstrate that our approach can effectively balance multiple performance metrics to achieve optimal QoS.

Mobile Robot Navigation Based on Noisy N-Step Dueling Double Deep Q-Network and Prioritized Experience Replay

Effective real-time autonomous navigation for mobile robots in static and dynamic environments has become a challenging and active research topic. Although the simultaneous localization and mapping (SLAM) algorithm offers a solution, it often heavily relies on complex global and local maps, resulting in significant computational demands, slower convergence rates, and prolonged training times. In response to these challenges, this paper presents a novel algorithm called PER-n2D3QN, which integrates prioritized experience replay, a noisy network with factorized Gaussian noise, n-step learning, and a dueling structure into a double deep Q-network. This combination enhances the efficiency of experience replay, facilitates exploration, and provides more accurate Q-value estimates, thereby significantly improving the performance of autonomous navigation for mobile robots. To further bolster the stability and robustness, meaningful improvements, such as target “soft” updates and the gradient clipping mechanism, are employed. Additionally, a novel and powerful target-oriented reshaping reward function is designed to expedite learning. The proposed model is validated through extensive experiments using the robot operating system (ROS) and Gazebo simulation environment. Furthermore, to more specifically reflect the complexity of the simulation environment, this paper presents a quantitative analysis of the simulation environment. The experimental results demonstrate that PER-n2D3QN exhibits heightened accuracy, accelerated convergence rates, and enhanced robustness in both static and dynamic scenarios.

Smart line planning method for power transmission based on D3QN‐PER algorithm

AbstractThe planning of power transmission line projects encompasses vast and complex geographical terrains. To address the complexity of transmission line planning and achieve lower line costs, this study proposes a novel intelligent line planning method. For the first time, it combines the Dueling Double Deep Q Network (D3QN) with the prioritized experience replay (PER) mechanism. First, correlate the reward function with metrics such as line length, number of corner points, and geographical environmental data, which are pertinent to the construction costs of power transmission line. Second, the D3QN algorithm is formulated by integrating Double DQN and Dueling DQN. The network's input information is divided into two components during training, aligning with the characteristics of power transmission line planning projects. Finally, the convergence efficiency of the algorithm is improved by using the PER mechanism for the problem of cost difference due to the different number of corner points in the planning path. In order to test the feasibility of the algorithm, we conducted experiments using real maps. Compared with the traditional ant colony optimization (ACO) algorithm, the D3QN‐PER deep reinforcement learning algorithm reduces the line length by more than 4% and the number of corner points by more than 60%.

Path planning of mobile robot based on improved TD3 algorithm in dynamic environment

This paper proposes an improved TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to address the flaws of low success rate and slow training speed, when using the original TD3 algorithm in mobile robot path planning in dynamic environment. Firstly, prioritized experience replay and transfer learning are introduced to enhance the learning efficiency, where the probability of beneficial experiences being sampled in the experience pool is increased, and the pre-trained model is applied in an obstacle-free environment as the initial model for training in a dynamic environment. Secondly, dynamic delay update strategy is devised and OU noise is added to improve the success rate of path planning, where the probability of missing high-quality value estimate is reduced through changing the delay update interval dynamically, and the correlated exploration of the mobile robot inertial navigation system in the dynamic environment is temporally improved. The algorithm is tested by simulation where the Turtlebot3 robot model as a training object, the ROS melodic operating system and Gazebo simulation software as an experimental environment. Meanwhile, the result shows that the improved TD3 algorithm has a 16.6 % increase in success rate and a 23.5 % reduction in algorithm training time. A generalization experiment was designed finally, and it indicates that superior generation performance has been acquired in mobile robot path planning with continuous action spaces through the improved TD3 algorithm.

An indoor blind area-oriented autonomous robotic path planning approach using deep reinforcement learning

Deep reinforcement learning (DRL) provides a new solution for autonomous robotic path planning in a known indoor environment. Previous studies mainly focused on robot path optimization but ignored blind areas in the indoor exploration, naturally result in low coverage rate and low exploration efficiency. The blind areas exploration is a crucial issue of the indoor environment. This work proposes an indoor blind area-oriented autonomous robotic path planning approach using DRL methods. First, the method optimization is based on a double deep Q-network (DDQN) with prioritized experience replay (PER). Then the Blocking and Blind Angle mechanism (BBA) is proposed to explore blind areas, assisted in selecting the optimal exploration points of next moment. Meanwhile it solves the common sparse reward problem in DRL. Finally, the presented method is successfully applied in simulation environment using a cleaning robot. Experiments show that the proposed BBA-PER-DDQN not only explores the blind areas, but also accelerates the convergence speed. The results show that the training time is reduced from more than one hour to 36 minutes, and the coverage rate is increased by 11.37% higher than that of the baseline algorithms.

Deep Reinforcement Learning-Based 3D Trajectory Planning for Cellular Connected UAV

To address the issue of limited application scenarios associated with connectivity assurance based on two-dimensional (2D) trajectory planning, this paper proposes an improved deep reinforcement learning (DRL) -based three-dimensional (3D) trajectory planning method for cellular unmanned aerial vehicles (UAVs) communication. By considering the 3D space environment and integrating factors such as UAV mission completion time and connectivity, we develop an objective function for path optimization and utilize the advanced dueling double deep Q network (D3QN) to optimize it. Additionally, we introduce the prioritized experience replay (PER) mechanism to enhance learning efficiency and expedite convergence. In order to further aid in trajectory planning, our method incorporates a simultaneous navigation and radio mapping (SNARM) framework that generates simulated 3D radio maps and simulates flight processes by utilizing measurement signals from the UAV during flight, thereby reducing actual flight costs. The simulation results demonstrate that the proposed approach effectively enable UAVs to avoid weak coverage regions in space, thereby reducing the weighted sum of flight time and expected interruption time.

Deep reinforcement learning based decision making for radar jamming suppression

Due to the need for the real-time participation of a large number of professionals, existing decision-making methods for radar interference suppression are characterized by a slow decision-making speed, unstable decision-making effects, and insufficient decision-making intelligence. In this paper, a deep reinforcement learning (DRL)-based decision-making method for radar interference suppression is proposed. This method has a fast decision speed and a stable and accurate decision effect, and can complete decision-making tasks by itself with high intelligence. To enhance the ability of the agent to acquire high-value experiences, the variable greedy algorithm (VGA) is proposed. The VGA adjusts the fixed greedy value in the original action strategy into a declining greedy curve that mimics the human learning process via a combination of the ideas in the win or learn fast–policy hill-climbing (WoLF-PHC) algorithm and the Ebbinghaus forgetting curve. To improve the efficiency of the agent in utilizing high-value experiences, the double-depth prioritized experience replay (DDPER) algorithm is proposed. The DDPER algorithm changes the uniform random experience replay into prioritized experience replay (PER), and performs sorting and extraction learning based on experience values in the form of additive trees to achieve better learning results. Further, the accuracy and speed of decision-making are improved via the double-depth experience replay system. The findings of a simulation experiment show that the agent can efficiently learn the most optimal radar interference suppression method by knowing that the interference suppression algorithm library contains algorithms that can deal with current environmental interference signals. Furthermore, compared to the PER–double deep Q-Network (PER-DDQN) presented by Zhang, the average accuracy, speed, and stability of decision-making are respectively increased by 6.4%, 2.51%, and 102.12%.

Blockchain-Based Zero-Trust Supply Chain Security Integrated with Deep Reinforcement Learning for Inventory Optimization

Modern supply chain systems face significant challenges, including lack of transparency, inefficient inventory management, and vulnerability to disruptions and security threats. Traditional optimization methods often struggle to adapt to the complex and dynamic nature of these systems. This paper presents a novel blockchain-based zero-trust supply chain security framework integrated with deep reinforcement learning (SAC-rainbow) to address these challenges. The SAC-rainbow framework leverages the Soft Actor–Critic (SAC) algorithm with prioritized experience replay for inventory optimization and a blockchain-based zero-trust mechanism for secure supply chain management. The SAC-rainbow algorithm learns adaptive policies under demand uncertainty, while the blockchain architecture ensures secure, transparent, and traceable record-keeping and automated execution of supply chain transactions. An experiment using real-world supply chain data demonstrated the superior performance of the proposed framework in terms of reward maximization, inventory stability, and security metrics. The SAC-rainbow framework offers a promising solution for addressing the challenges of modern supply chains by leveraging blockchain, deep reinforcement learning, and zero-trust security principles. This research paves the way for developing secure, transparent, and efficient supply chain management systems in the face of growing complexity and security risks.

Data-Based Optimal Consensus Control for Multiagent Systems With Time Delays: Using Prioritized Experience Replay

Data-Based Optimal Consensus Control for Multiagent Systems With Time Delays: Using Prioritized Experience Replay

Exploring the Efficacy of Learning Techniques in Model Extraction Attacks on Image Classifiers: A Comparative Study

In the rapidly evolving landscape of cybersecurity, model extraction attacks pose a significant challenge, undermining the integrity of machine learning models by enabling adversaries to replicate proprietary algorithms without direct access. This paper presents a comprehensive study on model extraction attacks towards image classification models, focusing on the efficacy of various Deep Q-network (DQN) extensions for enhancing the performance of surrogate models. The goal is to identify the most efficient approaches for choosing images that optimize adversarial benefits. Additionally, we explore synthetic data generation techniques, including the Jacobian-based method, Linf-projected Gradient Descent (LinfPGD), and Fast Gradient Sign Method (FGSM) aiming to facilitate the training of adversary models with enhanced performance. Our investigation also extends to the realm of data-free model extraction attacks, examining their feasibility and performance under constrained query budgets. Our investigation extends to the comparison of these methods under constrained query budgets, where the Prioritized Experience Replay (PER) technique emerges as the most effective, outperforming other DQN extensions and synthetic data generation methods. Through rigorous experimentation, including multiple trials to ensure statistical significance, this work provides valuable insights into optimizing model extraction attacks.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.