Deep Reinforcement Learning Techniques Research Articles

Deep Reinforcement Learning for UAV-Based SDWSN Data Collection

Recent advancements in Unmanned Aerial Vehicle (UAV) technology have made them effective platforms for data capture in applications like environmental monitoring. UAVs, acting as mobile data ferries, can significantly improve ground network performance by involving ground network representatives in data collection. These representatives communicate opportunistically with accessible UAVs. Emerging technologies such as Software Defined Wireless Sensor Networks (SDWSN), wherein the role/function of sensor nodes is defined via software, can offer a flexible operation for UAV data-gathering approaches. In this paper, we introduce the “UAV Fuzzy Travel Path”, a novel approach that utilizes Deep Reinforcement Learning (DRL) algorithms, which is a subfield of machine learning, for optimal UAV trajectory planning. The approach also involves the integration between UAV and SDWSN wherein nodes acting as gateways (GWs) receive data from the flexibly formulated group members via software definition. A UAV is then dispatched to capture data from GWs along a planned trajectory within a fuzzy span. Our dual objectives are to minimize the total energy consumption of the UAV system during each data collection round and to enhance the communication bit rate on the UAV-Ground connectivity. We formulate this problem as a constrained combinatorial optimization problem, jointly planning the UAV path with improved communication performance. To tackle the NP-hard nature of this problem, we propose a novel DRL technique based on Deep Q-Learning. By learning from UAV path policy experiences, our approach efficiently reduces energy consumption while maximizing packet delivery.

Leveraging the Synergy of Digital Twins and Artificial Intelligence for Sustainable Power Grids: A Scoping Review

As outlined by the International Energy Agency, 44% of carbon emissions in 2021 were attributed to electricity and heat generation. Under this critical scenario, the power industry has adopted technologies promoting sustainability in the form of smart grids, microgrids, and renewable energy. To overcome the technical challenges associated with these emerging approaches and to preserve the stability and reliability of the power system, integrating advanced digital technologies such as Digital Twins (DTs) and Artificial Intelligence (AI) is crucial. While existing research has explored DTs and AI in power systems separately, an overarching review of their combined, synergetic application in sustainable power systems is lacking. Hence, in this work, a comprehensive scoping review is conducted under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). The main results of this review analysed the breadth and relationships among power systems, DTs, and AI dynamics and presented an evolutionary timeline with three distinct periods of maturity. The prominent utilisation of deep learning, supervised learning, reinforcement learning, and swarm intelligence techniques was identified as mainly constrained to power system operations and maintenance functions, along with the potential for more sophisticated AI techniques in computer vision, natural language processing, and smart robotics. This review also discovered sustainability-related objectives addressed by AI-powered DTs in power systems, encompassing renewable energy integration and energy efficiency, while encouraging the investigation of more direct efforts on sustainable power systems.

3D Voxel-based Path Planning for AVs in Dynamic Complex Environments

Abstract. With the rapid advancement of science and technology, autonomous driving technology has been widely applied in various fields such as the automotive industry, intelligent warehousing, and emergency rescue. Traditional 2D path planning has limitations in dealing with the complex and dynamic environments of the real world, making it difficult to meet the high precision and complex path planning requirements of autonomous vehicles. With the development of data acquisition technology, large-scale point cloud data has become easily accessible, which can better extract the three-dimensional complex of the environment. Nevertheless, the unstructured nature and vast amount of point cloud data make path planning on it extremely challenging. Voxel models can effectively compress point cloud data and provide neighboring topological structures. In view of this, we propose a voxel-based framework for the path planning of autonomous vehicles in dynamic and complex 3D environments, which involves converting point cloud data into voxels and extracting navigation spaces. Subsequently, deep reinforcement learning techniques are utilized to achieve obstacle avoidance and navigation on the voxel map. It is expected that this framework will contribute to the development of 3D path planning and enhance the utilization of point cloud data in the navigation field.

TCP Congestion Management Using Deep Reinforcement Trained Agent for RED

ABSTRACTIncreasing data transmission volumes are causing more frequent and more severe network congestion. In order to handle spikes in network traffic, a substantially bigger buffer has been included into the system. Bufferbloat, which happens when a bigger buffer is implemented, exacerbates network congestion. Using the transfer control protocol (TCP) congestion management strategy with active queue management (AQM) can fix this issue. As congestion increases, it becomes increasingly difficult to forecast and fine‐tune dynamic AQM/TCP systems in order to achieve acceptable performance. To shed new light on the AQM system, we plan to use deep reinforcement learning (DRL) techniques. It is possible that AQM can learn about the appropriate drop policy the same way people do when using a model‐free technique like DRL‐AQM. After training in a simple network scenario, DRL‐AQM is able to recognize complex patterns in the data traffic model and apply them to improve performance in a wide variety of scenarios. Offline training precedes deployment in our approach. In many cases, the model does not require any further parameter tweaks after training. Even in the most complicated networks, AQM algorithms have proven to be effective, regardless of the network's complexity. Minimizing buffer capacity use is an important goal of DRL‐AQM. It automatically and continually adjusts to changes in network connectivity.

Deep Reinforcement Learning for Autonomous Driving Systems

Autonomous driving systems (ADS) are poised to revolutionize the future of transportation, promising increased safety, efficiency, and convenience. Deep Reinforcement Learning (DRL) has emerged as a powerful approach to solving complex decision-making tasks in dynamic environments, making it a promising candidate for the development of intelligent autonomous vehicles. This paper explores the application of DRL techniques in autonomous driving, focusing on the integration of perception, planning, and control. We review state-of-the-art DRL algorithms, including Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC), and examine their roles in enabling end-to-end learning for driving policies. Furthermore, we discuss the challenges inherent in deploying DRL in real-world autonomous driving scenarios, including sample inefficiency, safety constraints, and the sim-to-real gap. Finally, the paper presents case studies and experimental results that highlight the potential of DRL to improve autonomous vehicle performance in complex environments, while identifying future research directions to address open problems in the field.

Deep Deterministic Policy Gradient-Based Resource Allocation Considering Network Slicing and Device-to-Device Communication in Mobile Networks.

Next-generation mobile networks, such as those beyond the 5th generation (B5G) and 6th generation (6G), have diverse network resource demands. Network slicing (NS) and device-to-device (D2D) communication have emerged as promising solutions for network operators. NS is a candidate technology for this scenario, where a single network infrastructure is divided into multiple (virtual) slices to meet different service requirements. Combining D2D and NS can improve spectrum utilization, providing better performance and scalability. This paper addresses the challenging problem of dynamic resource allocation with wireless network slices and D2D communications using deep reinforcement learning (DRL) techniques. More specifically, we propose an approach named DDPG-KRP based on deep deterministic policy gradient (DDPG) with K-nearest neighbors (KNNs) and reward penalization (RP) for undesirable action elimination to determine the resource allocation policy maximizing long-term rewards. The simulation results show that the DDPG-KRP is an efficient solution for resource allocation in wireless networks with slicing, outperforming other considered DRL algorithms.

A Deep Reinforcement Learning Method Based on a Transformer Model for the Flexible Job Shop Scheduling Problem

The flexible job shop scheduling problem (FJSSP), which can significantly enhance production efficiency, is a mathematical optimization problem widely applied in modern manufacturing industries. However, due to its NP-hard nature, finding an optimal solution for all scenarios within a reasonable time frame faces serious challenges. This paper proposes a solution that transforms the FJSSP into a Markov Decision Process (MDP) and employs deep reinforcement learning (DRL) techniques for resolution. First, we represent the state features of the scheduling environment using seven feature vectors and utilize a transformer encoder as a feature extraction module to effectively capture the relationships between state features and enhance representation capability. Second, based on the features of the jobs and machines, we design 16 composite dispatching rules from multiple dimensions, including the job completion rate, processing time, waiting time, and manufacturing resource utilization, to achieve flexible and efficient scheduling decisions. Furthermore, we project an intuitive and dense reward function with the objective of minimizing the total idle time of machines. Finally, to verify the performance and feasibility of the algorithm, we evaluate the proposed policy model on the Brandimarte, Hurink, and Dauzere datasets. Our experimental results demonstrate that the proposed framework consistently outperforms traditional dispatching rules, surpasses metaheuristic methods on larger-scale instances, and exceeds the performance of existing DRL-based scheduling methods across most datasets.

Vehicular edge cloud computing content caching optimization solution based on content prediction and deep reinforcement learning

In conventional studies on vehicular edge computing, researchers frequently overlook the high-speed mobility of vehicles and the dynamic nature of the vehicular edge environment. Moreover, when employing deep reinforcement learning to address vehicular edge challenges, insufficient attention is given to the potential issue of the algorithm converging to a local optimal solution. This paper presents a content caching solution tailored for vehicular edge cloud computing, integrating content prediction and deep reinforcement learning techniques. Given the swift mobility of vehicles and the ever-changing nature of the vehicular edge environment, the study proposes a content prediction model based on Informer. Leveraging the Informer prediction model, the system anticipates the vehicular edge environment dynamics, thereby informing the caching of vehicle task content. Acknowledging the diverse time scales involved in policy decisions such as content updating, vehicle scheduling, and bandwidth allocation, the paper advocates a dual time-scale Markov modeling approach. Moreover, to address the local optimality issue inherent in the A3C algorithm, an enhanced A3C algorithm is introduced, incorporating an ɛ-greedy strategy to promote exploration. Recognizing the potential limitations posed by a fixed exploration rate ɛ, a dynamic baseline mechanism is proposed for updating ɛ dynamically. Experimental findings demonstrate that compared to alternative content caching approaches, the proposed vehicle edge computing content caching solution substantially mitigates content access costs. To support research in this area, we have publicly released the source code and pre-trained models at https://github.com/JYAyyyyyy/Informer.git.

Experimental Implementation of a TD3 Agent Based Speed Controller for Direct Torque Control of PMSM Drives

This manuscript presents a novel control approach for Permanent Magnet Synchronous Motors (PMSMs) by integrating the widely used Direct Torque Control (DTC) method with Deep Reinforcement Learning (DRL), specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The TD3 algorithm is well-suited to address the challenges posed by high-dimensional state and action spaces in PMSM drives. The conventional Proportional–Integral (PI) controller in the outer loop of DTC is replaced with the DRL based TD3 agent to overcome the limitations of traditional PI controllers, such as model dependency and complex parameter tuning. The DRL technique allows the agent to learn optimal control policies from experience, making it suitable for complex and nonlinear control problems. Extensive simulation studies demonstrate the effectiveness of the TD3-based control in achieving accurate speed tracking under varying operating conditions. Real-time experimental validation using a TMS320F28379D digital signal processor confirms the feasibility of the proposed DRL based approach. The research offers new insights into improving the performance of PMSM drives and paves the way for future advancements in electric drive control.

A hybrid Bi-level management framework for caching and communication in Edge-AI enabled IoT

The proliferation of IoT devices has led to a surge in network traffic, resulting in higher energy usage and response delays. In-network caching has emerged as a viable solution to address this issue. However, caching IoT data faces two key challenges: the transient nature of IoT content and the unknown spatiotemporal content popularity. Additionally, the use of a global view on dynamic IoT networks is problematic due to the high communication overhead involved. To tackle these challenges, this paper presents an adaptive management approach that jointly optimizes caching and communication in IoT networks using a novel bi-level control method called BC3. The approach employs two types of controllers: a global ILP-based optimal controller for long-term timeslots and local learning-based controllers for short-term timeslots. The long-term controller periodically establishes a global cache policy for the network and sends specific cache rules to each edge server. The local controller at each edge server solves the joint problem of bandwidth allocation and cache adaptation using deep reinforcement learning (DRL) technique. The main objective is to minimize energy consumption and system response time with utilizing the global and local observations. Experimental results demonstrate that the proposed approach increases cache hit rate by approximately 12% and uses 11% less energy compared to the other methods. Increasing the cache hit rate can lead to a reduction in about 17% in response time for user requests. Our bi-level control approach offers a promising solution for leveraging the network's global visibility while balancing communication overhead (as energy consumption) against system performance. Additionally, the proposed method has the lowest cache eviction, around 19% lower than the lowest eviction of the other comparison methods. The eviction metric is a metric to evaluate the effectiveness of adaptive caching approach designed for transient data.

Optimising shapes of multiple pin fins in a microchannel using deep reinforcement learning and mesh deformation techniques

The utilisation of advanced pin fin designs in microchannels is useful for enhancing cooling efficiency. Advancements in machine learning and processing power have sparked interest in shape optimisation techniques. This research employs a novel framework that integrates Deep Artificial Neural Networks and Reinforcement Learning with a Computational Fluid Dynamics (CFD) solver to optimise multiple pin fin shapes within a microchannel. By incorporating Radial Basis Function interpolation and Proximal Policy Optimisation alongside FLUENT, acting as the CFD environment, the reinforcement learning agent adeptly explores the design space to enhance the thermohydraulic performance factor (TPF), aiming to maximise Nusselt number while minimising pressure loss. Unlike previous heat transfer optimisation studies, which typically required mesh regeneration at each step, the proposed framework could bypass the meshing step and alter the geometry directly by relying on the RBF interpolation technique to deform the mesh directly. Three distinct scenarios investigated in this study are uniform deformation of all pin fins, deformation of all pin fins arranged in two rows, and individual deformation of each pin fin. Extensive simulations, exceeding 90,000 different cases, demonstrate that although the optimisation process for the individual deformation of each pin fin requires more iterations compared to others, it surpasses them in terms of TPF improvement. Notably, significant improvements are achieved, such as a 49 % enhancement in Nusselt number and a 33 % reduction in pressure drop, culminating in an impressive 63 % increase in TPF compared to the initial geometry.

Efficient adversarial attacks detection for deep reinforcement learning-based autonomous planetary landing GNC

Given the constraints of remote communication and the unpredictability of the environment, autonomous planetary landing mechanisms are expected to achieve the high criteria of autonomy and provide optimal trajectory in future space exploration missions. As the results, applying Deep Reinforcement Learning (DRL) techniques into autonomous landing has produced encouraging findings. Due to the black-box nature of deep learning algorithms, one of the main concerns regarding the robustness of DRL is its vulnerability to adversarial attacks. This constraint prevents the transfer of DRL-based autonomous landing schemes from simulation to real-world applications. In this article, we explore how the DRL-based autonomous landing will be impacted by adversarial attacks and how to protect the system effectively and efficiently. To achieve this, a Long Short Term Memory (LSTM) based adversarial attack detector is been proposed. The proposed method adopts the explainability measurement of the target DRL scheme and flag the detection of adversarial attacks when acting. The proposed method is built and tested on 3D digital terrain model of Candidate Landing Site for 2020 Mission in Jezero Crater to simulate the landing scenario on the Mars. The experimental results demonstrate the proposed methodology can effectively detect adversarial attacks when acting on DRL agent with a high confidence in detection accuracy.

Deep reinforcement learning based resource provisioning for federated edge learning

With the rapid development of mobile internet technology and increasing concerns over data privacy, Federated Learning (FL) has emerged as a significant framework for training machine learning models. Given the advancements in technology, User Equipment (UE) can now process multiple computing tasks simultaneously, and since UEs can have multiple data sources that are suitable for various FL tasks, multiple tasks FL could be a promising way to respond to different application requests at the same time. However, running multiple FL tasks simultaneously could lead to a strain on the device’s computation resource and excessive energy consumption, especially the issue of energy consumption challenge. Due to factors such as limited battery capacity and device heterogeneity, UE may fail to efficiently complete the local training task, and some of them may become stragglers with high-quality data. Aiming at alleviating the energy consumption challenge in a multi-task FL environment, we design an automatic Multi-Task FL Deployment (MFLD) algorithm to reach the local balancing and energy consumption goals. The MFLD algorithm leverages Deep Reinforcement Learning (DRL) techniques to automatically select UEs and allocate the computation resources according to the task requirement. Extensive experiments validate our proposed approach and showed significant improvements in task deployment success rate and energy consumption cost.

Deep reinforcement learning-based attitude control for spacecraft using control moment gyros

This paper addresses the development of an attitude control system that steers control moment gyros (CMGs) based on deep reinforcement learning (DRL) for agile spacecraft. The proposed DRL-based attitude control system learns CMG steering strategies to achieve the desired attitude, thus potentially bypassing the singularity issues inherent in the CMG cluster. In particular, it is designed in two-phases to apply the DRL technique efficiently. In the first phase, the attitude control is performed based on DRL up to a certain tolerance, after which it switches to conventional control and steering law for stabilization in the second phase. The rapid pointing capability of the proposed DRL-based attitude control system is demonstrated for an agile spacecraft equipped with pyramid-type single gimbal control moment gyros. Additionally, in realistic scenarios of pointing multiple targets on the ground, the momentum vector recovery that the CMG system needs to consider is also briefly discussed.

Tool Condition Monitoring in the Milling Process Using Deep Learning and Reinforcement Learning

Tool condition monitoring (TCM) is crucial in the machining process to confirm product quality as well as process efficiency and minimize downtime. Traditional methods for TCM, while effective to a degree, often fall short in real-time adaptability and predictive accuracy. This research work aims to advance the state-of-the-art methods in predictive maintenance for TCM and improve tool performance and reliability during the milling process. The present work investigates the application of Deep Learning (DL) and Reinforcement Learning (RL) techniques to monitor tool conditions in milling operations. DL models, including Long Short-Term Memory (LSTM) networks, Feed Forward Neural Networks (FFNN), and RL models, including Q-learning and SARSA, are employed to classify tool conditions from the vibration sensor. The performance of the selected DL and RL algorithms is evaluated through performance metrics like confusion matrix, recall, precision, F1 score, and Receiver Operating Characteristics (ROC) curves. The results revealed that RL based on SARSA outperformed other algorithms. The overall classification accuracies for LSTM, FFNN, Q-learning, and SARSA were 94.85%, 98.16%, 98.50%, and 98.66%, respectively. In regard to predicting tool conditions accurately and thereby enhancing overall process efficiency, SARSA showed the best performance, followed by Q-learning, FFNN, and LSTM. This work contributes to the advancement of TCM systems, highlighting the potential of DL and RL techniques to revolutionize manufacturing processes in the era of Industry 5.0.

AnimalEnvNet: A Deep Reinforcement Learning Method for Constructing Animal Agents Using Multimodal Data Fusion

Simulating animal movement has long been a central focus of study in the area of wildlife behaviour studies. Conventional modelling methods have difficulties in accurately representing changes over time and space in the data, and they generally do not effectively use telemetry data. Thus, this paper introduces a new and innovative deep reinforcement learning technique known as AnimalEnvNet. This approach combines historical trajectory data and remote sensing images to create an animal agent using deep reinforcement learning techniques. It overcomes the constraints of conventional modelling approaches. We selected pandas as the subject of our research and carried out research using GPS trajectory data, Google Earth images, and Sentinel-2A remote sensing images. The experimental findings indicate that AnimalEnvNet reaches convergence during supervised learning training, attaining a minimal mean absolute error (MAE) of 28.4 m in single-step prediction when compared to actual trajectories. During reinforcement learning training, the agent has the capability to replicate animal locomotion for a maximum of 12 iterations, while maintaining an error margin of 1000 m. This offers a novel approach and viewpoint for mimicking animal behaviour.

Efficient and practical quantum compiler towards multi-qubit systems with deep reinforcement learning ∗ ∗The authors list is in alphabetical order.

Efficient quantum compiling is essential for complex quantum algorithms realization. The Solovay–Kitaev (S–K) theorem offers a theoretical lower bound on the required operations for approaching any unitary operator. However, it is still an open question that this lower bound can be actually reached in practice. Here, we present an efficient quantum compiler which, for the first time, approaches the S–K lower bound in practical implementations, both for single-qubit and two-qubit scenarios, marking a significant milestone. Our compiler leverages deep reinforcement learning (RL) techniques to address current limitations in terms of optimality and inference time. Furthermore, we show that our compiler is versatile by demonstrating comparable performance between inverse-free basis sets, which is always the case in real quantum devices, and inverse-closed sets. Our findings also emphasize the often-neglected constant term in scaling laws, bridging the gap between theory and practice in quantum compiling. These results highlight the potential of RL-based quantum compilers, offering efficiency and practicality while contributing novel insights to quantum compiling theory.

Efficient deep reinforcement learning strategies for active flow control based on physics-informed neural networks

Reducing the reliance on intrusive flow probes is a critical task in active flow control based on deep reinforcement learning (DRL). Although a scarcity of flow data captured by probes adversely impacts the control proficiency of the DRL agent, leading to suboptimal flow modulation, minimizing the use of redundant probes significantly reduces the overall implementation costs, making the control strategy more economically viable. In this paper, we propose an active flow control method based on physics-informed DRL. This method integrates a physics-informed neural network into the DRL framework, harnessing the inherent physical characteristics of the flow field using strategically placed probes. We analyze the impact of probe placement, probe quantity, and DRL agent sampling strategies on the fidelity of flow predictions and the efficacy of flow control. Using the wake control of a two-dimensional cylinder flow with a Reynolds number of 100 as a case study, we position a specific number of flow probes within the flow field to gather pertinent information. When benchmarked against traditional DRL techniques, the results are unequivocal: in terms of training efficiency, physics-informed DRL reduces the training cycle by up to 30 rounds. Furthermore, by decreasing the number of flow probes in the flow field from 164 to just 4, the physics-based DRL achieves superior drag reduction through more precise control. Notably, compared to traditional DRL control, the drag reduction effect is enhanced by a significant 6%.

Improved STNNet, A benchmark for detection, tracking, and counting crowds using Drones

In computer vision, navigating multi-object tracking in crowded scenes poses a fundamental challenge with broad applications ranging from surveillance systems to autonomous vehicles. Traditional tracking methods encounter difficulties associating noisy object detections and maintaining consistent labels across frames, particularly in scenarios like video surveillance for crowd control and public safety. This paper introduces ’Improved Space-Time Neighbor-Aware Network (STNNet),’ an advanced framework for online Multi-Object Tracking (MOT) designed to address these challenges. Expanding upon the foundational STNNet architecture, our enhanced model incorporates deep reinforcement learning techniques to refine decision-making. By framing the online MOT problem as a Markov Decision Process (MDP), Improved STNNet learns a sophisticated policy for data association, adeptly handling complexities such as object birth/death and appearance/disappearance as state transitions within the MDP. Through extensive experimentation on benchmark datasets, including the MOT Challenge, our proposed Improved STNNet demonstrates superior performance, surpassing existing methods in demanding, crowded scenarios. This study showcases the effectiveness of our approach and lays the groundwork for advancing real-time video analysis applications, particularly in dynamic, crowded environments. Additionally, we utilize the dataset provided by STNNET for density map estimation, forming the basis for our research.•Develop an advanced framework for online Multi-Object Tracking (MOT) to address crowded scene challenges, particularly improving object association and label consistency across frames.•Explore integrating Deep Reinforcement learning techniques into the MOT framework, framing the problem as an MDP to refine decision-making and handle complexities such as object birth or death and appearance or disappearance transitions.

A reinforcement learning based autonomous vehicle control in diverse daytime and weather scenarios

Autonomous driving holds significant promise for substantially reducing road fatalities. Unlike traditional machine learning methods that have conventionally been applied to enhance the motion control of Autonomous Vehicles (AVs), recent attention has shifted toward the utilization of Deep Learning (DL) and Deep Reinforcement Learning (DRL) techniques. These advanced approaches have the potential to greatly improve AV vehicle control and empower vehicles to learn from their surroundings. However, the majority of existing research has concentrated on straightforward scenarios, often neglecting the intricate challenges posed by vulnerable road users such as pedestrians, cyclists, and motorcyclists, as well as the influence of varying weather conditions. In this study, we propose a novel model founded on DRL, specifically leveraging Deep-Q Networks (DQN), to effectively manage AVs in complex scenarios characterized by heavy traffic, diverse road users, and diverse weather conditions. Our approach involves training the model in diverse weather conditions, encompassing clear daytime and nighttime as well as challenging weather conditions like heavy rainfall during both the day and sunset. Through this comprehensive training, the AV becomes proficient in navigating safely through intersections and reaching its destination without any accidents. To rigorously evaluate and validate our proposed approach, extensive testing was conducted employing the CARLA simulator. The simulation results unequivocally demonstrate that our model not only reduces travel delays but also minimizes the occurrence of collisions, marking a significant step forward in achieving safer and more efficient autonomous driving.