Deep Reinforcement Learning Algorithm Research Articles

Automated position control of tunnel boring machine during excavation using deep reinforcement learning

Position control of tunnel boring machines (TBM) is critical to ensure precise construction and meets design specifications. However, the high dependence on human intervention largely affects the efficiency and reliability of the operations due to slow response, inconsistent judgment and high risk of errors. Targeting to automate the position control of the TBM to enable it to the planned route in a more efficient and reliable manner, this research proposes a deep reinforcement learning (DRL) method considering spatial-temporal dynamics to perform automated real-time TBM operations. The DRL algorithm is embedded with a time-series deep learning model to simulate the working environment with dynamic time-space variations. The whole process is coupled with the designed reward and evaluation metrics for model enhancement. To showcase the viability and applicability of the suggested approach, a project from Singapore is utilized as an illustrative example. The outcomes indicate that the suggested method exhibits robust capabilities in forecasting and controlling TBM positions, with an R2 of 0.86 and 0.92 in forecasting the horizontal and vertical deviations of the TBM tail three steps ahead and an overall improvement of 50.7 %. The proposed method is capable of providing an optimal strategy to intelligently forecast and control the TBM deviation. The novelty of this method comes from the feasible integration of time series prediction and DRL to simulate the spatial-temporal dynamics along with TBM excavation and obtain a position control model for optimal tunneling strategy, contributing to the effective facilitation of TBM tunneling to improve efficiency and reliability.

UAV-IRS-assisted energy harvesting for edge computing based on deep reinforcement learning

In the internet of everything (IoE) era, the proliferation of internet of things (IoT) devices is accelerating rapidly. Particularly, smaller devices are increasingly constrained by hardware limitations that impact their computational capacity, communication bandwidth, and battery longevity. Our research explores a multi-device, multi-access edge computing (MEC) environment within small cells to address the challenges posed by the hardware limitations of IoT devices in this environment. We employ wireless power transfer (WPT) to ensure these IoT devices have sufficient energy for task processing. We propose a system architecture in which an intelligent reflective surface (IRS) is carried by an unmanned aerial vehicle (UAV) to enhance communication conditions. For sustainable energy harvesting (EH), we integrate a normal distribution into the objective function. We utilize a softmax deep double deterministic policy gradients (SD3) algorithm, based on deep reinforcement learning (DRL), to optimize the computational and communication capabilities of IoT devices. Simulation experiments demonstrate that our SD3-based EH edge computing (EHEC-SD3) algorithm surpasses existing DRL algorithms in our explored environments, achieving more than 90% in overall optimization and EH performance.

Optimal dispatch of unbalanced distribution networks with phase-changing soft open points based on safe reinforcement learning

Distributed energy resources and uneven load allocation cause the three-phase unbalance in distribution networks, which may harm the health of power equipment and increase the operational costs. There is emerging opportunity to dispatch soft open points to improve the operation performance of active distribution network. This paper proposes an optimal dispatch strategy to improve the network balancing performance, where a new type of phase-changing soft open point is installed. First, a new type of phase-changing soft open point with full-phase changing ability is introduced to balance the three-phase power flow. Then, the optimization model is formulated for phase-changing soft open points dispatching to minimize the total cost of distribution network. Furthermore, the model is formed as a constrained Markov decision process and efficiently solved by the augmented Lagrangian-based safe deep reinforcement learning algorithm featuring the soft actor-critic method. Finally, numerical simulations are conducted to validate the effectiveness, accuracy, and efficiency of the proposed method.

Method of Motion Planning for Digital Twin Navigation and Cutting of Shearer.

To further enhance the intelligence level of coal mining faces and achieve the autonomous derivation, learning, and optimization of shearer navigation cutting, this paper proposes the methods of shearer digital twin navigation cutting motion planning based on the concept of shearer autonomous navigation cutting technology and intelligent coal mining face digital twins. This study includes the digital twin theory and the construction method of the shearer digital twin navigation cutting motion planning system based on this theory. Based on the digital twin theory, a shearer digital twin navigation cutting motion planning system was constructed. This system supports the service functions of the shearer cutting digital twin, dynamic navigation map digital twin, reinforcement learning environment construction, and motion planning through the physical perception layer, comprehensive data layer, and digital-model fusion analysis layer. Finally, by comparing the effects of the DQN-NAF and DDPG deep reinforcement learning algorithms in the shearer motion planning task within the constructed digital twin environment, the results show that the DQN-NAF algorithm demonstrates better performance and stability in solving the shearer digital twin motion planning task.

Research on priority scheduling strategy for smoothing power fluctuations of microgrid tie‐lines based on PER‐DDPG algorithm

AbstractThe variability of renewable energy within microgrids (MGs) necessitates the smoothing of power fluctuations through the effective scheduling of internal power equipment. Otherwise, significant power variations on the tie‐line connecting the MG to the main power grid could occur. This study introduces an innovative scheduling strategy that utilizes a data‐driven approach, employing a deep reinforcement learning algorithm to achieve this smoothing effect. The strategy prioritizes the scheduling of MG's internal power devices, taking into account the stochastic charging patterns of electric vehicles. The scheduling optimization model is initially described as a Markov decision process with the goal of minimizing power fluctuations on the interconnection lines and operational costs of the MG. Subsequently, after preprocessing the historical operational data of the MG, an enhanced scheduling strategy is developed through a neural network learning process. Finally, the results from four scheduling scenarios demonstrate the significant impact of the proposed strategy. Comparisons of reward curves before and after data preprocessing underscore its importance. In contrast to optimization results from deep deterministic policy gradient, soft actor‐critic, and particle swarm optimization algorithms, the superiority of the deep deterministic policy gradient algorithm with the addition of a priority experience replay mechanism is highlighted.

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance

Learning-based adaptive control methods hold the potential to empower autonomous agents in mitigating the impact of process variations with minimal human intervention. However, their application to autonomous underwater vehicles (AUVs) has been constrained by two main challenges: (1) the presence of unknown dynamics in the form of sea current disturbances, which cannot be modelled or measured due to limited sensor capability, particularly on smaller low-cost AUVs, and (2) the nonlinearity of AUV tasks, where the controller response at certain operating points must be excessively conservative to meet specifications at other points. Deep Reinforcement Learning (DRL) offers a solution to these challenges by training versatile neural network policies. Nevertheless, the application of DRL algorithms to AUVs has been predominantly limited to simulated environments due to their inherent high sample complexity and the distribution shift problem. This paper introduces a novel approach by combining the Maximum Entropy Deep Reinforcement Learning framework with a classic model-based control architecture to formulate an adaptive controller. In this framework, we propose a Sim-to-Real transfer strategy, incorporating a bio-inspired experience replay mechanism, an enhanced domain randomisation technique, and an evaluation protocol executed on a physical platform. Our experimental assessments demonstrate the effectiveness of this method in learning proficient policies from suboptimal simulated models of the AUV. When transferred to a real-world vehicle, the approach exhibits a control performance three times higher compared to its model-based nonadaptive but optimal counterpart.

Achieving Robust Learning Outcomes in Autonomous Driving with DynamicNoise Integration in Deep Reinforcement Learning

The advancement of autonomous driving technology is becoming increasingly vital in the modern technological landscape, where it promises notable enhancements in safety, efficiency, traffic management, and energy use. Despite these benefits, conventional deep reinforcement learning algorithms often struggle to effectively navigate complex driving environments. To tackle this challenge, we propose a novel network called DynamicNoise, which was designed to significantly boost the algorithmic performance by introducing noise into the deep Q-network (DQN) and double deep Q-network (DDQN). Drawing inspiration from the NoiseNet architecture, DynamicNoise uses stochastic perturbations to improve the exploration capabilities of these models, thus leading to more robust learning outcomes. Our experiments demonstrated a 57.25% improvement in the navigation effectiveness within a 2D experimental setting. Moreover, by integrating noise into the action selection and fully connected layers of the soft actor–critic (SAC) model in the more complex 3D CARLA simulation environment, our approach achieved an 18.9% performance gain, which substantially surpassed the traditional methods. These results confirmed that the DynamicNoise network significantly enhanced the performance of autonomous driving systems across various simulated environments, regardless of their dimensionality and complexity, by improving their exploration capabilities rather than just their efficiency.

A cognitive communication jamming strategy based on Transformer and Deep Reinforcement Learning

The advent of sophisticated communication technologies, such as cognitive radio and anti-jamming techniques, has significantly elevated the challenge of disrupting enemy communications. Nevertheless, the inherent openness of wireless communications remains a vulnerability that can be exploited to interfere with them. Some contemporary Reinforcement Learning (RL)-based jamming strategies examine methods for rapidly identifying the optimal jamming strategy for a specific modulated signal. However, such algorithms lack the flexibility and responsiveness required to effectively counter the enemy’s evolving communication strategies. To address this issue, we propose a Transformer and Deep Reinforcement Learning (DRL)-based jamming strategy that can be trained to identify jamming methods for multiple digital and analog signals. In particular, the Transformer Encoder is employed as a network for DRL to process the state information pertaining to the enemy communication. Subsequently, the decision module of the Double Deep Q Network (DDQN) is utilized to select the jamming action based on the processed information. Furthermore, we have devised a reward function and constructed an invalid jamming list, with the objective of selecting an action that requires low power consumption and enhances the convergence speed of the algorithm. The experimental results demonstrate that the algorithm proposed in this paper exhibits notable performance advantages in comparison to other networks and DRL algorithms.

An optimized heterogeneous multi-access edge computing framework based on transfer learning and artificial internet of things

In most practical applications, the feature space of the training datasets and the target domain datasets are inconsistent, or the data distribution between them is inconsistent, which leads to the problem of data starvation and makes it difficult for terminal devices to obtain high accurate results. Aiming at the problems of limited terminal device resources, low accuracy of data processing results, and unsatisfactory processing speed, a Heterogeneous Multi-access Edge Computing (MEC) Framework based on Transfer Learning (TL) is proposed, abbreviated as HMECF-TL. This framework adopts a cloud-edge-end three-layer architecture. It uses model transfer to optimize the Convolutional Neural Networks (CNN) model at each layer to achieve the goal of improving data processing speed and accuracy. Furthermore, a multi-agent Deep Reinforcement Learning Algorithm having Attention Mechanism (DRLAAM) is designed to further increase the timeliness performance of computation-intensive applications. The performance of HMECSF-TL framework is verified by simulation experiments, which not only reduces the delay by more than 24.66 %, but also improves the accuracy by more than 8.34 %. The framework not only increase the computing capacity to solve the shortage of terminal device resources, but also improve the quality of data processing to solve the problem of data starvation.

Dynamic difficulty adjustment using deep reinforcement learning: A review

With the evolution of the gaming industry, there is an increasing demand for games that offer immersive experiences beyond basic gratification. Traditional games with fixed difficulty levels often fail to cater to the diverse preferences and skills of players. To address these needs, dynamic difficulty adjustment (DDA) has emerged as a crucial element in game design. This review explores the application of Deep Reinforcement Learning (DRL) in achieving DDA, contrasting it with traditional methods. By examining various DRL algorithms and their effectiveness, as well as evaluating traditional models, this review identifies the potential of DRL in enhancing player experiences and outlines existing challenges. It also highlights future research directions, including the integration of DRL with Flow Theory and innovative evaluation methods. The goal is to provide a comprehensive overview of how DRL can advance game design and improve player satisfaction.

Deep Reinforcement Learning in Nonstationary Environments With Unknown Change Points.

Deep reinforcement learning (DRL) is a powerful tool for learning from interactions within a stationary environment where state transition and reward distributions remain constant throughout the process. Addressing the practical but challenging nonstationary environments with time-varying state transition or reward function changes during the interactions, ingenious solutions are essential for the stability and robustness of DRL agents. A key assumption to cope with nonstationary environments is that the change points between the previous and the new environments are known beforehand. Unfortunately, this assumption is impractical in many cases, such as outdoor robots and online recommendations. To address this problem, this article presents a robust DRL algorithm for nonstationary environments with unknown change points. The algorithm actively detects change points by monitoring the joint distribution of states and actions. A detection boosted, gradient-constrained optimization method then adapts the training of the current policy with the supporting knowledge of formerly well-trained policies. The previous policies and experience help the current policy adapt rapidly to environmental changes. Experiments show that the proposed method accumulates the highest reward among several alternatives and is the fastest to adapt to new environments. This work has compelling potential for increasing the environmental suitability of intelligent agents, such as drones, autonomous vehicles, and underwater robots.

A latent space method with maximum entropy deep reinforcement learning for data assimilation

Data assimilation aims to calibrate the uncertain state or parameter vectors of a system by matching simulation results with observations, which is crucial for uncertainty quantification and optimization. Although traditional methods of data assimilation have shown promising results in practical applications, they need well-designed iteration rules (i.e., gradient, covariance, search strategies). Deep reinforcement learning (DRL) can solve data assimilation problems by trial and error, which does not require convoluted iteration rules. However, previous DRL methods cannot tackle high-dimensional data assimilation problems efficiently. In this work, we propose a latent space method with maximum entropy DRL for data assimilation to extend DRL to complicated systems with lots of parameters. Solutions can be found in a reduced space, through the interaction between the agent represented by artificial neural networks and the environment of numerical simulation. The proposed method contains two key points. First, to make the method applicable to high-dimensional problems, we construct a latent space method by integrating a dimensionality reduction method with the state, action, and reward settings for the agent. Second, a maximum entropy DRL algorithm, Soft Actor–Critic (SAC), was employed to efficiently explore the parameter space. The proposed method overcomes the limitation that previous DRL algorithms are only suitable for small-scale cases, and enables DRL to deal with medium-scale or even large-scale data assimilation problems. The performance of the proposed method was validated by comparison with a deep reinforcement learning algorithm and traditional data assimilation algorithms on 2D synthetic and 3D reservoir cases.

Adaptive Path Planning for Subsurface Plume Tracing with an Autonomous Underwater Vehicle

Autonomous underwater vehicles (AUVs) have been increasingly applied in marine environmental monitoring. Their outstanding capability of performing tasks without human intervention makes them a popular tool for environmental data collection, especially in unknown and remote regions. This paper addresses the path planning problem when AUVs are used to perform plume source tracing in an unknown environment. The goal of path planning is to locate the plume source efficiently. The path planning approach is developed using the Double Deep Q-Network (DDQN) algorithm in the deep reinforcement learning (DRL) framework. The AUV gains knowledge by interacting with the environment, and the optimal direction is extracted from the mapping obtained by a deep neural network. The proposed approach was tested by numerical simulation and on a real ground vehicle. In the numerical simulation, several initial sampling strategies were compared on the basis of survey efficiency. The results show that direct learning based on the interaction with the environment could be an appropriate survey strategy for plume source tracing problems. The comparison with the canonical lawnmower path used in practice showed that path planning using DRL algorithms could be potentially promising for large-scale environment exploration.

Towards equitable infrastructure asset management: Scour maintenance strategy for aging bridge systems in flood-prone zones using deep reinforcement learning

Bridges play a critical role in transportation networks; however, they are vulnerable to deterioration, aging, and degradation, especially in the face of climate change and extreme weather events such as floodings. Furthermore, bridges can significantly affect social vulnerability; their damage or destruction can isolate communities, inhibit emergency responses, and disrupt essential services. Maintaining critical bridges in a cost-effective and sustainable manner is crucial to ensure their longevity and protect vulnerable communities. To address the maintenance optimization problem of bridge systems considering the effects of time deterioration, flood degradation, and social vulnerability, this study proposes a deep reinforcement learning algorithm to optimally allocate resources to bridges that are at expected cost of failure due to scour. The algorithm considers the effects of flood degradation with different return periods and is trained using a Markov Decision Process as the environment. The study conducts four flood simulation scenarios using Geographic Information System data. The findings suggest that the deep reinforcement learning algorithm proposes a sequence of repair actions that outperforms the status quo, currently employed by bridge managers. The significance of this study lies in its valuable insights for cities worldwide on how to effectively optimize their limited resources for the maintenance and rehabilitation of critical infrastructure systems to decrease portfolio cost and increase social equity.

A deep residual reinforcement learning algorithm based on Soft Actor-Critic for autonomous navigation

The problem of autonomous navigation has attracted significant attention from robotics research community in the last few decades. In this paper, we address the problem of low data utilization due to the large amount of episode experience value data. A maximum entropy algorithm based on prioritized experience replay (Learning Good Experience based on Soft Actor-Criti, LGE-SAC) is proposed to quickly reproduce past good experience episodes. As the deep reinforcement learning method is susceptible to failure to plan ahead and explore the target position in a long sequence environment, a deep Residual Soft Actor-Critic (RSAC) is proposed to alleviate this problem. The reinforcement learning policy is fused with the Artificial Potential Field method to improve the generalization ability of the proposed algorithm, thus improving robot adaptation in new test environments. In order to validate the effectiveness of the proposed algorithm, we conducted simulation experiments in Gazebo simulator environment and real experiments on a Turtlebot3 robot equipped with LiDAR sensor. Simulation and experiment results show that the proposed algorithm effectively avoids obstacles and succeeds in reaching the goal compared to other obstacle avoidance algorithms. In comparison with the Artificial Potential Field method, the planning success rate of the proposed RSAC algorithm in the test environment is increased by 30%, and at the same time, the number of planning steps is reduced by half, and the generalization ability is improved.

EnergyShare AI: Transforming P2P energy trading through advanced deep learning

Peer-to-peer (P2P) energy trading is an innovative concept poised to transform energy demand management and utilization. EnergyShare AI is a powerful peer-to-peer energy exchange system that operates on a P2P model that integrates advanced machine learning with distributed energy sharing. This paper presents EnergyShare AI, a technology that connects consumers and prosumers through solar arrays, energy storage systems (ESS), and electric vehicles (EVs). Using Deep Reinforcement Learning (DRL) algorithms, Energy Share AI significantly improves energy management efficiency and substantially reduces costs. Our approach offers several advantages over traditional linear integer programming models, particularly in optimizing bidirectional energy transfer involving EVs and highlighting the critical role of ESS and photovoltaic (PV) systems in facilitating efficient P2P energy trading. Our research results show that successful P2P exchange can lead to significant cost savings and improved sustainability, thereby increasing the amount of energy transferred between different household profiles and stages of human development.

Investigation of energy management strategy based on deep reinforcement learning algorithm for multi-speed pure electric vehicles

Investigation of energy management strategy based on deep reinforcement learning algorithm for multi-speed pure electric vehicles

Bio-inspired distributed load frequency control in Islanded Microgrids: A multi-agent deep reinforcement learning approach

To prevent errors in load frequency control (LFC) decision-making in an islanded microgrid and reduce the waste of frequency regulation resources, a fully distributed “starfish” load frequency control method is proposed. For this method, a novel deep reinforcement learning algorithm called the distributed decomposed multirole multiagent deep deterministic policy gradient (DDMR-MADDPG) algorithm is introduced. This algorithm treats each unit as an agent, enabling centralized training of all the agents. Through the coordination and control of multiple agents, a global optimal strategy can be obtained. During online application, the algorithm employs a distributed implementation strategy, enabling each unit to make decisions autonomously. This decision-making strategy is akin to the distributed neural network of a starfish, and it can fundamentally improve the efficiency and correctness of decision-making. In addition, it employs multiple techniques, including value function decomposition, multirole learning, imitation learning, and curriculum learning approaches. The performance of the proposed method compared with 24 existing methods is demonstrated on a simulated LFC model of the Zhuzhou Island microgrid in China.

An improved hierarchical deep reinforcement learning algorithm for multi-intelligent vehicle lane change

Lane change is a pivotal technology in autonomous driving, playing a significant role in improving traffic conditions. The control task of lane change becomes exceptionally complex for coordination of multiple intelligent vehicles, especially in unpredictable situations. To address this issue, the paper proposes a novel multi-intelligent vehicle lane change method (MVLC) based on Deep Reinforcement Learning (DRL). The MVLC features a hierarchical framework that includes a lower-layer lane change controller and an upper-layer reinforcement learning decision-making method. In the lower layer, a lane change controller is designed to execute motion control for a single vehicle, consisting of a lateral controller utilizing dueling double deep Q-network to make lane change decision and a longitudinal controller that employs Intelligent Driver Model (IDM) to follow the preceding vehicle. In the upper layer, a model-free DRL method is used for simultaneous control of multiple intelligent vehicles. To learn the optimal policy, a comprehensive reward function is designed. Finally, experiments are conducted in mixed traffic to valid the effectiveness of the proposed method. Results show that MVLC achieves secure and timely lane changes for multiple vehicles. Therefore, the method is expected to have significant potential for improving overall traffic efficiency and mitigating traffic congestion.

Comparative Analysis of Reinforcement Learning Approaches for Multi-Objective Optimization in Residential Hybrid Energy Systems

The rapid expansion of renewable energy in buildings has been expedited by technological advancements and government policies. However, including highly permeable intermittent renewables and energy storage presents significant challenges for traditional home energy management systems (HEMSs). Deep reinforcement learning (DRL) is regarded as the most efficient approach for tackling these problems because of its robust nonlinear fitting capacity and capability to operate without a predefined model. This paper presents a DRL control method intended to lower energy expenses and elevate renewable energy usage by optimizing the actions of the battery and heat pump in HEMS. We propose four DRL algorithms and thoroughly assess their performance. In pursuit of this objective, we also devise a new reward function for multi-objective optimization and an interactive environment grounded in expert experience. The results demonstrate that the TD3 algorithm excels in cost savings and PV self-consumption. Compared to the baseline model, the TD3 model achieved a 13.79% reduction in operating costs and a 5.07% increase in PV self-consumption. Additionally, we explored the impact of the feed-in tariff (FiT) on TD3’s performance, revealing its resilience even when the FiT decreases. This comparison provides insights into algorithm selection for specific applications, promoting the development of DRL-driven energy management solutions.