Deep Reinforcement Learning Method Research Articles

Robust motion planning for mobile robots under attacks against obstacle localization

Abstract Thanks to its real-time computation efficiency, deep reinforcement learning (DRL) has been widely applied in motion planning for mobile robots. In DRL-based methods, a DRL model computes an action for a robot based on the states of its surrounding obstacles, including other robots that may communicate with it. These methods always assume that the environment is attack-free and the obtained obstacles’ states are reliable. However, in the real world, a robot may suffer from obstacle localization attacks (OLAs), such as sensor attacks, communication attacks, and remote-control attacks, which cause the robot to retrieve inaccurate positions of the surrounding obstacles. In this paper, we propose a robust motion planning method ObsGAN-DRL, integrating a generative adversarial network (GAN) into DRL models to mitigate OLAs in the environment. First, ObsGAN-DRL learns a generator based on the GAN model to compute the approximation of obstacles’ accurate positions in benign and attack scenarios. Therefore, no detectors are required for ObsGAN-DRL. Second, by using the approximation positions of the surrounding obstacles, ObsGAN-DRL can leverage the state-of-the-art DRL methods to compute collision-free motion commands (e.g., velocity) efficiently. Comprehensive experiments show that ObsGAN-DRL can mitigate OLAs effectively and guarantee safety. We also demonstrate the generalization of ObsGAN-DRL.

Advancements in UAV Path Planning: A Deep Reinforcement Learning Approach with Soft Actor-Critic for Enhanced Navigation

This paper tackles the intricate challenge of autonomous navigation for Unmanned Aerial Vehicles (UAVs) through dynamically changing environments. We focus on a sophisticated Deep Reinforcement Learning (DRL) approach using the Soft Actor-Critic (SAC) algorithm, optimized for UAV path planning within a continuous action space. This methodology leverages environmental image data to enhance the precision of flight maneuvers and effective obstacle avoidance. Our approach, validated through extensive simulations in Gazebo and field tests, demonstrates the algorithm’s efficacy in enabling UAVs to adeptly navigate obstacles using depth maps. The study further explores the robustness of the SAC algorithm by comparing it with traditional DRL methods, emphasizing its superior performance in real-world applications. This research contributes significantly to advancing UAV technology, particularly in autonomous motion planning, by integrating cutting-edge machine learning techniques. The video link is: https://www.youtube.com/watch?v=Nd_aMzejNXY .

Optimal load distribution control for airport terminal chiller units based on deep reinforcement learning

The building sector consumes substantial energy, with HVAC systems accounting for nearly half of the total energy use. Optimizing chiller unit operations is crucial for reducing energy consumption. This research introduces a novel model-free control method for optimizing central chiller load distribution using deep reinforcement learning (DRL). The aim is to address the challenges posed by the high-dimensional complexity and parameter tuning in traditional meta-heuristic algorithms. The proposed method leverages deep neural networks combined with reinforcement learning to optimize chiller control based on real-world data. Case studies conducted in a large airport terminal demonstrate that this approach achieves a 7.71 % energy savings, closely matching the performance of model-based control methods with only a 0.26 % difference. The results indicate that the DRL method not only outperforms traditional expert control systems but also offers a feasible alternative for optimal control in complex, variable environments with limited data. This study contributes to the field by filling a research gap in applying DRL for optimal chiller load (OCL) control and demonstrates its potential for large-scale public building applications. The novelty of this research lies in its model-free approach, which provides a robust solution for energy optimization in dynamic and uncertain environments.

Automated position control of tunnel boring machine during excavation using deep reinforcement learning

Position control of tunnel boring machines (TBM) is critical to ensure precise construction and meets design specifications. However, the high dependence on human intervention largely affects the efficiency and reliability of the operations due to slow response, inconsistent judgment and high risk of errors. Targeting to automate the position control of the TBM to enable it to the planned route in a more efficient and reliable manner, this research proposes a deep reinforcement learning (DRL) method considering spatial-temporal dynamics to perform automated real-time TBM operations. The DRL algorithm is embedded with a time-series deep learning model to simulate the working environment with dynamic time-space variations. The whole process is coupled with the designed reward and evaluation metrics for model enhancement. To showcase the viability and applicability of the suggested approach, a project from Singapore is utilized as an illustrative example. The outcomes indicate that the suggested method exhibits robust capabilities in forecasting and controlling TBM positions, with an R2 of 0.86 and 0.92 in forecasting the horizontal and vertical deviations of the TBM tail three steps ahead and an overall improvement of 50.7 %. The proposed method is capable of providing an optimal strategy to intelligently forecast and control the TBM deviation. The novelty of this method comes from the feasible integration of time series prediction and DRL to simulate the spatial-temporal dynamics along with TBM excavation and obtain a position control model for optimal tunneling strategy, contributing to the effective facilitation of TBM tunneling to improve efficiency and reliability.

V2X Communication Computing Resource Collaboration Technology Based on Deep Reinforcement Learning

In traditional cloud computing, data needs to be transferred from distributed data sources to remote cloud data centers for high-performance computing. However, tasks often experience network transmission delays of hundreds of milliseconds or more from data sources to cloud data centers, which does not meet the need for low latency in V2X (Vehicle to Everything) systems. On the basis of the existing heterogeneous task scenario, this paper proposes a communication computing resource collaboration technology scheme based on heterogeneous task, which can meet the requirements of delay and cost when facing tasks with different delay requirements and different divisibility. This scheme uses the communication computing resources of the roadside unit and the service vehicle, so that the computing task of the task vehicle can be completed within the delay requirement and the cost is minimal. We use the deep reinforcement learning method to study, and finally the performance of the proposed scheme is verified by simulation.

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

Analyzing deep reinforcement learning model decisions with Shapley additive explanations for counter drone operations

As the use of drones continues to increase, their capabilities pose a threat to airspace safety when they are misused. Deploying AI models for intercepting these unwanted drones becomes crucial. However, these AI models, such as deep learning models, often operate as “black boxes”, making it hard to trust their decision-making system. This also affects end-users’ confidence in these AI systems. In this paper, the explainability of deep reinforcement learning is investigated and a deep reinforcement learning (DRL) method, double deep Q-network with dueling architecture and prioritized experience replay is applied to train the AI models. To make the AI model decisions more transparent and to understand the reasoning behind the AI decisions for counter-drone systems, Shapley Additive Explanations (SHAP) method is implemented. After training the DRL agent, experience replay is visualized, and the absolute SHAP values are calculated to explain the key factors that influence the deep reinforcement learning agent’s choices. The integration of DRL with explainable AI methods such as SHAP demonstrates significant potential for the advancement of robust and efficient counter-drone systems.

A multi-step on-policy deep reinforcement learning method assisted by off-policy policy evaluation

A multi-step on-policy deep reinforcement learning method assisted by off-policy policy evaluation

A NoisyNet deep reinforcement learning method for frequency regulation in power systems

AbstractThis study thoroughly investigates the NoisyNet Deep Deterministic Policy Gradient (DDPG) for frequency regulation. Compared with the conventional DDPG method, the suggested method can provide several benefits. First, the parameter noise will explore different strategies more thoroughly and can potentially discover better policies that it might miss if only action noise were used, which helps the actor achieve an optimal control strategy, resulting in enhanced dynamic response. Second, by employing the delayed policy update policy work with the proposed framework, the training process exhibits faster convergence, enabling rapid adaptation to changing disturbances. To substantiate its efficacy, the scheme is subjected to simulation tests on both an IEEE three‐area power system, an IEEE 39 bus power system, and an IEEE 68 bus system. A comprehensive performance comparison was performed against other DDPG‐based methods to validate and evaluate the performance of the proposed LFC scheme.

Optimizing the thermal performance of phase change materials in building applications using deep reinforcement learning and Bayesian optimization

This research presents a novel methodology for Deep Reinforcement Learning (DRL) and Bayesian Optimisation of the thermal performance of PCMs in building operations. The developed models utilise a unique and large dataset comprising 1500 building thermal profiles, simulated for various climates and building setups obtained from our industry partners and as open data. PCM-based systems are deployed for thermal insulation of building envelopes to regulate indoor temperature conditions and reduce the need for heating and cooling systems, resulting in enhanced energy efficiency. Through the real-time management of the thermal efficacy of PCMs using the DRL method trained on the large dataset and fine-tuning of the underlying model parameters using Bayesian Optimisation, the optimised system achieves energy saving in heating and cooling load of up to 45 percent, along with the induced reduction in CO2 emission. At the same time, DRL contributes to decreasing the thermal fluctuation in the indoor temperature and keeps it in the narrow range of 1.2 °C in case of high thermal variability scenarios. Currently, the best performance is reported in the literature. This research exemplifies the potential of DRL and Bayesian optimisation in sustainable building. It depicts the applications of advanced intelligent computing algorithms with big building energy data as a novel, robust and superior approach for optimising real-world building energy management systems. The methodology and the improvements in energy savings in thermal and energy management of buildings highlight the novelty and potential benefit of the implemented research as a new intellectual property towards sustainable building design.

Capacity planning in logistics corridors: Deep reinforcement learning for the dynamic stochastic temporal bin packing problem

This paper addresses the challenge of managing uncertainty in the daily capacity planning of a terminal in a corridor-based logistics system. Corridor-based logistics systems facilitate the exchange of freight between two distinct regions, usually involving industrial and logistics clusters. In this context, we introduce the dynamic stochastic temporal bin packing problem. It models the assignment of individual containers to carriers’ trucks over discrete time units in real-time. We formulate it as a Markov decision process (MDP). Two distinguishing characteristics of our problem are the stochastic nature of the time-dependent availability of containers, i.e., container delays, and the continuous-time, or dynamic, aspect of the planning, where a container announcement may occur at any time moment during the planning horizon. We introduce an innovative real-time planning algorithm based on Proximal Policy Optimization (PPO), a Deep Reinforcement Learning (DRL) method, to allocate individual containers to eligible carriers in real-time. In addition, we propose some practical heuristics and two novel rolling-horizon batch-planning methods based on (stochastic) mixed-integer programming (MIP), which can be interpreted as computational information relaxation bounds because they delay decision making. The results show that our proposed DRL method outperforms the practical heuristics and effectively scales to larger-sized problems as opposed to the stochastic MIP-based approach, making our DRL method a practically appealing solution.

A latent space method with maximum entropy deep reinforcement learning for data assimilation

Data assimilation aims to calibrate the uncertain state or parameter vectors of a system by matching simulation results with observations, which is crucial for uncertainty quantification and optimization. Although traditional methods of data assimilation have shown promising results in practical applications, they need well-designed iteration rules (i.e., gradient, covariance, search strategies). Deep reinforcement learning (DRL) can solve data assimilation problems by trial and error, which does not require convoluted iteration rules. However, previous DRL methods cannot tackle high-dimensional data assimilation problems efficiently. In this work, we propose a latent space method with maximum entropy DRL for data assimilation to extend DRL to complicated systems with lots of parameters. Solutions can be found in a reduced space, through the interaction between the agent represented by artificial neural networks and the environment of numerical simulation. The proposed method contains two key points. First, to make the method applicable to high-dimensional problems, we construct a latent space method by integrating a dimensionality reduction method with the state, action, and reward settings for the agent. Second, a maximum entropy DRL algorithm, Soft Actor–Critic (SAC), was employed to efficiently explore the parameter space. The proposed method overcomes the limitation that previous DRL algorithms are only suitable for small-scale cases, and enables DRL to deal with medium-scale or even large-scale data assimilation problems. The performance of the proposed method was validated by comparison with a deep reinforcement learning algorithm and traditional data assimilation algorithms on 2D synthetic and 3D reservoir cases.

A deep residual reinforcement learning algorithm based on Soft Actor-Critic for autonomous navigation

The problem of autonomous navigation has attracted significant attention from robotics research community in the last few decades. In this paper, we address the problem of low data utilization due to the large amount of episode experience value data. A maximum entropy algorithm based on prioritized experience replay (Learning Good Experience based on Soft Actor-Criti, LGE-SAC) is proposed to quickly reproduce past good experience episodes. As the deep reinforcement learning method is susceptible to failure to plan ahead and explore the target position in a long sequence environment, a deep Residual Soft Actor-Critic (RSAC) is proposed to alleviate this problem. The reinforcement learning policy is fused with the Artificial Potential Field method to improve the generalization ability of the proposed algorithm, thus improving robot adaptation in new test environments. In order to validate the effectiveness of the proposed algorithm, we conducted simulation experiments in Gazebo simulator environment and real experiments on a Turtlebot3 robot equipped with LiDAR sensor. Simulation and experiment results show that the proposed algorithm effectively avoids obstacles and succeeds in reaching the goal compared to other obstacle avoidance algorithms. In comparison with the Artificial Potential Field method, the planning success rate of the proposed RSAC algorithm in the test environment is increased by 30%, and at the same time, the number of planning steps is reduced by half, and the generalization ability is improved.

Deep Reinforcement Learning Method for Task Offloading in Mobile Edge Computing Networks Based on Parallel Exploration with Asynchronous Training

Deep Reinforcement Learning Method for Task Offloading in Mobile Edge Computing Networks Based on Parallel Exploration with Asynchronous Training

An improved hierarchical deep reinforcement learning algorithm for multi-intelligent vehicle lane change

Lane change is a pivotal technology in autonomous driving, playing a significant role in improving traffic conditions. The control task of lane change becomes exceptionally complex for coordination of multiple intelligent vehicles, especially in unpredictable situations. To address this issue, the paper proposes a novel multi-intelligent vehicle lane change method (MVLC) based on Deep Reinforcement Learning (DRL). The MVLC features a hierarchical framework that includes a lower-layer lane change controller and an upper-layer reinforcement learning decision-making method. In the lower layer, a lane change controller is designed to execute motion control for a single vehicle, consisting of a lateral controller utilizing dueling double deep Q-network to make lane change decision and a longitudinal controller that employs Intelligent Driver Model (IDM) to follow the preceding vehicle. In the upper layer, a model-free DRL method is used for simultaneous control of multiple intelligent vehicles. To learn the optimal policy, a comprehensive reward function is designed. Finally, experiments are conducted in mixed traffic to valid the effectiveness of the proposed method. Results show that MVLC achieves secure and timely lane changes for multiple vehicles. Therefore, the method is expected to have significant potential for improving overall traffic efficiency and mitigating traffic congestion.

Causal deconfounding deep reinforcement learning for mobile robot motion planning

Deep reinforcement learning (DRL) has emerged as an efficient approach for motion planning in mobile robot systems. It leverages the offline training process to enhance real-time computation efficiency. In DRL-based methods, the DRL models are trained to compute an action based on the current state of the robot and the surrounding obstacles. However, the trained models may capture spurious correlations through potential confounders, resulting in non-robust state representations, which limits the models’ robustness and generalizability. In this paper, we propose a Causal Deconfounding DRL method for Motion Planning, CD-DRL-MP, to address spurious correlations and learn robust and generalizable policies. Specifically, we formalize the temporal causal relationships between states and actions using a structural causal model. We then extract the minimal sufficient state representation set by blocking the backdoor paths in the causal model. Finally, using the representation set, CD-DRL-MP learns the causal effect between states and actions while mitigating the detrimental influence of potential confounders and computes motion commands for mobile robots. Comprehensive experiments show that the proposed method significantly outperforms non-causal DRL methods and existing causal methods, while guaranteeing good robustness and generalizability.

Optimizing Room Temperature Control: A Comparative Analysis of Deep Reinforcement Learning and Contextual Bayesian Optimization Methods

Building temperature control presents significant modeling challenges due to the complexity of internal systems and varied usage patterns. Consequently, black box modeling has become a prevalent approach in this domain. This technique broadly encompasses deep learning and Bayesian optimization methods, each offering unique mechanisms for deriving optimal control parameters. Deep learning methods iteratively refine control parameters using extensive datasets, whereas Bayesian optimization employs Gaussian process regression and acquisition functions to pinpoint optimal solutions. This paper conducts a comparative analysis of two innovative methods within these branches: the application of Deep Reinforcement Learning (DRL) for integrated control of household temperature and electric vehicle charging, and the Primal-Dual Contextual Bayesian Optimization (PDCBO) method for adaptive thermal control in buildings. The study reveals that the DRL approach, integrated with electric vehicle charging systems, excels in responsiveness and reducing energy consumption. In contrast, the PDCBO method enhances control by allowing for the artificial definition and continuous optimization of constraints on energy usage and comfort during the operation, facilitated by algorithmic design. By synthesizing the strengths of both methods and considering electric vehicle integration and tailored adaptive control constraints, enhanced outcomes are achievable. This paper proposes that specific control strategies be developed for different building types to address diverse real-world requirements effectively.

Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties

Improving the efficiency of home energy management (HEM) is of great significance in reducing resource waste and prompting renewable energy consumption. With the development of advanced HEM systems and internet-of-things technology, more residents are willing to participate in the electricity market through a demand response mechanism, which can not only reduce the electricity bill but also stabilize the operation of the main grid through peak shaving and valley filling. However, the uncertainties from household appliance parameters, user behaviors, penetrated renewable energy, and electricity prices render the HEM problem a non-trivial task. Although previous deep reinforcement learning (DRL) methods have no requirement for system dynamics due to their model-free and data-driven merits, the unrestricted action space may violate the physical constraints of various devices, and few works have concentrated on this area before. To solve the above challenges, this paper proposes a novel interior-point policy optimization-based multi-agent DRL algorithm to optimize the home energy scheduling procedure while guaranteeing safety during its operation. Besides, a time-series prediction model based on a non-stationary Transformer neural network is proposed to predict the future trend of solar generation and electricity price, which can provide the agents with abundant information to make better decisions. The superior performance of our proposed predictive-control coupled method is demonstrated through extensive numerical experiments, which achieves more precise prediction results, near-zero constraint violations, and better trade-offs between cost and safety than several benchmark methods.

Proximal evolutionary strategy: improving deep reinforcement learning through evolutionary policy optimization

Evolutionary Algorithms (EAs), including Evolutionary Strategies (ES) and Genetic Algorithms (GAs), have been widely accepted as competitive alternatives to Policy Gradient techniques for Deep Reinforcement Learning (DRL). However, they remain eclipsed by cutting-edge DRL algorithms in terms of time efficiency, sample complexity, and learning effectiveness. In this paper, aiming at advancing evolutionary DRL research, we develop an evolutionary policy optimization algorithm with three key technical improvements. First, we design an efficient layer-wise strategy for training DNNs through Covariance Matrix Adaptation Evolutionary Strategies (CMA-ES) in a highly scalable manner. Second, we establish a surrogate model based on proximal performance lower bound for fitness evaluations with low sample complexity. Third, we embed a gradient-based local search technique within the evolutionary policy optimization process to further improve the learning effectiveness. The three technical innovations jointly forge a new EA for DRL method named Proximal Evolutionary Strategies (PES). Our experiments on ten continuous control problems show that PES with layer-wise training can be more computationally efficient than CMA-ES; our surrogate model can remarkably reduce the sample complexity of PES in comparison to latest EAs for DRL including CMA-ES, OpenAI-ES, and Uber-GA; PES with gradient-based local search can significantly outperform several promising DRL algorithms including TRPO, AKCTR, PPO, OpenAI-ES, and Uber-GA.

Enhancing Highway Driving: High Automated Vehicle Decision Making in a Complex Multi-Body Simulation Environment

Automated driving is a promising development in reducing driving accidents and improving the efficiency of driving. This study focuses on developing a decision-making strategy for autonomous vehicles, specifically addressing maneuvers such as lane change, double lane change, and lane keeping on highways, using deep reinforcement learning (DRL). To achieve this, a highway driving environment in the commercial multi-body simulation software IPG Carmaker 11 version is established, wherein the ego vehicle navigates through surrounding vehicles safely and efficiently. A hierarchical control framework is introduced to manage these vehicles, with upper-level control handling driving decisions. The DDPG (deep deterministic policy gradient) algorithm, a specific DRL method, is employed to formulate the highway decision-making strategy, simulated in MATLAB software. Also, the computational procedures of both DDPG and deep Q-network algorithms are outlined and compared. A set of simulation tests is carried out to evaluate the effectiveness of the suggested decision-making policy. The research underscores the advantages of the proposed framework concerning its convergence rate and control performance. The results demonstrate that the DDPG-based overtaking strategy enables efficient and safe completion of highway driving tasks.