Double Deep Q-learning Algorithm Research Articles

Research on robot path planning and obstacle avoidance algorithm in dynamic environment based on deep reinforcement learning

Abstract. In dynamic environments, robot path planning and obstacle avoidance are critical tasks, especially in applications such as autonomous driving, industrial automation, and mobile robotics. These tasks are inherently challenging due to the unpredictability of the environment and the need for real-time decision-making. This paper seeks to address these challenges by developing and analyzing both traditional and optimized models for robot navigation. The initial model utilizes a basic Q-learning algorithm, which provides a straightforward approach to learning from the environment but often struggles with the complexity of dynamic scenarios. To this end, an optimized model is developed that combines the Double Deep Q-Learning algorithm (Double DQN) in conjunction with heuristic strategies. The research employs the MATLAB Reinforcement Learning Toolbox to implement and train these models, and utilizes a simulated environment with dynamic obstacles as a testing site. The simulation generates the necessary data to allow for comprehensive testing and evaluation of the models performance. The results show that the optimized model greatly exceeds the initial model in terms of path planning efficiency and obstacle avoidance capabilities, and that the combination of advanced reinforcement learning techniques and heuristic strategies is extremely important for enhancing the performance and reliability of robotic systems in complex, dynamic environments, offering valuable insights for future applications in various fields of robotics.

Achieving accurate trajectory predicting and tracking for autonomous vehicles via reinforcement learning-assisted control approaches

In complex urban traffic scenarios, autonomous vehicles face significant challenges in adapting to diverse and dynamic traffic conditions. Reward-based reinforcement learning has emerged as an effective approach to tackle these challenges. This paper presents a novel method that combines deep reinforcement learning with automotive dynamics systems. Building upon the Double Deep Q-learning algorithm, our approach integrates a Recurrent Neural Network with Gated Recurrent Units to enhance the environmental exploration capabilities of autonomous vehicles. To obtain more precise reward values, we introduce a trajectory tracking algorithm based on a combination of proportional-integral-derivative control and feedforward control within the automotive dynamics system. The proportional-integral-derivative controller is utilized for longitudinal control, while the Error-Optimized feedforward controller enhances lateral control, thereby improving trajectory tracking accuracy. Finally, extensive simulation experiments are conducted to evaluate the proposed method, comparing it against other baseline methods in terms of vehicle following and lane-changing scenarios. The results demonstrate that our approach significantly improves both the reward values and control performance of the algorithm.

Retrospective-Based Deep Q-Learning Method for Autonomous Pathfinding in Three-Dimensional Curved Surface Terrain

Path planning in complex environments remains a challenging task for unmanned vehicles. In this paper, we propose a decoupled path-planning algorithm with the help of a deep reinforcement learning algorithm that separates the evaluation of paths from the planning algorithm to facilitate unmanned vehicles in real-time consideration of environmental factors. We use a 3D surface map to represent the path cost, where the elevation information represents the integrated cost. The peaks function simulates the path cost, which is processed and used as the algorithm’s input. Furthermore, we improved the double deep Q-learning algorithm (DDQL), called retrospective-double DDQL (R-DDQL), to improve the algorithm’s performance. R-DDQL utilizes global information and incorporates a retrospective mechanism that employs fuzzy logic to evaluate the quality of selected actions and identify better states for inclusion in the memory. Our simulation studies show that the proposed R-DDQL algorithm has better training speed and stability compared to the deep Q-learning algorithm and double deep Q-learning algorithm. We demonstrate the effectiveness of the R-DDQL algorithm under both static and dynamic tasks.

Joint Secure Offloading and Resource Allocation for Vehicular Edge Computing Network: A Multi-Agent Deep Reinforcement Learning Approach

The mobile edge computing (MEC) technology can simultaneously provide high-speed computing services for multiple vehicular users (VUs) in vehicular edge computing (VEC) networks. Nevertheless, due to the open feature of the wireless offloading channels and the high mobility of the vehicles, the security and stability of the offloading process would be seriously degraded. In this paper, by utilizing the physical layer security (PLS) technique and spectrum sharing architecture, we propose a deep reinforcement learning based joint secure offloading and resource allocation (SORA) scheme to improve the secrecy performance and resource efficiency of the multi-user VEC networks, where the VU offloading links share the frequency spectrum preoccupied with the vehicle-to-vehicle (V2V) communication links. We use Wyner’s wiretap coding scheme to obtain the achievable secrecy rate and guarantee that confidential information cannot be decoded by multiple mobile eavesdroppers. We aim at minimizing the system processing delay while securing the wireless offloading process, by jointly optimizing the transmit power, the frequency spectrum selection and the computation resource allocation. We formulate the optimization problem as a multi-agent collaborative optimal decision problem and solve it with a double deep Q-learning algorithm. Besides, we set a punishment mechanism for the rate degradation to guarantee the communication quality of each V2V link. Simulation results demonstrate that multiple VU agents adopting the SORA scheme can rapidly adapt to the highly dynamic VEC networks and cooperate to improve the system delay performance while increasing the secrecy probability.

Deep learning in the development of energy Management strategies of hybrid electric Vehicles: A hybrid modeling approach

The Energy Management Strategy (EMS) in an HEV is the key for improving fuel economy and simultaneously reducing pollutant emissions. This paper presents a methodology for developing hybrid models that enable EMS testing as well as the evaluation of fuel consumption, CO2 and pollutant emissions (CO, NOx and THC). In this context, pollutant emissions are hard to quantify with static models such as the well-known map-based approach which is mainly due to the pronounced impact of transient effects. The novelty of this paper primarily comes from the characterization of pollutant emissions through Convolutional Neural Networks (CNN), providing high accuracy for both, instantaneous and cumulative values. The input parameters are classical Internal Combustion Engine (ICE) measurements such as engine speed, air mass flow, torque and exhaust temperature. The proposed CNNs are reduced to a minimum for low complexity and fast computability. These models are developed with experimental data from chassis dyno testing of a conventional turbo-charged gasoline engine vehicle. The pollutant emission models are used in conjunction with physical models of the remaining powertrain allowing for real time simulations of the complete HEV vehicle. The Double Deep-Q learning algorithm is proposed for the EMS and compared to the Dynamic programming (DP) solution.The introduced methodology is developed in a co-simulation framework between MATLAB-Simulink and AMESIM. The resulting model runs between 8 and 10 times faster than real time in an off-the-shelf PC. This provides the capability for developing models suitable for HIL (hardware-in-the-loop) and SIL (software-in-the-loop) applications. The final error in predicted CO2 remains below 2.5% while the final cumulative error for pollutants is below 8.5% in the case of CO and HC emissions.

Deep Reinforcement Learning Approach for Automated Vehicle Mandatory Lane Changing

This paper proposes a reinforcement learning-based framework for mandatory lane changing of automated vehicles in a non-cooperative environment. The objective is to create a reinforcement learning (RL) agent that is able to perform lane-changing maneuvers successfully and efficiently and with minimal impact on traffic flow in the target lane. For this purpose, this study utilizes the double deep Q-learning algorithm structure, which takes relevant traffic states as input and outputs the optimal actions (policy) for the automated vehicle. We put forward a realistic approach for dealing with this problem where, for instance, actions selected by the automated vehicle include steering angles and acceleration/deceleration values. We show that the RL agent is able to learn optimal policies for the different scenarios it encounters and performs the lane-changing task safely and efficiently. This work illustrates the potential of RL as a flexible framework for developing superior and more comprehensive lane-changing models that take into consideration multiple aspects of the road environment and seek to improve traffic flow as a whole.

Successful Pass Schedule Design in Open-Die Forging Using Double Deep Q-Learning

In order to not only produce an open-die forged part with the desired final geometry but to also maintain economic production, precise process planning is necessary. However, due to the incremental forming of the billet, often with several hundred strokes, the process design is arbitrarily complicated and, even today, often only based on experience or simple mathematical models describing the geometry development. Hence, in this paper, fast process models were merged with a double deep Q-learning algorithm to enable a pass schedule design including multi-objective optimization. The presented implementation of a double deep Q-learning algorithm was successfully trained on an industrial-scale forging process and converged stably against high reward values. The generated pass schedules reliably produced the desired final ingot geometry, utilized the available press force well without exceeding plant limits, and, at the same time, minimized the number of passes. Finally, a forging experiment was performed at the institute of metal forming to validate the generated results. Overall, a proof of concept for the pass schedule design in open-die forging via double deep Q-learning was achieved which opens various starting points for future work.

Deep Q-learning for the selection of optimal isocratic scouting runs in liquid chromatography

An important challenge in chromatography is the development of adequate separation methods. Accurate retention models can significantly simplify and expedite the development of adequate separation methods for complex mixtures. The purpose of this study was to introduce reinforcement learning to chromatographic method development, by training a double deep Q-learning algorithm to select optimal isocratic scouting runs to generate accurate retention models. These scouting runs were fit to the Neue-Kuss retention model, which was then used to predict retention factors both under isocratic and gradient conditions. The quality of these predictions was compared to experimental data points, by computing a mean relative percentage error (MRPE) between the predicted and actual retention factors. By providing the reinforcement learning algorithm with a reward whenever the scouting runs led to accurate retention models and a penalty when the analysis time of a selected scouting run was too high (> 1h); it was hypothesized that the reinforcement learning algorithm should by time learn to select good scouting runs for compounds displaying a variety of characteristics. The reinforcement learning algorithm developed in this work was first trained on simulated data, and then evaluated on experimental data for 57 small molecules – each run at 10 different fractions of organic modifier (0.05 to 0.90) and four different linear gradients. The results showed that the MRPE of these retention models (3.77% for isocratic runs and 1.93% for gradient runs), mostly obtained via 3 isocratic scouting runs for each compound, were comparable in performance to retention models obtained by fitting the Neue-Kuss model to all (10) available isocratic datapoints (3.26% for isocratic runs and 4.97% for gradient runs) and retention models obtained via a “chromatographer's selection” of three scouting runs (3.86% for isocratic runs and 6.66% for gradient runs). It was therefore concluded that the reinforcement learning algorithm learned to select optimal scouting runs for retention modeling, by selecting 3 (out of 10) isocratic scouting runs per compound, that were informative enough to successfully capture the retention behavior of each compound.

Task Scheduling in Cloud Using Deep Reinforcement Learning

Cloud computing is an emerging technology used in many applications such as data analysis, storage, and Internet of Things (IoT). Due to the increasing number of users in the cloud and the IoT devices that are being integrated with the cloud, the amount of data generated by these users and these devices is increasing ceaselessly. Managing this data over the cloud is no longer an easy task. It is almost impossible to move all data to the cloud datacenters, and this will lead to excessive bandwidth usage, latency, cost, and energy consumption. This makes it evident that allocating resources to users’ tasks is an essential quality feature in cloud computing. This is because it provides the customers or the users with high Quality of Service (QoS) with the best response time, and it also respects the established Service Level Agreement. Therefore, there is a great importance of efficient utilization of computing resources for which an optimal strategy for task scheduling is required. This paper focuses on the problem of task scheduling of cloud-based applications and aims to minimize the computational cost under resource and deadline constraints. Towards this end, we propose a clipped double deep Q-learning algorithm utilizing the target network and experience relay techniques, as we as using the reinforcement learning approach.

Применение алгоритмов Deep Q-learning и Double Deep Q-learning к задаче управления перевернутым маятником

Today, such a branch of science as «artificial intelligence» is booming in the world. Systems built on the basis of artificial intelligence methods have the ability to perform functions that are traditionally considered the prerogative of man. Artificial intelligence has a wide range of research areas. One such area is machine learning. This article discusses the algorithms of one of the approaches of machine learning – reinforcement learning (RL), according to which a lot of research and development has been carried out over the past seven years. Development and research on this approach is mainly carried out to solve problems in Atari 2600 games or in other similar ones. In this article, reinforcement training will be applied to one of the dynamic objects – an inverted pendulum. As a model of this object, we consider a model of an inverted pendulum on a cart taken from the Gym library, which contains many models that are used to test and analyze reinforcement learning algorithms. The article describes the implementation and study of two algorithms from this approach, Deep Q-learning and Double Deep Q-learning. As a result, training, testing and training time graphs for each algorithm are presented, on the basis of which it is concluded that it is desirable to use the Double Deep Q-learning algorithm, because the training time is approximately 2 minutes and provides the best control for the model of an inverted pendulum on a cart.

Comparative Study of NeuroEvolution Algorithms in Reinforcement Learning for Self-Driving Cars

Neuroevolution has been used to train neural networks for challenging deep Reinforcement Learning (RL) problems like Atari, image hard maze, and humanoid locomotion. The performance is comparable to the performance of neural networks trained by algorithms like Q-learning and policy gradients. This work conducts a detailed comparative study of using neuroevolution algorithms in solving the self-driving car problem. Different neuroevolution algorithms are used to train deep neural networks to predict the steering angle of a car in a simulated environment. Neuroevolution algorithms are compared to the Double Deep Q-Learning (DDQN) algorithm. Based on the experimental results, the neuroevolution algorithms show better performance than DDQN algorithm. The Evolutionary Strategies (ES) algorithm outperforms the rest in accuracy in driving in the middle of the lane, with the best average result of 97.13%. Moreover, the Random Search (RS) algorithm outperforms the rest in terms of driving the longest while keeping close to the middle of the lane, with the best average result of 403.54m. These results confirm that the entire family of genetic and evolutionary algorithms with all their performance optimization techniques, are available to train and develop self driving cars.

Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle

An energy management strategy, based on double deep Q-learning algorithm, is proposed for a dual-motor driven hybrid electric tracked-vehicle. Typical model framework of tracked-vehicle is established where the lateral dynamic can be taken into consideration. For the propose of optimizing the fuel consumption performance, a double deep Q-learning-based control structure is put forward. Compared to conventional deep Q-learning, the proposed strategy prevents training process falling into the overoptimistic estimate of policy value and highlights its significant advantages in terms of the iterative convergence rate and optimization performance. Unique observation states are selected as input variables of reinforcement learning algorithm in view of revealing tracked-vehicles characteristic. The conventional deep Q-learning and dynamic programming are also employed and compared with the proposed strategy for different driving schedules. Simulation results demonstrate the fuel economy of proposed methodology achieves 7.1% better than that of conventional deep Q learning-based strategy and reaches 93.2% level of Dynamic programing benchmark. Moreover, the designed algorithm has a good performance in battery SOC retention with different initial values.