Deep Q-network Algorithm Research Articles

A Deep Reinforcement Learning-Based Speed Optimization System to Reduce Fuel Consumption and Emissions for Smart Cities

Increasing traffic density in cities exacerbates air pollution, threatens human health and worsens the global climate crisis. Urgent solutions for sustainable and eco-friendly urban transportation are needed. Innovative technologies like artificial intelligence, particularly Deep Reinforcement Learning (DRL), play a crucial role in reducing fuel consumption and emissions. This study presents an effective approach using DRL to minimize waiting times at traffic lights, thus reducing fuel consumption and emissions. DRL can evaluate complex traffic scenarios and learn optimal solutions. Unlike other studies focusing solely on optimizing traffic light durations, this research aims to choose the optimal vehicle acceleration based on traffic conditions. This method provides safer, more comfortable travel while lowering emissions and fuel consumption. Simulations with various scenarios prove the Deep Q-Network (DQN) algorithm’s success in adjusting speed according to traffic lights. Although the findings confirmed that the DRL algorithms used were effective in reducing fuel consumption and emissions, the DQN algorithm outperformed other DRL algorithms in reducing fuel consumption and emissions in complex city traffic scenarios, and in reducing waiting times at traffic lights. It provides better contributions to creating a sustainable environment by reducing fuel consumption and emissions.

Microgrid cooperative scheduling optimization based on quality of service constraint and deep Q-network algorithm

Microgrid cooperative scheduling optimization based on quality of service constraint and deep Q-network algorithm

Lagrangian analysis on mechanism of forced convection heat transfer control for cylinder via deep reinforcement learning

Forced convection heat transfer plays a vital role in engineering; however, its control presents significant complexity. In this paper, a closed-loop deep reinforcement learning framework is introduced to optimize cooling tasks in a heat exchanger, where a cylindrical heat source is immersed in a narrow cavity. The online learning deep Q-networks (DQN) algorithm is implemented, and an offline learning conservative Q-learning soft actor-critic (CQL-SAC) algorithm is first proposed to learn based solely on preexisting databases without interacting with the environment. Taking the continuous blowing mode as the baseline, the CQL-SAC control obtains a temperature reduction of 49.1% greater than that under the optimal human control and consumes only 0.53% of the time required by online learning. While the DQN control achieves the best cooling performance, with a temperature reduction exceeding that of the optimal human control by 91.4%. The underlying mechanisms are analyzed. Particle tracking technology and Lagrangian coherence structures (LCS) are employed to identify regions of sufficient heat exchange and precisely map from where a cold particle can be captured to undergo sufficient heat exchange or swiftly escapes with inadequate heat exchange. The mechanism of the enhanced cooling effect under the DQN control is clarified from a particle capturing and escaping perspective. The greater overlap between the cold particle capture region and hotspots correlating with the more saddle points of the LCS within this region indicates more intense heat exchange in areas closer to hotspots, thus resulting in better cooling performance.

Optimization method of task uninstallation in mobile edge computing environment combining improved deep Q-learning and transmission learning

Traditional task uninstallation methods are difficult to cope with the continuous changes in network environment and device status, therefore a more intelligent solution is urgently needed. This article studied the optimization method of task uninstallation in mobile edge computing (MEC) environment, and studied the optimization method of task uninstallation in combination with improved deep Q-learning and transmission learning (TL). Firstly, it provided an overview of the basic concepts and optimization problems of task uninstallation in MEC environments, and then delved into the application of improved deep Q-learning in task uninstallation optimization, namely the application of deep Q-network (DQN) in task uninstallation optimization. Next, this article designed the TL-DQN method by combining DQN and TL. This method utilizes the DQN algorithm to construct a model that can handle complex states and action spaces. Meanwhile, by introducing TL, the model can quickly transfer knowledge and achieve adaptive optimization of tasks in new environments, thereby improving the generalization ability and efficiency of the decision-making process. Experiments have shown that TL-DQN performs better in convergence speed compared to other algorithms. For delay sensitive users, the delay growth of TL-DQN algorithm is relatively slow. When the task volume reached 10, the delay of TL-DQN was 7.97 s, which was 1.99 s lower than reinforcement learning. When the number of unloading tasks increased to 500, the energy consumption of TL-DQN was 554 J, which was 282 J lower than the energy consumption required by the task uninstallation method constructed by reinforcement learning algorithms. The task uninstallation method studied in this article has achieved significant results in reducing latency, improving energy efficiency, and adapting to network dynamic changes, bringing users a more stable and reliable service experience.

Adaptive Control Strategies for Enhanced Integration of Solar Power in Smart Grids Using Reinforcement Learning

Integrating solar power into smart grids is challenging because of the variable nature of solar energy. This study focuses on implementing reinforcement learning (RL) using a Deep Q-Network algorithm to enhance the stability and efficiency of a grid. A custom environment was designed using OpenAI Gym, in which real-time simulation of grid operations was conducted using real-time data on solar power, weather, and other grid metrics. The trained RL agent exhibited high predictability in optimally distributing the load and managing the battery storage, with R-squared = 0.886, mean average error = 1,173,046.55 Wh, and root mean squared error = 2,075,515.10 Wh. The model effectively captured the seasonality and daily variations in solar power generation. Forecasting using the proposed model provides insights into future energy trends and uncertainties. The reward function will be further refined and scaled for more complex energy systems by incorporating additional variables and hybrid approaches. This study highlights the potential of RL-based adaptive control strategies for developing more efficient and resilient integration of renewable energy sources into smart grids.

An Enhanced Deep Q Network Algorithm for Localized Obstacle Avoidance in Indoor Robot Path Planning

Path planning is a key task in mobile robots, and the application of Deep Q Network (DQN) algorithm for mobile robot path planning has become a hotspot and challenge in current research. In order to solve the obstacle avoidance limitations faced by the DQN algorithm in indoor robot path planning, this paper proposes a solution based on an improved DQN algorithm. In view of the low learning efficiency of the DQN algorithm, the Duel DQN structure is introduced to enhance the performance and combined with a Prioritized Experience Replay (PER) mechanism to ensure the stability of the robot during the learning process. In addition, the idea of Munchausen Deep Q Network (M-DQN) is incorporated to guide the robot to learn the optimal policy more effectively. Based on the above improvements, the PER-D2MQN algorithm is proposed in this paper. In order to validate the effectiveness of the proposed algorithm, we conducted multidimensional simulation comparison experiments of the PER-D2MQN algorithm with DQN, Duel DQN, and the existing methodology PMR-DQN in the Gazebo simulation environment and examined the cumulative and average rewards for reaching the goal point, the number of convergent execution steps, and the time consumed by the robot in reaching the goal point. The simulation results show that the PER-D2MQN algorithm obtains the highest reward in both static and complex environments, exhibits the best convergence, and finds the goal point with the lowest average number of steps and the shortest elapsed time.

GSADDQN: Combining GraphSAGE and reinforcement learning for routing optimization in software-defined optical transport network

As a result of rapid development of network communication technology, optical transport network (OTN) traffic has also experienced rapid growth in terms of information volume scale, traffic complexity, and spatiotemporal distribution dynamics. Because the OTN traffic demand has complex spatiotemporal fluctuations, the traditional deep reinforcement learning (DRL) algorithm has been applied to the routing optimization of software-defined OTNs. However, the traditional DRL algorithm has problems such as slow convergence, weak generalization ability, and load imbalance when performing routine tasks. To address these issues, we propose a graph sampling and aggregation (GraphSAGE)-based dueling deep Q-network (GSADDQN) algorithm for software-defined OTN routing optimization. First, we design a DRL-based routing decision model to find the best routing strategy for each optical network’s source–destination traffic demand. Second, considering the sparse connection characteristics of optical network nodes, we use sampling neighbors and a deep aggregation mechanism as the neural network model of the Dueling Deep Q-Network (Dueling DQN) algorithm so that the reinforcement learning agent can consciously aggregate important network information and improve the model’s convergence performance and generalization ability. Finally, we design simulation routing experiments based on Gym and evaluate the algorithm’s load balancing and generalization capabilities for different network topologies. The experimental results show that the GSADDQN algorithm has good convergence performance and load balancing ability in routing optimization of optical transmission networks and can generalize new network structures, maintaining good decision-making ability even during network node failures.

Research on Path Planning of Agricultural UAV Based on Improved Deep Reinforcement Learning

Traditional manual or semi-mechanized pesticide spraying methods often suffer from issues such as redundant coverage and cumbersome operational steps, which fail to meet current pest and disease control requirements. Therefore, there is an urgent need to develop an efficient pest control technology system. This paper builds upon the Deep Q-Network algorithm by integrating the Bi-directional Long Short-Term Memory structure to propose the BL-DQN algorithm. Based on this, a path planning framework for pest and disease control using agricultural drones is designed. This framework comprises four modules: remote sensing image acquisition via the Google Earth platform, task area segmentation using a deep learning U-Net model, rasterized environmental map creation, and coverage path planning. The goal is to enhance the efficiency and safety of pesticide application by drones in complex agricultural environments. Through simulation experiments, the BL-DQN algorithm achieved a 41.68% improvement in coverage compared with the traditional DQN algorithm. The repeat coverage rate for BL-DQN was 5.56%, which is lower than the 9.78% achieved by the DQN algorithm and the 31.29% of the Depth-First Search (DFS) algorithm. Additionally, the number of steps required by BL-DQN was only 80.1% of that of the DFS algorithm. In terms of target point guidance, the BL-DQN algorithm also outperformed both DQN and DFS, demonstrating superior performance.

Improved exploration–exploitation trade-off through adaptive prioritized experience replay

Experience replay is an indispensable part of deep reinforcement learning algorithms that enables the agent to revisit and reuse its past and recent experiences to update the network parameters. In many baseline off-policy algorithms, such as deep Q-networks (DQN), transitions in the replay buffer are typically sampled uniformly. This uniform sampling is not optimal for accelerating the agent’s training towards learning the optimal policy. A more selective and prioritized approach to experience sampling can yield improved learning efficiency and performance. In this regard, this work is devoted to the design of a novel prioritizing strategy to adaptively adjust the sampling probabilities of stored transitions in the replay buffer. Unlike existing sampling methods, the proposed algorithm takes into consideration the exploration–exploitation trade-off (EET) to rank transitions, which is of utmost importance in learning an optimal policy. Specifically, this approach utilizes temporal difference and Bellman errors as criteria for sampling priorities. To maintain balance in EET throughout training, the weights associated with both criteria are dynamically adjusted when constructing the sampling priorities. Additionally, any bias introduced by this sample prioritization is mitigated through assigning importance-sampling weight to each transition in the buffer. The efficacy of this prioritization scheme is assessed through training the DQN algorithm across various OpenAI Gym environments. The results obtained underscore the significance and superiority of our proposed algorithm over state-of-the-art methods. This is evidenced by its accelerated learning pace, greater cumulative reward, and higher success rate.

A Novel FBG Placement Optimization Method for Tunnel Monitoring Based on WOA and Deep Q-Network

By employing the whale optimization algorithm’s (WOA) capability to reduce the probability of being stuck in a locally optimal solution, this study proposed an improved WOA-DQN algorithm based on the Deep Q-Network algorithm (DQN). Firstly, the mathematical model of Fiber Bragg Grating (FBG) sensor placement was established to calculate the reward of DQN. Secondly, the effectiveness and applicability of WOA-DQN were validated through experiments in nine cases. It indicated that the algorithm is far superior to other methods (Noisy DQN, Prioritized DQN, DQN, WOA), especially with the learning rate of 0.001, the initial noise 0.4, the hidden layer 3–512, and the updated frequency of 20. Finally, the FBG sensors were placed at [0°, 27°, 30°, 47°, 51°, 111°, 126°, 219°, 221°, 289°] to detect the accurate deformation of the tunnel with the maximum error 8.66 mm, which is better than the traditional placement. In conclusion, the algorithm provides a theoretical foundation for sensor placement and improves monitoring accuracy. It further shows great promise for deformation monitoring in tunnels.

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

Active power balance control of wind-photovoltaic-storage power system based on transfer learning double deep Q-network approach

IntroductionThis study addresses the challenge of active power (AP) balance control in wind-photovoltaic-storage (WPS) power systems, particularly in regions with a high proportion of renewable energy (RE) units. The goal is to effectively manage the AP balance to reduce the output of thermal power generators, thereby improving the overall efficiency and sustainability of WPS systems.MethodsTo achieve this objective, we propose the transfer learning double deep Q-network (TLDDQN) method for controlling the energy storage device within WPS power systems. The TLDDQN method leverages the benefits of transfer learning to quickly adapt to new environments, thereby enhancing the training speed of the double deep Q-network (DDQN) algorithm. Additionally, we introduce an adaptive entropy mechanism integrated with the DDQN algorithm, which is further improved to enhance the training capability of agents.ResultsThe proposed TLDDQN algorithm was applied to a regional WPS power system for experimental simulation of AP balance control. The results indicate that the TLDDQN algorithm trains agents more rapidly compared to the standard DDQN algorithm. Furthermore, the AP balance control method based on TLDDQN can more accurately manage the storage device, thereby reducing the output of thermal power generators more effectively than the particle swarm optimization-based method.DiscussionOverall, the TLDDQN algorithm proposed in this study can provide some insights and theoretical references for research in related fields, especially those requiring decision making.

Addressing unpredictable movements of dynamic obstacles with deep reinforcement learning to ensure safe navigation for omni-wheeled mobile robot

The safe and efficient navigation of mobile robots in the presence of unknown dynamic obstacles remains a complex and unresolved challenge. This paper presents collision-free path planning for a mobile robot that safely deals with multi-directional obstacles, that is, randomly moving dynamic obstacles, using a Deep Reinforcement Learning (DRL) algorithm named Deep Q-Network (DQN) with inflated robot reward functions. The robot moves in a time-efficient and collision-free route while maintaining a safe distance with both static and unpredictable dynamic obstacles. The modified DQN algorithm takes RGB images of the environment as input for training a Convolution Neural Network (CNN) and provides a safe and short path for navigation. The robot used for training is an omni-wheeled mobile robot exploring outdoors, that is, concourse environment, and indoors, that is, home environment. The Closed-Loop Inverse Kinematics (CLIK) algorithm is employed to control a mobile robot to follow the desired path. The simulation results indicate that the proposed algorithm with inflated robot reward functions demonstrates remarkable performance as compared to recently used Reinforcement Learning (RL) algorithms when dealing with both stationary and randomly moving obstacles in the given environment.

Gain-Based Feedback for Shape Control of Antenna Reflectors Using Deep Q-Network Algorithm

An intelligent shape control method for antenna reflectors is proposed using gain feedback instead of displacement feedback, considering the limitations of high-precision deformation measurement on orbit. Moreover, the finite element method is used to establish an analysis model of a grid reflector with embedded piezoelectric actuators, and the antenna gain formula is derived. To solve the multidimensional control laws from a single gain dimension, an active shape control model training framework is proposed based on the deep Q-network algorithm, including the state space, action space, and reward function. Numerical simulations, with errors stemming from two typical thermal loads as the initial errors, are conducted to illustrate the effect of the proposed control method. The influence of the voltage step size on the shape control precision is analyzed, and further online training is discussed for implementing control. The results indicate that the proposed method reduces the root-mean-square (RMS) error by more than 50% The control effect of the trained control model on other error conditions does not perform as desired. However, online training can effectively solve this problem, achieving faster convergence and a significant reduction of the RMS error.

Intelligent Traffic Control Decision-Making Based on Type-2 Fuzzy and Reinforcement Learning

Intelligent traffic control decision-making has long been a crucial issue for improving the efficiency and safety of the intelligent transportation system. The deficiencies of the Type-1 fuzzy traffic control system in dealing with uncertainty have led to a reduced ability to address traffic congestion. Therefore, this paper proposes a Type-2 fuzzy controller for a single intersection. Based on real-time traffic flow information, the green timing of each phase is dynamically determined to achieve the minimum average vehicle delay. Additionally, in traffic light control, various factors (such as vehicle delay and queue length) need to be balanced to define the appropriate reward. Improper reward design may fail to guide the Deep Q-Network algorithm to learn the optimal strategy. To address these issues, this paper proposes a deep reinforcement learning traffic control strategy combined with Type-2 fuzzy control. The output action of the Type-2 fuzzy control system replaces the action of selecting the maximum output Q-value of the target network in the DQN algorithm, reducing the error caused by the use of the max operation of the target network. This approach improves the online learning rate of the agent and increases the reward value of the signal control action. The simulation results using the Simulation of Urban MObility platform show that the traffic signal optimization control proposed in this paper has achieved significant improvement in traffic flow optimization and congestion alleviation, which can effectively improve the traffic efficiency in front of the signal light and improve the overall operation level of traffic flow.

Deep reinforcement learning using deep-Q-network for Global Maximum Power Point tracking: Design and experiments in real photovoltaic systems

This paper presents a methodology for integrating Deep Reinforcement Learning (DRL) using a Deep-Q-Network (DQN) agent into real-time experiments to achieve the Global Maximum Power Point (GMPP) of Photovoltaic (PV) systems under various environmental conditions. Conventional methods, such as the Perturb and Observe (P&O) algorithm, often become stuck at the Local Maximum Power Point (LMPP) and fail to reach the GMPP under Partial Shading Conditions (PSC). The main contribution of this work is the experimental validation of the DQN agent's implementation in a synchronous DC-DC Buck converter (step-down converter) un-der both uniform and PSC conditions. Additionally, we establish a testing pipeline for DRL models. The DQN agent's performance is evaluated alongside the P&O algorithm. Results consistently indicate the DQN agent's superiority over the P&O algorithm in all simulated scenarios. Although this trend does not entirely replicate in real-world test setups, significant results are observed. Specifically, in PSC sce-narios where the P&O algorithm becomes trapped at an LMPP, the DQN algorithm extracts up to 63.5 % more power than the P&O algorithm. An open repository is available, containing PCB schematic designs and layouts, along with the code used for model training and deployment.

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

SPA-GPT: General Pulse Tailor for Simple Power Analysis Based on Reinforcement Learning

In side-channel analysis of public-key algorithms, we usually classify operations based on the differences in power traces produced by different basic operations (such as modular square or modular multiplication) to recover secret information like private keys. The more accurate the segmentation of power traces, the higher the efficiency of their classification. There exist two commonly used methods: one is equidistant segmentation, which requires a fixed number of basic operations and similar trace lengths for each type of operation, leading to limited application scenarios; the other is peak-based segmentation, which relies on personal experience to configure parameters, resulting in insufficient flexibility and poor universality. In this paper, we propose an automated trace segmentation method based on reinforcement learning applicable to a wide range of common implementation of public-key algorithms. The introduction of reinforcement learning, which doesn’t need labels, into trace processing for side-channel analysis marks its debut in this field. Our method has good universality on the traces with varying segment lengths and differing peak heights. By using prioritized experience replay optimized Deep Q-Network algorithm, we reduce the required number of parameters to one, which is the key length. We also employ various techniques to improve the segmentation effectiveness, such as clustering algorithm and enveloped-based feature enhancement. We validate the effectiveness of the new method in nine scenarios involving hardware and software implementations of different public-key algorithms executed on diverse platforms such as microcontrollers, SAKURA-G, and smart cards. Specifically, one of these implementations is protected by time randomization countermeasures. Experimental results show that a basic version of our method can correctly segment most traces. The enhanced version is capable of reconstructing the sequence of operations during trace segmentation, achieving an accuracy rate of 100% for the majority of the traces. For traces that cannot be entirely restored, we utilize reward values of reinforcement learning to correct errors and achieve fully recovery. We also conducted comparative experiments with supervised seq2seq methods, revealing our approach’s 42% higher accuracy in operation recovery and 96% faster time efficiency. In addition, we applied our method to the post-quantum cryptography Kyber, and successfully recovered an intermediate value crucial for deriving the secret key. Besides, power traces collected from these devices have been uploaded as open databases, which are available for researchers engaged in public-key algorithms to conduct related experiments or verify our method.

Towards various occupants with different thermal comfort requirements: A deep reinforcement learning approach combined with a dynamic PMV model for HVAC control in buildings

Reinforcement learning (RL) has great potential in achieving energy-efficient, comfortable and intelligent control of heating, ventilation and air conditioning (HVAC) systems. Although research on RL-based HVAC control has attracted increasing interest, current studies generally use simple building simulation as the environment to train agents, and the definition of thermal comfort is limited to a wide temperature range, which cannot meet the different thermal comfort requirements of various occupants. This study proposes a deep reinforcement learning (DRL) control framework based on the Dueling Deep Q-network (DQN) algorithm, combined with a self-designed environmental model and reward function, for HVAC control meeting different thermal comfort requirements. Specifically, based on the theory of building thermal dynamics, a nonlinear equation modified by experimental data is used for the environmental model that reflects the actual thermal change of building. Different thermal comfort requirements are considered and analysed through a dynamic predicted mean vote (PMV) model that focuses on the metabolic rate and clothing level of occupants. By systematically exploring different heating modes for occupants and control time intervals, the proposed framework demonstrates that heating energy consumption can be reduced by 4.8%-39.58% under various conditions compared to rule-based control. In addition, the study found that the HVAC control based on DRL has greater potential in saving energy when the heating demand of building is higher. Our study is helpful for researchers to make HVAC control more energy-efficient and user-friendly with the help of artificial intelligence.