Deep Q-network Research Articles

Advancing multi-port container stowage efficiency: A novel DQN-LNS algorithmic solution

Maritime shipping handles approximately 90% of global cargo transport, with 60% utilizing steel containers, playing a pivotal role in the global supply chain. As the global economy recovers in the post-pandemic era, the demand for and efficiency requirements of container transportation are continually increasing. This study introduces a new method to enhance the efficiency and safety of inland river container transport, which has seen significant growth in feeder services. We propose a novel mathematical model specifically tailored for river-sea direct container ships. Unlike traditional staged stowage planning approaches, our model optimizes multi-port stowage planning within a single stage. Additionally, we introduce the Deep Q-Network-based Large Neighborhood Search (DQN-LNS) algorithm, a novel advancement rooted in the Adaptive Large Neighborhood Search (ALNS) framework. This algorithm harnesses the power of Deep Q-Networks (DQN) to dynamically select the optimal ruin and recreate strategies in each iteration, enhancing the stowage planning’s efficiency and accuracy. Extensive numerical experiments validate our contributions, showcasing the algorithm’s capacity to markedly improve operational efficiency and environmental performance in inland river logistics.

Advanced Primary Frequency Regulation Optimization in Wind Storage Systems with DC Integration Using Double Deep Q-Networks

With the gradual increase in wind power installation capacity, the proportion of traditional synchronous generators driven by fossil fuel is gradually declining. Due to the fact that wind turbines are connected to the grid through power electronic converters, which decouple rotor speeds from the system frequency and reduce system inertia levels, inadequate inertia levels can pose a threat to frequency stability when disturbances occur. To address this issue, this paper proposes a frequency regulation optimization strategy for the direct current (DC) transmission of a wind storage system. This strategy incorporates virtual inertia control and virtual droop control to adjust wind power output based on frequency deviation and rate of change. Fuzzy logic control is employed for energy storage, adaptively adjusting active power based on frequency deviation and the rate of change. Additionally, under the context of multi-DC transmission in renewable energy systems, an optimization strategy for proportion and integration (PI) parameters of the frequency limit controller (FLC) is proposed. Considering frequency deviation and DC regulation power simultaneously, the double deep Q-network (DDQN) algorithm is adopted in the simulation model to attain the optimal parameters of FLC. Simulation results conducted using MATLAB/Simulink 2022a indicate that this strategy increases the lowest frequency by 0.28 Hz and decreases the response time by 1.04 s compared with the non-optimized strategy.

Research on Decomposition and Offloading Strategies for Complex Divisible Computing Tasks in Computing Power Networks

With the continuous emergence of intelligent network applications and complex tasks for mobile terminals, the traditional single computing model often fails to meet the greater requirements of computing and network technology, thus promoting the formation of a new computing power network architecture, of ‘cloud, edge and end’ three-level heterogeneous computing. For complex divisible computing tasks in the network, task decomposition and offloading help to realize a distributed execution of tasks, thus reducing the overall running time and improving the utilization of fragmented resources in the network. However, in the process of task decomposition and offloading, there are problems, such as there only being a single method of task decomposition; that too large or too small decomposition granularity will lead to an increase in transmission delay; and the pursuit of low-delay and low-energy offloading requirements. Based on this, a complex divisible computing task decomposition and offloading scheme is proposed. Firstly, the computational task is decomposed into multiple task elements based on code partitioning, and then a density-peak-clustering algorithm with an improved adaptive truncation distance and clustering center (ATDCC-DPC) is proposed to cluster the task elements into subtasks based on the task elements themselves and the dependencies between the task elements. Secondly, taking the subtasks as the offloading objects, the improved Double Deep Q-Network subtask offloading algorithm (ISO-DDQN) is proposed to find the optimal offloading scheme that minimizes the delay and energy consumption. Finally, the proposed algorithms are verified by simulation experiments, and the scheme in this paper can effectively reduce the task delay and energy consumption and improve the service experience.

Rolling bearing fault diagnosis based on DRS frequency spectrum image and improved DQN

Deep learning may encounter challenges in fault diagnosis, such as exploration capability, long-term planning, and handling non-deterministic factors. The paper introduces a novel fault diagnosis method for rolling bearings combining deep Q-network (DQN) with discrete random separation (DRS) frequency spectrum images. DRS can be used to separate the deterministic components and random components of the fault signal and extract the fault feature very effectively. The improved DQN incorporates the atrous spatial pyramid pooling module for multiscale contextual fault feature extraction, enhancing diagnosis accuracy. Various deep networks, including convolutional neural network, ResNet18, traditional DQN, and the improved DQN, are employed with different frequency spectrum images (power spectral density, cepstrum, and DRS) for diagnosing bearing faults. Simulation results demonstrate that combining DRS frequency spectrum images with the improved DQN enhances fault diagnosis accuracy across diverse conditions. Generalization tests reveal strong capability of the proposed method in handling conditions different from the training data. Validation tests on difficult-to-diagnose fault data from the CWRU-bearing dataset demonstrate commendable performance of the improved DQN, even on difficult datasets. Finally, validation tests using the XJTU–SY bearing dataset reaffirm the excellent performance and robust adaptability of the improved DQN in conjunction with DRS frequency spectrum images.

A Comparative Study of Traffic Signal Control Based on Reinforcement Learning Algorithms

In recent years, the increasing production and sales of automobiles have led to a notable rise in congestion on urban road traffic systems, particularly at ramps and intersections with traffic signals. Intelligent traffic signal control represents an effective means of addressing traffic congestion. Reinforcement learning methods have demonstrated considerable potential for addressing complex traffic signal control problems with multidimensional states and actions. In this research, the team propose Q-learning and Deep Q-Network (DQN) based signal control frameworks that use variable phase sequences and cycle times to adjust the order and the duration of signal phases to obtain a stable traffic signal control strategy. Experiments are simulated using the traffic simulator Simulation of Urban Mobility (SUMO) to test the average speed and the lane occupancy rate of vehicles entering the ramp to evaluate its safety performance and test the vehicle’s traveling time to assess its stability. The simulation results show that both reinforcement learning algorithms are able to control cars in dynamic traffic environments with higher average speed and lower lane occupancy rate than the no-control method and that the DQN control model improves the average speed by about 10% and reduces the lane occupancy rate by about 30% compared to the Q-learning control model, providing a higher safety performance.

Ftdho Zfnet: Block chain based Fractional Tasmanian Devil Harris optimization enabled deep learning using attack detection and mitigation

Block chain technology is regarded for enhancing the characteristics of security because of decentralized design, safe distributed storage, and privacy. However, in recent times the present situation of block chain technology has experienced some crisis that may delay the quick acceptance and utilization in real-time applications. To conquer this subdues, a blockchain based system for attack detection and mitigation with Deep Learning (DL) named Fractional Tasmanian Devil Harris Optimization_Zeiler and Fergus network (FTDHO_ZFNet) is introduced. In this investigation, the entities utilized are owner, block chain, server, trusted authority and user. Here, authentication phase is done by means of Ethereum block chain by Key Exchange module and privacy preserved data sharing and communication is also done. Then, recorded log file creation is executed by the below mentioned stages. At first, a log file is generated with the basis of communication to record the events. After wards, the features are extracted by BoT-IoT database. Then, feature fusion is done by overlap coefficient utilizing Deep Q-Network (DQN). Moreover, data augmentation (DA) is doneusing bootstrapping method. At last, attack detection is observed by ZFNet tuned by FTDHO. Here, FTDHO is unified by Fractional Tasmanian Devil Optimization (FTDO) and Harris Hawks Optimization (HHO). Additionally, FTDO is integrated by Fractional Calculus (FC) concept and Tasmanian devil optimization (TDO). Furthermore, attack mitigation is performed. The performance measures applied for FTDHO_ZFNet are accuracy, and True Negative rate (TNR), observed supreme values with 92.9%, 93.8% and 92.9%.

DinoDroid: Testing Android Apps Using Deep Q-Networks

The large demand of mobile devices creates significant concerns about the quality of mobile applications (apps). Developers need to guarantee the quality of mobile apps before it is released to the market. There have been many approaches using different strategies to test the GUI of mobile apps. However, they still need improvement due to their limited effectiveness. In this article, we propose DinoDroid, an approach based on deep Q-networks to automate testing of Android apps. DinoDroid learns a behavior model from a set of existing apps and the learned model can be used to explore and generate tests for new apps. DinoDroid is able to capture the fine-grained details of GUI events (e.g., the content of GUI widgets) and use them as features that are fed into deep neural network, which acts as the agent to guide app exploration. DinoDroid automatically adapts the learned model during the exploration without the need of any modeling strategies or pre-defined rules. We conduct experiments on 64 open-source Android apps. The results showed that DinoDroid outperforms existing Android testing tools in terms of code coverage and bug detection.

Performance evaluation and improvement of deep Q network for lunar landing task

Reinforcement learning is now being applied more and more in a variety of scenarios, the majority of which are based on the deep Q network (DQN) technology. However, the algorithm is heavily influenced by multiple factors. In this paper, we take the lunar lander as a case to study how various hyper-parameters affect the performance of the DQN algorithm, based on which we tune to get a model with better performance. At present, it is known that the DQN model has an average reward of 280+ on 100 test episodes, and the reward value of the model in this article can reach 290+. Meanwhile, its robustness is tested and verified by introducing additional uncertainty tests into the original problem. In addition, to speed up the training process, imitation learning is incorporated in our model, using heuristic function model guidance method to obtain demonstration data, which accelerates training speed and improves performance. Simulation results have proven the effectiveness of this method.

Hierarchical Multiview Top-k Pooling With Deep-Q-Networks

Hierarchical Multiview Top-k Pooling With Deep-Q-Networks

Reinforcement Contract Design for Vehicular-Edge Computing Scheduling and Energy Trading via Deep Q-Network With Hybrid Action Space

Reinforcement Contract Design for Vehicular-Edge Computing Scheduling and Energy Trading via Deep Q-Network With Hybrid Action Space

LSTM-QDCNN: long short-term memory and quantum dilated convolutional neural network enabled occlusion percentage prediction

ABSTRACT Occlusion detection is vital in longer video sequences of real-time scenarios, as deformable objects generally occlude themselves or be occluded by exterior objects eventually. Occlusion can affect accuracy of the entire model significantly, because optical flow may reveal random attributes in occluded as well as neighbourhood areas. Occlusion is a challenging issue in real-world visual object tracking. Several techniques learn imprecise appearance of target while it becomes occluded by another object in the scene. Occlusion is a complicated issue that causes target loss in tracking procedure. Here, long short-term memory fused quantum dilated convolutional neural network (LSTM-QDCNN) is presented for occlusion percentage prediction. Initially, video frames are extracted from considered input video. The extracted frame is pre-processed by adaptive fuzzy filtering, and object localisation is accomplished by employing sparse fuzzy c-means with local optimal oriented pattern feature. Thereafter, occlusion category detection is conducted by deep Q-network. Lastly, occlusion percentage prediction is done by utilising LSTM-QDCNN. Additionally, LSTM-QDCNN is designed by merging deep LSTM (DLSTM) with quantum dilated convolutional neural network (QDCNN). Furthermore LSTM-QDCNN achieved maximum multiple-object tracking precision of 0.892 and minimum tracking distance of 3.714. In addition, LSTM-QDCNN achieved tracking number about 18.

AN ENSEMBLE ADAPTIVE REINFORCEMENT LEARNING BASED EFFICIENT LOAD BALANCING IN MOBILE AD HOC NETWORKS

his research work introduces an Ensemble Adaptive Reinforcement Learning (EARL) approach for efficient load balancing in Mobile Ad Hoc Networks (MANETs). Traditional methods often fail to adapt to the dynamic nature of MANETs, leading to congestion and inefficiency. EARL leverages multiple reinforcement learning agents, trained with Q-learning and Deep Q-Networks (DQN), to optimize routing decisions based on real-time network conditions. The ensemble mechanism combines the strengths of individual agents, enhancing adaptability and performance. Simulation results demonstrate that EARL significantly outperforms traditional methods like AODV and DSR, achieving higher packet delivery ratios, lower end-to-end delays, increased throughput, better energy efficiency, and reduced packet loss, thereby proving its effectiveness in dynamic network environments.

Design and application of deep reinforcement learning algorithms based on unbiased exploration strategies for value functions

Deep Q-networks, as a representation of several classical techniques, have emerged as one of the primary branches in the field of value function-based reinforcement learning. The paper addresses two issues that come up in the realm of reinforcement learning for value function solving: estimating bias and maximizing projected action value function evaluation. By treating the estimation of the highest expected action value as a random selection estimation problem, the suggested approach addresses the estimation bias issue from the standpoint of random selection. A random choice estimate procedure forms the basis of the technique. Firstly, a proposed random choice estimator is presented and its theoretical fairness is established. Second, the estimator is applied to create a reinforcement learning method in a different application. Two techniques, namely stochastic two-depth Q-networks and double-Q learning, are suggested based on the random choice estimation technique. The main parameters of the suggested algorithms are then investigated, and parameter formulas for both predictable and unpredictable scenarios are created. Lastly, a random choice estimation perspective suggests a stochastic two-depth Q-network. The new approach may effectively remove bias in value function estimate, enhance learning performance, and stabilise the learning process, according to simulation findings on Grid World and Atari games.

P‐235: Late‐News Poster: Optimization of Display Production Scheduling with Reinforcement Learning

The display module process is characterized by having to deal with a wide range of customers and product groups with a short turnaround time (TAT). In this study, a new concept of simulation tool reflecting the characteristic of the module process was developed. This tool reflects the TAT of the equipment using the cumulative distribution function of the process time, and the frequency and time of down of the equipment are modeled using statistical techniques. This tool has confirmed a high accuracy of 92% in comparison with actual production. The tool provides rule‐based operation guidance to manufacturing engineers. However, rule‐based operation is not easy to adequately respond to all manufacturing situations in the actual field. Thus, reinforcement learning (RL) was applied to the module scheduler system for optimal operation. We applied RL algorithm such as deep Q‐Networks (DQN), double DQN, dueling network. Through a test comparing reinforcement learning‐based operations and rule‐based operations under the same conditions, it was found that reinforcement learning‐based operations are better in terms of productivity and efficiency than rule‐based operations. This result shows that the paradigm of module scheduling has shifted from traditional rule‐based operations to reinforcement learning‐based AI operations.

A Fault Diagnosis Method Based on an Improved Deep Q-Network for the Interturn Short Circuits of a Permanent Magnet Synchronous Motor

A Fault Diagnosis Method Based on an Improved Deep Q-Network for the Interturn Short Circuits of a Permanent Magnet Synchronous Motor

Automatic depth matching method of well log based on deep reinforcement learning

In the traditional well log depth matching tasks, manual adjustments are required, which means significantly labor-intensive for multiple wells, leading to low work efficiency. This paper introduces a multi-agent deep reinforcement learning (MARL) method to automate the depth matching of multi-well logs. This method defines multiple top-down dual sliding windows based on the convolutional neural network (CNN) to extract and capture similar feature sequences on well logs, and it establishes an interaction mechanism between agents and the environment to control the depth matching process. Specifically, the agent selects an action to translate or scale the feature sequence based on the double deep Q-network (DDQN). Through the feedback of the reward signal, it evaluates the effectiveness of each action, aiming to obtain the optimal strategy and improve the accuracy of the matching task. Our experiments show that MARL can automatically perform depth matches for well-logs in multiple wells, and reduce manual intervention. In the application to the oil field, a comparative analysis of dynamic time warping (DTW), deep Q-learning network (DQN), and DDQN methods revealed that the DDQN algorithm, with its dual-network evaluation mechanism, significantly improves performance by identifying and aligning more details in the well log feature sequences, thus achieving higher depth matching accuracy.

Erosion-safe operation using double deep Q-learning

Leading edge erosion on wind turbine blades can reduce aerodynamic efficiency and cause increased maintenance costs, potentially impacting the overall economic viability. Erosion-safe operation is the concept of reducing the blade tip speed during periods of heavy rain, thereby significantly reducing the erosion development and progression. This study explores the application of reinforcement learning, namely using a double deep Q-network, to implement erosion-safe operation. The proposed methodology involves learning a policy for tip speed control that maximizes revenue over a specific period of time. We demonstrate the concept based on 5 years of simulation of the DTU 10MW reference turbine and mesoscale weather simulation from Horns Rev. The trained model was found to increase the cumulative revenue by 1.6 % compared to not using erosion-safe operation. The model was able to effectively adapt to varying weather conditions and stochastic damage progression. Based on 10,000 random simulations, the trained model outperforms two baseline models in more than 98 % of the simulations.

Intelligent emergency traffic signal control system with pedestrian access

With the integration of artificial intelligence and traffic systems, intelligent traffic systems are utilizing enhanced perception coverage and computational capabilities to provide data-intensive solutions, achieving higher levels of performance than traditional systems. This paper combines the D3QN algorithm from deep reinforcement learning with practical issues and proposes an intelligent emergency traffic signal control system based on Deep Reinforcement Learning (DRL). The system takes into account pedestrian movement and utilizes real-time traffic data and environmental information to model traffic flow and road conditions within a novel state space. It employs the Dueling Double Deep Q-Network (D3QN) to optimize signal control strategies. The system dynamically adjusts signal timings to enhance operational efficiency at intersections. By using the Weibull distribution to simulate realistic traffic congestion and actual traffic data from Shanyin Road in Hangzhou for validation, the results demonstrate that this method converges faster and is more stable compared to other methods, significantly reducing traffic congestion. Furthermore, by incorporating pedestrian movement, this method reduces pedestrian waiting times by 44.736% during peak periods and 22.95% during off-peak periods, while maintaining comparable vehicle queue lengths, delay times, and carbon dioxide emissions. This approach shows the potential improvement of smart urban mobility and resolving intersection congestion challenges.

A Service-Caching Strategy Assisted by Double DQN in LEO Satellite Networks.

Satellite fog computing (SFC) achieves computation, caching, and other functionalities through collaboration among fog nodes. Satellites can provide real-time and reliable satellite-to-ground fusion services by pre-caching content that users may request in advance. However, due to the high-speed mobility of satellites, the complexity of user-access conditions poses a new challenge in selecting optimal caching locations and improving caching efficiency. Motivated by this, in this paper, we propose a real-time caching scheme based on a Double Deep Q-Network (Double DQN). The overarching objective is to enhance the cache hit rate. The simulation results demonstrate that the algorithm proposed in this paper improves the data hit rate by approximately 13% compared to methods without reinforcement learning assistance.

Gate Assignment Algorithm for Airport Peak Time Based on Reinforcement Learning

In existing airport gate allocation studies, little consideration has been given to situations where gate resources are limited during peak periods. Under such circumstances, some flights may not be able to make regular stops. In this paper, the airport gate assignment problem under peak time is investigated. We propose a gate pre-assignment model to maximize the gate matching degree and the near gate passenger allocation rate. Besides, to minimize the pre-assignment gate change rate, we propose a dynamic reassignment model based on the pre-assignment model. By considering the non-deterministic polynomial hard (NP-hard) property of this problem, a gate assignment algorithm based on proximal policy optimization (GABPPO) is proposed. The simulation results show that the algorithm can effectively solve the gate shortage problem during the airport peak period. Compared with the adaptive parallel genetic, deep Q-network, and policy gradient algorithms, the target value of solutions obtained by the proposed algorithm in the near gate passenger allocation rate is increased by 5.7%, 3.6%, and 7.9%, respectively, and the target value in the gate matching degree is increased by 10.6%, 4.9%, and 11.5% respectively.