Double Deep Q-Network Research Articles

Cooperative multi-agent reinforcement learning for multi-area integrated scheduling in wafer fabs

The existing scheduling methods of wafer fabs focus on single area, achieving local optimisation while failing to realise global optimisation due to neglecting the coordination of multi-area. Therefore, it is necessary to consider the complex opposing relationships between multi-area caused by constraints such as batch processing, re-entrance, and multiple residency times within and between areas to conduct integrated scheduling and shorten the production cycle time. For this issue, this paper proposes a cooperative multi-agent reinforcement learning for multi-area integrated scheduling. Aiming at the dynamic batching and scheduling considering the dynamic arrival lots in multi-area, a multi-agent reinforcement learning algorithm is presented to learn the optimal dynamic batching and scheduling policy firstly. Subsequently, a cooperative multi-agent framework is raised to achieve the global optimisation and coordination of multi-area. Furthermore, an adaptive exploration strategy is constructed to enhance the global exploration capability of the complex solution space caused by residency time constraints and re-entrant property. Moreover, a policy share enhanced Double DQN is employed to improve the generalisation and adaptability of the multi-agent. Finally, the experiments demonstrate that the proposed integrated scheduling method has better comprehensive performance compared to the previous area-separated scheduling methods.

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

Active power balance control of wind-photovoltaic-storage power system based on transfer learning double deep Q-network approach

IntroductionThis study addresses the challenge of active power (AP) balance control in wind-photovoltaic-storage (WPS) power systems, particularly in regions with a high proportion of renewable energy (RE) units. The goal is to effectively manage the AP balance to reduce the output of thermal power generators, thereby improving the overall efficiency and sustainability of WPS systems.MethodsTo achieve this objective, we propose the transfer learning double deep Q-network (TLDDQN) method for controlling the energy storage device within WPS power systems. The TLDDQN method leverages the benefits of transfer learning to quickly adapt to new environments, thereby enhancing the training speed of the double deep Q-network (DDQN) algorithm. Additionally, we introduce an adaptive entropy mechanism integrated with the DDQN algorithm, which is further improved to enhance the training capability of agents.ResultsThe proposed TLDDQN algorithm was applied to a regional WPS power system for experimental simulation of AP balance control. The results indicate that the TLDDQN algorithm trains agents more rapidly compared to the standard DDQN algorithm. Furthermore, the AP balance control method based on TLDDQN can more accurately manage the storage device, thereby reducing the output of thermal power generators more effectively than the particle swarm optimization-based method.DiscussionOverall, the TLDDQN algorithm proposed in this study can provide some insights and theoretical references for research in related fields, especially those requiring decision making.

Optimization of the age of correlated information in V2X networks with edge computing

Vehicle-to-Everything (V2X) communication has the potential to revolutionize the travel experience in intelligent transportation, which has received the great attention recently. However, ensuring the freshness of information from multiple sources is critical for the real-time and reliable communication in vehicular networks, especially for timely updates of service centers. To address this issue, we use a promising metric called Age of Correlated Information (AoCI), which can characterize the freshness of multi-source information. Therefore, we propose a novel model that can dynamically regulate the channel activation matching and edge computing collaboration strategy to minimize AoCI in V2X vehicular networks. Firstly, we describe the system model of a V2X network with edge computing, including definitions and assumptions for freshness of information, edge co-computing, etc. Secondly, we formulate the joint optimization problem as a source-related age minimization (SRAM) problem, which is NP-complete. A heuristic algorithm is proposed to solve it under fast-fading channel. Finally, since traditional graph models cannot capture the changing correlation between nodes in dynamic networks, we use graph convolutional networks(GCN) to extract the features of multi-source correlation. The features extracted by GCN include relevant attributes of the sources and its communication links. The features are provided as input to a double deep Q network (DDQN) for training the model that can adapt to a dynamic network environment. Extensive simulation experiments in different network scenarios validate that our proposed method can effectively and efficiently reduce the average AoCI and the computational resources.

МЕТОД ОБУЧЕНИЯ ИНТЕЛЛЕКТУАЛЬНОГО АГЕНТА С ПОМОЩЬЮ СЕТЕЙ DOUBLE DQN, ПУТЕВЫХ ТОЧЕК И ФУНКЦИИ ВОЗНАГРАЖДЕНИЯ

The problems of increasing the control efficiency of autonomous vehicles are considered. The problem of reducing the required computational resources for the intelligent vehicle control module is highlighted. A neural network training algorithm for Double DQN architecture with modified reward functions is proposed. The basis of the proposed solution is the use of lane segmentation, reward function and the use of additional waypoints in training. A software model has been developed and simulation of the learning process has been performed. The results obtained from a comparative analysis with known solutions show a stable increase in episode duration, and effective training in a realistic urban simulation. The study points to the possibility of reducing the need for high computing power, which will enable the use of central processing units (CPUs) for basic functions of unmanned vehicles instead of graphics processing units (GPUs).

Identification of Anomalies in Lung and Colon Cancer Using Computer Vision-Based Swin Transformer with Ensemble Model on Histopathological Images.

Lung and colon cancer (LCC) is a dominant life-threatening disease that needs timely attention and precise diagnosis for efficient treatment. The conventional diagnostic techniques for LCC regularly encounter constraints in terms of efficiency and accuracy, thus causing challenges in primary recognition and treatment. Early diagnosis of the disease can immensely reduce the probability of death. In medical practice, the histopathological study of the tissue samples generally uses a classical model. Still, the automated devices that exploit artificial intelligence (AI) techniques produce efficient results in disease diagnosis. In histopathology, both machine learning (ML) and deep learning (DL) approaches can be deployed owing to their latent ability in analyzing and predicting physically accurate molecular phenotypes and microsatellite uncertainty. In this background, this study presents a novel technique called Lung and Colon Cancer using a Swin Transformer with an Ensemble Model on the Histopathological Images (LCCST-EMHI). The proposed LCCST-EMHI method focuses on designing a DL model for the diagnosis and classification of the LCC using histopathological images (HI). In order to achieve this, the LCCST-EMHI model utilizes the bilateral filtering (BF) technique to get rid of the noise. Further, the Swin Transformer (ST) model is also employed for the purpose of feature extraction. For the LCC detection and classification process, an ensemble deep learning classifier is used with three techniques: bidirectional long short-term memory with multi-head attention (BiLSTM-MHA), Double Deep Q-Network (DDQN), and sparse stacked autoencoder (SSAE). Eventually, the hyperparameter selection of the three DL models can be implemented utilizing the walrus optimization algorithm (WaOA) method. In order to illustrate the promising performance of the LCCST-EMHI approach, an extensive range of simulation analyses was conducted on a benchmark dataset. The experimentation results demonstrated the promising performance of the LCCST-EMHI approach over other recent methods.

A two-stage subsynchronous oscillation assessment method for DFIG-based wind farm grid-connected system

In the power system, the wind farm based on Doubly-Fed Induction Generator (DFIG) may lead to Subsynchronous Oscillation (SSO), which poses a challenge to the stability of the power grid. In order to accurately evaluate SSO, this paper proposes a new evaluation method. It is divided into two main stages: firstly, the interference level of Phasor Measurement Unit (PMU) data is identified by using the classification model based on Upper Confidence Bound (UCB) and Double Deep Q Network (DDQN). Then, an SSO parameter estimation model based on Local Feature Fusion Transformer (LFF-Transformer) network is designed for data with different interference levels. Experimental results show that the errors of eRMSE-f and EMAPE-F are 0.001 and 0.003 respectively, and the errors of eRMSE-δ and EMAPE-δ are 0.009 and 0.015 respectively. In terms of training and testing time, this method is 90 s and 18 s respectively, which is significantly better than Multi-SVR and Multi-CNN. After application, the frequency deviation decreased from 0.05 to 0.02 Hz, the voltage deviation decreased from 3.5 to 1.5%, the power fluctuation decreased from 10 to 5 MW, the SSO frequency decreased from 1.5 Hz to less than 0.5 Hz, and the SSO damping ratio increased from 0.08 to 0.15. This shows that the proposed method effectively increases the stability of the power grid.

DT-DDQNR: A digital twin assisted direct-to-cell satellite network intelligent routing algorithm

This paper presents a novel routing algorithm designed for direct-to-cell satellite constellations in satellite networks. The algorithm aims to ensure stable Quality of Service (QoS) for users despite the highly dynamic nature of satellite links and network conditions. It comprises two key components: a terrestrial-satellite link selection model and an inter-satellite multi-hop routing decision model. To enhance decision-making, the algorithm incorporates a mechanism that interacts with large-scale digital twin constellations, offering agents with high-precision state data inputs and feedback for training and application support. The terrestrial-satellite link selection process is modeled as a constraint satisfaction problem, maximizing the Link Quality Indicator (LQI). Meanwhile, inter-satellite data forwarding decisions are determined utilizing an enhanced dynamic reward mechanism and a re-designed neural network in the DDQN model. Simulation results demonstrate the superiority of the Digital Twin-Assisted Double Deep Q-Network Routing (DT-DDQNR) algorithm over traditional routing algorithms in terms of overall network QoS performance, offering a promising solution for optimizing data transmission in integrated terrestrial-satellite networks.

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

Analyzing deep reinforcement learning model decisions with Shapley additive explanations for counter drone operations

As the use of drones continues to increase, their capabilities pose a threat to airspace safety when they are misused. Deploying AI models for intercepting these unwanted drones becomes crucial. However, these AI models, such as deep learning models, often operate as “black boxes”, making it hard to trust their decision-making system. This also affects end-users’ confidence in these AI systems. In this paper, the explainability of deep reinforcement learning is investigated and a deep reinforcement learning (DRL) method, double deep Q-network with dueling architecture and prioritized experience replay is applied to train the AI models. To make the AI model decisions more transparent and to understand the reasoning behind the AI decisions for counter-drone systems, Shapley Additive Explanations (SHAP) method is implemented. After training the DRL agent, experience replay is visualized, and the absolute SHAP values are calculated to explain the key factors that influence the deep reinforcement learning agent’s choices. The integration of DRL with explainable AI methods such as SHAP demonstrates significant potential for the advancement of robust and efficient counter-drone systems.

Achieving Robust Learning Outcomes in Autonomous Driving with DynamicNoise Integration in Deep Reinforcement Learning

The advancement of autonomous driving technology is becoming increasingly vital in the modern technological landscape, where it promises notable enhancements in safety, efficiency, traffic management, and energy use. Despite these benefits, conventional deep reinforcement learning algorithms often struggle to effectively navigate complex driving environments. To tackle this challenge, we propose a novel network called DynamicNoise, which was designed to significantly boost the algorithmic performance by introducing noise into the deep Q-network (DQN) and double deep Q-network (DDQN). Drawing inspiration from the NoiseNet architecture, DynamicNoise uses stochastic perturbations to improve the exploration capabilities of these models, thus leading to more robust learning outcomes. Our experiments demonstrated a 57.25% improvement in the navigation effectiveness within a 2D experimental setting. Moreover, by integrating noise into the action selection and fully connected layers of the soft actor–critic (SAC) model in the more complex 3D CARLA simulation environment, our approach achieved an 18.9% performance gain, which substantially surpassed the traditional methods. These results confirmed that the DynamicNoise network significantly enhanced the performance of autonomous driving systems across various simulated environments, regardless of their dimensionality and complexity, by improving their exploration capabilities rather than just their efficiency.

A cognitive communication jamming strategy based on Transformer and Deep Reinforcement Learning

The advent of sophisticated communication technologies, such as cognitive radio and anti-jamming techniques, has significantly elevated the challenge of disrupting enemy communications. Nevertheless, the inherent openness of wireless communications remains a vulnerability that can be exploited to interfere with them. Some contemporary Reinforcement Learning (RL)-based jamming strategies examine methods for rapidly identifying the optimal jamming strategy for a specific modulated signal. However, such algorithms lack the flexibility and responsiveness required to effectively counter the enemy’s evolving communication strategies. To address this issue, we propose a Transformer and Deep Reinforcement Learning (DRL)-based jamming strategy that can be trained to identify jamming methods for multiple digital and analog signals. In particular, the Transformer Encoder is employed as a network for DRL to process the state information pertaining to the enemy communication. Subsequently, the decision module of the Double Deep Q Network (DDQN) is utilized to select the jamming action based on the processed information. Furthermore, we have devised a reward function and constructed an invalid jamming list, with the objective of selecting an action that requires low power consumption and enhances the convergence speed of the algorithm. The experimental results demonstrate that the algorithm proposed in this paper exhibits notable performance advantages in comparison to other networks and DRL algorithms.

Adaptive Path Planning for Subsurface Plume Tracing with an Autonomous Underwater Vehicle

Autonomous underwater vehicles (AUVs) have been increasingly applied in marine environmental monitoring. Their outstanding capability of performing tasks without human intervention makes them a popular tool for environmental data collection, especially in unknown and remote regions. This paper addresses the path planning problem when AUVs are used to perform plume source tracing in an unknown environment. The goal of path planning is to locate the plume source efficiently. The path planning approach is developed using the Double Deep Q-Network (DDQN) algorithm in the deep reinforcement learning (DRL) framework. The AUV gains knowledge by interacting with the environment, and the optimal direction is extracted from the mapping obtained by a deep neural network. The proposed approach was tested by numerical simulation and on a real ground vehicle. In the numerical simulation, several initial sampling strategies were compared on the basis of survey efficiency. The results show that direct learning based on the interaction with the environment could be an appropriate survey strategy for plume source tracing problems. The comparison with the canonical lawnmower path used in practice showed that path planning using DRL algorithms could be potentially promising for large-scale environment exploration.

EETS: An energy-efficient task scheduler in cloud computing based on improved DQN algorithm

The huge energy consumption of data centers in cloud computing leads to increased operating costs and high carbon emissions to the environment. Deep Reinforcement Learning (DRL) technology combines of deep learning and reinforcement learning, which has an obvious advantage in solving complex task scheduling problems. Deep Q Network(DQN)-based task scheduling has been employed for objective optimization. However, training the DQN algorithm may result in value overestimation, which can negatively impact the learning effectiveness. The replay buffer technique, while increasing sample utilization, does not distinguish between sample importance, resulting in limited utilization of valuable samples. This study proposes an enhanced task scheduling algorithm based on the DQN framework, which utilizes a more optimized Dueling-network architecture as well as Double DQN strategy to alleviate the overestimation bias and address the shortcomings of DQN. It also incorporates a prioritized experience replay technique to achieve importance sampling of experience data, which overcomes the problem of low utilization due to uniform sampling from replay memory. Based on these improved techniques, we developed an energy-efficient task scheduling algorithm called EETS (Energy-Efficient Task Scheduling). This algorithm automatically learns the optimal scheduling policy from historical data while interacting with the environment. Experimental results demonstrate that EETS exhibits faster convergence rates and higher rewards compared to both DQN and DDQN. In scheduling performance, EETS outperforms other baseline algorithms in key metrics, including energy consumption, average task response time, and average machine working time. Particularly, it has a significant advantage when handling large batches of tasks.

A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs

Efficient maintenance has always been essential for the successful application of engineering systems. However, the challenges to be overcome in the implementation of Industry 4.0 necessitate new paradigms of maintenance optimization. Machine learning techniques are becoming increasingly used in engineering and maintenance, with reinforcement learning being one of the most promising. In this paper, we propose a gamma degradation process together with a novel maintenance model in which repairs are increasingly imperfect, i.e., the beneficial effect of system repairs decreases as more repairs are performed, reflecting the degradational behavior of real-world systems. To generate maintenance policies for this system, we developed a reinforcement-learning-based agent using a Double Deep Q-Network architecture. This agent presents two important advantages: it works without a predefined preventive threshold, and it can operate in a continuous degradation state space. Our agent learns to behave in different scenarios, showing great flexibility. In addition, we performed an analysis of how changes in the main parameters of the environment affect the maintenance policy proposed by the agent. The proposed approach is demonstrated to be appropriate and to significatively improve long-run cost as compared with other common maintenance strategies.

An improved hierarchical deep reinforcement learning algorithm for multi-intelligent vehicle lane change

Lane change is a pivotal technology in autonomous driving, playing a significant role in improving traffic conditions. The control task of lane change becomes exceptionally complex for coordination of multiple intelligent vehicles, especially in unpredictable situations. To address this issue, the paper proposes a novel multi-intelligent vehicle lane change method (MVLC) based on Deep Reinforcement Learning (DRL). The MVLC features a hierarchical framework that includes a lower-layer lane change controller and an upper-layer reinforcement learning decision-making method. In the lower layer, a lane change controller is designed to execute motion control for a single vehicle, consisting of a lateral controller utilizing dueling double deep Q-network to make lane change decision and a longitudinal controller that employs Intelligent Driver Model (IDM) to follow the preceding vehicle. In the upper layer, a model-free DRL method is used for simultaneous control of multiple intelligent vehicles. To learn the optimal policy, a comprehensive reward function is designed. Finally, experiments are conducted in mixed traffic to valid the effectiveness of the proposed method. Results show that MVLC achieves secure and timely lane changes for multiple vehicles. Therefore, the method is expected to have significant potential for improving overall traffic efficiency and mitigating traffic congestion.

An Active Learning Semantic Segmentation Model Based on an Improved Double Deep Q-Network

Image semantic segmentation is essential in fields such as computer vision, autonomous driving, and human-computer interaction due to its ability to accurately identify and classify each pixel in an image. However, this task is fraught with challenges, including the difficulty of obtaining detailed pixel labels and the problem of class imbalance in segmentation datasets. These challenges can hinder the effectiveness and efficiency of segmentation models. To address these issues, we propose an active learning semantic segmentation model named CG_D3QN, which is designed and implemented based on an enhanced Double Deep Q-Network (D3QN). The proposed CG_D3QN model incorporates a hybrid network structure that combines a dueling network architecture with Gated Recurrent Units (GRUs). This novel approach improves policy evaluation accuracy and computational efficiency by mitigating a Q-value overestimation and making better use of historical state information. Our experiments, conducted on the CamVid and Cityscapes datasets, reveal that the CG_D3QN model significantly reduces the number of required sample annotations by 65.0% compared to traditional methods. Additionally, it enhances the mean Intersection over Union (IoU) for underrepresented categories by approximately 1% to 3%. These results highlight the model’s effectiveness in lowering annotation costs, addressing class imbalance, and its versatility across different segmentation networks.

DDQN-SFCAG: A service function chain recovery method against network attacks in 6G networks

Service Function Chain (SFC) as an effective solution can satisfy the diverse service requirements of six application scenarios in Network Function Virtualization (NFV)-enabled 6G networks. Resilience as a new key capability indicator in 6G networks puts forward higher requirements for the Quality of Service (QoS) of SFCs. In this article, we study the resilience recovery method in network attack scenarios. We make full use of the monitoring and early warning capability of the monitoring function and propose a proactive recovery method called Double Deep Q-Network based on SFC Attack Graph (DDQN-SFCAG). We fully consider the characteristics of network attacks to generate SFC attack graphs and determine the recovery strategy of SFC according to the service security requirements to provide guidance for recovery. Among them, we design three recovery modes for the recovered Virtual Network Functions (VNFs) to determine the optimal recovery strategy and avoid resource waste. We formalize the SFC recovery problem, which aims to minimize the recovery cost while meeting the recovery strategy. In order to shorten the interruption time, we use DDQN to quickly solve the recovery solution to ensure optimal recovery performance. Our extensive evaluation shows DDQN-SFCAG has excellent recovery performance in network attack scenarios and can reduce the recovery cost by at least 31% compared to the state-of-the-art methods.

Task-Importance-Oriented Task Selection and Allocation Scheme for Mobile Crowdsensing

In Mobile Crowdsensing (MCS), sensing tasks have different impacts and contributions to the whole system or specific targets, so the importance of the tasks is different. Since resources for performing tasks are usually limited, prioritizing the allocation of resources to more important tasks can ensure that key data or information can be collected promptly and accurately, thus improving overall efficiency and performance. Therefore, it is very important to consider the importance of tasks in the task selection and allocation of MCS. In this paper, a task queue is established, the importance of tasks, the ability of participants to perform tasks, and the stability of the task queue are considered, and a novel task selection and allocation scheme (TSAS) in the MCS system is designed. This scheme introduces the Lyapunov optimization method, which can be used to dynamically keep the task queue stable, balance the execution ability of participants and the system load, and perform more important tasks in different system states, even when the participants are limited. In addition, the Double Deep Q-Network (DDQN) method is introduced to improve on the traditional solution of the Lyapunov optimization problem, so this scheme has a certain predictive ability and foresight on the impact of future system states. This paper also proposes action-masking and iterative training methods for the MCS system, which can accelerate the training process of the neural network in the DDQN and improve the training effect. Experiments show that the TSAS based on the Lyapunov optimization method and DDQN performs better than other algorithms, considering the long-term stability of the queue, the number and importance of tasks to be executed, and the congestion degree of tasks.