Double Deep Q-Network Research Articles

OptiPower AI: A deep reinforcement learning framework for intelligent cluster energy management and V2X optimization in industrial applications

Peer-to-peer (P2P) energy trading has emerged as a practical solution for efficient energy management, particularly with the rising affordability of renewable energy and electric vehicles (EVs). This paper introduces the Optimal Power Artificial Intelligence (OptiPower AI) algorithm, a significant advancement in Intelligent Cluster Energy Management (ICEM). OptiPower AI combines Double Deep Q-Network (DDQN)-based Deep Reinforcement Learning (DRL), Vehicle-to-Everything (V2X) technology, and P2P energy trading to optimize energy distribution among clusters of prosumers and consumers. The system efficiently manages Renewable Energy Sources (RES) and EVs, achieving a 19.18% reduction in energy costs and a 50.02% decrease in average energy prices across V2X and P2P scenarios.OptiPower AI uses DRL to dynamically allocate energy and implement real-time pricing, enhancing energy efficiency and user satisfaction. Simulations based on meteorological data from Tunisia validate the system’s ability to improve thermal comfort, increase energy savings, and lower costs. The model’s parameters enable accurate forecasting and allocation, showcasing OptiPower AI’s reliability in variable demand conditions. This work advances the application of DRL in decentralized, sustainable P2P energy management systems for industrial clusters, addressing critical challenges in energy distribution, efficiency, and cost reduction.

Double Deep Q-Networks Based Game-Theoretic Equilibrium Control of Automated Vehicles at Autonomous Intersection

Double Deep Q-Networks Based Game-Theoretic Equilibrium Control of Automated Vehicles at Autonomous Intersection

Effectiveness study of learning algorithms in supply chains of a dairy business

Objective: To develop a control system to prevent over-response of the supply chain of a dairy business. Methodology: The following methods were used: DQN, Double DQN, Dueling DQN, and Dueling Double DQN to determine the distribution of demand: normal and uniform. Results: Results were calculated based on stability in learning (the last 10,000 episodes). It was observed that the means of DQN and DDQN were very similar. To validate whether the performance of the Dueling DQN algorithm is better than that of the DQN algorithm, a non-parametric test was performed to compare the mean rank of two related samples and to determine if there are differences between them. The p values ​​were 5.83e−38 and 0.000 for the Normal and Uniform distributions, respectively. Conclusions: The algorithm with the best results is Dueling DQN, with an average total cost of 151.27 units for the demand with a normal distribution and an average of 155.3 units for the demand with a uniform distribution. This method has less variability once convergence is achieved

Research on robot path planning and obstacle avoidance algorithm in dynamic environment based on deep reinforcement learning

Abstract. In dynamic environments, robot path planning and obstacle avoidance are critical tasks, especially in applications such as autonomous driving, industrial automation, and mobile robotics. These tasks are inherently challenging due to the unpredictability of the environment and the need for real-time decision-making. This paper seeks to address these challenges by developing and analyzing both traditional and optimized models for robot navigation. The initial model utilizes a basic Q-learning algorithm, which provides a straightforward approach to learning from the environment but often struggles with the complexity of dynamic scenarios. To this end, an optimized model is developed that combines the Double Deep Q-Learning algorithm (Double DQN) in conjunction with heuristic strategies. The research employs the MATLAB Reinforcement Learning Toolbox to implement and train these models, and utilizes a simulated environment with dynamic obstacles as a testing site. The simulation generates the necessary data to allow for comprehensive testing and evaluation of the models performance. The results show that the optimized model greatly exceeds the initial model in terms of path planning efficiency and obstacle avoidance capabilities, and that the combination of advanced reinforcement learning techniques and heuristic strategies is extremely important for enhancing the performance and reliability of robotic systems in complex, dynamic environments, offering valuable insights for future applications in various fields of robotics.

Dynamic Path Planning Scheme for OHT in AMHS Based on Map Information Double Deep Q-Network

Dynamic Path Planning Scheme for OHT in AMHS Based on Map Information Double Deep Q-Network

Research on Bionic Robot Motion Control Based on Reinforcement Learning

This research explores the application of reinforcement learning (RL) to enhance the motion control of bio-inspired robots across various environments. Focusing on underwater, terrestrial, and aerial robotic models, the study integrates model-free Q-learning, Deep Q Network (DQN), State-Action-Reward-State-Action (SARSA), and Double Deep Q Network (DDQN) algorithms. These methods are employed to achieve adaptive and efficient movement strategies without relying on pre-defined environmental models. The methodologies range from real-time data capture using live camera feeds for terrestrial robots to simulating aquatic and flight dynamics in controlled environments. Experimental results confirm the efficacy of these RL methods, demonstrating significant improvements in the robots' ability to adapt to dynamic and unknown environments, optimize movement efficiency, and navigate complex scenarios. The findings suggest a promising direction for future robotic applications, emphasizing the need for further research on optimizing these algorithms and expanding their real-world applicability. This study highlights the potential of RL to revolutionize robotic motion control, making robots more versatile and capable in varied and unpredictable settings.

Double Deep Q- energy aware Service allocation based on Dynamic fractional frequency reusable technique for lifetime maximization in HetNet-LTE network

The development of mobile communication in heterogeneous networks is incredible in providing various services through wireless cellular communication through advanced long-term evaluation networks. Increasing multi-concern services and frequencies in spectrum channels are highly layered to select the bandwidth to provide the fastest network without interference. Selecting the channel through macro cell selection is essential to improve network communication and provide the quickest service. Most frequency reuse techniques use service optimality and route selection-based protocols to enrich the packet flow. Still, the improper spectrum delights create more delay tolerance due to short-range service optimality due to energy loss by selecting the short spectrum signal to reuse, which doesn't support the lifetime improvement of the LTE network. To resolve these problems, we propose a Double Deep Q- energy-aware Service allocation based on a Dynamic fractional frequency reusable technique for lifetime maximization in the HetNet-LTE network. Initially, the heterogenous communication environment and node deplanement were carried out to construct the LTE network under the WCC. The communication logs are Route Table (RT), and its services are taken by all node LTE Communication Impact Rate (LTE-CIR). Then, the Backhaul Traffic Algorithm (BTA) is applied to predict the interference on traffic rate from the channel frequency margin. Select the balanced node using the Channel Interference Macro Cell Selection (CIMCS) technique. Considering frequency limits with the Double Deep Q- Network (DDQN) approach, energy-aware selects the optimal route to reuse the frequency level using Frequency Domain Packet Scheduling (FDPS) to improve communication. The proposed system improves the overall throughput by up to 97.8 % with adopted channel selection from the macro unit to improve the latency performance. Also, the interference frequency limits are dynamically reused at an energy optimal level with low-level delay tolerance to improve the link stability by up to 98.4 % with higher lifetime maximation in the LTE network.

Cooperative multi-agent reinforcement learning for multi-area integrated scheduling in wafer fabs

The existing scheduling methods of wafer fabs focus on single area, achieving local optimisation while failing to realise global optimisation due to neglecting the coordination of multi-area. Therefore, it is necessary to consider the complex opposing relationships between multi-area caused by constraints such as batch processing, re-entrance, and multiple residency times within and between areas to conduct integrated scheduling and shorten the production cycle time. For this issue, this paper proposes a cooperative multi-agent reinforcement learning for multi-area integrated scheduling. Aiming at the dynamic batching and scheduling considering the dynamic arrival lots in multi-area, a multi-agent reinforcement learning algorithm is presented to learn the optimal dynamic batching and scheduling policy firstly. Subsequently, a cooperative multi-agent framework is raised to achieve the global optimisation and coordination of multi-area. Furthermore, an adaptive exploration strategy is constructed to enhance the global exploration capability of the complex solution space caused by residency time constraints and re-entrant property. Moreover, a policy share enhanced Double DQN is employed to improve the generalisation and adaptability of the multi-agent. Finally, the experiments demonstrate that the proposed integrated scheduling method has better comprehensive performance compared to the previous area-separated scheduling methods.

Probing-aided spectrum sensing-based hybrid access strategy for energy harvesting CRNs

To address the issues of energy supply and spectrum scarcity in Internet of Things (IoT), energy harvesting (EH) and cognitive radio (CR) technologies have been proposed and widely applied. In EH-CR networks (EH-CRNs), miss detection causes significant energy and time wastage of IoT devices, especially secondary users (SUs), and causes serious interference to primary users (PUs). To alleviate this concern, we propose a probing-aided spectrum sensing (PaSS) model for EH-CRNs, where M pairs of PUs and one pair of SUs coexist. The secondary transmitter (ST) harvests energy from the radio frequency (RF) signals of PUs for opportunistic spectrum access. In the PaSS model, probing operation is employed to further confirm the real state of the spectrum that has been sensed as free in order to avoid the waste of time and energy resulting from miss detection. Based on the PaSS model, we propose a novel hybrid access strategy, where the ST’s actions (i.e., sensing, probing, EH, underlay/overlay transmission mode) depend on the belief vector of M channels, energy state and data buffer state of the ST. By developing an adjusted double deep Q-network (ADDQN) reinforcement learning algorithm, we aim to find the optimal strategy that minimizes the long-term average number of packet losses (ANPL) and the ANPL minimization problem is an integer programming problem. Simulation results validate the ANPL performance of the ST in the ADDQN-PaSS model, and reveal impacts of network parameters on the performance of the ST, and find that at least 7.9% reduction of ANPL is achieved by using the ADDQN-PaSS model.

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

Active power balance control of wind-photovoltaic-storage power system based on transfer learning double deep Q-network approach

IntroductionThis study addresses the challenge of active power (AP) balance control in wind-photovoltaic-storage (WPS) power systems, particularly in regions with a high proportion of renewable energy (RE) units. The goal is to effectively manage the AP balance to reduce the output of thermal power generators, thereby improving the overall efficiency and sustainability of WPS systems.MethodsTo achieve this objective, we propose the transfer learning double deep Q-network (TLDDQN) method for controlling the energy storage device within WPS power systems. The TLDDQN method leverages the benefits of transfer learning to quickly adapt to new environments, thereby enhancing the training speed of the double deep Q-network (DDQN) algorithm. Additionally, we introduce an adaptive entropy mechanism integrated with the DDQN algorithm, which is further improved to enhance the training capability of agents.ResultsThe proposed TLDDQN algorithm was applied to a regional WPS power system for experimental simulation of AP balance control. The results indicate that the TLDDQN algorithm trains agents more rapidly compared to the standard DDQN algorithm. Furthermore, the AP balance control method based on TLDDQN can more accurately manage the storage device, thereby reducing the output of thermal power generators more effectively than the particle swarm optimization-based method.DiscussionOverall, the TLDDQN algorithm proposed in this study can provide some insights and theoretical references for research in related fields, especially those requiring decision making.

Optimization of the age of correlated information in V2X networks with edge computing

Vehicle-to-Everything (V2X) communication has the potential to revolutionize the travel experience in intelligent transportation, which has received the great attention recently. However, ensuring the freshness of information from multiple sources is critical for the real-time and reliable communication in vehicular networks, especially for timely updates of service centers. To address this issue, we use a promising metric called Age of Correlated Information (AoCI), which can characterize the freshness of multi-source information. Therefore, we propose a novel model that can dynamically regulate the channel activation matching and edge computing collaboration strategy to minimize AoCI in V2X vehicular networks. Firstly, we describe the system model of a V2X network with edge computing, including definitions and assumptions for freshness of information, edge co-computing, etc. Secondly, we formulate the joint optimization problem as a source-related age minimization (SRAM) problem, which is NP-complete. A heuristic algorithm is proposed to solve it under fast-fading channel. Finally, since traditional graph models cannot capture the changing correlation between nodes in dynamic networks, we use graph convolutional networks(GCN) to extract the features of multi-source correlation. The features extracted by GCN include relevant attributes of the sources and its communication links. The features are provided as input to a double deep Q network (DDQN) for training the model that can adapt to a dynamic network environment. Extensive simulation experiments in different network scenarios validate that our proposed method can effectively and efficiently reduce the average AoCI and the computational resources.

МЕТОД ОБУЧЕНИЯ ИНТЕЛЛЕКТУАЛЬНОГО АГЕНТА С ПОМОЩЬЮ СЕТЕЙ DOUBLE DQN, ПУТЕВЫХ ТОЧЕК И ФУНКЦИИ ВОЗНАГРАЖДЕНИЯ

The problems of increasing the control efficiency of autonomous vehicles are considered. The problem of reducing the required computational resources for the intelligent vehicle control module is highlighted. A neural network training algorithm for Double DQN architecture with modified reward functions is proposed. The basis of the proposed solution is the use of lane segmentation, reward function and the use of additional waypoints in training. A software model has been developed and simulation of the learning process has been performed. The results obtained from a comparative analysis with known solutions show a stable increase in episode duration, and effective training in a realistic urban simulation. The study points to the possibility of reducing the need for high computing power, which will enable the use of central processing units (CPUs) for basic functions of unmanned vehicles instead of graphics processing units (GPUs).

Identification of Anomalies in Lung and Colon Cancer Using Computer Vision-Based Swin Transformer with Ensemble Model on Histopathological Images.

Lung and colon cancer (LCC) is a dominant life-threatening disease that needs timely attention and precise diagnosis for efficient treatment. The conventional diagnostic techniques for LCC regularly encounter constraints in terms of efficiency and accuracy, thus causing challenges in primary recognition and treatment. Early diagnosis of the disease can immensely reduce the probability of death. In medical practice, the histopathological study of the tissue samples generally uses a classical model. Still, the automated devices that exploit artificial intelligence (AI) techniques produce efficient results in disease diagnosis. In histopathology, both machine learning (ML) and deep learning (DL) approaches can be deployed owing to their latent ability in analyzing and predicting physically accurate molecular phenotypes and microsatellite uncertainty. In this background, this study presents a novel technique called Lung and Colon Cancer using a Swin Transformer with an Ensemble Model on the Histopathological Images (LCCST-EMHI). The proposed LCCST-EMHI method focuses on designing a DL model for the diagnosis and classification of the LCC using histopathological images (HI). In order to achieve this, the LCCST-EMHI model utilizes the bilateral filtering (BF) technique to get rid of the noise. Further, the Swin Transformer (ST) model is also employed for the purpose of feature extraction. For the LCC detection and classification process, an ensemble deep learning classifier is used with three techniques: bidirectional long short-term memory with multi-head attention (BiLSTM-MHA), Double Deep Q-Network (DDQN), and sparse stacked autoencoder (SSAE). Eventually, the hyperparameter selection of the three DL models can be implemented utilizing the walrus optimization algorithm (WaOA) method. In order to illustrate the promising performance of the LCCST-EMHI approach, an extensive range of simulation analyses was conducted on a benchmark dataset. The experimentation results demonstrated the promising performance of the LCCST-EMHI approach over other recent methods.

A two-stage subsynchronous oscillation assessment method for DFIG-based wind farm grid-connected system

In the power system, the wind farm based on Doubly-Fed Induction Generator (DFIG) may lead to Subsynchronous Oscillation (SSO), which poses a challenge to the stability of the power grid. In order to accurately evaluate SSO, this paper proposes a new evaluation method. It is divided into two main stages: firstly, the interference level of Phasor Measurement Unit (PMU) data is identified by using the classification model based on Upper Confidence Bound (UCB) and Double Deep Q Network (DDQN). Then, an SSO parameter estimation model based on Local Feature Fusion Transformer (LFF-Transformer) network is designed for data with different interference levels. Experimental results show that the errors of eRMSE-f and EMAPE-F are 0.001 and 0.003 respectively, and the errors of eRMSE-δ and EMAPE-δ are 0.009 and 0.015 respectively. In terms of training and testing time, this method is 90 s and 18 s respectively, which is significantly better than Multi-SVR and Multi-CNN. After application, the frequency deviation decreased from 0.05 to 0.02 Hz, the voltage deviation decreased from 3.5 to 1.5%, the power fluctuation decreased from 10 to 5 MW, the SSO frequency decreased from 1.5 Hz to less than 0.5 Hz, and the SSO damping ratio increased from 0.08 to 0.15. This shows that the proposed method effectively increases the stability of the power grid.

Test data generation using deep reinforcement learning

The process of creating test data for software is one of the most complex and labor-intensive stages in the software development cycle. It requires significant resources and effort, especially when it comes to achieving high test coverage. Search-Based Software Testing (SBST) is an approach that automates this process by using metaheuristic algorithms to generate test data. Metaheuristic algorithms, such as genetic algorithms or simulated annealing, operate on the principle of systematically exploring possible options and selecting the most effective solutions based on feedback from a fitness function. This process is similar to reinforcement learning, where an agent receives rewards for correct actions and penalties for incorrect ones, with the goal of optimizing overall performance. We have proposed a new approach that explores the feasibility of applying reinforcement learning in SBST, with the aim of replacing traditional, human-designed metaheuristic algorithms. To this end, we developed the GunPowder platform, which transforms the software testing process (STP) into a learning environment. In this environment, the agent uses the Double Deep Q-Networks (DDQN) method, which incorporates deep neural networks to learn the optimal strategy for interacting with the software under test. Our study involved a series of empirical experiments to evaluate the effectiveness of the proposed approach. The results indicate that agents trained using GunPowder can effectively learn and apply metaheuristic algorithms, achieving high branch coverage in training functions. In particular, our agents achieved 100% branch coverage in the case of training functions, demonstrating their ability to adapt to different test scenarios. These findings highlight the potential of using deep neural networks and reinforcement learning in SBST, which could significantly improve the quality and efficiency of the software testing process in the future. Thus, our work opens up new opportunities for integrating modern machine learning methods into automated software testing, potentially reducing the time and resources required for testing and improving the quality of final products.

DT-DDQNR: A digital twin assisted direct-to-cell satellite network intelligent routing algorithm

This paper presents a novel routing algorithm designed for direct-to-cell satellite constellations in satellite networks. The algorithm aims to ensure stable Quality of Service (QoS) for users despite the highly dynamic nature of satellite links and network conditions. It comprises two key components: a terrestrial-satellite link selection model and an inter-satellite multi-hop routing decision model. To enhance decision-making, the algorithm incorporates a mechanism that interacts with large-scale digital twin constellations, offering agents with high-precision state data inputs and feedback for training and application support. The terrestrial-satellite link selection process is modeled as a constraint satisfaction problem, maximizing the Link Quality Indicator (LQI). Meanwhile, inter-satellite data forwarding decisions are determined utilizing an enhanced dynamic reward mechanism and a re-designed neural network in the DDQN model. Simulation results demonstrate the superiority of the Digital Twin-Assisted Double Deep Q-Network Routing (DT-DDQNR) algorithm over traditional routing algorithms in terms of overall network QoS performance, offering a promising solution for optimizing data transmission in integrated terrestrial-satellite networks.

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

Learning-based hierarchical cooperative eco-driving with traffic flow prediction for hybrid electric vehicles

The integration of autonomous driving and hybrid electric vehicle technologies presents a promising solution for achieving environmental sustainability. This paper introduces an innovative energy-efficient driving strategy for hybrid electric vehicles that incorporates real-time traffic flow prediction. The study delves into the impact of both lateral and longitudinal vehicle maneuvers on energy consumption within dynamic traffic environments, offering novel insights into optimizing energy utilization. Firstly, a multi-lane traffic flow state rolling predictor is constructed based on the Hankel dynamic mode decomposition algorithm. Subsequently, a vehicle longitudinal and lateral coordinated control strategy is established by integrating the prioritized experience replay double deep Q-network algorithm. Finally, a novel energy management strategy is proposed that leverages Simulink dynamic model and the deep deterministic policy gradient algorithm to address the vehicle dynamic decision-making planning results. Within a hierarchical cooperative optimization framework, this research comprehensively considers safety, comfort, traffic efficiency, and fuel economy. By introducing a novel hierarchical collaborative ecological driving framework, we have achieved a substantial improvement in environmental sustainability, with traffic efficiency increasing by 10.27%-14.41% and fuel economy rising by 9.44%-10.47%. Hardware-in-the-loop validation has confirmed the proposed approach’s real-time capabilities and promising practical applications.

Analyzing deep reinforcement learning model decisions with Shapley additive explanations for counter drone operations

As the use of drones continues to increase, their capabilities pose a threat to airspace safety when they are misused. Deploying AI models for intercepting these unwanted drones becomes crucial. However, these AI models, such as deep learning models, often operate as “black boxes”, making it hard to trust their decision-making system. This also affects end-users’ confidence in these AI systems. In this paper, the explainability of deep reinforcement learning is investigated and a deep reinforcement learning (DRL) method, double deep Q-network with dueling architecture and prioritized experience replay is applied to train the AI models. To make the AI model decisions more transparent and to understand the reasoning behind the AI decisions for counter-drone systems, Shapley Additive Explanations (SHAP) method is implemented. After training the DRL agent, experience replay is visualized, and the absolute SHAP values are calculated to explain the key factors that influence the deep reinforcement learning agent’s choices. The integration of DRL with explainable AI methods such as SHAP demonstrates significant potential for the advancement of robust and efficient counter-drone systems.