Double Deep Research Articles

Dynamic Path Planning Scheme for OHT in AMHS Based on Map Information Double Deep Q-Network

Dynamic Path Planning Scheme for OHT in AMHS Based on Map Information Double Deep Q-Network

Research on robot path planning and obstacle avoidance algorithm in dynamic environment based on deep reinforcement learning

Abstract. In dynamic environments, robot path planning and obstacle avoidance are critical tasks, especially in applications such as autonomous driving, industrial automation, and mobile robotics. These tasks are inherently challenging due to the unpredictability of the environment and the need for real-time decision-making. This paper seeks to address these challenges by developing and analyzing both traditional and optimized models for robot navigation. The initial model utilizes a basic Q-learning algorithm, which provides a straightforward approach to learning from the environment but often struggles with the complexity of dynamic scenarios. To this end, an optimized model is developed that combines the Double Deep Q-Learning algorithm (Double DQN) in conjunction with heuristic strategies. The research employs the MATLAB Reinforcement Learning Toolbox to implement and train these models, and utilizes a simulated environment with dynamic obstacles as a testing site. The simulation generates the necessary data to allow for comprehensive testing and evaluation of the models performance. The results show that the optimized model greatly exceeds the initial model in terms of path planning efficiency and obstacle avoidance capabilities, and that the combination of advanced reinforcement learning techniques and heuristic strategies is extremely important for enhancing the performance and reliability of robotic systems in complex, dynamic environments, offering valuable insights for future applications in various fields of robotics.

Research on Bionic Robot Motion Control Based on Reinforcement Learning

This research explores the application of reinforcement learning (RL) to enhance the motion control of bio-inspired robots across various environments. Focusing on underwater, terrestrial, and aerial robotic models, the study integrates model-free Q-learning, Deep Q Network (DQN), State-Action-Reward-State-Action (SARSA), and Double Deep Q Network (DDQN) algorithms. These methods are employed to achieve adaptive and efficient movement strategies without relying on pre-defined environmental models. The methodologies range from real-time data capture using live camera feeds for terrestrial robots to simulating aquatic and flight dynamics in controlled environments. Experimental results confirm the efficacy of these RL methods, demonstrating significant improvements in the robots' ability to adapt to dynamic and unknown environments, optimize movement efficiency, and navigate complex scenarios. The findings suggest a promising direction for future robotic applications, emphasizing the need for further research on optimizing these algorithms and expanding their real-world applicability. This study highlights the potential of RL to revolutionize robotic motion control, making robots more versatile and capable in varied and unpredictable settings.

Application of hyperspectral band selection method based on deep reinforcement learning to low-value recyclable waste classification

Hyperspectral images (HSIs) have proven effective for classification of Low-value Recyclable Waste (LVRW). However, the high correlation between bands in HSIs introduces redundant information. In this paper, to overcome the challenge of low applicability of existing band selection methods of LVRW HSIs, we propose a Dueling Double Deep Q Network based on Supervised Band Selection method (D3QN-SBS). Specifically, we formulate band selection as a reinforcement learning problem, treating it as a combinatorial optimization task that explores band combinations within a discrete space, using overall accuracy as the reward to tune the policy and optimise the state-action value function. The results of comparative experiments show that D3QN-SBS outperforms other methods when selecting 2–10 bands, where achieves an accuracy of about 99.24 %, 99.10 %, 99.05 %, and 99.16 % in k-NN, SVM-RBF, RF, and MLP based on 10 bands, the precision, recall, and F1-score are nearly 100 % for OTHERS and are more than 99 % for PS. In K-fold cross-validation, the majority of the folds under four classifiers achieves above 98 % for four evaluation metrics and the average F1-scores are 99.25 %, 99.01 %, 99.06 %, and 99.17 %. This approach can be deployed in LVRW sorting equipment, contributing to the advancement of hyperspectral imaging technologies in plastic waste sorting field.

Double Deep Q- energy aware Service allocation based on Dynamic fractional frequency reusable technique for lifetime maximization in HetNet-LTE network

The development of mobile communication in heterogeneous networks is incredible in providing various services through wireless cellular communication through advanced long-term evaluation networks. Increasing multi-concern services and frequencies in spectrum channels are highly layered to select the bandwidth to provide the fastest network without interference. Selecting the channel through macro cell selection is essential to improve network communication and provide the quickest service. Most frequency reuse techniques use service optimality and route selection-based protocols to enrich the packet flow. Still, the improper spectrum delights create more delay tolerance due to short-range service optimality due to energy loss by selecting the short spectrum signal to reuse, which doesn't support the lifetime improvement of the LTE network. To resolve these problems, we propose a Double Deep Q- energy-aware Service allocation based on a Dynamic fractional frequency reusable technique for lifetime maximization in the HetNet-LTE network. Initially, the heterogenous communication environment and node deplanement were carried out to construct the LTE network under the WCC. The communication logs are Route Table (RT), and its services are taken by all node LTE Communication Impact Rate (LTE-CIR). Then, the Backhaul Traffic Algorithm (BTA) is applied to predict the interference on traffic rate from the channel frequency margin. Select the balanced node using the Channel Interference Macro Cell Selection (CIMCS) technique. Considering frequency limits with the Double Deep Q- Network (DDQN) approach, energy-aware selects the optimal route to reuse the frequency level using Frequency Domain Packet Scheduling (FDPS) to improve communication. The proposed system improves the overall throughput by up to 97.8 % with adopted channel selection from the macro unit to improve the latency performance. Also, the interference frequency limits are dynamically reused at an energy optimal level with low-level delay tolerance to improve the link stability by up to 98.4 % with higher lifetime maximation in the LTE network.

Noisy Dueling Double Deep Q-Network algorithm for autonomous underwater vehicle path planning.

How to improve the success rate of autonomous underwater vehicle (AUV) path planning and reduce travel time as much as possible is a very challenging and crucial problem in the practical applications of AUV in the complex ocean current environment. Traditional reinforcement learning algorithms lack exploration of the environment, and the strategies learned by the agent may not generalize well to other different environments. To address these challenges, we propose a novel AUV path planning algorithm named the Noisy Dueling Double Deep Q-Network (ND3QN) algorithm by modifying the reward function and introducing a noisy network, which generalizes the traditional D3QN algorithm. Compared with the classical algorithm [e.g., Rapidly-exploring Random Trees Star (RRT*), DQN, and D3QN], with simulation experiments conducted in realistic terrain and ocean currents, the proposed ND3QN algorithm demonstrates the outstanding characteristics of a higher success rate of AUV path planning, shorter travel time, and smoother paths.

Station keeping control method based on deep reinforcement learning for stratospheric aerostat in dynamic wind field

Aiming at the problem of long-endurance station keeping of stratospheric aerostats under energy constraints in dynamic wind environment, a stratospheric aerostat station keeping controller was designed using the D3QN (Dueling Double Deep Q Network) algorithm of deep reinforcement learning in this article. The performance of the station keeping controllers are evaluated by the average station keeping radius and station keeping effective time ratio. Meanwhile, the effects of different reward functions on the performance of station keeping controller are studied. The simulation results of typical scenarios with European Centre for Medium-Range Weather Forecasts (ECMWF) data show that: under the task constraints of three days of station keeping time and 50 km of station keeping radius, the average station keeping radius of the stratospheric aerostat without power propulsion is 28.25 km, the station keeping time ratio is 90.56%, and the State of Charge (SOC) of the battery is more than 50%. Under the condition of propulsion, the control effect of the average station keeping radius of stratospheric aerostat is significantly improved. The control effect of adding dynamic propulsion in the north-south direction is better than that in the east-west direction, which is close to the dual-channel control effect. At the same time, this study verifies that the station keeping controller based on deep reinforcement learning has strong robustness, and demonstrates that the station keeping controller can be designed with different reward functions for the requirements of different station keeping task.

Routing cost-integrated intelligent handover strategy for multi-layer LEO mega-constellation networks

Low Earth Orbit (LEO) mega-constellation networks, exemplified by Starlink, are poised to play a pivotal role in future mobile communication networks, due to their low latency and high capacity. With the massively deployed satellites, ground users now can be covered by multiple visible satellites, but also face complex handover issues with such massive high-mobility satellites in multi-layer. The end-to-end routing is also affected by the handover behavior. In this paper, we propose an intelligent handover strategy dedicated to multi-layer LEO mega-constellation networks. Firstly, an analytic model is utilized to rapidly estimate the end-to-end propagation latency as a key handover factor to construct a multi-objective optimization model. Subsequently, an intelligent handover strategy is proposed by employing the Dueling Double Deep Q Network (D3QN)-based deep reinforcement learning algorithm for single-layer constellations. Moreover, an optimal cross-layer handover scheme is proposed by predicting the latency-jitter and minimizing the cross-layer overhead. Simulation results demonstrate the superior performance of the proposed method in the multi-layer LEO mega-constellation, showcasing reductions of up to 8.2% and 59.5% in end-to-end latency and jitter respectively, when compared to the existing handover strategies.

Identification of Anomalies in Lung and Colon Cancer Using Computer Vision-Based Swin Transformer with Ensemble Model on Histopathological Images.

Lung and colon cancer (LCC) is a dominant life-threatening disease that needs timely attention and precise diagnosis for efficient treatment. The conventional diagnostic techniques for LCC regularly encounter constraints in terms of efficiency and accuracy, thus causing challenges in primary recognition and treatment. Early diagnosis of the disease can immensely reduce the probability of death. In medical practice, the histopathological study of the tissue samples generally uses a classical model. Still, the automated devices that exploit artificial intelligence (AI) techniques produce efficient results in disease diagnosis. In histopathology, both machine learning (ML) and deep learning (DL) approaches can be deployed owing to their latent ability in analyzing and predicting physically accurate molecular phenotypes and microsatellite uncertainty. In this background, this study presents a novel technique called Lung and Colon Cancer using a Swin Transformer with an Ensemble Model on the Histopathological Images (LCCST-EMHI). The proposed LCCST-EMHI method focuses on designing a DL model for the diagnosis and classification of the LCC using histopathological images (HI). In order to achieve this, the LCCST-EMHI model utilizes the bilateral filtering (BF) technique to get rid of the noise. Further, the Swin Transformer (ST) model is also employed for the purpose of feature extraction. For the LCC detection and classification process, an ensemble deep learning classifier is used with three techniques: bidirectional long short-term memory with multi-head attention (BiLSTM-MHA), Double Deep Q-Network (DDQN), and sparse stacked autoencoder (SSAE). Eventually, the hyperparameter selection of the three DL models can be implemented utilizing the walrus optimization algorithm (WaOA) method. In order to illustrate the promising performance of the LCCST-EMHI approach, an extensive range of simulation analyses was conducted on a benchmark dataset. The experimentation results demonstrated the promising performance of the LCCST-EMHI approach over other recent methods.

A two-stage subsynchronous oscillation assessment method for DFIG-based wind farm grid-connected system

In the power system, the wind farm based on Doubly-Fed Induction Generator (DFIG) may lead to Subsynchronous Oscillation (SSO), which poses a challenge to the stability of the power grid. In order to accurately evaluate SSO, this paper proposes a new evaluation method. It is divided into two main stages: firstly, the interference level of Phasor Measurement Unit (PMU) data is identified by using the classification model based on Upper Confidence Bound (UCB) and Double Deep Q Network (DDQN). Then, an SSO parameter estimation model based on Local Feature Fusion Transformer (LFF-Transformer) network is designed for data with different interference levels. Experimental results show that the errors of eRMSE-f and EMAPE-F are 0.001 and 0.003 respectively, and the errors of eRMSE-δ and EMAPE-δ are 0.009 and 0.015 respectively. In terms of training and testing time, this method is 90 s and 18 s respectively, which is significantly better than Multi-SVR and Multi-CNN. After application, the frequency deviation decreased from 0.05 to 0.02 Hz, the voltage deviation decreased from 3.5 to 1.5%, the power fluctuation decreased from 10 to 5 MW, the SSO frequency decreased from 1.5 Hz to less than 0.5 Hz, and the SSO damping ratio increased from 0.08 to 0.15. This shows that the proposed method effectively increases the stability of the power grid.

DT-DDQNR: A digital twin assisted direct-to-cell satellite network intelligent routing algorithm

This paper presents a novel routing algorithm designed for direct-to-cell satellite constellations in satellite networks. The algorithm aims to ensure stable Quality of Service (QoS) for users despite the highly dynamic nature of satellite links and network conditions. It comprises two key components: a terrestrial-satellite link selection model and an inter-satellite multi-hop routing decision model. To enhance decision-making, the algorithm incorporates a mechanism that interacts with large-scale digital twin constellations, offering agents with high-precision state data inputs and feedback for training and application support. The terrestrial-satellite link selection process is modeled as a constraint satisfaction problem, maximizing the Link Quality Indicator (LQI). Meanwhile, inter-satellite data forwarding decisions are determined utilizing an enhanced dynamic reward mechanism and a re-designed neural network in the DDQN model. Simulation results demonstrate the superiority of the Digital Twin-Assisted Double Deep Q-Network Routing (DT-DDQNR) algorithm over traditional routing algorithms in terms of overall network QoS performance, offering a promising solution for optimizing data transmission in integrated terrestrial-satellite networks.

Dynamic flexible job shop scheduling based on deep reinforcement learning

The unpredictable dynamic events in smart factory seriously influence the scheduling schemes and production efficiency. To minimize the total tardiness of orders, this paper proposes a Deep Reinforcement Learning (DRL) method to solve the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) with random job arrival. In the scheduling process, the intelligent agent can select the operations to be processed on the available machines according to the job shop state at each scheduling point by transforming DFJSP into a Markov Decision Process (MDP). For reflecting the job shop environment, eight state features are extracted, and six composite dispatching rules are designed as the action space. Moreover, a reward function is developed to enhance the learning efficiency and solution quality of the intelligent agent, the Soft-max selection strategy is applied to balance exploration and exploitation. The Dueling Double Deep Q-Network (Dueling DDQN) algorithm is introduced to train the intelligent agent, which effectively alleviates the overestimation problem. Numerical simulation results show that the proposed Dueling DDQN method effectively solves the DFJSP. Compared with existing scheduling strategies, the proposed approach not only exhibits superior performance but also maintains its advantages among different scale instances.

A cognitive communication jamming strategy based on Transformer and Deep Reinforcement Learning

The advent of sophisticated communication technologies, such as cognitive radio and anti-jamming techniques, has significantly elevated the challenge of disrupting enemy communications. Nevertheless, the inherent openness of wireless communications remains a vulnerability that can be exploited to interfere with them. Some contemporary Reinforcement Learning (RL)-based jamming strategies examine methods for rapidly identifying the optimal jamming strategy for a specific modulated signal. However, such algorithms lack the flexibility and responsiveness required to effectively counter the enemy’s evolving communication strategies. To address this issue, we propose a Transformer and Deep Reinforcement Learning (DRL)-based jamming strategy that can be trained to identify jamming methods for multiple digital and analog signals. In particular, the Transformer Encoder is employed as a network for DRL to process the state information pertaining to the enemy communication. Subsequently, the decision module of the Double Deep Q Network (DDQN) is utilized to select the jamming action based on the processed information. Furthermore, we have devised a reward function and constructed an invalid jamming list, with the objective of selecting an action that requires low power consumption and enhances the convergence speed of the algorithm. The experimental results demonstrate that the algorithm proposed in this paper exhibits notable performance advantages in comparison to other networks and DRL algorithms.

Resource allocation scheduling scheme for task migration and offloading in 6G Cybertwin internet of vehicles based on DRL

AbstractAs vehicular technology advances, intelligent vehicles generate numerous computation‐intensive tasks, challenging the computational resources of both the vehicles and the Internet of Vehicles (IoV). Traditional IoV struggles with fixed network structures and limited scalability, unable to meet the growing computational demands and next‐generation mobile communication technologies. In congested areas, near‐end Mobile Edge Computing (MEC) resources are often overtaxed, while far‐end MEC servers are underused, resulting in poor service quality. A novel network framework utilizing sixth‐generation mobile communication (6G) and digital twin technologies, combined with task migration, promises to alleviate these inefficiencies. To address these challenges, a task migration and re‐offloading model based on task attribute classification is introduced, employing a hybrid deep reinforcement learning (DRL) algorithm—Dueling Double Q Network DDPG (QDPG). This algorithm merges the strengths of the Deep Deterministic Policy Gradient (DDPG) and the Dueling Double Deep Q‐Network (D3QN), effectively handling continuous and discrete action domains to optimize task migration and re‐offloading in IoV. The inclusion of the Mini Batch K‐Means algorithm enhances learning efficiency and optimization in the DRL algorithm. Experimental results show that QDPG significantly boosts task efficiency and computational performance, providing a robust solution for resource allocation in IoV.

Adaptive Path Planning for Subsurface Plume Tracing with an Autonomous Underwater Vehicle

Autonomous underwater vehicles (AUVs) have been increasingly applied in marine environmental monitoring. Their outstanding capability of performing tasks without human intervention makes them a popular tool for environmental data collection, especially in unknown and remote regions. This paper addresses the path planning problem when AUVs are used to perform plume source tracing in an unknown environment. The goal of path planning is to locate the plume source efficiently. The path planning approach is developed using the Double Deep Q-Network (DDQN) algorithm in the deep reinforcement learning (DRL) framework. The AUV gains knowledge by interacting with the environment, and the optimal direction is extracted from the mapping obtained by a deep neural network. The proposed approach was tested by numerical simulation and on a real ground vehicle. In the numerical simulation, several initial sampling strategies were compared on the basis of survey efficiency. The results show that direct learning based on the interaction with the environment could be an appropriate survey strategy for plume source tracing problems. The comparison with the canonical lawnmower path used in practice showed that path planning using DRL algorithms could be potentially promising for large-scale environment exploration.

A Variation of the Double Deep Fibular Nerve with Multiple Connections to the Superficial Fibular Nerve on the Dorsal Aspect of the Foot

In this case report, we aim to present our observation of a rare variation on the dorsum of the foot, including the double deep fibular (peroneal) nerve and its connections to superficial nerves. The cutaneous nerves of both feet ' dorsum were examined during a routine dissection on a formalin-fixed adult male cadaver. During the dissection of the dorsum of the right foot, it was observed that the deep fibular nerve (DFN) surfaced at two distinct points by piercing the investing fascia. Then, these two branches converged, and DFN had connections at three different points with the superficial fibular (peroneal) nerve (SFN). However, the superficial nerve anatomy of the dorsum of the left foot exhibited the anatomical structure commonly described in classical textbooks. To the best of our knowledge, the variation of double DFN with multiple connections to the SFN has not been previously described in the literature. Understanding the innervation of the dorsum of the foot may be clinically substantial in preventing nerve injuries during surgical interventions. The knowledge regarding rare superficial nerve variations may contribute to the success of anaesthesia applications, especially in the ankle and dorsum of the foot.

A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs

Efficient maintenance has always been essential for the successful application of engineering systems. However, the challenges to be overcome in the implementation of Industry 4.0 necessitate new paradigms of maintenance optimization. Machine learning techniques are becoming increasingly used in engineering and maintenance, with reinforcement learning being one of the most promising. In this paper, we propose a gamma degradation process together with a novel maintenance model in which repairs are increasingly imperfect, i.e., the beneficial effect of system repairs decreases as more repairs are performed, reflecting the degradational behavior of real-world systems. To generate maintenance policies for this system, we developed a reinforcement-learning-based agent using a Double Deep Q-Network architecture. This agent presents two important advantages: it works without a predefined preventive threshold, and it can operate in a continuous degradation state space. Our agent learns to behave in different scenarios, showing great flexibility. In addition, we performed an analysis of how changes in the main parameters of the environment affect the maintenance policy proposed by the agent. The proposed approach is demonstrated to be appropriate and to significatively improve long-run cost as compared with other common maintenance strategies.

An Active Learning Semantic Segmentation Model Based on an Improved Double Deep Q-Network

Image semantic segmentation is essential in fields such as computer vision, autonomous driving, and human-computer interaction due to its ability to accurately identify and classify each pixel in an image. However, this task is fraught with challenges, including the difficulty of obtaining detailed pixel labels and the problem of class imbalance in segmentation datasets. These challenges can hinder the effectiveness and efficiency of segmentation models. To address these issues, we propose an active learning semantic segmentation model named CG_D3QN, which is designed and implemented based on an enhanced Double Deep Q-Network (D3QN). The proposed CG_D3QN model incorporates a hybrid network structure that combines a dueling network architecture with Gated Recurrent Units (GRUs). This novel approach improves policy evaluation accuracy and computational efficiency by mitigating a Q-value overestimation and making better use of historical state information. Our experiments, conducted on the CamVid and Cityscapes datasets, reveal that the CG_D3QN model significantly reduces the number of required sample annotations by 65.0% compared to traditional methods. Additionally, it enhances the mean Intersection over Union (IoU) for underrepresented categories by approximately 1% to 3%. These results highlight the model’s effectiveness in lowering annotation costs, addressing class imbalance, and its versatility across different segmentation networks.

DDQN-SFCAG: A service function chain recovery method against network attacks in 6G networks

Service Function Chain (SFC) as an effective solution can satisfy the diverse service requirements of six application scenarios in Network Function Virtualization (NFV)-enabled 6G networks. Resilience as a new key capability indicator in 6G networks puts forward higher requirements for the Quality of Service (QoS) of SFCs. In this article, we study the resilience recovery method in network attack scenarios. We make full use of the monitoring and early warning capability of the monitoring function and propose a proactive recovery method called Double Deep Q-Network based on SFC Attack Graph (DDQN-SFCAG). We fully consider the characteristics of network attacks to generate SFC attack graphs and determine the recovery strategy of SFC according to the service security requirements to provide guidance for recovery. Among them, we design three recovery modes for the recovered Virtual Network Functions (VNFs) to determine the optimal recovery strategy and avoid resource waste. We formalize the SFC recovery problem, which aims to minimize the recovery cost while meeting the recovery strategy. In order to shorten the interruption time, we use DDQN to quickly solve the recovery solution to ensure optimal recovery performance. Our extensive evaluation shows DDQN-SFCAG has excellent recovery performance in network attack scenarios and can reduce the recovery cost by at least 31% compared to the state-of-the-art methods.

Machine Learning Methods for Precision Dosing in Anticancer Drug Therapy: A Scoping Review.

In the last decade, various Machine Learning techniques have been proposed aiming to individualise the dose of anticancer drugs mostly based on a presumed drug effect or measured effect biomarkers. The aim of this scoping review was to comprehensively summarise the research status on the use of Machine Learning for precision dosing in anticancer drug therapy. This scoping review was conducted in accordance with the interim guidance by Cochrane and the Joanna Briggs Institute. We systematically searched the databases Medline (via PubMed), Embase and the Cochrane Library for research articles and reviews including results published after 2016. Results were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist. A total of 17 relevant studies was identified. In 12 of the included studies, Reinforcement Learning methods were used, including Classical, Deep, Double Deep and Conservative Q-Learning and Fuzzy Reinforcement Learning. Furthermore, classical Machine Learning methods were compared in terms of their performance and an artificial intelligence platform based on parabolic equations was used to guide dosing prospectively and retrospectively, albeit only in a limited number of patients. Due to the significantly different algorithm structures, a meaningful comparison between the various Machine Learning approaches was not possible. Overall, this review emphasises the clinical relevance of Machine Learning methods for anticancer drug dose optimisation, as many algorithms have shown promising results enabling model-free predictions with the potential to maximise efficacy and minimise toxicity when compared to standard protocols.