Attention Mechanism Research Articles

Intelligent road surface state recognition method based on multi-layer attention residual network

Abstract Data-driven road surface state recognition enhances the efficiency and accuracy of road management, contributing to increased safety and reliability in road traffic. However, traditional machine learning and deep learning-based road surface state recognition typically rely on extensive data for model training, making it challenging to adapt to complex tasks in diverse scenarios. Therefore, this paper proposes a Multi-layer Attention Residual Network (MARN)-based intelligent road surface state recognition method. First, a Residual Convolutional Neural Network (ResNet) is constructed as the backbone model of MARN to mitigate the gradient vanishing problem, allowing the network to extract deeper features. Subsequently, an adaptive multi-layer attention mechanism is introduced in each convolutional layer, enabling adaptive weighting of each feature channel in the dataset to enhance the model’s focus on different features for better feature extraction. Furthermore, a cosine annealing learning rate adjuster is designed to improve the accuracy, robustness, and convergence during the model training process. Finally, the proposed MARN is validated using an image dataset containing six different road surface states. Comparative studies are conducted on the recognition accuracy of the proposed MARN, original ResNet, Visual Geometry Group network (VGG16), and Convolutional Neural Network (CNN). The impact of different batch sizes on the convergence speed of road surface state recognition under MARN is also analyzed. Results demonstrate that MARN achieves a training set accuracy of over 95%, surpassing VGG16 and CNN with accuracies below 85%. Compared to ResNet, MARN exhibits a 1.3% higher training set accuracy and a 0.25 lower validation set loss, showcasing superior accuracy and robustness in road surface state recognition.

Adaptive occlusion object detection algorithm based on OL-IoU.

The continuous advancement of autonomous driving technology imposes higher demands on the accuracy of target detection in complex environments, particularly when traffic targets are occluded. Existing algorithms still face significant challenges in detection accuracy and real-time performance under such conditions. To address this issue, this paper proposes an improved YOLOX algorithm based on adaptive deformable convolution, named OCC-YOLOX. This algorithm enhances the feature extraction network's ability to focus on occluded targets by incorporating a coordinate attention mechanism. Additionally, it introduces the Overlapping IoU (OL-IoU) loss function to optimize the overlap between predicted and ground truth bounding boxes, thereby improving detection accuracy. Furthermore, the adoption of Fast Spatial Pyramid Pooling (Fast SPP) reduces computational complexity while maintaining real-time performance. Experiments on fused public datasets demonstrate that OCC-YOLOX achieves improvements in accuracy, recall, and average precision by 2.76%, 1.25%, and 1.92%, respectively. In addition to testing on the KITTI, CityPersons, and BDD100K datasets, the effectiveness of the OCC-YOLOX algorithm is further validated through comparisons with self-collected occlusion scene data. The experimental results indicate that OCC-YOLOX outperforms existing mainstream detection algorithms, particularly in handling complex occlusion scenarios, significantly enhancing the accuracy and efficiency of object detection. This study provides new insights for addressing the challenges of occluded target detection in intelligent transportation systems.

Submarine Landslide Identification Based on Improved DeepLabv3 with Spatial and Channel Attention

As one of the most destructive, hazardous, and frequent marine geohazards, correctly recognizing submarine landslides holds substantial importance for regional risk assessment, disaster prevention, and marine resource development. Many conventional approaches to prediction and mapping necessitate the involvement of expert insights, oversight, and extensive field investigations, which can result in significant time and effort invested in the prediction process. This paper focuses on employing a deep neural network semantic segmentation technique to detect submarine landslides to replace previous methods, such as numerical analysis and physical modeling, to predict and identify the landslide areas quickly. The peripheral zone of the western Iberian Sea is selected as the study area. Since the neural network image recognition task usually requires RGB images as input data, factors such as slope, hillshade, and elevation extracted from digital elevation model (DEM) data are used to synthesize RGB images through band synthesis methods, and the number and diversity of data are increased utilizing data enhancement. Based on the classical semantic segmentation model DeepLabV3, this paper proposes an improved deep learning method, which strengthens the ability of model feature extraction for complex situations by adding an attention mechanism module, improving the spatial pyramid pooling module, and improving the landslide intersection over union metric from 0.4257 to 0.5219 and the F1-score metric from 0.609 to 0.6631 to achieve effective identification of submarine landslides.

Semantic Segmentation of Corn Leaf Blotch Disease Images Based on U-Net Integrated with RFB Structure and Dual Attention Mechanism

Northern corn leaf blight (NCLB) is caused by a fungus and can be susceptible to the disease throughout the growing period of corn, posing a significant impact on corn yield. Aiming at the problems of under-segmentation, over-segmentation, and low segmentation accuracy in the traditional segmentation model of northern corn leaf blight, this study proposes a segmentation method based on an improved U-Net network model. By introducing a convolutional layer and maximum pooling layer to a VGG19 network, the channel attention module and spatial attention module (CBAM) are fused, and the squeeze excitation (SE) attention mechanism is combined. This enhances image feature decoding, integrates feature maps of each layer, strengthens the feature extraction process, expands the sensory fields and aggregates context information, and reduces the loss of location and dense semantic information caused by the pooling operation. Findings from the study show that the proposed NCLB-Net has significantly improved the MIoU and PA indexes, reaching 92.43% and 94.71%, respectively. Compared with the traditional methods, U-Net, SETR, DAnet, OCnet, PSPNet, etc., the MIoU is improved by 20.81%, 16.10%, 9.79%, 5.27%, and 11.06%, and the PA is improved by 11.49%, 8.18%, 9.54%, 13.11%, and 6.26%, respectively.

An Innovative Attention-based Triplet Deep Hashing Approach to Retrieve Histopathology Images.

Content-based histopathology image retrieval (CBHIR) can assist in the diagnosis of different diseases. The retrieval procedure can be complex and time-consuming if high-dimensional features are required. Thus, hashing techniques are employed to address these issues by mapping the feature space into binary values of varying lengths. The performance of deep hashing approaches in image retrieval is often superior to that of traditional hashing methods. Among deep hashing approaches, triplet-based models are typically more effective than pairwise ones. Recent studies have demonstrated that incorporating the attention mechanism into a deep hashing approach can improve its effectiveness in retrieving images. This paper presents an innovative triplet deep hashing strategy based on the attention mechanism for retrieving histopathology images, called histopathology attention triplet deep hashing (HATDH). Three deep attention-based hashing models with identical architectures and weights are employed to produce binary values. The proposed attention module can aid the models in extracting features more efficiently. Moreover, we introduce an improved triplet loss function considering pair inputs separately in addition to triplet inputs for increasing efficiency during the training and retrieval steps. Based on experiments conducted on two public histopathology datasets, BreakHis and Kather, HATDH significantly outperforms state-of-the-art hashing algorithms.

DeepMEns: an ensemble model for predicting sgRNA on-target activity based on multiple features.

The CRISPR/Cas9 system developed from Streptococcus pyogenes (SpCas9) has high potential in gene editing. However, its successful application is hindered by the considerable variability in target efficiencies across different single guide RNAs (sgRNAs). Although several deep learning models have been created to predict sgRNA on-target activity, the intrinsic mechanisms of these models are difficult to explain, and there is still scope for improvement in prediction performance. To overcome these issues, we propose an ensemble interpretable model termed DeepMEns based on deep learning to predict sgRNA on-target activity. By using five different training and validation datasets, we constructed five sub-regressors, each comprising three parts. The first part uses one-hot encoding, wherein 0-1 representation of the secondary structure is used as the input to the convolutional neural network (CNN) with Transformer encoder. The second part uses the DNA shape feature matrix as the input to the CNN with Transformer encoder. The third part uses positional encoding feature matrices as the proposed input into a long short-term memory network with an attention mechanism. These three parts are concatenated through the flattened layer, and the final prediction result is the average of the five sub-regressors. Extensive benchmarking experiments indicated that DeepMEns achieved the highest Spearman correlation coefficient for 6 of 10 independent test datasets as compared to previous predictors, this finding confirmed that DeepMEns can accomplish state-of-the-art performance. Moreover, the ablation analysis also indicated that the ensemble strategy may improve the performance of the prediction model.

An Automated Diagnosis of Parkinson's Disease from MRI Scans Based on Enhanced Residual Dense Network with Attention Mechanism.

The increasing prevalence of neurodegenerative diseases has recently heightened interest in research on early diagnosis of these diseases. Parkinson's disease (PD), among the most prominent of these conditions, is a neurological disorder causing the loss of nerve cells and significantly affecting movement control. Detection of PD in early stages is of critical importance to prevent the progression of the disease and improve treatment processes. The aim of the current study is to develop a deep learning model that can perform accurate classification for early diagnosis of PD from MRI images. In this study, a densely connected feature fusion network with residual learning is designed to diagnose PD patients. The designed network consists of a serial dense block with skip connections and efficient attention mechanisms. In this architecture, squeeze-excitation (SE) blocks with ResNeXt (SE-ResNeXt block) modules are utilized to extract distinctive and high-level features. In the experiments, a publicly available T2-weighted MRI dataset is used, and an offline augmentation process is applied to limited data to increase the generalization ability and classification performance. The proposed method is evaluated and compared with current state-of-the-art deep learning methods. The obtained results show that the proposed model gives higher classification performance with an overall accuracy of 94.44%, precision of 91.67%, sensitivity of 91.67%, specificity of 95.83%, F1-score of 91.67%, and Matthew's correlation coefficient of 87.50% for the PD and healthy control subjects.

A Lightweight Cotton Field Weed Detection Model Enhanced with EfficientNet and Attention Mechanisms

Cotton is a crucial crop in the global textile industry, with major production regions including China, India, and the United States. While smart agricultural mechanization technologies, such as automated irrigation and precision pesticide systems, have improved crop management, weeds remain a significant challenge. These weeds not only compete with cotton for nutrients but can also serve as hosts for diseases, affecting both cotton yield and quality. Existing weed detection models perform poorly in the complex environment of cotton fields, where the visual features of weeds and crops are similar and often overlap, resulting in low detection accuracy. Furthermore, real-time deployment on edge devices is difficult. To address these issues, this study proposes an improved lightweight weed detection model, YOLO-WL, based on the YOLOv8 architecture. The model leverages EfficientNet to reconstruct the backbone, reducing model complexity and enhancing detection speed. To compensate for any performance loss due to backbone simplification, CA (cross-attention) is introduced into the backbone, improving feature sensitivity. Finally, AFPN (Adaptive Feature Pyramid Network) and EMA (efficient multi-scale attention) mechanisms are integrated into the neck to further strengthen feature extraction and improve weed detection accuracy. At the same time, the model maintains a lightweight design suitable for deployment on edge devices. Experiments on the CottonWeedDet12 dataset show that the YOLO-WL model achieved an mAP of 92.30%, reduced the detection time per image by 75% to 1.9 ms, and decreased the number of parameters by 30.3%. After TensorRT optimization, the video inference time was reduced from 23.134 ms to 2.443 ms per frame, enabling real-time detection in practical agricultural environments.

Real-time fault detection for IIoT facilities using GA-Att-LSTM based on edge-cloud collaboration

With the rapid development of Industrial Internet of Things (IIoT) technology, various IIoT devices are generating large amounts of industrial sensor data that are spatiotemporally correlated and heterogeneous from multi-source and multi-domain. This poses a challenge to current detection algorithms. Therefore, this paper proposes an improved long short-term memory (LSTM) neural network model based on the genetic algorithm, attention mechanism and edge-cloud collaboration (GA-Att-LSTM) framework is proposed to detect anomalies of IIoT facilities. Firstly, an edge-cloud collaboration framework is established to real-time process a large amount of sensor data at the edge node in real time, which reduces the time of uploading sensor data to the cloud platform. Secondly, to overcome the problem of insufficient attention to important features in the input sequence in traditional LSTM algorithms, we introduce an attention mechanism to adaptively adjust the weights of important features in the model. Meanwhile, a genetic algorithm optimized hyperparameters of the LSTM neural network is proposed to transform anomaly detection into a classification problem and effectively extract the correlation of time-series data, which improves the recognition rate of fault detection. Finally, the proposed method has been evaluated on a publicly available fault database. The results indicate an accuracy of 99.6%, an F1-score of 84.2%, a precision of 89.8%, and a recall of 77.6%, all of which exceed the performance of five traditional machine learning methods.

I-DINO: High-Quality Object Detection for Indoor Scenes

Object Detection in Complex Indoor Scenes is designed to identify and categorize objects in indoor settings, with applications in areas such as smart homes, security surveillance, and home service robots. It forms the basis for advanced visual tasks including visual question answering, video description generation, and instance segmentation. Nonetheless, the task faces substantial hurdles due to background clutter, overlapping objects, and significant size differences. To tackle these challenges, this study introduces an indoor object detection approach utilizing an enhanced DINO framework. To cater to the needs of indoor object detection, an Indoor-COCO dataset was developed from the COCO object detection dataset. The model incorporates an advanced Res2net as the backbone feature extraction network, complemented by a deformable attention mechanism to better capture detailed object features. An upgraded Bi-FPN module is employed to replace the conventional feature fusion module, and SIoU loss is utilized to expedite convergence. The experimental outcomes indicate that the refined model attains an mAP of 62.3%, marking a 5.2% improvement over the baseline model. These findings illustrate that the DINO-based indoor object detection model exhibits robust generalization abilities and practical utility for multi-scale object detection in complex environments.

Research on recognition of slippery road surface and collision warning system based on deep learning.

Aiming at the problems of slow detection speed, large prediction error and weak environmental adaptability of current vehicle collision warning system, this paper proposes a recognition method of slippery road surface and collision warning system based on deep learning. Firstly, this paper uses the on-board camera to monitor the environment and road conditions in front of the vehicle in real time, and a residual network model FS-ResNet50 is proposed, which integrated SE attention mechanism and multi-level feature information based on the traditional ResNet50 model. The FS-ResNet50 model is used to identify the slippery states of the current road, such as wet and snowy. Secondly, the yolov5 algorithm is used to detect the position of the vehicle in front, and a driving safety distance model with adaptive traffic environment characteristics is established based on different road conditions and driving conditions, and an early warning area that dynamically changed with the speed and the road slippery states is generated. Finally, according to the relationship between the warning area and the position of the vehicle, the possible collision is predicted and timely warned. Experimental results show that the method proposed in this paper improves the overall warning accuracy by 6.72% and reduces the warning false alarm rate for oncoming traffic on both sides by 16.67% compared with the traditional collision warning system. It can ensure safe driving, especially in bad weather conditions and has a high application value.

Multi-Scale Feature Fusion Enhancement for Underwater Object Detection

Underwater object detection (UOD) presents substantial challenges due to the complex visual conditions and the physical properties of light in underwater environments. Small aquatic creatures often congregate in large groups, further complicating the task. To address these challenges, we develop Aqua-DETR, a tailored end-to-end framework for UOD. Our method includes an align-split network to enhance multi-scale feature interaction and fusion for small object identification and a distinction enhancement module using various attention mechanisms to improve ambiguous object identification. Experimental results on four challenging datasets demonstrate that Aqua-DETR outperforms most existing state-of-the-art methods in the UOD task, validating its effectiveness and robustness.

MonoFG: Monocular 3D Object Detection with Knowledge Distillation for Human-Centric Autonomous Driving Systems

Monocular 3D object detection is essential for identifying objects in road images, thus offering valuable environmental perception data that are crucial for human-centric autonomous driving systems. However, due to the inherent limitations of camera imaging, obtaining precise depth information from images alone is challenging, which hampers the accuracy of within-scene object localization. In this paper, we introduce a monocular 3D object detection method called MonoFG that uses knowledge distillation with separated foreground and background components to improve the accuracy of object localization. First, detached foreground and background distillation processes can strategically leverage the distinct positional information acquired from each location to optimize the produced global distillation effects. This step serves as the foundation for the subsequent feature and response distillation process, which focuses on the distilled foreground and background rather than isolated object distillation. Second, triple attention mechanism-based feature distillation intensifies the feature imitation and feature representation capabilities of the student network. Spatial and channel attention mechanisms encourage the student network to capture crucial pixels and channels from the teacher network, whereas a self-attention mechanism globally transfers the learned relationships between pixels. Third, localization error-based response distillation facilitates a clearer transfer of positional information from the teacher network to the student network. Only when the positioning ability of the teacher network exceeds that of the student network can knowledge be comprehensively distilled across both the foreground and background. Therefore, the distillation process is constrained to specific content, which is delineated by positioning errors that serve as the boundaries. Finally, experiments conducted on the KITTI benchmark dataset demonstrate that our method outperforms many well-known baseline methods in several representative evaluation tasks (e.g., 3D object detection and bird's-eye view (BEV) detection) involving human-centric autonomous driving systems.

Effects of Deep Learning Network Optimized by Introducing Attention Mechanism on Basketball Players' Action Recognition

Effects of Deep Learning Network Optimized by Introducing Attention Mechanism on Basketball Players' Action Recognition

AFF_CGE: Combined Attention-Aware Feature Fusion and Communication Graph Embedding Learning for Detecting Encrypted Malicious Traffic

While encryption enhances data security, it also presents significant challenges for network traffic analysis, especially in detecting malicious activities. To tackle this challenge, this paper introduces combined Attention-aware Feature Fusion and Communication Graph Embedding Learning (AFF_CGE), an advanced representation learning framework designed for detecting encrypted malicious traffic. By leveraging an attention mechanism and graph neural networks, AFF_CGE extracts rich semantic information from encrypted traffic and captures complex relations between communicating nodes. Experimental results reveal that AFF_CGE substantially outperforms traditional methods, improving F1-scores by 5.3% through 22.8%. The framework achieves F1-scores ranging from 0.903 to 0.929 across various classifiers, exceeding the performance of state-of-the-art techniques. These results underscore the effectiveness and robustness of AFF_CGE in detecting encrypted malicious traffic, demonstrating its superior performance.

DESENet: a bilateral network with detail-enhanced semantic encoder for real-time semantic segmentation

Abstract Real-time semantic segmentation is widely used in various domains such as autonomous driving. Most real-time semantic segmentation networks adopt an encoder-decoder structure. During encoding, feature maps undergo multiple downsampling stages to enhance computational efficiency and enlarge the receptive field. However, this process may lead to the loss of detailed information, which may not be fully recovered during upsampling. Moreover, upsampling semantic features may cause blurring due to the lack of spatial detail information. To mitigate these issues, we use spatially guided cross-resolution self-attention to improve the upsampling of the semantic features by supplementing detailed information from the spatial branch. Furthermore, we incorporate an inductive bias into the cross-resolution attention mechanism to enhance its ability to learn generalized features. Additionally, we design a semantic feature extraction block, and a spatial feature extraction branch to construct a lightweight backbone. The results on Cityscapes and CamVid show that the proposed model achieves a good balance between accuracy and efficiency. Specifically, it obtains 74.2 and 71.5 mIoU on the two test datasets with 1.31M parameters, respectively. Code is available on https://github.com/clearwater753/DECENet.

FA-HRNet: A New Fusion Attention Approach for Vegetation Semantic Segmentation and Analysis

Semantic segmentation of vegetation in aerial remote sensing images is a critical aspect of vegetation mapping. Accurate vegetation segmentation effectively informs real-world production and construction activities. However, the presence of species heterogeneity, seasonal variations, and feature disparities within remote sensing images poses significant challenges for vision tasks. Traditional machine learning-based methods often struggle to capture deep-level features for the segmentation. This work proposes a novel deep learning network named FA-HRNet that leverages the fusion of attention mechanism and a multi-branch network structure for vegetation detection and segmentation. Quantitative analysis from multiple datasets reveals that our method outperforms existing approaches, with improvements in MIoU and PA by 2.17% and 4.85%, respectively, compared with the baseline network. Our approach exhibits significant advantages over the other methods regarding cross-region and cross-scale capabilities, providing a reliable vegetation coverage ratio for ecological analysis.

PCNet: a human pose compensation network based on incremental learning for sports actions estimation

Human pose estimation has a wide range of applications. Existing methods perform well in conventional domains, but there are certain defects when they are applied to sports activities. The first is lack of estimation of the extremity posture, making it impossible to comprehensively evaluate the movement posture; the second is insufficient occlusion handling. Therefore, we propose a human pose compensation network based on incremental learning, which obtains shared weights to extract detailed features under the premise of limited extremity training data. We propose a higher-order feature compensator (HOF-compensator) to embed the attributes of the extremity into the torso and limbs topology structure, building a complete higher-order feature. In addition, to improve the occlusion handling performance, we propose an occlusion feature enhancement attention mechanism (OFE-attention) that can identify occluded keypoints and enhance attention to occlusion areas. We design comparative experiments on three public datasets and a self-built sports dataset, achieving the highest mean accuracy among all comparative methods. In addition, we design a series of ablation analysis and visualization displays to verify that our method performs best in sports pose estimation.

Pulmonary CT Registration Network Based on Deformable Cross Attention.

Current Transformer structure utilizes the self-attention mechanism to model global contextual relevance within image, which makes an impact on medical image registration. However, the use of Transformer in handling large deformation lung CT registration is relatively straightforwardly. These models only focus on single image feature representation neglecting to employ attention mechanism to capture the across image correspondence. This hinders further improvement in registration performance. To address the above limitations, we propose a novel registration method in a cascaded manner, Cascaded Swin Deformable Cross Attention Transformer based U-shape structure (SD-CATU), to address the challenge of large deformations in lung CT registration. In SD-CATU, we introduce a Cross Attention-based Transformer (CAT) block that incorporates the Shifted Regions Multihead Cross-attention (SR-MCA) mechanism to flexibly exchange feature information and thus reduce the computational complexity. Besides, a consistency constraint in the loss function is used to ensure the preservation of topology and inverse consistency of the transformations. Experiments with public lung datasets demonstrate that the Cascaded SD-CATU outperforms current state-of-the-art registration methods (Dice Similarity Coefficient of 93.19% and Target registration error of 0.98mm). The results further highlight the potential for obtaining excellent registration accuracy while assuring desirable smoothness and consistency in the deformed images.

Attention-Based Hydrogen Refueling Imputation Model for Efficient Hydrogen Refueling Stations

During hydrogen refueling, the data values determining the state of charge (SoC) of a vehicle can be missing due to internal and external factors. This causes inaccurate SoC estimation, resulting in oversupply or undersupply. To overcome this issue, an attention-based hydrogen refueling imputation (AHRI) model, which restores missing values, is proposed in this paper. In particular, considering that data variables can vary depending on the environmental conditions and equipment in a hydrogen refueling station (HRS), we use the attention mechanism. It determines the primary features, which improves the predictive performance and helps adapt to new conditions. Using the observed data during hydrogen refueling, we train the proposed AHRI model and verify its efficacy. Experimental results show that the proposed AHRI model outperforms existing imputation models significantly. Here, AHRI achieves 0.95 and 0.82 in terms of R2 when 20% and 40% of the values are missing, respectively. These results indicate that the proposed model can be used to solve the data missing problems in HSRs.