Attention Mechanism Research Articles

A Multi-scale Subconvolutional U-Net with Time-Frequency Attention Mechanism for Single Channel Speech Enhancement

A Multi-scale Subconvolutional U-Net with Time-Frequency Attention Mechanism for Single Channel Speech Enhancement

A multi-head self-attention autoencoder network for fault detection of wind turbine gearboxes under random loads*

Wind turbine gearboxes work under random load for extended periods of time, and the fault detection indicator constructed by the existing deep learning models fluctuate constantly due to the load, which is easy to cause frequent false alarms. Therefore, a multihead self-attention autoencoder network is proposed and combined with a dynamic alarm threshold to detect faults in a wind turbine gearbox subjected to random loads. The multiheaded attention mechanism layer enhances the feature-extraction capability of the proposed network by extracting global and local features from input data. Furthermore, to suppress the influence of the random load, a dynamic warning threshold was designed based on the reconstruction error between the inputs and outputs of the proposed network. Finally, the effectiveness of the proposed method was verified using the vibration data of wind turbine gearboxes from an actual wind farm.

MCCM: multi-scale feature extraction network for disease classification and recognition of chili leaves

Currently, foliar diseases of chili have significantly impacted both yield and quality. Despite effective advancements in deep learning techniques for the classification of chili leaf diseases, most existing classification models still face challenges in terms of accuracy and practical application in disease identification. Therefore, in this study, an optimized and enhanced convolutional neural network model named MCCM (MCSAM-ConvNeXt-MSFFM) is proposed by introducing ConvNeXt. The model incorporates a Multi-Scale Feature Fusion Module (MSFFM) aimed at better capturing disease features of various sizes and positions within the images. Moreover, adjustments are made to the positioning, activation functions, and normalization operations of the MSFFM module to further optimize the overall model. Additionally, a proposed Mixed Channel Spatial Attention Mechanism (MCSAM) strengthens the correlation between non-local channels and spatial features, enhancing the model’s extraction of fundamental characteristics of chili leaf diseases. During the training process, pre-trained weights are obtained from the Plant Village dataset using transfer learning to accelerate the model’s convergence. Regarding model evaluation, the MCCM model is compared with existing CNN models (Vgg16, ResNet34, GoogLeNet, MobileNetV2, ShuffleNet, EfficientNetV2, ConvNeXt), and Swin-Transformer. The results demonstrate that the MCCM model achieves average improvements of 3.38%, 2.62%, 2.48%, and 2.53% in accuracy, precision, recall, and F1 score, respectively. Particularly noteworthy is that compared to the original ConvNeXt model, the MCCM model exhibits significant enhancements across all performance metrics. Furthermore, classification experiments conducted on rice and maize disease datasets showcase the MCCM model’s strong generalization performance. Finally, in terms of application, a chili leaf disease classification website is successfully developed using the Flask framework. This website accurately identifies uploaded chili leaf disease images, demonstrating the practical utility of the model.

Sperm YOLOv8E-TrackEVD: A Novel Approach for Sperm Detection and Tracking

Male infertility is a global health issue, with 40–50% attributed to sperm abnormalities. The subjectivity and irreproducibility of existing detection methods pose challenges to sperm assessment, making the design of automated semen analysis algorithms crucial for enhancing the reliability of sperm evaluations. This paper proposes a comprehensive sperm tracking algorithm (Sperm YOLOv8E-TrackEVD) that combines an enhanced YOLOv8 small object detection algorithm (SpermYOLOv8-E) with an improved DeepOCSORT tracking algorithm (SpermTrack-EVD) to detect human sperm in a microscopic field of view and track healthy sperm in a sample in a short period effectively. Firstly, we trained the improved YOLOv8 model on the VISEM-Tracking dataset for accurate sperm detection. To enhance the detection of small sperm objects, we introduced an attention mechanism, added a small object detection layer, and integrated the SPDConv and Detect_DyHead modules. Furthermore, we used a new distance metric method and chose IoU loss calculation. Ultimately, we achieved a 1.3% increase in precision, a 1.4% increase in recall rate, and a 2.0% improvement in mAP@0.5:0.95. We applied SpermYOLOv8-E combined with SpermTrack-EVD for sperm tracking. On the VISEM-Tracking dataset, we achieved 74.303% HOTA and 71.167% MOTA. These results show the effectiveness of the designed Sperm YOLOv8E-TrackEVD approach in sperm tracking scenarios.

Biomedical named entity recognition based on multi-cross attention feature fusion.

Currently, in the field of biomedical named entity recognition, CharCNN (Character-level Convolutional Neural Networks) or CharRNN (Character-level Recurrent Neural Network) is typically used independently to extract character features. However, this approach does not consider the complementary capabilities between them and only concatenates word features, ignoring the feature information during the process of word integration. Based on this, this paper proposes a method of multi-cross attention feature fusion. First, DistilBioBERT and CharCNN and CharLSTM are used to perform cross-attention word-char (word features and character features) fusion separately. Then, the two feature vectors obtained from cross-attention fusion are fused again through cross-attention to obtain the final feature vector. Subsequently, a BiLSTM is introduced with a multi-head attention mechanism to enhance the model's ability to focus on key information features and further improve model performance. Finally, the output layer is used to output the final result. Experimental results show that the proposed model achieves the best F1 values of 90.76%, 89.79%, 94.98%, 80.27% and 88.84% on NCBI-Disease, BC5CDR-Disease, BC5CDR-Chem, JNLPBA and BC2GM biomedical datasets respectively. This indicates that our model can capture richer semantic features and improve the ability to recognize entities.

Fine Segmentation of Chinese Character Strokes Based on Coordinate Awareness and Enhanced BiFPN

Considering the complex structure of Chinese characters, particularly the connections and intersections between strokes, there are challenges in low accuracy of Chinese character stroke extraction and recognition, as well as unclear segmentation. This study builds upon the YOLOv8n-seg model to propose the YOLOv8n-seg-CAA-BiFPN Chinese character stroke fine segmentation model. The proposed Coordinate-Aware Attention mechanism (CAA) divides the backbone network input feature map into four parts, applying different weights for horizontal, vertical, and channel attention to compute and fuse key information, thus capturing the contextual regularity of closely arranged stroke positions. The network’s neck integrates an enhanced weighted bi-directional feature pyramid network (BiFPN), enhancing the fusion effect for features of strokes of various sizes. The Shape-IoU loss function is adopted in place of the traditional CIoU loss function, focusing on the shape and scale of stroke bounding boxes to optimize the bounding box regression process. Finally, the Grad-CAM++ technique is used to generate heatmaps of segmentation predictions, facilitating the visualization of effective features and a deeper understanding of the model’s focus areas. Trained and tested on the public Chinese character stroke datasets CCSE-Kai and CCSE-HW, the model achieves an average accuracy of 84.71%, an average recall rate of 83.65%, and a mean average precision of 80.11%. Compared to the original YOLOv8n-seg and existing mainstream segmentation models like SegFormer, BiSeNetV2, and Mask R-CNN, the average accuracy improved by 3.50%, 4.35%, 10.56%, and 22.05%, respectively; the average recall rates improved by 4.42%, 9.32%, 15.64%, and 24.92%, respectively; and the mean average precision improved by 3.11%, 4.15%, 8.02%, and 19.33%, respectively. The results demonstrate that the YOLOv8n-seg-CAA-BiFPN network can accurately achieve Chinese character stroke segmentation.

Periodic Transformer Encoder for Multi-Horizon Travel Time Prediction

In the domain of Intelligent Transportation Systems (ITS), ensuring reliable travel time predictions is crucial for enhancing the efficiency of transportation management systems and supporting long-term planning. Recent advancements in deep learning have demonstrated the ability to effectively leverage large datasets for accurate travel time predictions. These innovations are particularly vital as they address both short-term and long-term travel demands, which are essential for effective traffic management and scheduled routing planning. Despite advances in deep learning applications for traffic analysis, the dynamic nature of traffic patterns frequently challenges the forecasting capabilities of existing models, especially when forecasting both immediate and future traffic conditions across various time horizons. Additionally, the area of long-term travel time forecasting still remains not fully explored in current research due to these complexities. In response to these challenges, this study introduces the Periodic Transformer Encoder (PTE). PTE is a Transformer-based model designed to enhance traffic time predictions by effectively capturing temporal dependencies across various horizons. Utilizing attention mechanisms, PTE learns from long-range periodic traffic data for handling both short-term and long-term fluctuations. Furthermore, PTE employs a streamlined encoder-only architecture that eliminates the need for a traditional decoder, thus significantly simplifying the model’s structure and reducing its computational demands. This architecture enhances both the training efficiency and the performance of direct travel time predictions. With these enhancements, PTE effectively tackles the challenges presented by dynamic traffic patterns, significantly improving prediction performance across multiple time horizons. Comprehensive evaluations on an extensive real-world traffic dataset demonstrate PTE’s superior performance in predicting travel times over multiple horizons compared to existing methods. PTE is notably effective in adapting to high-variability road segments and peak traffic hours. These results prove PTE’s effectiveness and robustness across diverse traffic environments, indicating its significant contribution to advancing traffic prediction capabilities within ITS.

YOLOv8-PD: an improved road damage detection algorithm based on YOLOv8n model.

Road damage detection is an crucial task to ensure road safety. To tackle the issues of poor performance on multi-scale pavement distresses and high costs in detection task, this paper presents an improved lightweight road damage detection algorithm based on YOLOv8n, named YOLOv8-PD (pavement distress). Firstly, a BOT module that can extract global information of road damage images is proposed to adapt to the large-span features of crack objects. Secondly, the introduction of the large separable kernel attention (LKSA) mechanism enhances the detection accuracy of the algorithm. Then, a C2fGhost block is constructed in the neck network to strengthen the feature extraction of complex road damages while reducing the computational load. Furthermore, we introduced lightweight shared convolution detection head (LSCD-Head) to improve feature expressiveness and reduce the number of parameters. Finally, extensive experiments on the RDD2022 dataset yield a model with parametric and computational quantities of 2.3M and 6.1 GFLOPs, which are only 74.1% and 74.3% of the baseline, and the mAP reaches an improvement of 1.4 percentage points from the baseline. In addition, experimental results on the RoadDamage dataset show that the mAP increased by 4.2% and this algorithm has good robustness. This method can provide a reference for the automatic detection method of pavement distress.

Self-Supervised Image Denoising of Third Harmonic Generation Microscopic Images of Human Glioma Tissue by Transformer-based Blind Spot (TBS) Network.

Third harmonic generation (THG) microscopy shows great potential for instant pathology of brain tumor tissue during surgery. However, due to the maximal permitted exposure of laser intensity and inherent noise of the imaging system, the noise level of THG images is relatively high, which affects subsequent feature extraction analysis. Denoising THG images is challenging for modern deep-learning based methods because of the rich morphologies contained and the difficulty in obtaining the noise-free counterparts. To address this, in this work, we propose an unsupervised deep-learning network for denoising of THG images which combines a self-supervised blind spot method and a U-shape Transformer using a dynamic sparse attention mechanism. The experimental results on THG images of human glioma tissue show that our approach exhibits superior denoising performance qualitatively and quantitatively compared with previous methods. Our model achieves an improvement of 2.47-9.50 dB in SNR and 0.37-7.40 dB in CNR, compared to six recent state-of-the-art unsupervised learning models including Neighbor2Neighbor, Blind2Unblind, Self2Self+, ZS-N2N, Noise2Info and SDAP. To achieve an objective evaluation of our model, we also validate our model on public datasets including natural and microscopic images, and our model shows a better denoising performance than several recent unsupervised models such as Neighbor2Neighbor, Blind2Unblind and ZS-N2N. In addition, our model is nearly instant in denoising a THG image, which has the potential for real-time applications of THG microscopy.

Structural acceleration response reconstruction based on BiLSTM network and multi-head attention mechanism

The effectiveness of structural health monitoring often relies on the precise analysis of structural response data, with data completeness directly impacting the performance of monitoring systems. Addressing the common issue of data loss in structural health monitoring systems, this study proposes a Bidirectional Long Short-Term Memory (BiLSTM) network based on a multi-head attention mechanism for the reconstruction of missing structural acceleration responses. Firstly, employing a two-layer BiLSTM network to establish an encoder-decoder framework for extracting spatio-temporal correlation features among sensor data. Subsequently, innovatively embedding a multi-head attention mechanism into the network framework enhances the modeling capability of long-distance input elements and establishes dependencies among different sensors, thereby enabling the network to better capture the global features of input information. The proposed method undergoes validation through a numerical example of a simply supported beam, practical monitoring data on the Hardanger Bridge from the Norwegian University of Science and Technology, and an experimental study of steel frame structure from our laboratory. These validations are compared with alternative response reconstruction models. The experimental results make obvious that the proposed method demonstrates superior accuracy in response reconstruction. Additionally, the suitability of this method for modal identification is validated. Through the utilization of reconstructed responses, the method effectively discerns the structural natural frequencies with accuracy.

Emotion Recognition Based on Handwriting Using Generative Adversarial Networks and Deep Learning

The quality of people’s lives is closely related to their emotional state. Positive emotions can boost confidence and help overcome difficulties, while negative emotions can harm both physical and mental health. Research has shown that people’s handwriting is associated with their emotions. In this study, audio-visual media were used to induce emotions, and a dot-matrix digital pen was used to collect neutral text data written by participants in three emotional states: calm, happy, and sad. To address the challenge of limited samples, a novel conditional table generative adversarial network called conditional tabular-generative adversarial network (CTAB-GAN) was used to increase the number of task samples, and the recognition accuracy of task samples improved by 4.18%. The TabNet (a neural network designed for tabular data) with SimAM (a simple, parameter-free attention module) was employed and compared with the original TabNet and traditional machine learning models; the incorporation of the SimAm attention mechanism led to a 1.35% improvement in classification accuracy. Experimental results revealed significant differences between negative (sad) and nonnegative (calm and happy) emotions, with a recognition accuracy of 80.67%. Overall, this study demonstrated the feasibility of emotion recognition based on handwriting with the assistance of CTAB-GAN and SimAm-TabNet. It provides guidance for further research on emotion recognition or other handwriting-based applications.

A fiber recognition framework based on multi-head attention mechanism

Convolutional neural networks have been extensively studied in textile fiber recognition. However, the down-sampling performed by the convolutional and pooling layers on the extracted features result in a significant loss of fine-grained fiber features. To address this issue, a multi-head attention framework for fiber recognition has been proposed. First, textile surface images are captured using optical magnifiers and smart devices such as smartphones. Second, fiber features and label embeddings are learned separately by multi-head self-attention. Finally, the label embeddings are used to query the presence of fiber types, and pool type-related features in the multi-head cross-attention module to classify fibers. The experimental results demonstrate that the proposed method performs exceptionally well on the textile surface image dataset, with a mean average precision accuracy improvement of 6% compared with the convolutional neural network-based fiber recognition method Cu-Net, and a 5% improvement compared with the fiber recognition method FiberCT combining convolutional neural networks and attention mechanisms. It is worth mentioning that the fiber images captured at 200× magnification in the collected dataset are most favorable for models to recognize fiber. The proposed method achieves a mean average precision fiber recognition accuracy of 80.2% on images at 200× magnification.

Meta learning based residual network for industrial production quality prediction with limited data

Due to the challenge of collecting a substantial amount of production-quality data in real-world industrial settings, the implementation of production quality prediction models based on deep learning is not effective. To achieve the goal of predicting production quality with limited data and address the issue of model degradation in the training process of deep learning networks, we propose Meta-Learning based on Residual Network (MLRN) models for production quality prediction with limited data. Firstly, the MLRN model is trained on a variety of learning tasks to acquire knowledge for predicting production quality. Furthermore, to obtain more features with limited data and avoid the issues of gradient disappearing or exploding in deep network training, the enhanced residual network with the effective channel attention (ECA) mechanism is chosen as the basic network structure of MLRN. Additionally, a multi-batch and multi-task data input approach is implemented to prevent overfitting. Finally, the availability of the MLRN model is demonstrated by comparing it with other models using both numerical and graphical datasets.

An Interpolation and Prediction Algorithm for XCO2 Based on Multi-Source Time Series Data

Carbon satellites are an important observation tool for analyzing ground carbon emission. From the perspective of the Earth’s scale, the spatiotemporal sparse characteristics of raw data observed from carbon satellite requires the accurate interpolation of data, and based on only this work, people predict future carbon emission trends and formulate appropriate management and conservation strategies. The existing research work has not fully considered the close correlation between data and seasons, as well as the characteristics accumulated over a long time scale. In this paper, firstly, by employing extreme random forests and auxiliary data, we reconstruct a daily average CO2 dataset at a resolution of 0.25°, and achieve a validated determination coefficient of 0.92. Secondly, introducing technologies such as Time Convolutional Networks (TCN), Channel Attention Mechanism (CAM), and Long Short-Term Memory networks (LSTM), we conduct atmospheric CO2 concentration interpolation and predictions. When conducting predictive analysis for the Yangtze River Delta region, we train the model by using quarterly data from 2016 to 2020; the correlation coefficient in summer is 0.94, and in winter it is 0.91. These experimental data indicate that compared to other algorithms, this algorithm has a significantly better performance.

An interpretable and high-precision method for predicting landslide displacement using evolutionary attention mechanism

An interpretable and high-precision method for predicting landslide displacement using evolutionary attention mechanism

Application of an Improved Convolutional Neural Network Algorithm in Text Classification

This paper proposes a text classification model based on a combination of a convolutional neural network (CNN) and a support vector machine (SVM) using Amazon review polarity, TREC, and Kaggle as experimental data. By adding an attention mechanism to simplify the parameters and using the classifier based on SVM to replace the Softmax layer, the extraction effect of feature words is improved and the problem of weak generalization ability of the CNN model is solved. Simulation experiments show that the proposed algorithm performs better in precision rate, recall rate, F1 value, and training time compared with CNN, RNN, BERT and term frequency-inverse document frequency (TF-IDF).

Ultrasound segmentation analysis via distinct and completed anatomical borders.

Segmenting ultrasound images is important for precise area and/or volume calculations, ensuring reliable diagnosis and effective treatment evaluation for diseases. Recently, many segmentation methods have been proposed and shown impressive performance. However, currently, there is no deeper understanding of how networks segment target regions or how they define the boundaries. In this paper, we present a new approach that analyzes ultrasound segmentation networks in terms of learned borders because border delimitation is challenging in ultrasound. We propose a way to split the boundaries for ultrasound images into distinct and completed. By exploiting the Grad-CAM of the split borders, we analyze the areas each network pays attention to. Further, we calculate the ratio of correct predictions for distinct and completed borders. We conducted experiments on an in-house leg ultrasound dataset (LEG-3D-US) as well as on two additional public datasets of thyroid, nerves, and one private for prostate. Quantitatively, the networks exhibit around 10% improvement in handling completed borders compared to distinct borders. Similar to doctors, the network struggles to define the borders in less visible areas. Additionally, the Seg-Grad-CAM analysis underscores how completion uses distinct borders and landmarks, while distinct focuses mainly on the shiny structures. We also observe variations depending on the attention mechanism of each architecture. In this work, we highlight the importance of studying ultrasound borders differently than other modalities such as MRI or CT. We split the borders into distinct and completed, similar to clinicians, and show the quality of the network-learned information for these two types of borders. Additionally, we open-source a 3D leg ultrasound dataset to the community https://github.com/Al3xand1a/segmentation-border-analysis .

A Federated Social Recommendation Approach with Enhanced Hypergraph Neural Network

In recent years, the development of online social network platforms has led to increased research efforts in social recommendation systems. Unlike traditional recommendation systems, social recommendation systems utilize both user-item interactions and user-user social relations to recommend relevant items, taking into account social homophily and social influence. Graph neural network (GNN) based social recommendation methods have been proposed to model these item interactions and social relations effectively. However, existing GNN-based methods rely on centralized training, which raises privacy concerns and faces challenges in data collection due to regulations and privacy restrictions. Federated learning has emerged as a privacy-preserving alternative. Combining federated learning with GNN-based methods for social recommendation can leverage their respective advantages, but it also introduces new challenges: 1) existing federated recommendation systems often lack the capability to process heterogeneous data, such as user-item interactions and social relations; 2) due to the sparsity of data distributed across different clients, capturing the higher-order relationship information among users becomes challenging and is often overlooked by most federated recommendation systems. To overcome these challenges, we propose a federated social recommendation approach with enhanced hypergraph neural network. We introduce hypergraph graph neural networks (HGNN) to learn user and item embeddings in federated recommendation systems, leveraging the hypergraph structure to address the heterogeneity of data. Based on carefully crafted triangular motifs, we merge user and item nodes to construct hypergraphs on local clients, capturing specific triangular relations. Multiple HGNN channels are used to encode different categories of high-order relations, and an attention mechanism is applied to aggregate the embedded information from these channels. Our experiments on real-world social recommendation datasets demonstrate the effectiveness of the proposed approach. Extensive experiment results on three publicly available datasets validate the effectiveness of the proposed method.

Research on Portable Intelligent Terminal and APP Application Analysis and Intelligent Monitoring Method of College Students' Health Status

As a carrier of college students' health status monitoring, portable intelligent terminal APP, the study of its APP application analysis not only provides a new way for college students' extracurricular physical exercise, guides college students to actively participate in extracurricular physical activities using intelligent terminal APP software, but also promotes college students' physical health monitoring and enhancement in various aspects. Aiming at the current portable terminal APP college students' health monitoring application analysis method research exists low precision, real-time poor and other problems, through the analysis of the basic functional framework and functional characteristics of the portable intelligent terminal APP, the establishment of the portable intelligent terminal APP analysis index system applied to college students' health monitoring, combined with the heuristic optimisation algorithm and the improvement of deep learning algorithms, the construction of the marine predator based heuristic optimisation algorithm and the attention mechanism to improve the gating control loop. Combining the heuristic optimisation algorithm and the improved deep learning algorithm, we construct the portable intelligent terminal APP application analysis method for college students' health monitoring based on the marine predator heuristic optimisation algorithm and the attention mechanism improved gated recurrent unit neural network. Through simulation analysis, the results show that the proposed method meets the real-time requirements while improving the prediction accuracy of the portable smart terminal APP application analysis method, and significantly improves the efficiency of portable smart terminal APP analysis.

Time–Frequency Domain Seismic Signal Denoising Based on Generative Adversarial Networks

Existing deep learning-based seismic signal denoising methods primarily operate in the time domain. Those methods are ineffective when noise overlaps with the seismic signal in the time domain. Time–frequency domain-based deep learning methods are relatively rare and usually employ single loss function, resulting in suboptimal performance on low SNR signals and potential damage to P wave. This paper proposes a method based on generative adversarial networks (GANs). Compared to convolutional neural networks, the discriminator in GANs helps retain more true signal details by judging denoising performance. Additionally, an attention mechanism is introduced to fully extract signal features, and a perceptual loss is employed to evaluate the difference between the denoised result and the target’s high-level features. Experimental results show that this method can effectively improve SNR and ensure that the denoised result is close to the true signal. Furthermore, by comparing DeepDenoiser and ARDU, it is proven that the proposed method achieves better denoising performance, especially for low SNR signals, while causing less damage to the seismic signals.