Attention Mechanism Research Articles

Rolling Bearing Fault Diagnosis Model Based on External Attention Integrated Convolutional Neural Network under Imbalanced Data Conditions

Abstract Rolling bearings are essential components in numerous mechanical systems, and their failure can result in considerable downtime and expensive repairs. Therefore, accurate and timely fault diagnosis is vital for effective predictive maintenance and overall reliability. Traditional diagnostic methods often struggle with complex and non-stationary signals, compounded by issues of data imbalance in 
 realworld scenarios. A method for diagnosing rolling bearing faults has been developed in this paper utilizing External Attention (EA), Convolutional Neural Networks (CNN), and Continuous Wavelet Transform (CWT), specifically addressing the challenge of imbalanced sample data. This approach offers significant advantages, including a reduction in complexity by eliminating the need for data augmentation and leveraging external attention for enhanced feature extraction from samples. Compared to other attention mechanisms, this method demonstrates outstanding performance on both training and testing sets with imbalanced samples, exhibiting minimal overfitting tendencies. The proposed CWT-EACNN method effectively addresses the challenge of imbalanced sample data in rolling bearing fault diagnosis, demonstrating exceptional performance and reduced complexity.

Large-Kernel Central Block Masked Convolution and Channel Attention-Based Reconstruction Network for Anomaly Detection of High-Resolution Hyperspectral Imagery

In recent years, the rapid advancement of drone technology has led to an increasing use of drones equipped with hyperspectral sensors for ground imaging. Hyperspectral data captured via drones offer significantly higher spatial resolution, but this also introduces more complex background details and larger target scales in high-resolution hyperspectral imagery (HRHSI), posing substantial challenges for hyperspectral anomaly detection (HAD). Mainstream reconstruction-based deep learning methods predominantly emphasize spatial local information in hyperspectral images (HSIs), relying on small spatial neighborhoods for reconstruction. As a result, large anomalous targets and background details are often well reconstructed, leading to poor anomaly detection performance, as these targets are not sufficiently distinguished from the background. To address these limitations, we propose a novel HAD network for HRHSI based on large-kernel central block masked convolution and channel attention, termed LKCMCA. Specifically, we first employ the pixel-shuffle technique to reduce the size of anomalous targets without losing image information. Next, we design a large-kernel central block masked convolution to make the network pay more attention to the surrounding background information, enabling better fusion of the information between adjacent bands. This, coupled with an efficient channel attention mechanism, allows the network to capture deeper spectral features, enhancing the reconstruction of the background while suppressing anomalous targets. Furthermore, we introduce an adaptive loss function by down-weighting anomalous pixels based on the mean absolute error. This loss function is specifically designed to suppress the reconstruction of potentially anomalous pixels during network training, allowing our model to be considered an excellent background reconstruction network. By leveraging reconstruction error, the model effectively highlights anomalous targets. Meanwhile, we produced four benchmark datasets specifically for HAD tasks using existing HRHSI data, addressing the current shortage of HRHSI datasets in the HAD field. Extensive experiments demonstrate that our LKCMCA method achieves superior detection performance, outperforming ten state-of-the-art HAD methods on all datasets.

YOLOX-S-TKECB: A Holstein Cow Identification Detection Algorithm

Accurate identification of individual cow identity is a prerequisite for the construction of digital farms and serves as the basis for optimized feeding, disease prevention and control, breed improvement, and product quality traceability. Currently, cow identification faces challenges such as poor recognition accuracy, large data volumes, weak model generalization ability, and low recognition speed. Therefore, this paper proposes a cow identification method based on YOLOX-S-TKECB. (1) Based on the characteristics of Holstein cows and their breeding practices, we constructed a real-time acquisition and preprocessing platform for two-dimensional Holstein cow images and built a cow identification model based on YOLOX-S-TKECB. (2) Transfer learning was introduced to improve the convergence speed and generalization ability of the cow identification model. (3) The CBAM attention mechanism module was added to enhance the model’s ability to extract features from cow torso patterns. (4) The alignment between the apriori frame and the target size was improved by optimizing the clustering algorithm and the multi-scale feature fusion method, thereby enhancing the performance of object detection at different scales. The experimental results demonstrate that, compared to the traditional YOLOX-S model, the improved model exhibits a 15.31% increase in mean average precision (mAP) and a 32-frame boost in frames per second (FPS). This validates the feasibility and effectiveness of the proposed YOLOX-S-TKECB-based cow identification algorithm, providing valuable technical support for the application of dairy cow identification in farms.

Text Detection on Industrial Barrel Label with Convolutional Attention and Dual‐Branch Feature Network

Industrial barrel labels generally have low visual contrast, uneven lighting, and cluttered background, making it challenging to accurately locate text regions. This paper proposes a text detection network to solve the inaccurate localization problem based on DBNet. First, a convolutional attention mechanism is applied to the feature extraction network to get more valuable text feature maps. Then, a dual‐branch convolutional feature module is proposed in the feature pyramid to enrich contextual information. Besides, during the probability map generation stage, using a feature remodeling enhancement module to further distinguish text and text boundaries. This paper designs comparative experiments on ILTD, ICDAR2015 and MSRA‐TD500 datasets, achieve F‐measure of 92.3%, 86.0% and 84.1%, which are 2.2%, 2.3%, and 1.9% higher than DBNet, respectively. They demonstrate that our proposed method exhibits competitive performance and strong robustness. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation

Residual channel attention based sample adaptation few-shot learning for hyperspectral image classification.

Few-shot learning (FSL) uses prior knowledge and supervised experience to effectively classify hyperspectral images (HSIs), thereby reducing the cost of large numbers of labeled samples. However, existing few-shot methods ignore the correlation between cross-domain feature channels, and the feature representation ability is insufficient. To address above issue, this paper proposes a novel Residual Channel Attention Based Sample Adaptation Few-Shot Learning for Hyperspectral Image Classification(RCASA-FSL) for hyperspectral image classification (HSIC), which can capture and enhance cross-domain dependencies through multi-layer residual connection and random-based feature recalibration. Specifically, a Deep Residual Feature Channel Attention Mechanism (DRFCAM) is designed to obtain cross-domain dependencies by residual concatenation, and further the residual structure is stacked for mining depth discrimination information. Furthermore, a new Random-based Feature Recalibration Module (RFRM) is proposed to reassign the feature weights via random matrix, which fully explore feature weight relationships to guide the sample adaptation process. Besides, we design a joint loss function with combining the FSL loss and domain adaptive loss for further optimization model. Experiments conducted on several standard hyperspectral datasets demonstrate that the proposed RCASA-FSL is superior to other FSL techniques in both quantitative and qualitative aspects.

Enhancing the Rainfall Forecasting Accuracy of Ensemble Numerical Prediction Systems via Convolutional Neural Networks

Abstract Ensemble prediction systems are commonly used to demonstrate the potential uncertainty of weather forecasts. These systems also help provide weather predictions to prepare for disasters. Specifically, a type of prediction called a quantitative precipitation forecast (QPF) is calculated from the average of multiple forecasts. The probability-matched mean (PMM) method is used to improve these QPF predictions when they underestimate rainfall. However, the PMM method can exhibit limitations when dealing with certain types of rain patterns. To address this issue, this study focuses on improving short-term heavy rainfall predictions using an artificial intelligence (AI) algorithm that combines deep neural networks (DNNs), including convolutional neural networks (CNNs), with a space-based attention mechanism (convolutional block attention module [CBAM]). Four experiments were conducted to evaluate the new method, including those performed to optimize the timing of a 24-hour QPF model, adjust the training dataset, and refine the training algorithm. Two specific weather events were assessed: rainfall influenced by southwest winds after typhoons and afternoon thunderstorm rainfall. A comparison of the results obtained via the root mean square error (RMSE) and structural similarity index measure (SSIM) reveals that the accuracy and distribution of the QPF predictions significantly improved over those of the PMM method. Compared with the PMM method, our approach can reduce the RMSE from 77.51 to 19.52 in the Mei-yu case, an improvement of approximately 74.82%. The SSIM also increased from 0.33 to 0.56, indicating a 70.9% enhancement. Overall, the new approach successfully enhances rainfall prediction accuracy via AI techniques and has the potential to be applied in disaster preparedness operations.

WCAY object detection of fractures for X-ray images of multiple sites

The WCAY (weighted channel attention YOLO) model, which is meticulously crafted to identify fracture features across diverse X-ray image sites, is presented herein. This model integrates novel core operators and an innovative attention mechanism to enhance its efficacy. Initially, leveraging the benefits of dynamic snake convolution (DSConv), which is adept at capturing elongated tubular structural features, we introduce the DSC-C2f module to augment the model’s fracture detection performance by replacing a portion of C2f. Subsequently, we integrate the newly proposed weighted channel attention (WCA) mechanism into the architecture to bolster feature fusion and improve fracture detection across various sites. Comparative experiments were conducted, to evaluate the performances of several attention mechanisms. These enhancement strategies were validated through experimentation on public X-ray image datasets (FracAtlas and GRAZPEDWRI-DX). Multiple experimental comparisons substantiated the model’s efficacy, demonstrating its superior accuracy and real-time detection capabilities. According to the experimental findings, on the FracAtlas dataset, our WCAY model exhibits a notable 8.8% improvement in mean average precision (mAP) over the original model. On the GRAZPEDWRI-DX dataset, the mAP reaches 64.4%, with a detection accuracy of 93.9% for the “fracture” category alone. The proposed model represents a substantial improvement over the original algorithm compared to other state-of-the-art object detection models. The code is publicly available at https://github.com/cccp421/Fracture-Detection-WCAY .

Multilevel attention mechanism for motion fatigue recognition based on sEMG and ACC signal fusion.

This study aims to develop a cost-effective and reliable motion monitoring device capable of comprehensive fatigue analysis. It achieves this objective by integrating surface electromyography (sEMG) and accelerometer (ACC) signals through a feature fusion strategy. The study introduces a multi-level attention mechanism for classification, leveraging convolutional neural networks (CNNs). The preprocessing phase involves a local feature attention mechanism that enhances local waveform features using the amplitude envelope. A dual-scale attention mechanism, operating at both channel and neuron levels, is employed to enhance the model's learning from high-dimensional fused data, improving feature extraction and generalization. The local feature attention mechanism significantly improves the model's classification accuracy and convergence, as demonstrated in ablation experiments. The model, optimized with multi-level attention mechanisms, excels in accuracy and generalization, particularly in handling data with pseudo-artifacts. Computational analysis indicates that the proposed optimization algorithm has minimal impact on CNN's training and testing times. The study achieves recognition accuracies of 92.52%, 92.38%, and 92.30%, as well as F1-scores of 91.92%, 92.13%, and 92.29% for the three fatigue states, affirming its reliability. This research provides technical support for the development of affordable and dependable wearable motion monitoring devices.

A Comparative Study on the Monitoring of Abnormal Power Consumption Behavior by Different Models

This paper investigates and compares the performance of three distinct models for monitoring abnormal electricity consumption behavior: Support Vector Regression (SVR), Long Short-Term Memory (LSTM), and a novel model that incorporates an attention mechanism known as PSO-ATT-LSTM (PAL), which integrates a Particle Swarm Optimization (PSO) algorithm with an attention mechanism into the LSTM framework. The PAL model is specifically designed to enhance forecasting accuracy by optimizing the network parameters through PSO and focusing on significant temporal features via the attention mechanism. This design allows PAL to outperform the other two models in predicting future electricity consumption, particularly in identifying anomalous patterns. The study utilizes a dataset with hourly electricity consumption values and evaluates the models using metrics such as Mean Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE). The experimental results demonstrate the superiority of the PAL model in terms of predictive accuracy and its ability to capture abnormal consumption behaviors more effectively than SVR and LSTM.

RecGOBD: Accurate Recognition of Gene Ontology Related Brain Development Protein Functions through Multi-Feature Fusion and Attention Mechanisms

Abstract Motivation Protein function prediction is important in bioinformatics, driven by the increasing availability of protein sequence data from high-throughput sequencing technologies. Traditional methods for determining protein functions are costly and time-consuming, highlighting the need for computational approaches. Deep learning models are powerful tools in this area, but many are not optimized for brain development-related datasets. Understanding protein functions in brain development is essential for studying neurodevelopmental disorders. To address this, we developed RecGOBD (Recognition of Gene Ontology-related Brain Development protein function), designed to predict protein functions related to key brain development processes. Result RecGOBD focuses on ten critical Gene Ontology (GO) terms associated with brain development, extracting and embedding the protein sequences linked to these terms. Using advanced pre-trained models, we generated embeddings that capture both sequence and structural information, and applied attention mechanisms to align them with relevant GO terms. RecGOBD’s category attention layer improves prediction accuracy for brain development-related terms. Evaluated against five models using AUROC, AUPR, and Fmax metrics, RecGOBD outperformed in all measures. We also applied the model to predict protein functions related to autism spectrum disorder and analyzed how protein site mutations affect GO term changes. These results highlight RecGOBD’s potential for advancing protein function prediction, particularly in brain development, offering valuable insights into neurodevelopmental disorder research. Availability and implementation All Python codes associated with this study are available at https://github.com/ZL-Xia/RECGOBD.git. Supplementary information Supplementary data are available at Bioinformatics Advance online

A Unified Model for Chinese Cyber Threat Intelligence Flat Entity and Nested Entity Recognition

In recent years, as cybersecurity threats have become increasingly severe and cyberattacks have occurred frequently, higher requirements have been put forward for cybersecurity protection. Therefore, the Named Entity Recognition (NER) technique, which is the cornerstone of Cyber Threat Intelligence (CTI) analysis, is particularly important. However, most existing NER studies are limited to recognizing single-layer flat entities, ignoring the possible nested entities in CTI. On the other hand, most of the existing studies focus on English CTIs, and the existing models performed poorly in a limited number of Chinese CTI studies. Given the above challenges, we propose in this paper a novel unified model, RBTG, which aims to identify flat and nested entities in Chinese CTI effectively. To overcome the difficult boundary recognition problem and the direction-dependent and distance-dependent properties in Chinese CTI NER, we use Global Pointer as the decoder and TENER as the encoder layer, respectively. Specifically, the Global Pointer layer solves the problem of the insensitivity of general NER methods to entity boundaries by utilizing the relative position information and the multiplicative attention mechanism. The TENER layer adapts to the Chinese CTI NER task by introducing an attention mechanism with direction awareness and distance awareness. Meanwhile, to cope with the complex feature capture of hierarchical structure and dependencies among Chinese CTI nested entities, the TENER layer solves the problem by following the structure of multiple self-attention layers and feed-forward network layers superimposed on each other in the Transformer. In addition, to fill the gap in the Chinese CTI nested entity dataset, we further apply the Large Language Modeling (LLM) technique and domain knowledge to construct a high-quality Chinese CTI nested entity dataset, CDTinee, which consists of six entity types selected from STIX, including nearly 4000 entity types extracted from more than 3000 threatening sentences. In the experimental session, we conduct extensive experiments on multiple datasets, and the results show that the proposed model RBTG outperforms the baseline model in both flat NER and nested NER.

A Review of Deep Learning-Based Remote Sensing Image Caption: Methods, Models, Comparisons and Future Directions

Remote sensing images contain a wealth of Earth-observation information. Efficient extraction and application of hidden knowledge from these images will greatly promote the development of resource and environment monitoring, urban planning and other related fields. Remote sensing image caption (RSIC) involves obtaining textual descriptions from remote sensing images through accurately capturing and describing the semantic-level relationships between objects and attributes in the images. However, there is currently no comprehensive review summarizing the progress in RSIC based on deep learning. After defining the scope of the papers to be discussed and summarizing them all, the paper begins by providing a comprehensive review of the recent advancements in RSIC, covering six key aspects: encoder–decoder framework, attention mechanism, reinforcement learning, learning with auxiliary task, large visual language models and few-shot learning. Subsequently a brief explanation on the datasets and evaluation metrics for RSIC is given. Furthermore, we compare and analyze the results of the latest models and the pros and cons of different deep learning methods. Lastly, future directions of RSIC are suggested. The primary objective of this review is to offer researchers a more profound understanding of RSIC.

Automatic visual recognition for leaf disease based on enhanced attention mechanism

Recognition methods have made significant strides across various domains, such as image classification, automatic segmentation, and autonomous driving. Efficient identification of leaf diseases through visual recognition is critical for mitigating economic losses. However, recognizing leaf diseases is challenging due to complex backgrounds and environmental factors. These challenges often result in confusion between lesions and backgrounds, limiting information extraction from small lesion targets. To tackle these challenges, this article proposes a visual leaf disease identification method based on an enhanced attention mechanism. By integrating multi-head attention mechanisms, this method accurately identifies small targets of tomato lesions and demonstrates robustness in complex conditions, such as varying illumination. Additionally, the method incorporates Focaler-SIoU to enhance learning capabilities for challenging classification samples. Experimental results showcase that the proposed algorithm enhances average detection accuracy by 10.3% compared to the baseline model, while maintaining a balanced identification speed. This method facilitates rapid and precise identification of tomato diseases, offering a valuable tool for disease prevention and economic loss reduction.

Classification of EEG event-related potentials based on channel attention mechanism

Classification of EEG event-related potentials based on channel attention mechanism

Real-life boxing activity recognition with smartphones using attention assisted deep learning models

Mobile computing technologies play a pivotal role in narrowing the gap between athletes and sports monitoring systems, while also advancing towards intelligent and automatic supervision of individual sports activities. The widespread presence of smartphones equipped with multifunctional sensors has enabled the acquisition and analysis of data, facilitating Human Activity Recognition (HAR). Although recognising sports activities using a smart mobile phone’s accelerometer is widely used, traditional methods struggle with intricate and real time activities due to the high-dimensional sensor data. This study introduces a smartphone-based architecture for HAR utilising inertial accelerometers to record athlete’s game actions data sequences, extract relevant features and derive the player’s activity data through multiple three-axis accelerometers. In this paper an Attention assisted Deep Convolution Neural Network based Bi-LSTM (AT-DCNN-BL) model is proposed. The raw data undergoes sliding window pre-processing technique, the deep learning framework of deep convolution neural network layers helps in automatic feature extraction, the bi-LSTM structure process the boxing activity data and the attention is focussed over the boxing activity classification data. The attention mechanism redistributes the weights of the extracted features which aids in increasing the classification accuracy. Other classification models like Deep Convolutional Neural Network (DCNN), Bi-directional Long Short-Term Memory (bi-LSTM), Multi-Layer Perceptron Neural Network (MLPNN) and Random Forest (RF) are also applied to the dataset collected from a mobile sensor. The study explores training of deep learning methods and illustrates the superiority of the proposed approach over others using the collected mobile sensor dataset. The AT-DCNN-BL model achieved a higher classification accuracy compared to the other models. The AT-DCNN-BL model outperformed all the other models achieving a higher accuracy rate (92.71%).

Reconstruction of Typhoon-Induced Ocean Thermal Structures Using Deep Learning and Multi-Source Satellite Data with News Impact Analysis

Reconstructing the three-dimensional thermal structure of the ocean under typhoon conditions presents significant challenges due to the scarcity of observational data, particularly in subsurface regions, and the limitations of existing observation methods in terms of spatial and temporal resolution. Accurate reconstruction of these structures is crucial for understanding the energy exchange between the ocean and typhoons, as this exchange directly influences typhoon intensity and trajectory. To address these challenges, this study introduces a fully connected transformer network (FCT), which integrates fully connected layers with a transformer model. The FCT model leverages the attention mechanisms inherent in the transformer architecture to effectively extract and integrate multi-scale ocean dynamical features. Using data from Typhoon Lekima in 2019, this study reconstructs ocean thermal structures at various depths and achieves an RMSE of 1.03 °C and an MAE of 0.83 °C when validated against Argo data. Furthermore, the model’s robustness was demonstrated through five-fold cross-validation, with the validation loss exhibiting minor fluctuations across folds but remaining stable overall, with an average validation loss of 0.986 °C, indicating the model’s generalizability. Sensitivity analysis also revealed the model’s resilience to variations in key input variables, showing minimal impact on output even with perturbations of up to 10% in input data. In addition, the study incorporates content analysis of typhoon-related news reports from 2011 to 2020, revealing a predominance of political topics, which underscores the central role of government in disaster response, with economic and ecological topics following. This integrated approach not only enhances our understanding of the interactions between ocean thermal structures and typhoon dynamics but also provides valuable insights into the societal impacts of typhoons, as reflected in media coverage, contributing to improved disaster management strategies.

ATGAN: attention-based temporal GAN for EEG data augmentation in personal identification

With the development of brain–computer interfaces in recent years, deep learning models have been widely used in EEG-based personal identification. The training of complex models requires a large amount of data, but the acquisition of EEG signals is very time-consuming and energy-consuming. EEG data augmentation can reduce the overhead of EEG data acquisition and provide the data volume required by various complex models, which is conducive to applying EEG identification in practice. The generative adversarial network (GAN) has been applied to augment EEG data features. But the augmentation of temporal EEG signals is less studied. Therefore, this paper proposes a method of temporal EEG data augmentation based on an attention mechanism time series generative adversarial network (ATGAN), which can expand the original EEG dataset to facilitate the training of downstream deep learning classifiers. The experiment is conducted on the BCI Competition IV 2a dataset. The ATGAN model is trained with the temporal EEG signals after preprocessing, and the generated augmentation data by ATGAN is used to expand the original dataset. The accuracy of the EEG-based identification task on the augmented dataset is 7.78% higher than that on the original dataset, which proves the effectiveness of the proposed method.

Inversion of the planetary boundary layer height from lidar by combining UNet++ and coordinate attention mechanism

The Arctic plays a significant role in global climate, and the planetary boundary layer height (PBLH) is one of the important parameters for studying Arctic climate. The Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) North Slope of Alaska (NSA) atmospheric observatory is an important location for studying the Arctic. However, the weather at the NSA site is complicated and varied. Arctic Haze frequently appears in this region from late autumn to early summer, while low clouds are prone to occur in summer. Meanwhile, due to the consistently low temperatures on the Arctic surface, the frequency of stable boundary layer occurrence is much higher than that in mid-latitude regions. All of these will increase the difficulty of PBLH detection. To address these challenges, we propose a PBLH inversion method based on deep-learning called Coord-UNet++. This method is based on UNet++ and introduces coordinate attention mechanism which can gather features in both horizontal and vertical directions, so it can more effectively capture spatial information in images to cope with complex weather conditions. The training set for the algorithm comes from the micropulse lidar at the NSA site, and the PBLH is labeled by using the microwave radiation profiler at the same site. This algorithm can achieve accurate inversion of the PBLH in complex weather conditions such as cloudy, haze and aerosol layer interference, R2 reaches 0.87, and it performs well in long-term inversion, with much higher stability and accuracy than traditional methods.

Temporal attention fusion network with custom loss function for EEG--fNIRS classification.

Objective.Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artefacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.Approach.To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG--fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.Main resultsRigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.Significance.This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.