Attention Mechanism Research Articles

Label-Guided relation prototype generation for Continual Relation Extraction

Continual relation extraction (CRE) aims to extract relations towards the continuous and iterative arrival of new data. To address the problem of catastrophic forgetting, some existing research endeavors have focused on exploring memory replay methods by storing typical historical learned instances or embedding all observed relations as prototypes by averaging the hidden representation of samples and replaying them in the subsequent training process. However, this prototype generation method overlooks the rich semantic information within the label namespace and are also constrained by memory size, resulting in inadequate descriptions of relation semantics by relation prototypes. To this end, we introduce an approach termed Label-Guided Relation Prototype Generation. Initially, we enhance the representations of label embeddings through a technique named label knowledge infusion. Following that, we utilize the multi-head attention mechanism to form relation prototypes, allowing them to capture diverse aspects of typical instances. The embeddings of relation labels are utilized at this stage, leveraging their contained semantics. Additionally, we propose a feature-based distillation loss function called multi-similarity distillation, to ensure the model retains prior knowledge after learning new tasks. The experimental results indicate that our method has achieved competitive performance compared to the state-of-the-art baseline models in CRE.

Advanced ENF Region Classification Using UniTS-SinSpec: A Novel Approach Integrating Sinusoidal Activation Function and Spectral Attention

The electric network frequency (ENF), often referred to as the industrial heartbeat, plays a crucial role in the power system. In recent years, it has found applications in multimedia evidence identification for court proceedings and audio–visual temporal source identification. This paper introduces an ENF region classification model named UniTS-SinSpec within the UniTS framework. The model integrates the sinusoidal activation function and spectral attention mechanism while also redesigning the model framework. Training is conducted using a public dataset on the open science framework (OSF) platform, with final experimental results demonstrating that, after parameter optimization, the UniTS-SinSpec model achieves an average validation accuracy of 97.47%, surpassing current state-of-the-art and baseline models. Accurate classification can significantly aid in ENF temporal source identification. Future research will focus on expanding dataset coverage and diversity to verify the model’s generality and robustness across different regions, time spans, and data sources. Additionally, it aims to explore the extensive application potential of ENF region classification in preventing crimes such as telecommunications fraud, terrorism, and child pornography.

Conditional diffusion model for arbitrary-size synthetic aperture radar image despeckling with multi-scale and dual attention mechanism

Conditional diffusion model for arbitrary-size synthetic aperture radar image despeckling with multi-scale and dual attention mechanism

A Heterogeneous Information Networks with an Attention Mechanism for Improving Recommender Systems

A Heterogeneous Information Networks with an Attention Mechanism for Improving Recommender Systems

A Novel Trajectory Prediction Method Based on CNN, BiLSTM, and Multi-Head Attention Mechanism

A four-dimensional (4D) trajectory is a multi-dimensional time series that embodies rich spatiotemporal features. However, its high complexity and inherent uncertainty pose significant challenges for accurate prediction. In this paper, we present a novel 4D trajectory prediction model that integrates convolutional neural networks (CNNs), bidirectional long short-term memory networks (BiLSTMs), and multi-head attention mechanisms. This model effectively addresses the characteristics of aircraft flight trajectories and the difficulties associated with simultaneously extracting spatiotemporal features using existing prediction methods. Specifically, we leverage the local feature extraction capabilities of CNNs to extract key spatial and temporal features from the original trajectory data, such as geometric shape information and dynamic change patterns. The BiLSTM network is employed to consider both forward and backward temporal orders in the trajectory data, allowing for a more comprehensive capture of long-term dependencies. Furthermore, we introduce a multi-head attention mechanism that enhances the model’s ability to accurately identify key information in the trajectory data while minimizing the interference of redundant information. We validated our approach through experiments conducted on a real ADS-B trajectory dataset. The experimental results demonstrate that the proposed method significantly outperforms comparative approaches in terms of trajectory estimation accuracy.

Image-Based Fitness Yoga Pose Recognition: Using Ensemble Learning and Multi-head Attention

With the increasing awareness of fitness, more and more people are choosing to participate in fitness activities. Yoga, as a form of exercise that improves both physical and mental health, is becoming increasingly popular worldwide. In order to assist yoga practitioners in more effective training through automated or semi automated systems, improve training effectiveness, assist professional athletes in training through intelligent recognition systems, correct movements, and improve athletic performance. This paper proposes a method that addresses the low accuracy issue of current yoga pose recognition algorithms by integrating multi-head attention mechanism and ensemble learning. Firstly, the Mixup algorithm is used to enhance yoga movement images. Subsequently, convolutional features are extracted from the images using the ResNet101 and VGGNet19 transfer learning models. Finally, the extracted convolutional features are combined and stacked using a multi-head attention mechanism. Model training, validation, and testing are performed using the Soft target cross-entropy loss function. Experimental results demonstrate that the proposed method achieves a training accuracy of 100%, a validation accuracy of 89.94%, a testing accuracy of 93.79%, and a detection speed of 297 frames per second. Overall, this method demonstrates high stability and robustness, providing a technological foundation for intelligent recognition of yoga poses.

Graph node classification algorithm based on similarity random walk aggregation

Aiming at the relatively low accuracy of methods such as MLP and GCN in heterogeneous graph node classification tasks, this paper proposes a graph neural network based on similarity random walk aggregation (SRW-GNN). Most existing node classification methods usually take neighbor nodes as neighborhoods, but the target node and its neighbors in heterogeneous graphs usually belong to different categories. To reduce the impact of heterogeneity on node embedding, SRW-GNN uses the similarity between nodes as probability to perform random walks and takes the sampled paths as neighborhoods to obtain more homogeneous information. The order in which nodes appear in the path is particularly critical for capturing neighborhood information. However, most existing GNN aggregators are insensitive to node order. This paper introduces a path aggregator based on recurrent neural network (RNN) to simultaneously extract the features and order information of nodes in the path. In addition, nodes have different preferences for different paths. In order to adaptively learn the importance of different paths in node encoding, an attention mechanism is used to dynamically adjust the contribution of each path to the final embedding. Experimental results on multiple commonly used heterogeneous graph datasets show that the accuracy of this method is significantly better than that of MLP, GCN, H2GCN, HOG-GCN and other methods, verifying its effectiveness in heterogeneous graph node classification tasks.

Power equipment defect prediction based on time series knowledge graph

With the increasing scale of power facilities, the operation monitoring of power equipment has become an important part of power system management. Defect prediction of power equipment is a key step in power system operation monitoring. In order to solve the problem of defect prediction of power equipment in large-scale power systems, this paper proposes a power equipment defect prediction model based on time series knowledge graph in combination with artificial intelligence technology. Multimodal information is fused through the attention mechanism, and then the time series representation of entities and relations is obtained by using the relationship-aware graph neural network and recurrent neural network. Finally, the defects of power equipment are predicted based on the time series representation. The method proposed in this paper can make full use of multimodal information and improve the accuracy of power equipment defect prediction. Experiments show that the performance of the model proposed in this paper is better than that of the baseline model.

Advancements in prostate zone segmentation: integrating attention mechanisms into the nnU-Net framework

Abstract Prostate cancer is one of the most lethal cancers in the world. Early diagnosis is essential for successful treatment of prostate cancer. Segmentation of prostate zones in magnetic resonance images is an important task in the diagnosis of prostate cancer. Currently, the state-of-the-art method for this task is no-new U-Net. In this paper, a method to incorporate the attention U-Net architecture into no-new U-Net is proposed and compared with a classical U-net architecture as research. The experimental results indicate that there is no significant statistical difference between the proposed modification of no-new U-Net with the generalizability of the attention mechanism or the ability to achieve more accurate results. Moreover, two novel workflows are proposed for prostate segmentation, transitional zone segmentation and peripheral zone calculation workflow, and separate models for peripheral zone and transitional zone segmentation workflow. These workflows are compared with a baseline single peripheral zone and transitional zone segmentation model workflow. The experimental results indicate that separate models for peripheral zone and transitional zone segmentation workflow generalizes better than the baseline between data sets of different sources. In peripheral zone segmentation separate models for peripheral zone and transitional zone segmentation workflow achieves 1.9% higher median Dice score coefficient than the baseline workflow when using the attention U-Net architecture and 5.6% higher median Dice score coefficient when using U-Net architecture. Moreover, in transitional zone segmentation separate models for peripheral zone and transitional zone segmentation workflow achieves 0.4% higher median Dice score coefficient than the baseline workflow when using the attention U-Net architecture and 0.7% higher median Dice score coefficient when using U-Net architecture. Meanwhile, prostate segmentation, transitional zone segmentation and peripheral zone calculation workflow generalizes worse than the baseline. In peripheral zone segmentation prostate segmentation, transitional zone segmentation and peripheral zone calculation workflow achieves 4.6% lower median Dice score coefficient than the baseline workflow when using the attention U-Net architecture and 3.6% lower median Dice score coefficient when using U-Net architecture. In transitional zone segmentation prostate segmentation, transitional zone segmentation and peripheral zone calculation workflow achieves a similar median Dice score coefficient to the baseline workflow.

Oxygen Content Control in the Electroslag Remelting Process: An Incremental Learning Strategy Based on Optimized Wasserstein Generative Adversarial Network with Gradient Penalty Data Augmentation

Electroslag remelting (ESR) is essential for producing high‐end special steel, but its complex process and numerous influencing factors make quality control challenging. This study addresses oxygen content control during ESR using a big data machine learning approach. An incremental learning strategy is proposed based on an optimized Wasserstein generative adversarial network with gradient penalty (WGAN‐GP) for data enhancement, focusing on G20Cr2Ni4A bearing steel. The WGAN‐GP model enhances time‐series data and metadata, utilizing long short‐term memory networks, fully connected networks, and attention mechanisms. The effectiveness of data enhancement is verified using a deep neural network classifier and statistical methods. Data is divided into historical and data streams, with an incremental learning strategy based on histogram gradient boosting regression trees to prevent catastrophic forgetting and improve efficiency through knowledge distillation and real‐time hyperparameter adjustment. Results show that the data augmentation method significantly improves model generalization and accuracy in small sample metallurgy. The incremental learning strategy enhances prediction accuracy for oxygen content, contributing to better cleanliness quality of electroslag steel. This study offers a novel approach for addressing small sample challenges in metallurgical processes.

Identifying Re-identification Challenges: Past, Current and Future Trends

Person and vehicle re-identification has been a popular subject in the field of the computer vision technologies. Existing closed-set re-identification surpasses human-level accuracies on commonly used benchmarks, and the research focus for re-identification is shifting to the open world-setting. The latter setting is more suitable for practical applications, however, is less developed due to its challenges. On the other hand, existing research is more focused on person re-identification, even though both, person and vehicle, are important components for smart city applications. This review attempts to combine for the first time the problem of person and vehicle re-identification under closed and open settings, its challenges, and the existing research. Specifically, we start from the origin of the re-identification task and then summarize state-of-the-art research based on deep learning in different scenarios: person or vehicle or unified re-identification in closed- and open-world settings. Additionally, we analyse a new method for solving the re-identification task using the Transformer, a model architecture that relies entirely on an attention mechanism, which shows promising results. This survey facilitates future research by providing a summary on past and present trends, and aids to improve the usability of re-ID techniques.

SOD‐YOLO: Small Object Detection Network for UAV Aerial Images

With the rapid development of the UAV industry, object detection using UAV has become a research hotspot. However, most current object detection models based on deep learning have large parameter counts and are difficult to deploy on embedded devices with limited memory and computational power. To address this problem, a Small Object Detection network for UAV aerial images SOD‐YOLO based on YOLOv8 is proposed, which can meet the application requirements of resource‐constrained devices while ensuring the detection accuracy. First, cross‐domain fusion attention (CDFA) mechanism is proposed to build the C2f‐Attention module in this paper, which is embedded in the backbone network in order to improve the extraction capability of key object features. Meanwhile, the AIFI_LSPE feature fusion module with improved RT‐DETR and the IoU‐aware query selection mechanism are added to the path aggregation network to improve the accuracy of multi‐scale object detection. In addition, in order to balance the sample size ratio and improve the robustness of the network model, we make a new UAV image dataset named VisDrone2019 Extended Edition (VDEE) using images from the VisDrone2019 and UAVDT public datasets. Finally, Shape‐IoU is used as a loss function to reduce the difference between the object GT frame and the detection frame. Experiments show that SOD‐YOLO has a mAP@0.5 of 42.8% in the VDEE dataset, which is increased by 5.1% over YOLOv8. In the VisDrone2019 dataset mAP@0.5 is 39.2%, an improvement of 5.8% over YOLOv8. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Multi-Scale Feature Attention Fusion for Image Splicing Forgery Detection

Image splicing is a widely occurrence image tampering technology. With the rapid development of digital image processing technology, detecting image splicing forgery has become significantly challenging. Although various methods have been devised to identify such tampered images, existing approaches have not achieved optimal performance due to limitations in effectively leveraging feature maps of different scales. To address this issue, we propose a novel method for image splicing forgery detection called multi-scale feature attention fusion network (MFAF-Net). We propose a multi-scale atrous feature attention (MAFA) module designed to capture rich contextual features for multi-scale high-level feature fusion. Additionally, we present the multi-branch attention mechanism (MBAM) module to fuse contextual information from various branches for low-level features. This integration enhances the capability of low-level features to produce more refined pixel-level attention. We employ the weighted binary cross-entropy loss and dice loss in the MFAF-Net to overcome the imbalance between positive and negative samples. Extensive experiments demonstrate that the proposed MFAF-Net outperforms state-of-the-art methods. Robustness experiments also show our model exhibits image splicing forgery detection robustness under common attacks.

Attention-Enhanced Defensive Distillation Network for Channel Estimation in V2X mm-Wave Secure Communication

Millimeter-wave (mm-wave) technology, crucial for future networks and vehicle-to-everything (V2X) communication in intelligent transportation, offers high data rates and bandwidth but is vulnerable to adversarial attacks, like interference and eavesdropping. It is crucial to protect V2X mm-wave communication from cybersecurity attacks, as traditional security measures often fail to counter sophisticated threats and complex attacks. To tackle these difficulties, the current study introduces an attention-enhanced defensive distillation network (AEDDN) to improve robustness and accuracy in V2X mm-wave communication under adversarial attacks. The AEDDN model combines the transformer algorithm with defensive distillation, leveraging the transformer’s attention mechanism to focus on critical channel features and adapt to complex conditions. This helps mitigate adversarial examples by filtering misleading data. Defensive distillation further strengthens the model by smoothing decision boundaries, making it less sensitive to small perturbations. To evaluate and validate the AEDDN model, this study uses a publicly available dataset called 6g-channel-estimation and a proprietary dataset named MMMC, comparing the simulation results with the convolutional neural network (CNN) model. The findings from the experiments indicate that the AEDDN, especially in the complex V2X mm-wave environment, demonstrates enhanced performance.

Deep learning models for the prediction of acute postoperative pain in PACU for video‐assisted thoracoscopic surgery

BackgroundPostoperative pain is a prevalent symptom experienced by patients undergoing surgical procedures. This study aims to develop deep learning algorithms for predicting acute postoperative pain using both essential patient details and real-time vital sign data during surgery.MethodsThrough a retrospective observational approach, we utilized Graph Attention Networks (GAT) and graph Transformer Networks (GTN) deep learning algorithms to construct the DoseFormer model while incorporating an attention mechanism. This model employed patient information and intraoperative vital signs obtained during Video-assisted thoracoscopic surgery (VATS) surgery to anticipate postoperative pain. By categorizing the static and dynamic data, the DoseFormer model performed binary classification to predict the likelihood of postoperative acute pain.ResultsA total of 1758 patients were initially included, with 1552 patients after data cleaning. These patients were then divided into training set (n = 931) and testing set (n = 621). In the testing set, the DoseFormer model exhibited significantly higher AUROC (0.98) compared to classical machine learning algorithms. Furthermore, the DoseFormer model displayed a significantly higher F1 value (0.85) in comparison to other classical machine learning algorithms. Notably, the attending anesthesiologists' F1 values (attending: 0.49, fellow: 0.43, Resident: 0.16) were significantly lower than those of the DoseFormer model in predicting acute postoperative pain.ConclusionsDeep learning model can predict postoperative acute pain events based on patients' basic information and intraoperative vital signs.

Generation of NIR Spectral Band from RGB Image with Wavelet Domain Spectral Extrapolation Generative Adversarial Network

Near-infrared (NIR) imaging exhibits outstanding penetration capabilities and robust anti-interference characteristics. However, acquiring high-resolution and high-fidelity NIR images is difficult due to the limited mobility, high cost, and low resolution of NIR imaging hardware. In this paper, we proposed a new end-to-end Wavelet Domain Spectral Extrapolation Generative Adversarial Network (WSEGAN) to generate highly realistic NIR images from RGB images. Since RGB and NIR images have different types of noise and artifacts, which affects the quality of NIR images generation. We design a generator with a discrete wavelet transform and an attention mechanism to capture multi-resolution contextual information, and a multi-scale discriminator to capture detailed and global features. Then, the proposed approach is evaluated by three different datasets to achieve optimal results regarding both visual effects and quantitative evaluation. The normalized difference vegetation index (NDVI) is utilized to validate the effectiveness of the generated NIR images. The result demonstrates a strong correlation between the generated images and the actual vegetation distribution. More importantly, the proposed network is testified to generate NIR images to be fused with the RGB image source for agricultural target detection tasks. In dark conditions, results show that using multi-modal data instead of RGB images improves mAP0.5 detection accuracy by 4% on the CAPSICUM dataset and by 8% on the KIWI dataset across five object detection methods. This follows the physical significance of the NIR imaging and demonstrates the potential use of the NIR images generated by the proposed method.

Text-aware small sample remote sensing image classification based on contrastive learning

In view of the fact that existing methods mainly use a single modality of remote sensing images to solve the problem of low similarity of the same category, a remote sensing image classification method based on multimodal learning is proposed. Firstly, the spatial features of the image are corrected and the image encoder is pre-trained using contrastive learning to generate image features, and the text encoder is used to generate text features. Secondly, a feature decoder is introduced to obtain text-aware visual features, and a new attention mechanism method is proposed in the feature fusion stage. Then, a new image encoder is designed to improve the classification accuracy. Finally, the similarity between the support set and the query set is calculated to further predict the category. Experiments are conducted on the NWPU-RESISC45, AID, and UC Merced datasets. Its 5-way 5-shot accuracy reaches 86.46%、85.89%and 80.32%outperforms the existing small-sample remote sensing image classification methods, respectively.

Remaining Useful Life of the Rolling Bearings Prediction Method Based on Transfer Learning Integrated with CNN-GRU-MHA

To accurately predict the remaining useful life (RUL) of rolling bearings under limited data and fluctuating load conditions, we propose a new method for constructing health indicators (HI) and a transfer learning prediction framework, which integrates Convolutional Neural Networks (CNN), Gated Recurrent Units (GRU), and Multi-head attention (MHA). Firstly, we combined Convolutional Neural Networks (CNN) with Gated Recurrent Units (GRU) to fully extract temporal and spatial features from vibration signals. Then, the Multi-head attention mechanism (MHA) was added for weighted processing to improve the expression ability of the model. Finally, a new method for constructing Health indicators (HIs) was proposed in which the noise reduction and normalized vibration signals were taken as a HI, the L1 regularization method was added to avoid overfitting, and the model-based transfer learning method was used to realize the RUL prediction of bearings under small samples and variable load conditions. Experiments were conducted using the PHM2012 dataset from the FEMTO-ST research institute and XJTU-SY dataset. Three sets of 12 migration experiments were conducted under three different operating conditions on the PHM2012 dataset. The results show that the average RMSE of the proposed method was 0.0443, indicating high prediction accuracy under variable loads and small sample conditions. Three different operating conditions and two sets of four migration experiments were conducted on the XJTU-SY dataset, and the results show that the average RMSE of the proposed method was 0.0693, verifying the good generalization of the model under variable load conditions. In summary, the proposed HI construction method and prediction framework can effectively reduce the differences between features, with high stability and good generalizability.

An Improved YOLOv8-Based Foreign Detection Algorithm for Transmission Lines

This research aims to overcome three major challenges in foreign object detection on power transmission lines: data scarcity, background noise, and high computational costs. In the improved YOLOv8 algorithm, the newly introduced lightweight GSCDown (Ghost Shuffle Channel Downsampling) module effectively captures subtle image features by combining 1 × 1 convolution and GSConv technology, thereby enhancing detection accuracy. CSPBlock (Cross-Stage Partial Block) fusion enhances the model’s accuracy and stability by strengthening feature expression and spatial perception while maintaining the algorithm’s lightweight nature and effectively mitigating the issue of vanishing gradients, making it suitable for efficient foreign object detection in complex power line environments. Additionally, PAM (pooling attention mechanism) effectively distinguishes between background and target without adding extra parameters, maintaining high accuracy even in the presence of background noise. Furthermore, AIGC (AI-generated content) technology is leveraged to produce high-quality images for training data augmentation, and lossless feature distillation ensures higher detection accuracy and reduces false positives. In conclusion, the improved architecture reduces the parameter count by 18% while improving the mAP@0.5 metric by a margin of 5.5 points when compared to YOLOv8n. Compared to state-of-the-art real-time object detection frameworks, our research demonstrates significant advantages in both model accuracy and parameter size.

Improving Effectiveness by Reducing Overconfidence in Large Catalogue Sequential Recommendation with gBCE loss

A large catalogue size is one of the central challenges in training recommendation models: a large number of items makes them memory and computationally inefficient to compute scores for all items during training, forcing these models to deploy negative sampling. However, negative sampling increases the proportion of positive interactions in the training data, and therefore, models trained with negative sampling tend to overestimate the probabilities of positive interactions – a phenomenon we call overconfidence . While the absolute values of the predicted scores/probabilities are not important for the ranking of retrieved recommendations, overconfident models may fail to estimate nuanced differences in the top-ranked items, resulting in degraded performance. In this paper, we show that overconfidence explains why the popular SASRec model underperforms when compared to BERT4Rec. This is contrary to the BERT4Rec authors’ explanation that the difference in performance is due to the bi-directional attention mechanism. To mitigate overconfidence, we propose a novel Generalised Binary Cross-Entropy Loss function (gBCE) and theoretically prove that it can mitigate overconfidence. We further propose the gSASRec model, an improvement over SASRec that deploys an increased number of negatives and the gBCE loss. Through detailed experiments on three datasets, we show that gSASRec does not exhibit the overconfidence problem. As a result, gSASRec can outperform BERT4Rec (e.g. +9.47% NDCG on the MovieLens-1M dataset), while requiring less training time (e.g. -73% training time on MovieLens-1M). Moreover, in contrast to BERT4Rec, gSASRec is suitable for large datasets that contain more than 1 million items. Finally, we show how addressing overconfidence can improve model calibration – the ability of a model to predict actual interaction probabilities accurately. By applying gBCE to the SASRec model on MovieLens-1M dataset, we reduce the models’ expected calibration error by 98.9% (from 0.966 to 0.01).