Self-attention Mechanism Research Articles

Transformer-based approach for printing quality recognition in fused filament fabrication

Ensuring high-quality prints in additive manufacturing is a critical challenge due to the variability in materials, process parameters, and equipment. Machine learning models are increasingly being employed for real-time quality monitoring, enabling the detection and classification of defects such as under-extrusion and over-extrusion. Vision Transformers (ViTs), with their global self-attention mechanisms, offer a promising alternative to traditional convolutional neural networks (CNNs). This paper presents a transformer-based approach for print quality recognition in additive manufacturing technologies, with a focus on fused filament fabrication (FFF), leveraging advanced self-supervised representation learning techniques to enhance the robustness and generalizability of ViTs. We show that the ViT model effectively classifies printing quality into different levels of extrusion, achieving exceptional performance across varying dataset scales and noise levels. Training evaluations show a steady decrease in cross-entropy loss, with prediction accuracy, precision, recall, and the harmonic mean of precision and recall (F1) scores reaching close to 1 within 40 epochs, demonstrating excellent performance across all classes. The macro and micro F1 scores further emphasize the ability of ViT to handle both class imbalance and instance-level accuracy effectively. Our results also demonstrate that ViT outperforms CNN in all scenarios, particularly in noisy conditions and with small datasets. Comparative analysis reveals ViT advantages, particularly in leveraging global self-attention and robust feature extraction methods, enhancing its ability to generalize effectively and remain resilient with limited data. These findings underline the potential of the transformer-based approach as a scalable, interpretable, and reliable solution to real-time quality monitoring in FFF, addressing key challenges in additive manufacturing defect detection and ensuring process efficiency.

BMSNet: Brain-inspired mixed ultrasound image segmentation network in cancer diagnosis

BMSNet: Brain-inspired mixed ultrasound image segmentation network in cancer diagnosis

Power evolution prediction of bidirectional Raman amplified WDM system based on a transformer

This study presents a transformer-based power prediction model for Raman-amplified wavelength division multiplexing (WDM) systems. Leveraging the transformer's self-attention mechanism, the model effectively captures global channel interactions, achieving a maximum prediction error of only 0.122 dB in a 200 km bidirectional Raman system with 47 channels and 12 pumps. The differential evolution (DE) technique was employed to optimize the hyperparameters of the models, further enhancing their performance. The comparative analysis was conducted to evaluate the performance of the two models under their respective optimal configurations. The model demonstrates robust generalization under varying pump powers and surpasses convolutional neural network (CNN)-based approaches in accuracy and interpretability.

Deciphering Muscular Dynamics: A Dual-Attention Framework for Predicting Muscle Contraction from Activation Patterns.

Quantitatively deciphering the relationship between muscle activation and thickness deformation is essential for diagnosing muscle-related diseases and monitoring muscle health (e.g., Facioscapulohumeral Dystrophy). Despite the potential of ultrasound (US) imaging and sensing to measure changes in muscle thickness during movements, it remains challenging to make a fully portable device, considering the wiring and data collection. On the other hand, surface electromyography (sEMG) can record muscle bioelectrical signals and measure muscle activations, offering a unique perspective that correlates with underlying changes in muscle thickness. This paper introduces a deep-learning-based approach that used sEMG signals to infer muscle deformation. Using a hierarchical combination of self-attention and cross-attention mechanisms, this method predicted muscle deformation directly from sEMG data, eliminating the dependency on applying ultrasound imaging techniques. The experimental results on six healthy subjects indicated that our approach could accurately predict muscle excursion with an average precision of 0.923$\pm$0.900mm, showing benefits in measuring muscle deformation only with a sEMG device. This technique facilitates real-time portable muscle health monitoring by sEMG to provide bioelectrical signals and biomechanical information. It indicates the great potential of using this technique in clinical diagnostics, sports science, and rehabilitation.

Interactive recommendation of social network communication between cities based on GNN and user preferences

Abstract To further enhance the effectiveness of interactive recommendation for intercity social network communication, the study proposes a novel interactive recommendation model for intercity social network communication. This model focuses on point-of-interest preference migration and recommendation modeling by integrating the self-attention mechanism, cross-attention mechanism, and graph neural network. The method effectively solves the problem of cross-city users’ interest point preference change and constructs a recommendation framework based on interest point heat. The experimental results showed that the recommendation accuracy of the new model was up to 93.52%. Compared with the existing more advanced recommendation methods, the recommendation coverage of Chengdu–Chongqing dining and food points of interest could be up to 96.72%. The recommendation coverage of Shanghai–Beijing business and residential and science, education, and culture points of interest could be up to 95.17%, and the maximum time reduction was 2.07 s. The contribution of this study is the introduction of a novel graph attention mechanism, which improves the accuracy and stability of recommendations. This provides an effective technical solution for intercity social network communication recommendations.

Flight Trajectory Prediction Based on Automatic Dependent Surveillance-Broadcast Data Fusion with Interacting Multiple Model and Informer Framework

Aircraft trajectory prediction is challenging because of the flight process with uncertain kinematic motion and varying dynamics, which is characterized by intricate temporal dependencies of the flight surveillance data. To address these challenges, this study proposes a novel hybrid prediction framework, the IMM-Informer, which integrates an interacting multiple model (IMM) approach with the deep learning-based Informer model. The IMM processes flight tracking with multiple typical motion models to produce the initial state predictions. Within the Informer framework, the encoder captures the temporal features with the ProbSparse self-attention mechanism, and the decoder generates trajectory deviation predictions. A final fusion combines the IMM estimators with Informer correction outputs and leverages their respective strengths to achieve accurate and robust predictions. The experiments are conducted using real flight surveillance data received from automatic dependent surveillance-broadcast (ADS-B) sensors to validate the effectiveness of the proposed method. The results demonstrate that the IMM-Informer framework has notable prediction error reductions and significantly outperforms the prediction accuracies of the standalone sequence prediction network models.

A Data-Driven Model for Tunnel Fire Scenario Deduction Based on Optimized Informer

Abstract The prediction models for tunnel fires require exceptionally high accuracy and responsiveness due to the complex enclosed environment and associated high-risk characteristics. However, existing temporal prediction models demonstrate limitations in processing multidimensional data and managing long-term dependencies, making them inadequate for effectively addressing the prediction requirements of tunnel fires. This study introduces the Bidirectional Down-sampling Informer (BiDS-Informer) model, which is built upon innovative components, including the Bidirectional Down-sampling Module, dynamic attention mechanisms, and encoder dimensionality reduction optimization. (1) Within the encoder, the BiDS-Informer utilizes the Bidirectional Down-sampling Module for feature extraction, thereby obtaining a more comprehensive representation of features; (2) The traditional self-attention mechanism is restructured into dynamic attention mechanisms, allowing the model to adaptively identify and process the characteristics of multidimensional time series data; (3) Finally, dimensionality reduction is applied to the encoder's input structure to eliminate temporal encoding information, resulting in a lightweight encoder design. Experimental results indicate that the proposed model outperforms existing time series prediction models regarding predictive accuracy across multiple real-world datasets, particularly on the tunnel fire dataset. Compared to the Informer model, the proposed model shows improvements of 18.56% and 27.49% in average MSE and MAE, respectively, on the tunnel fire dataset. This model provides substantial technical support for enhancing the reliability of tunnel fire early warning systems and holds promising potential for widespread application.

Interpretable capsule networks via self attention routing on spatially invariant feature surfaces

The accurate and efficient evaluation and classification of situational images is fundamental to making informed and effective decisions. However, current classification approaches based on convolutional neural networks often suffer from limited generalization and robustness, particularly when processing data characterized by abstract class features and pronounced spatial attributes. Additionally, the “black-box” nature of deep neural network architectures poses significant challenges to their application in fields with stringent security requirements. To address these limitations, this paper introduces a novel Spatially Invariant Self-Attention Capsule Network (SISA-CapsNet), designed to encode interpretable spatial features for classification tasks. SISA-CapsNet employs capsules to encode spatial features from specific image regions and classifies these features through a self-attention routing mechanism. Specifically, spatially invariant feature surfaces with dimensions identical to the input image are generated and stacked to form feature capsules, each encoding spatial features from distinct regions. The self-attention mechanism calculates coupling coefficients, clustering feature capsules into class capsules. This architecture integrates a spatially invariant feature extraction structure, facilitating pixel-level encoding of regional spatial features, and leverages self-attention to effectively capture the relative importance of different spatial regions for classification. Together, these two mechanisms constitute an interpretable classification framework. Experimental validation on benchmark datasets and battlefield situational image datasets with pronounced spatial characteristics demonstrates that the proposed method not only achieves superior classification performance but also offers interpretability closely aligned with human cognitive processes. Furthermore, comparative analyses with existing visual interpretability method underscore the enhanced interpretability of SISA-CapsNet.

Hadamard product in deep learning: Introduction, Advances and Challenges.

While convolution and self-attention mechanisms have dominated architectural design in deep learning, this survey examines a fundamental yet understudied primitive: the Hadamard product. Despite its widespread implementation across various applications, the Hadamard product has not been systematically analyzed as a core architectural primitive. We present the first comprehensive taxonomy of its applications in deep learning, identifying four principal domains: higher-order correlation, multimodal data fusion, dynamic representation modulation, and efficient pairwise operations. The Hadamard product's ability to model nonlinear interactions with linear computational complexity makes it particularly valuable for resource-constrained deployments and edge computing scenarios. We demonstrate its natural applicability in multimodal fusion tasks, such as visual question answering, and its effectiveness in representation masking for applications including image inpainting and pruning. This systematic review not only consolidates existing knowledge about the Hadamard product's role in deep learning architectures but also establishes a foundation for future architectural innovations. Our analysis reveals the Hadamard product as a versatile primitive that offers compelling trade-offs between computational efficiency and representational power, positioning it as a crucial component in the deep learning toolkit.

Integrating transformer and graph attention network for circRNA-miRNA interaction prediction.

CircRNA-miRNA interaction (CMI) plays a crucial role in the gene regulatory network of the cell. Numerous experiments have shown that abnormalities in CMI can impact molecular functions and physiological processes, leading to the occurrence of specific diseases. Current computational models for predicting CMI typically focus on local molecular entity relationships, thereby neglecting inherent molecular attributes and global structural information. To address these limitations, we propose a multi-feature fusion prediction model based on the transformer and graph attention network, named EGATCMI. Specifically, EGATCMI combines the transformer architecture with Word2vec to pre-train the sequence of circRNA and miRNA, capturing their sequence feature representation and sequence similarity. By leveraging the self-attention mechanism, EGATCMI extracts global structural feature from the CMI network. EGATCMI effectively integrates the obtained multi-feature for prediction, achieving AUC values of 0.9106 and 0.9470 on the CMI-9905 and CircBank datasets, respectively, outperforming existing methods. In case studies that the prediction of interactions between three miRNAs that are closely related to diseases and circRNAs, 8 out of 10 pairs were accurately predicted and validated. Extensive experimental results demonstrate the potential of EGATCMI as a reliable tool for candidate screening in biological investigations.

End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models

In this work, we propose a multi-modal speaker change detection (SCD) approach with focal loss, which integrates both audio and text features to enhance detection performance. The proposed approach utilizes pre-trained large-scale models for feature extraction and incorporates a self-attention mechanism to optimize useful features related to speaker change. The extracted features are fused and processed through a fully connected classification network, with layer normalization and dropout for stability and generalization. To address class imbalance, we apply focal loss, which reduces errors for the difficult samples, leading to better balanced performance. Extensive experiments on a multi-talker meeting dataset demonstrate that the proposed multi-modal approach consistently outperforms single-modal models, proving the complementary nature of audio and text for SCD. Fine-tuning pre-trained models (Wav2Vec2 and Bert) for audio and text significantly boosts accuracy, achieving a 21% improvement over frozen models. The self-attention mechanism further improves performance by 2%, highlighting its ability to capture speaker transition cues effectively. Additionally, focal loss enhances the model’s performance, making it more robust to imbalanced data.

Crystal Structure Prediction Using a Self-Attention Neural Network and Semantic Segmentation.

The development of new materials is a time-consuming and resource-intensive process. Deep learning has emerged as a promising approach to accelerate this process. However, accurately predicting crystal structures using deep learning remains a significant challenge due to the complex, high-dimensional nature of atomic interactions and the scarcity of comprehensive training data that captures the full diversity of possible crystal configurations. This work developed a neural network model based on a data set comprising thousands of crystallographic information files from existing crystal structure databases. The model incorporates a self-attention mechanism to enhance prediction accuracy by learning and extracting both local and global features of three-dimensional structures, treating the atoms in each crystal as point sets. This approach enables effective semantic segmentation and accurate unit cell prediction. Experimental results demonstrate that for unit cells containing up to 500 atoms, the model achieves a structure prediction accuracy of 89.78%.

Improved Deep Convolutional Generative Adversarial Network for Data Augmentation of Gas Polyethylene Pipeline Defect Images

Gas polyethylene (PE) pipes have an become essential component of the urban gas pipeline network due to their long service life and corrosion resistance. To prevent safety incidents, regular monitoring of gas pipelines is crucial. Traditional inspection methods face significant challenges, including low efficiency, high costs, and limited applicability. Machine vision-based inspection methods have emerged as a key solution to these issues. Despite this, the method also encounters the problem of scarcity of defect samples and uneven data distribution in gas pipeline defect detection. For this reason, an improved Deep Convolutional Generative Adversarial Network (DCGAN) is proposed. By integrating the Minibatch Discrimination (MD), Spectral Normalization (SN), Self-Attention Mechanism (SAM) and Two-Timescale Update Rule (TTUR), the proposed approach overcomes the original DCGAN’s limitations, including mode collapse, low resolution of generated images, and unstable training, the data augmentation of defective images inside the pipeline is realized. Experimental results demonstrate the superiority of the improved algorithm in terms of image generation quality and diversity, while the ablation study validates the positive impact of the improvement in each part. Additionally, the relationship between the number of augmented images and classification accuracy, showing that classifier performance improved across all scenarios when generated defect images were included. The findings indicate that the images produced by the improved model significantly enhance defect detection accuracy and hold considerable potential for practical application.

Inter-sentential translation and language perspective in Neural Machine Translation: insights from ChatGPT as a transformer-based model

ABSTRACT This study examines the challenges and advancements in inter-sentential translation and language perspective adaptation within Neural Machine Translation (NMT) systems, using ChatGPT as a representative of transformer-based models. Earlier NMT approaches, including Statistical Machine Translation (SMT) and initial neural architectures, often failed to maintain coherence across sentences or adapt cultural and linguistic nuances, leading to contextually inadequate translations. Leveraging self-attention mechanisms, ChatGPT demonstrates significant improvements, achieving 100% accuracy in resolving inter-sentential dependencies, such as gendered pronoun disambiguation, and effectively managing language perspective shifts by capturing intricate cultural, stylistic, and syntactic elements. Focusing on Indonesian-to-English translations, this study highlights the model’s ability to bridge linguistic and cultural disparities, offering a deeper understanding of language perspective. The findings highlight the importance of integrating contextual datasets and gender-balanced training materials to enhance translation quality and fairness. While ChatGPT exemplifies the potential of advanced NMT systems, its performance represents a case study and may not be generalizable to all NMT models. This research advances the discourse on NMT by emphasizing inter-sentential translation and language perspective as critical components for achieving human-like translation capabilities across diverse linguistic and cultural scenarios.

BERT-Based Code Learning for Exception Localization and Type Prediction

Exception handling is crucial but challenging in program development. It needs to identify and handle all potential exceptions within programs to ensure system security and stabilization. Traditional exception handling relies on the expertise and experience of programmers, which often leads to oversights. Therefore, identifying exceptional code and recommending handling solutions are hot research topics with significant practical value. This paper presents a model called CodeHunter for exception localization and type prediction. The model first utilizes BERT-based model to represent code features and then uses Bi-LSTM for sequence labeling to pinpoint exceptional code. Additionally, this model also considers contextual features of the exception code and learns weights for the code within the try block and its context through the self-attention mechanism. Subsequently, it performs exception localization and predicts exception types. We conduct experiments on three different datasets. The results demonstrate that in the task of exception localization, our model can achieve a maximum accuracy of 98.6%, exceeding SOTA baselines by 11.2%. In the task of exception type prediction, our model can surpass the accuracy of SOTA baselines by a maximum of 18.7%, achieving 92.0% Top-1 accuracy. The rationality of techniques used in our model is also proved by the ablation testing. The model is implemented as an IDE plugin for programming convenience.

VersaFusion: A Versatile Diffusion-Based Framework for Fine-Grained Image Editing and Enhancement

Text-to-image (T2I) diffusion models have achieved remarkable progress in generating realistic images from textual descriptions. However, ensuring consistent high-quality image generation with complete backgrounds, object appearance, and optimal texture rendering remains challenging. This paper presents a novel fine-grained pixel-level image editing method based on pre-trained diffusion models. The proposed dual-branch architecture, consisting of Guidance and Generation branches, employs U-Net Denoisers and Self-Attention mechanisms. An improved DDIM-like inversion method obtains the latent representation, followed by multiple denoising steps. Cross-branch interactions, such as KV Replacement, Classifier Guidance, and Feature Correspondence, enable precise control while preserving image fidelity. The iterative refinement and reconstruction process facilitates finegrained editing control, supporting attribute modification, image outpainting, style transfer, and face synthesis with Clickand-Drag style editing using masks. Experimental results demonstrate the effectiveness of the proposed approach in enhancing the quality and controllability of T2I-generated images, surpassing existing methods while maintaining attractive computational complexity for practical real-world applications.

Faithful and Accurate Self-Attention Attribution for Message Passing Neural Networks via the Computation Tree Viewpoint

The self-attention mechanism has been adopted in various popular message passing neural networks (MPNNs), enabling the model to adaptively control the amount of information that flows along the edges of the underlying graph. Such attention-based MPNNs (Att-GNNs) have also been used as a baseline for multiple studies on explainable AI (XAI) since attention has steadily been seen as natural model interpretations, while being a viewpoint that has already been popularized in other domains (e.g., natural language processing and computer vision). However, existing studies often use naive calculations to derive attribution scores from attention, undermining the potential of attention as interpretations for Att-GNNs. In our study, we aim to fill the gap between the widespread usage of Att-GNNs and their potential explainability via attention. To this end, we propose GAtt, edge attribution calculation method for self-attention MPNNs based on the computation tree, a rooted tree that reflects the computation process of the underlying model. Despite its simplicity, we empirically demonstrate the effectiveness of GAtt in three aspects of model explanation: faithfulness, explanation accuracy, and case studies by using both synthetic and real-world benchmark datasets. In all cases, the results demonstrate that GAtt greatly improves edge attribution scores, especially compared to the previous naive approach.

DocMamba: Efficient Document Pre-training with State Space Model

In recent years, visually-rich document understanding has attracted increasing attention. Transformer-based pre-trained models have become the mainstream approach, yielding significant performance gains in this field. However, the self-attention mechanism's quadratic computational complexity hinders their efficiency and ability to process long documents. In this paper, we present DocMamba, a novel framework based on the state space model. It is designed to reduce computational complexity to linear while preserving global modeling capabilities. To further enhance its effectiveness in document processing, we introduce the Segment-First Bidirectional Scan (SFBS) to capture contiguous semantic information. Experimental results demonstrate that DocMamba achieves new state-of-the-art results on downstream datasets such as FUNSD, CORD, and SORIE, while significantly improving speed and reducing memory usage. Notably, experiments on the HRDoc confirm DocMamba's potential for length extrapolation.

Semi-Supervised Multi-View Multi-Label Learning with View-Specific Transformer and Enhanced Pseudo-Label

Multi-view multi-label learning has become a research focus for describing objects with rich expressions and annotations. However, real-world data often contains numerous unlabeled instances, due to the high cost and technical limitations of manual labeling. This crucial problem involves three main challenges: i) How to extract advanced semantics from available views? ii) How to build a refined classification framework with limited labeled space? iii) How to provide more high-quality supervisory information? To address these problems, we propose a Semi-Supervised Multi-View Multi-Label Learning Method with View-Specific Transformer and Enhanced Pseudo-Label named SMVTEP. Specifically, Generative Adversarial Networks are employed to extract informative shared and specific representations and their consistency and distinctiveness are ensured through the adversarial mechanism and information theory based contrastive learning. Then we build specific classifiers for each extracted feature and apply instance-level manifold constraints to reduce bias across classifiers. Moreover, we design a transformer-style fusion approach that simultaneously captures the imbalance of expressive power among views, mapping effects on specific labels, and label dependencies by incorporating confidence scores and category semantics into the self-attention mechanism. Furthermore, after using Mixup for data augmentation, category-enhanced pseudo-labels are leveraged to improve the reliability of additional annotations by aligning the label distribution of unlabeled samples with the true distribution. Finally, extensive experimental results validate the effectiveness of SMVTEP against state-of-the-art methods.

Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition

Wearable Human Activity Recognition (WHAR) is a prominent research area within ubiquitous computing. Multi-sensor synchronous measurement has proven to be more effective for WHAR than using a single sensor. However, existing WHAR methods use shared convolutional kernels for indiscriminate temporal feature extraction across each sensor variable, which fails to effectively capture spatio-temporal relationships of intra-sensor and inter-sensor variables. We propose the DecomposeWHAR model consisting of a decomposition phase and a fusion phase to better model the relationships between modality variables. The decomposition creates high-dimensional representations of each intra-sensor variable through the improved Depth Separable Convolution to capture local temporal features while preserving their unique characteristics. The fusion phase begins by capturing relationships between intra-sensor variables and fusing their features at both the channel and variable levels. Long-range temporal dependencies are modeled using the State Space Model (SSM), and later cross-sensor interactions are dynamically captured through a self-attention mechanism, highlighting inter-sensor spatial correlations. Our model demonstrates superior performance on three widely used WHAR datasets, significantly outperforming state-of-the-art models while maintaining acceptable computational efficiency.