Self-attention Module Research Articles

GITPose: going shallow and deeper using vision transformers for human pose estimation

In comparison to convolutional neural networks (CNN), the newly created vision transformer (ViT) has demonstrated impressive outcomes in human pose estimation (HPE). However, (1) there is a quadratic rise in complexity with respect to image size, which causes the traditional ViT to be unsuitable for scaling, and (2) the attention process at the transformer encoder as well as decoder also adds substantial computational costs to the detector’s overall processing time. Motivated by this, we propose a novel Going shallow and deeper with vIsion Transformers for human Pose estimation (GITPose) without CNN backbones for feature extraction. In particular, we introduce a hierarchical transformer in which we utilize multilayer perceptrons to encode the richest local feature tokens in the initial phases (i.e., shallow), whereas self-attention modules are employed to encode long-term relationships in the deeper layers (i.e., deeper), and a decoder for keypoint detection. In addition, we offer a learnable deformable token association module (DTA) to non-uniformly and dynamically combine informative keypoint tokens. Comprehensive evaluation and testing on the COCO and MPII benchmark datasets reveal that GITPose achieves a competitive average precision (AP) on pose estimation compared to its state-of-the-art approaches.

Spatio-Temporal Self-Attention Network for Origin–Destination Matrix Prediction in Urban Rail Transit

Short-term origin–destination (OD) prediction in urban rail transit (URT) is vital for improving URT operation. However, due to the problems such as the unavailability of the OD matrix of the current day, high dimension and long-range spatio-temporal dependencies, it is difficult to further improve the prediction accuracy of an OD matrix. In this paper, a novel spatio-temporal self-attention network (SSNet) for OD matrix prediction in URT is proposed to further improve the prediction accuracy. In the proposed SSNet, a lightweight yet effective spatio-temporal self-attention module (STSM) is proposed to capture complex long-range spatio-temporal dependencies, thus helping improve the prediction accuracy of the proposed SSNet. Additionally, the finished OD matrices on previous days are used as the only data source without the passenger flow data on the current day in the proposed SSNet, which makes it possible to predict the OD matrices of all time intervals on the current day before the operation of the current day. It is demonstrated by experiments that the proposed SSNet outperforms three advanced deep learning methods for short-term OD prediction in URT, and the proposed STSM plays an important role in improving the prediction accuracy.

Post-Operative Brain MRI Resection Cavity Segmentation Model and Follow-Up Treatment Assistance

Post-operative brain magnetic resonance imaging (MRI) segmentation is inherently challenging due to the diverse patterns in brain tissue, which makes it difficult to accurately identify resected areas. Therefore, there is a crucial need for a precise segmentation model. Due to the scarcity of post-operative brain MRI scans, it is not feasible to use complex models that require a large amount of training data. This paper introduces an innovative approach for accurately segmenting and quantifying post-operative brain resection cavities in MRI scans. The proposed model, named Attention-Enhanced VGG-U-Net, integrates VGG16 initial weights in the encoder section and incorporates a self-attention module in the decoder, offering improved accuracy for postoperative brain MRI segmentation. The attention mechanism enhances its accuracy by concentrating on a specific area of interest. The VGG16 model is comparatively lightweight, has pre-trained weights, and allows the model to extract incredibly detailed information from the input. The model is trained on publicly available post-operative brain MRI data and achieved a Dice coefficient value of 0.893. The model is then assessed using a clinical dataset of postoperative brain MRIs. The model facilitates the quantification of the resected regions and enables comparisons with each brain region based on pre-operative images. The capabilities of the model assist radiologists in evaluating surgical success and directing follow-up procedures.

Hyperspectral Image Classification with the Orthogonal Self-Attention ResNet and Two-Step Support Vector Machine

Hyperspectral image classification plays a crucial role in remote sensing image analysis by classifying pixels. However, the existing methods require more spatial–global information interaction and feature extraction capabilities. To overcome these challenges, this paper proposes a novel model for hyperspectral image classification using an orthogonal self-attention ResNet and a two-step support vector machine (OSANet-TSSVM). The OSANet-TSSVM model comprises two essential components: a deep feature extraction network and an improved support vector machine (SVM) classification module. The deep feature extraction network incorporates an orthogonal self-attention module (OSM) and a channel attention module (CAM) to enhance the spatial–spectral feature extraction. The OSM focuses on computing 2D self-attention weights for the orthogonal dimensions of an image, resulting in a reduced number of parameters while capturing comprehensive global contextual information. In contrast, the CAM independently learns attention weights along the channel dimension. The CAM autonomously learns attention weights along the channel dimension, enabling the deep network to emphasise crucial channel information and enhance the spectral feature extraction capability. In addition to the feature extraction network, the OSANet-TSSVM model leverages an improved SVM classification module known as the two-step support vector machine (TSSVM) model. This module preserves the discriminative outcomes of the first-level SVM subclassifier and remaps them as new features for the TSSVM training. By integrating the results of the two classifiers, the deficiencies of the individual classifiers were effectively compensated, resulting in significantly enhanced classification accuracy. The performance of the proposed OSANet-TSSVM model was thoroughly evaluated using public datasets. The experimental results demonstrated that the model performed well in both subjective and objective evaluation metrics. The superiority of this model highlights its potential for advancing hyperspectral image classification in remote sensing applications.

MCFNet: Multi-Attentional Class Feature Augmentation Network for Real-Time Scene Parsing

For real-time scene parsing tasks, capturing multi-scale semantic features and performing effective feature fusion is crucial. However, many existing solutions ignore stripe-shaped things like poles, traffic lights and are so computationally expensive that cannot meet the high real-time requirements. This article presents a novel model, the Multi-Attention Class Feature Augmentation Network (MCFNet) to address this challenge. MCFNet is designed to capture long-range dependencies across different scales with low computational cost and to perform a weighted fusion of feature maps. It features the BAM (Strip Matrix Based Attention Module) for extracting strip objects in images. The BAM module replaces the conventional self-attention method using square matrices with strip matrices, which allows it to focus more on strip objects while reducing computation. Additionally, MCFNet has a parallel branch that focuses on global information based on self-attention to avoid wasting computation. The two branches are merged to enhance the performance of traditional self-attention modules. Experimental results on two mainstream datasets demonstrate the effectiveness of MCFNet. On the Camvid and Cityscapes test sets, MCFNet achieved 207.5 FPS/73.5% mIoU and 136.1 FPS/71.63% mIoU, respectively. The experiments show that MCFNet outperforms other models on the Camvid dataset and can significantly improve the performance of real-time scene parsing tasks.

Convergence of polarized self-attention with consistent rank Chinese text classification

Utilizing the powerful feature extraction capabilities of deep learning, a text classification algorithm with multi-dimensional and high-domain adaptability is designed in this study. This method enhances the models understanding of topics and content by incorporating the Polarized Self-Attention (PSA) module, which strengthens the spatial structure and semantic features of textual information. The loss function is redesigned to assign smaller losses to misclassifications of neighboring categories, allowing the model to optimize classification accuracy while learning hierarchical structural information between categories. Finally, experimental verification is conducted on a publicly available news dataset, demonstrating improved results in text classification achieved by the proposed algorithm.

Independent Dual Graph Attention Convolutional Network for Skeleton-Based Action Recognition

Graph convolutional networks (GCNs) have been widely adopted in skeleton-based action recognition, achieving impressive outcomes. However, the convolution operations in GCNs fail to make full use of the original input data, which restricts its ability to accurately capture the correlation within the skeleton. To solve this issue, this study introduces an independent dual graph attention convolutional network (IDGAN). Specifically, IDGAN additionally incorporates an instinctive attention module that leverages self-attention to capture the correlation among the joints in the original input skeleton. In addition, two independent convolutional operations are used to process two self-attention modules, respectively, to further refine the relationship between skeleton joints. Extensive experiments on several publicly available datasets show that IDGAN outperforms most state-of-the-art algorithms.

Spatial deformable transformer for 3D point cloud registration

Deformable attention only focuses on a small group of key sample-points around the reference point and make itself be able to capture dynamically the local features of input feature map without considering the size of the feature map. Its introduction into point cloud registration will be quicker and easier to extract local geometric features from point cloud than attention. Therefore, we propose a point cloud registration method based on Spatial Deformable Transformer (SDT). SDT consists of a deformable self-attention module and a cross-attention module where the deformable self-attention module is used to enhance local geometric feature representation and the cross-attention module is employed to enhance feature discriminative capability of spatial correspondences. The experimental results show that compared to state-of-the-art registration methods, SDT has a better matching recall, inlier ratio, and registration recall on 3DMatch and 3DLoMatch scene, and has a better generalization ability and time efficiency on ModelNet40 and ModelLoNet40 scene.

A novel Swin transformer approach utilizing residual multi-layer perceptron for diagnosing brain tumors in MRI images

Serious consequences due to brain tumors necessitate a timely and accurate diagnosis. However, obstacles such as suboptimal imaging quality, issues with data integrity, varying tumor types and stages, and potential errors in interpretation hinder the achievement of precise and prompt diagnoses. The rapid identification of brain tumors plays a pivotal role in ensuring patient safety. Deep learning-based systems hold promise in aiding radiologists to make diagnoses swiftly and accurately. In this study, we present an advanced deep learning approach based on the Swin Transformer. The proposed method introduces a novel Hybrid Shifted Windows Multi-Head Self-Attention module (HSW-MSA) along with a rescaled model. This enhancement aims to improve classification accuracy, reduce memory usage, and simplify training complexity. The Residual-based MLP (ResMLP) replaces the traditional MLP in the Swin Transformer, thereby improving accuracy, training speed, and parameter efficiency. We evaluate the Proposed-Swin model on a publicly available brain MRI dataset with four classes, using only test data. Model performance is enhanced through the application of transfer learning and data augmentation techniques for efficient and robust training. The Proposed-Swin model achieves a remarkable accuracy of 99.92%, surpassing previous research and deep learning models. This underscores the effectiveness of the Swin Transformer with HSW-MSA and ResMLP improvements in brain tumor diagnosis. This method introduces an innovative diagnostic approach using HSW-MSA and ResMLP in the Swin Transformer, offering potential support to radiologists in timely and accurate brain tumor diagnosis, ultimately improving patient outcomes and reducing risks.

Deep Multi-Instance Conv-Transformer Frameworks for Landmark-Based Brain MRI Classification

For brain diseases, e.g., autism spectrum disorder (ASD), with unclear biological characteristics, the detection of imaging-based biomarkers is a critical task for diagnosis. Several landmark-based categorization approaches have been developed for the computer-aided diagnosis of brain diseases, such as Alzheimer’s disease (AD), utilizing structural magnetic resonance imaging (sMRI). With the automatic detection of the landmarks of brain disease, more detailed brain features were identified for clinical diagnosis. Multi-instance learning is an effective technique for classifying brain diseases based on landmarks. The multiple-instance learning approach relies on the assumption of independent distribution hypotheses and is mostly focused on local information, thus the correlation among different brain regions may be ignored. However, according to previous research on ASD and AD, the abnormal development of different brain regions is highly correlated. Vision Transformers, with self-attention modules to capture the relationship between embedded patches from a whole image, have recently demonstrated superior performances in many computer vision tasks. Nevertheless, the utilization of 3D brain MRIs imposes a substantial computational load, especially while training with Vision Transformer. To address the challenges mentioned above, in this research, we proposed a landmark-based multi-instance Conv-Transformer (LD-MILCT) framework as a solution to the aforementioned issues in brain disease diagnosis. In this network, a two-stage multi-instance learning strategy was proposed to explore both spatial and morphological information between different brain regions; the Vision Transformer utilizes a multi-instance learning head (MIL head) to fully utilize the features that are not involved in the ultimate classification. We assessed our proposed framework using T1-weighted MRI images from both AD and ASD databases. Our method outperformed existing deep learning and landmark-based methods in terms of brain MRI classification tasks.

A Self-Attention CycleGAN for Cross-Domain Semi-Supervised Contactless Palmprint Recognition

Nowadays, there is a growing concern about contactless palmprint recognition because of its high-recognition rate, efficiency, and convenience. With the development of image acquisition equipment, it is an often case that the palmprint images for identification and for registration are captured by different devices. At the same time, a large amount of well-labeled palmprint images are difficult to collect. Therefore, the performance of most existing contactless palmprint recognition methods will be poor in real-life applications. To address these issues, we proposed a self-attention CycleGAN for cross-domain semi-supervised palmprint recognition. Based on CycleGAN, the styles of contactless palmprint images in source domain and target domain can be swapped. Specifically, the spatial features are captured through self-attention modules by modeling long-range dependencies. In addition, an extra source domain classifier is trained with the labeled source domain images to give the unlabeled images in target domain a pseudo-label, by which images in target domain are efficiently utilized. The experiment results showed that our method achieved competitive performance.

Hyperspectral image change detection based on an improved multi-scale and spectral-wise transformer

ABSTRACT Change detection (CD) has a significant application in the remote sensing field. Because of the popularity of hyperspectral image (HSI) and the application of deep learning methods, hyperspectral image change detection (HSI-CD) techniques have been greatly developed. Among them, convolutional neural network (CNN) has garnered the greatest interest in HSI-CD due to their superior feature learning capabilities. However, current CNN-based algorithms have trouble capturing spectral similarity and long-range dependency owing to their intrinsic structural restrictions. Recently, transformers have been shown to extract global dependency from nature images in an extremely efficient way. But it has some difficulties in handling high-dimensional data, such as HSI. To address these issues, we propose an improved multi-scale and spectral-wise transformer (MS-SWT). The proposed MS-SWT is capable of capturing spectral similarity and long-range dependence between bands to enhance the efficiency of the HSI-CD task. Furthermore, to maximize the utilization of spatial information, we present a multi-scale feature fusion module (MFFM) to extract and fuse different dimensions of spatial features. More importantly, a locality self-attention (LSA) module is employed to alleviate the problem of smoothing the distribution of attention scores due to the large number of spectral embeddings. Moreover, we design a channel self-supervised loss function that can capture intrinsic information from the spectral channels to further strengthen the robustness of model training when the training samples are scarce. Lastly, comprehensive experiments present the high performance of our MS-SWT on four bitemporal HSI datasets and demonstrate the superiority of MS-SWT over state-of-the-art approaches.

Time Interval-Enhanced Graph Neural Network for Shared-Account Cross-Domain Sequential Recommendation.

Shared-account cross-domain sequential recommendation (SCSR) task aims to recommend the next item via leveraging the mixed user behaviors in multiple domains. It is gaining immense research attention as more and more users tend to sign up on different platforms and share accounts with others to access domain-specific services. Existing works on SCSR mainly rely on mining sequential patterns via recurrent neural network (RNN)-based models, which suffer from the following limitations: 1) RNN-based methods overwhelmingly target discovering sequential dependencies in single-user behaviors and they are not expressive enough to capture the relationships among multiple entities in SCSR; 2) all existing methods bridge two domains via knowledge transfer in the latent space and ignore the explicit cross-domain graph structure; and 3) none existing studies consider the time interval information among items, which is essential in the sequential recommendation for characterizing different items and learning discriminative representations for them. In this work, we propose a new graph-based solution, namely, time interval-enhanced domain-aware graph convolutional network (TiDA-GCN), to address the above challenges. Specifically, we first link users and items in each domain as a graph. Then, we devise a domain-aware graph convolution network to learn user-specific node representations. To fully account for users' domain-specific preferences on items, two effective attention mechanisms are further developed to selectively guide the message-passing process. Moreover, to further enhance item- and account-level representation learning, we incorporate the time interval into the message passing and design an account-aware self-attention module for learning items' interactive characteristics. Experiments demonstrate the superiority of our proposed method from various aspects.

Intra- and inter-sector contextual information fusion with joint self-attention for file fragment classification

File fragment classification (FFC) aims to identify the file type of file fragments in memory sectors, which is of great importance in memory forensics and information security. Existing works focused on processing the bytes within sectors separately and ignoring contextual information between adjacent sectors. In this paper, we introduce a joint self-attention network (JSANet) for FFC to learn intra-sector local features and inter-sector contextual features. Specifically, we propose an end-to-end network with the byte, channel, and sector self-attention modules. Byte self-attention adaptively recognizes the intra-sector significant bytes, and channel self-attention re-calibrates the features between channels. Based on the insight that adjacent memory sectors are most likely to store a file fragment, sector self-attention leverages contextual information in neighboring sectors to enhance inter-sector feature representation. Extensive experiments on seven FFC benchmarks show the superiority of our method compared with state-of-the-art methods. Moreover, we construct VFF-16, a variable-length file fragment dataset to reflect file fragmentation. Integrated with sector self-attention, our method improves accuracy by more than 16.3% against the baseline on VFF-16, and the runtime achieves 5.1 s/GB with GPU acceleration. In addition, we extend our model to malware detection and show its applicability.

SCANeXt: Enhancing 3D medical image segmentation with dual attention network and depth-wise convolution

Existing approaches to 3D medical image segmentation can be generally categorized into convolution-based or transformer-based methods. While convolutional neural networks (CNNs) demonstrate proficiency in extracting local features, they encounter challenges in capturing global representations. In contrast, the consecutive self-attention modules present in vision transformers excel at capturing long-range dependencies and achieving an expanded receptive field. In this paper, we propose a novel approach, termed SCANeXt, for 3D medical image segmentation. Our method combines the strengths of dual attention (Spatial and Channel Attention) and ConvNeXt to enhance representation learning for 3D medical images. In particular, we propose a novel self-attention mechanism crafted to encompass spatial and channel relationships throughout the entire feature dimension. To further extract multiscale features, we introduce a depth-wise convolution block inspired by ConvNeXt after the dual attention block. Extensive evaluations on three benchmark datasets, namely Synapse, BraTS, and ACDC, demonstrate the effectiveness of our proposed method in terms of accuracy. Our SCANeXt model achieves a state-of-the-art result with a Dice Similarity Score of 95.18% on the ACDC dataset, significantly outperforming current methods.

An Orientation-Aware Attention Network for Person Re-Identification

Humans always identify persons through their characteristics, salient attributes, and these attributes’ locations on the body. Most person re-identification methods focus on global and local features corresponding to the former two discriminations, cropping person images into horizontal strips to obtain coarse locations of body parts. However, discriminative clues corresponding to location differences cannot be discovered, so persons with similar appearances are often confused because of their alike components. To address the above problem, we introduce pixel-wise relative positions for the invariance of their orientations in viewpoint changes. To cope with the scale change of relative position, we combine relative positions with self-attention modules that perform on multi-level features. Moreover, in the data augmentation stage, mirrored images are given new labels due to the conversion of the relative position along a horizontal orientation and change in visual chirality. Extensive experiments on four challenging benchmarks demonstrate that the proposed approach shows its superiority and effectiveness in discovering discriminating features.

Exploring Brain Dynamics via EEG and Steady-State Activation Map Networks in Music Composition.

In recent years, the integration of brain-computer interface technology and neural networks in the field of music generation has garnered widespread attention. These studies aimed to extract individual-specific emotional and state information from electroencephalogram (EEG) signals to generate unique musical compositions. While existing research has focused primarily on brain regions associated with emotions, this study extends this research to brain regions related to musical composition. To this end, a novel neural network model incorporating attention mechanisms and steady-state activation mapping (SSAM) was proposed. In this model, the self-attention module enhances task-related information in the current state matrix, while the extended attention module captures the importance of state matrices over different time frames. Additionally, a convolutional neural network layer is used to capture spatial information. Finally, the ECA module integrates the frequency information learned by the model in each of the four frequency bands, mapping these by learning their complementary frequency information into the final attention representation. Evaluations conducted on a dataset specifically constructed for this study revealed that the model surpassed representative models in the emotion recognition field, with recognition rate improvements of 1.47% and 3.83% for two different music states. Analysis of the attention matrix indicates that the left frontal lobe and occipital lobe are the most critical brain regions in distinguishing between 'recall and creation' states, while FP1, FPZ, O1, OZ, and O2 are the electrodes most related to this state. In our study of the correlations and significances between these areas and other electrodes, we found that individuals with musical training exhibit more extensive functional connectivity across multiple brain regions. This discovery not only deepens our understanding of how musical training can enhance the brain's ability to work in coordination but also provides crucial guidance for the advancement of brain-computer music generation technologies, particularly in the selection of key brain areas and electrode configurations. We hope our research can guide the work of EEG-based music generation to create better and more personalized music.

MSNSegNet: attention-based multi-shape nuclei instance segmentation in histopathology images.

In clinical research, the segmentation of irregularly shaped nuclei, particularly in mesenchymal areas like fibroblasts, is crucial yet often neglected. These irregular nuclei are significant for assessing tissue repair in immunotherapy, a process involving neovascularization and fibroblast proliferation. Proper segmentation of these nuclei is vital for evaluating immunotherapy's efficacy, as it provides insights into pathological features. However, the challenge lies in the pronounced curvature variations of these non-convex nuclei, making their segmentation more difficult than that of regular nuclei. In this work, we introduce an undefined task to segment nuclei with both regular and irregular morphology, namely multi-shape nuclei segmentation. We propose a proposal-based method to perform multi-shape nuclei segmentation. By leveraging the two-stage structure of the proposal-based method, a powerful refinement module with high computational costs can be selectively deployed only in local regions, improving segmentation accuracy without compromising computational efficiency. We introduce a novel self-attention module to refine features in proposals for the sake of effectiveness and efficiency in the second stage. The self-attention module improves segmentation performance by capturing long-range dependencies to assist in distinguishing the foreground from the background. In this process, similar features get high attention weights while dissimilar ones get low attention weights. In the first stage, we introduce a residual attention module and a semantic-aware module to accurately predict candidate proposals. The two modules capture more interpretable features and introduce additional supervision through semantic-aware loss. In addition, we construct a dataset with a proportion of non-convex nuclei compared with existing nuclei datasets, namely the multi-shape nuclei (MsN) dataset. Our MSNSegNet method demonstrates notable improvements across various metrics compared to the second-highest-scoring methods. For all nuclei, the score improved by approximately 1.66 , by about 2.15 , and by roughly 0.65 . For non-convex nuclei, which are crucial in clinical applications, our method's improved significantly by approximately 3.86 and by around 2.54 . These enhancements underscore the effectiveness of our approach on multi-shape nuclei segmentation, particularly in challenging scenarios involving irregularly shaped nuclei.

Night construction site detection based on ghost-YOLOX

Existing target detection algorithms, when applied to night-time site monitoring, are limited by site hardware conditions, lighting conditions, fuzzy and small targets to be detected, etc., which makes it difficult to achieve good detection results. In this paper, we propose the Ghost-YOLOX algorithm for night-time site monitoring using YOLOX-X as a baseline model. In order to reduce the number of model parameters and improve the detection speed, the algorithm is based on the Ghost convolution and SimAM self-attention modules build the Sim-Ghost residual module according to the gradient path design strategy, and uses it to reconstruct the backbone and neck networks; In order to improve the detection ability of the network for fuzzy targets and small targets, based on Involution, a Cross Involution Attention (CIA) module is constructed by double cross convolution, and added to the neck network to enable the network to obtain more efficient channel and spatial attention. The experimental results show that compared with the original YOLOX-X algorithm, the Ghost-YOLOX algorithm reduces the number of parameters to 16.7% of the original model, and the detection speed increases by 2.3 times, at the same time, the average accuracy of the model has increased by 2.55%.

Automatic high-precision crack detection of post-earthquake structure based on self-supervised transfer learning method and SegCrackFormer

Accurate crack detection is essential for structural damage assessment after earthquake disasters. However, due to the gap between the target domain of the detected structure and the source domain, it is challenging to achieve high-precision crack segmentation when performing crack detection based on deep learning (DL) in actual engineering. This article proposes a crack segmentation transfer learning method based on a self-supervised learning mechanism and a high-quality pseudo-label generation method, which can significantly improve the detection accuracy in the target domain without pre-made annotations. Besides, to improve the crack segmentation model’s ability to extract local and global features, this article proposes a SegCrackFormer model, which embeds convolutional layers and multi-head self-attention modules. An experiment of the crack segmentation transfer learning method is performed on two open-source crack datasets, METU and Crack500, and a newly proposed LD dataset. The experimental results show that the crack segmentation transfer learning method proposed in this article can improve the mean intersection over union (mIoU) by 38.41% and 15.66% on the Crack500 and LD datasets, respectively. The proposed SegCrackFormer is evaluated through comparative experiments, which demonstrate its superiority over existing crack segmentation models on the METU dataset. Additionally, the proposed method is shown to require significantly less computational resources than other existing models, which highlights the potential of SegCrackFormer as a powerful and efficient model for crack segmentation in practical applications.