Self-attention Model Research Articles

Transformer for one stop interpretable cell type annotation

Consistent annotation transfer from reference dataset to query dataset is fundamental to the development and reproducibility of single-cell research. Compared with traditional annotation methods, deep learning based methods are faster and more automated. A series of useful single cell analysis tools based on autoencoder architecture have been developed but these struggle to strike a balance between depth and interpretability. Here, we present TOSICA, a multi-head self-attention deep learning model based on Transformer that enables interpretable cell type annotation using biologically understandable entities, such as pathways or regulons. We show that TOSICA achieves fast and accurate one-stop annotation and batch-insensitive integration while providing biologically interpretable insights for understanding cellular behavior during development and disease progressions. We demonstrate TOSICA’s advantages by applying it to scRNA-seq data of tumor-infiltrating immune cells, and CD14+ monocytes in COVID-19 to reveal rare cell types, heterogeneity and dynamic trajectories associated with disease progression and severity.

Remaining Useful Life Prediction for Aero-Engines Using a Time-Enhanced Multi-Head Self-Attention Model

Data-driven Remaining Useful Life (RUL) prediction is one of the core technologies of Prognostics and Health Management (PHM). Committed to improving the accuracy of RUL prediction for aero-engines, this paper proposes a model that is entirely based on the attention mechanism. The attention model is divided into the multi-head self-attention and timing feature enhancement attention models. The multi-head self-attention model employs scaled dot-product attention to extract dependencies between time series; the timing feature enhancement attention model is used to accelerate and enhance the feature selection process. This paper utilises Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) turbofan engine simulation data obtained from NASA Ames’ Prognostics Center of Excellence and compares the proposed algorithm to other models. The experiments conducted validate the superiority of our model’s approach.

A mutual embedded self-attention network model for code search

To improve the efficiency of program implementation, developers can selectively reuse the previously written code by searching the open-source codebase. To date, many code search methods have been proposed to actively push the limit of code search accuracy, where the methods designed using Self-Attention mechanism are particularly promising. However, while existing methods can improve the efficiency to capture textual semantics by attending significant words in the code component unit, they typically fail to capture the structural dependencies between the code components which may produce suboptimal search accuracy. In this paper, we propose a novel Self-Attention model termed MESN-CS which considers both word-level attention and code unit-level attention for code search. MESN-CS not only the attention weight of each word in the code component unit is calculated, but also the weight of the embedding between the code combination units is calculated. To verify the effectiveness of the proposed model, three benchmark models were compared on a large-scale code data and CodesearchNet. The experimental results show that the MESN-CS has better Recall@k, NDCG and MRR performance than baseline methods. the experiments also show that the semantic syntactic information between sequences can be effectively characterized in MESN-CS.

Unsupervised self-attention lightweight photo-to-sketch synthesis with feature maps

Face-sketch synthesis is important for gaining a clear portrait photo of suspects when solving crimes. Recent research has made a great process in self-attention generative adversarial networks. We propose a method of unsupervised learning in the synthesis of face sketch-to-photo using a new attention module. The method of processing on a small reference set of photo-sketch pairs adds to the attention module, a focus on the regions distinguishing photos from sketches on the basis of the feature maps obtained by the auxiliary classifier. Unlike previous attention-based methods, which cannot handle the geometric changes between domains, our model can translate images requiring holistic changes. At the same time, we reduce the layers of the discriminator according to different residual layers to optimize our network. With the proposed approach, we can train our networks using a small reference set of photo-sketch pairs together with a large number of face-photo datasets and more distinguishing facial-feature regions in the self-attention model. Experiments have shown the superiority of the proposed method to existing face sketch-to-photo synthesis models using fixed network architectures and hyper-parameters.

Combining convolutional neural networks and self-attention for fundus diseases identification

Early detection of lesions is of great significance for treating fundus diseases. Fundus photography is an effective and convenient screening technique by which common fundus diseases can be detected. In this study, we use color fundus images to distinguish among multiple fundus diseases. Existing research on fundus disease classification has achieved some success through deep learning techniques, but there is still much room for improvement in model evaluation metrics using only deep convolutional neural network (CNN) architectures with limited global modeling ability; the simultaneous diagnosis of multiple fundus diseases still faces great challenges. Therefore, given that the self-attention (SA) model with a global receptive field may have robust global-level feature modeling ability, we propose a multistage fundus image classification model MBSaNet which combines CNN and SA mechanism. The convolution block extracts the local information of the fundus image, and the SA module further captures the complex relationships between different spatial positions, thereby directly detecting one or more fundus diseases in retinal fundus image. In the initial stage of feature extraction, we propose a multiscale feature fusion stem, which uses convolutional kernels of different scales to extract low-level features of the input image and fuse them to improve recognition accuracy. The training and testing were performed based on the ODIR-5k dataset. The experimental results show that MBSaNet achieves state-of-the-art performance with fewer parameters. The wide range of diseases and different fundus image collection conditions confirmed the applicability of MBSaNet.

Wetland classification based on sentinel-2 and 3D multi source domain self-attention model

Wetland classification based on sentinel-2 and 3D multi source domain self-attention model

Few-Shot Radar Jamming Recognition Network via Time-Frequency Self-Attention and Global Knowledge Distillation

Radar jamming recognition aims to accurately recognize the type of jamming to provide guidance for radar countermeasures. Although previous deep learning-based methods have made promising performance, they mainly rely on convolutional neural network (CNN) based on local processing, which ignore the global information in the time-frequency domain data of the jamming signal and also require much inference time to obtain the final recognition results. In this paper, a novel few-shot jamming recognition network via time-frequency self-attention and global knowledge distillation (JR-TFSAD) is proposed by jointly considering the global information in the time-frequency spectrum of the jamming signal and the real-time performance of the recognition network. A time-frequency self-attention model (TFSA) is proposed to extract the global deep features of radar jamming signals by learning the correlation between two arbitrary points in the time-frequency spectrum of the jamming signal, thus improving the recognition accuracy. Moreover, to effectively reduce the inference time of the method while preserving the recognition accuracy, a global knowledge distillation model (GKD) is further constructed to perform jamming recognition by distilling the global knowledge from the TFSA model. The experimental results on the simulated and measured mixed dataset verify that the proposed method has higher recognition accuracy and shorter inference time compared to the state-of-the-art methods.

A self-attentive model for tracing knowledge and engagement in parallel

• An effective model for tracing knowledge state and engagement in parallel is designed. • An adaptive heterogeneous graph embedding methods is proposed to extract the latent information of question-concept graph. • An engagement encoder is proposed model students’ engage state according to their learning behaviors. • State-of-the-art results on the benchmark datasets for knowledge tracing task. Knowledge tracing refers to the task of modeling students’ evolving knowledge state according to their historical learning trajectories. Although many research has devoted to exploiting the latent information of question-concept bipartite graph, most of them need to split the graph into question-question and concept-concept graph to obtain questions’ and concepts’ representation respectively. This operation will cause information loss of original bipartite graph. To this end, an adaptive heterogeneous graph embedding module is proposed to utilize the latent information of graph adequately. Besides, many works ignore that students’ engagement could also impact students’ learning performance. Therefore, a student engagement encoder and a student knowledge state encoder are designed to capture students’ engagement and knowledge respectively. To be specific, the question-concept bipartite graph is input to graph embedding module directly to obtain each question’s and concept’s representation, the two encoders use these representation and students’ learning history to encode their engagement and knowledge state.

Image restoration fabric defect detection based on the dual generative adversarial network patch model

The training of supervised learning requires the use of ground truth, which is difficult to obtain in large quantities in production practice. Unsupervised learning requires only flawless and anomalous images of fabrics, but inevitably generates a great deal of background noise when performing result generation, which reduces the quality of results. To overcome these limitations, we propose a new approach: image restoration fabric defect detection based on the dual generative adversarial network patch model (DGPM). We train with a modified generative adversarial network using only flawless and anomalous images of the fabric. We propose the patch model to directly obtain specific information about fabric defects and add a self-attentive model to reduce the generation of background noise. The performance of the DGPM is evaluated on box-, star-, and dot-patterned fabric databases. The true positive rate (TPR) of the box type is 81.56% and the f-measure is 62.69%, the TPR of the dot type is 83.72% and the f-measure is 67.33%, and the TPR of the star type is 79.79% and the f-measure is 64.65%.

MCGRAM: Linking Multi-Scale CNN With a Graph-Based Recurrent Attention Model for Subject-Independent ERP Detection

Electroencephalogram (EEG)-based event-related potential (ERP) detection is expected to play an important role in many applications. However, limited by the low signal-to-noise ratio and large inter-subject variability of EEG signals, it is still challenging to design a high-precision ERP detection method in subject-independent scenarios. In this brief, a novel graph-based multi-scale convolutional recurrent attention model (MCGRAM) is proposed to extract the underlying invariant features of EEG signals for subject-independent ERP detection. Specifically, considering the frequency factors of ERP signals, the multi-scale convolution module is designed for learning the frequency representations. Then, the graph convolution module is developed by constructing the links between each EEG node and designing trainable weights of the links to extract the discriminative spatial features. Besides, two-layer LSTM combined with a self-attention model is utilized to extract temporal representations of different EEG slices. Compared to the existing state-of-the-art methods, MCGRAM achieves the best performance in terms of AUC and F1-Score on a rapid serial visual presentation benchmark dataset. Lastly, the ablation studies demonstrate the superiority of each component in MCGRAM.

MTBI-DSANet: A deep self-attention model for diagnosing mild traumatic brain injury using multi-level functional connectivity networks

The main approach for analyzing resting-state functional magnetic resonance imaging (rs-fMRI) is the low-order functional connectivity network (LoFCN) based on the correlation between two brain regions. Based on LoFCN, researchers recently proposed the topographical high-order FCN (tHoFCN) and the associated high-order FCN (aHoFCN) to explore the high-order interactions among brain regions. In this work, we designed a Deep Self-Attention (DSA) framework called mTBI-DSANet to diagnose mild traumatic brain injury (mTBI) using multi-level FCNs, including LoFCN, tHoFCN, and aHoFCN. The multilayer perceptron and self-attention mechanism in mTBI-DSANet were designed to capture important features for the mTBI diagnosis. We evaluated the mTBI-DSANet’s performance on the real rs-fMRI dataset, which was collected by Third Xiangya Hospital of Central South University from April 2014 to February 2021. We compared the performance of mTBI-DSANet with distinct FCNs and their combinations under 10-fold cross-validation. Based on the LoFCN+aHoFCN combination, the average performance of mTBI-DSANet achieved the best accuracy of 0.834, which is significantly better than peer methods. The experiments demonstrated the potential of the mTBI-DSANet in assisting mTBI diagnosis.

Mineral Prospectivity Mapping Using Deep Self-Attention Model

Mineral Prospectivity Mapping Using Deep Self-Attention Model

The Learning Behavior Analysis of Online Vocational Education Students and Learning Resource Recommendation Based on Big Data

The Learning Behavior Analysis (LBA) of students is a major function provided by most online education platforms. Based on the results of LBA, students can get customized learning path and recommended learning resources that suit their own situations, which can help them plan their individual training programs of Online Vocational Education (OVE). At first, this paper studied the resource recommendation algorithm based on the modelling of students’ online learning behavior sequences, and solved the problem of conventional learning resource recommendation algorithms of ignoring the dynamic changes in students’ learning preferences. Then, this paper presented the process of LBA applicable for OVE students, and gave a diagram of the flow of using big data to perform LBA. After that, this paper developed a bidirectional encoder based on self-attention model, and described the co-occurrence characteristics and dependencies of students’ online learning behavior sequences. At last, the model pre-training and fine-tuning processes were introduced in detail, and the experimental results verified the effectiveness of the proposed analysis and recommendation methods.

Spatio-temporal convolutional emotional attention network for spotting macro- and micro-expression intervals in long video sequences

Emotional detection based on facial micro-expressions is essential in high-risk tasks such as criminal investigation or lie detection. However, micro-expressions often occur in high-risk tasks when people often use facial expressions to conceal their actual emotional states. Therefore, spotting macro- and micro-expression intervals in long video sequences has become hot research. Considering the difference in duration and facial muscle movement intensity between macro- and micro-expression, we propose a novel Spatio-temporal Convolutional Emotional Attention Network (STCEAN) for spotting macro- and micro-expression intervals in long video sequences. The spatial features of each frame in the video sequence are extracted through the convolution neural network. Then the emotional self-attention model is used to analyze the temporal weights of spatial features in different emotional dimensions. The emotional weights in the temporal dimension are filtered for spotting macro- and micro-expressions intervals. Finally, the STCEAN model is jointly optimized by the dual emotional focal loss of macro- and micro-expression to solve the problem of sample unbalance. The experimental results on the CAS(ME)2 and SAMM-LV datasets show that the STCEAN model achieves competitive results in the Facial Micro-Expression Challenge 2021.

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification.

For CNN-based visual action recognition, the accuracy may be increased if local key action regions are focused on. The task of self-attention is to focus on key features and ignore irrelevant information. So, self-attention is useful for action recognition. However, current self-attention methods usually ignore correlations among local feature vectors at spatial positions in CNN feature maps. In this paper, we propose an effective interaction-aware self-attention model which can extract information about the interactions between feature vectors to learn attention maps. Since the different layers in a network capture feature maps at different scales, we introduce a spatial pyramid with the feature maps at different layers for attention modeling. The multi-scale information is utilized to obtain more accurate attention scores. These attention scores are used to weight the local feature vectors of the feature maps and then calculate attentional feature maps. Since the number of feature maps input to the spatial pyramid attention layer is unrestricted, we easily extend this attention layer to a spatio-temporal version. Our model can be embedded in any general CNN to form a video-level end-to-end attention network for action recognition. Several methods are investigated to combine the RGB and flow streams to obtain accurate predictions of human actions. Experimental results show that our method achieves state-of-the-art results on the datasets UCF101, HMDB51, Kinetics-400, and untrimmed Charades.

A self-attention hybrid emoji prediction model for code-mixed language: (Hinglish)

A self-attention hybrid emoji prediction model for code-mixed language: (Hinglish)

Novel hybrid multi-head self-attention and multifractal algorithm for non-stationary time series prediction

Traditional time series prediction methods have shown their outstanding capabilities in time series prediction. However, due to essential differences in volatility characteristics among diverse types of non-stationary multivariate time series (NSMTS), it is difficult for traditional methods to maintain robust prediction performance. This study proposes a novel dynamic recurrent neural network to achieve stable and robust prediction performance. First, a multifractal gated recurrent unit (MF-GRU) based on the multifractal method is proposed to extract volatility characteristics. Meanwhile, to strengthen the parameters of the historical hidden layer state that has a more significant impact on the output, a self-attention mechanism is introduced into the MF-GRU, leading to a multifractal gated recurrent unit multi-head self-attention model. The efficiency of the proposed model was verified on public datasets. The experimental results show that the proposed model outperforms the traditional methods, such as long short-term memory (LSTM), the gated recurrent unit (GRU), and the minimal gated unit (MGU). etc.

Multi-scaled self-attention for drug–target interaction prediction based on multi-granularity representation

BackgroundDrug–target interaction (DTI) prediction plays a crucial role in drug discovery. Although the advanced deep learning has shown promising results in predicting DTIs, it still needs improvements in two aspects: (1) encoding method, in which the existing encoding method, character encoding, overlooks chemical textual information of atoms with multiple characters and chemical functional groups; as well as (2) the architecture of deep model, which should focus on multiple chemical patterns in drug and target representations.ResultsIn this paper, we propose a multi-granularity multi-scaled self-attention (SAN) model by alleviating the above problems. Specifically, in process of encoding, we investigate a segmentation method for drug and protein sequences and then label the segmented groups as the multi-granularity representations. Moreover, in order to enhance the various local patterns in these multi-granularity representations, a multi-scaled SAN is built and exploited to generate deep representations of drugs and targets. Finally, our proposed model predicts DTIs based on the fusion of these deep representations. Our proposed model is evaluated on two benchmark datasets, KIBA and Davis. The experimental results reveal that our proposed model yields better prediction accuracy than strong baseline models.ConclusionOur proposed multi-granularity encoding method and multi-scaled SAN model improve DTI prediction by encoding the chemical textual information of drugs and targets and extracting their various local patterns, respectively.

Multi-modal knowledge graphs representation learning via multi-headed self-attention

Traditional knowledge graphs (KG) representation learning focuses on the link information between entities, and the effectiveness of learning is influenced by the complexity of KGs. Considering a multi-modal knowledge graph (MKG), due to the introduction of considerable other modal information(such as images and texts), the complexity of KGs further increases, which degrades the effectiveness of representation learning. To resolve this solve the problem, this study proposed the multi-modal knowledge graphs representation learning via multi-head self-attention (MKGRL-MS) model, which improved the effectiveness of link prediction by adding rich multi-modal information to the entity. We first generated a single-modal feature vector corresponding to each entity. Then, we used multi-headed self-attention to obtain the attention degree of different modal features of entities in the process of semantic synthesis. In this manner, we learned the multi-modal feature representation of entities. New knowledge representation is the sum of traditional knowledge representation and an entity’s multi-modal feature representation. Simultaneously, we successfully train our model on two existing models and two different datasets and verified its versatility and effectiveness on the link prediction task.

DAFuse: a fusion for infrared and visible images based on generative adversarial network

Infrared and visible image fusion is a popular research hotspot in the field of image processing. However, the existing fusion methods still have some limitations, such as insufficient use of intermediate information and inability to focus on features that are meaningful for fusion. To solve these problems, we propose an infrared and visible image fusion method based on generative adversarial networks with dense connection and attention mechanism (DAFuse). Since infrared and visible image are different modalities, we design two branches to extract the features in infrared and visible image, respectively. To make full use of the features extracted from the middle layer and make the model focus on useful information, we introduce the dense block, channel attention mechanism, and spatial attention mechanism into the generator. The self-attention model is incorporated into the discriminator. The proposed method not only retains rich texture detail features and sufficient contrast information but also conforms to human visual perception. Extensive qualitative and quantitative experimental results show that the proposed method has better performance in visual perception and quantitative evaluation than the existing state-of-the-art methods.