Types Of Attention Mechanisms Research Articles

An efficient multimodal attentional principal component analysis for continual learning-based dynamic process monitoring

Traditional multimode process monitoring methods extract features from time series data. Due to the catastrophic forgetting effect, data-driven multimode dynamic process monitoring is challenging based on a single monitoring model paradigm, i.e. the learned knowledge from previous modes may diminish as operating conditions undergo changes between modes, yet it is impractical to access all past data to retrain the model. In this work, a novel efficient method of multimodal attentional principal component analysis (M-APCA) with continual learning ability is introduced. Under the assumption that data from successive modes are received sequentially, dynamic process data are modeled using an attention mechanism to capture the relationship between data and the latent space, whereby meaningful information is concentrated as dynamic features which are extracted via a vector autoregressive model. In order to overcome the catastrophic forgetting problem, the idea of replay continual learning is employed. Specifically, past modes’ data which are significant to reflect the operating conditions, are selected and stored. These are repeatedly used in tandem with sequential data as replay data. Two types of attention mechanisms are considered and analyzed, each of which is specifically designed to learn from data in an unsupervised manner, so the overall algorithm is efficient both in time and storage costs. The proposed attentional principal component analysis and M-APCA are analyzed against several state-of-the-art methods to highlight the virtues of the proposed method. Compared with multimode monitoring methods, the effectiveness is demonstrated through case studies of: a continuous stirred tank heater, the Tennessee Eastman process and a practical coal pulverizing system.

Imaging segmentation mechanism for rectal tumors using improved U-Net

ObjectiveIn radiation therapy, cancerous region segmentation in magnetic resonance images (MRI) is a critical step. For rectal cancer, the automatic segmentation of rectal tumors from an MRI is a great challenge. There are two main shortcomings in existing deep learning-based methods that lead to incorrect segmentation: 1) there are many organs surrounding the rectum, and the shape of some organs is similar to that of rectal tumors; 2) high-level features extracted by conventional neural networks often do not contain enough high-resolution information. Therefore, an improved U-Net segmentation network based on attention mechanisms is proposed to replace the traditional U-Net network.MethodsThe overall framework of the proposed method is based on traditional U-Net. A ResNeSt module was added to extract the overall features, and a shape module was added after the encoder layer. We then combined the outputs of the shape module and the decoder to obtain the results. Moreover, the model used different types of attention mechanisms, so that the network learned information to improve segmentation accuracy.ResultsWe validated the effectiveness of the proposed method using 3773 2D MRI datasets from 304 patients. The results showed that the proposed method achieved 0.987, 0.946, 0.897, and 0.899 for Dice, MPA, MioU, and FWIoU, respectively; these values are significantly better than those of other existing methods.ConclusionDue to time savings, the proposed method can help radiologists segment rectal tumors effectively and enable them to focus on patients whose cancerous regions are difficult for the network to segment.SignificanceThe proposed method can help doctors segment rectal tumors, thereby ensuring good diagnostic quality and accuracy.

Investigating attention mechanisms for plant disease identification in challenging environments

There is an increasing demand for efficient and precise plant disease detection methods that can quickly identify disease outbreaks. For this, researchers have developed various machine learning and image processing techniques. However, real-field images present challenges due to complex backgrounds, similarities between different disease symptoms, and the need to detect multiple diseases simultaneously. These obstacles hinder the development of a reliable classification model. The attention mechanisms emerge as a critical factor in enhancing the robustness of classification models by selectively focusing on relevant regions or features within infected regions in an image. This paper provides details about various types of attention mechanisms and explores the utilization of these techniques for the machine learning solutions created by researchers for image segmentation, feature extraction, object detection, and classification for efficient plant disease identification. Experiments are conducted on three models: MobileNetV2, EfficientNetV2, and ShuffleNetV2, to assess the effectiveness of attention modules. For this, Squeeze and Excitation layers, the Convolutional Block Attention Module, and transformer modules have been integrated into these models, and their performance has been evaluated using different metrics. The outcomes show that adding attention modules enhances the original models' functionality.

Adaptive Attention Module for Image Recognition Systems in Autonomous Driving

Lightweight, high‐performance networks are important in vision perception systems. Recent research on convolutional neural networks has shown that attention mechanisms can significantly improve the network performance. However, existing approaches either ignore the significance of using both types of attention mechanisms (channel and space) simultaneously or increase the model complexity. In this study, we propose the adaptive attention module (AAM), which is a truly lightweight yet effective module that comprises channel and spatial submodules to balance model performance and complexity. The AAM initially utilizes the channel submodule to generate intermediate channel‐refined features. In this module, an adaptive mechanism enables the model to autonomously learn the weights between features extracted by global max pooling and global average pooling to adapt to different stages of the model, thus enhancing performance. The spatial submodule employs a group‐interact‐aggregate strategy to enhance the expression of important features. It groups the intermediate channel‐refined features along the channel dimension into multiple subfeatures for parallel processing and generates spatial attention feature descriptors and channelwise refined subfeatures for each subfeature; subsequently, it aggregates all the refined subfeatures and employs a “channel shuffle” operator to transfer information between different subfeatures, thereby generating the final refined features and adaptively emphasizing important regions. Additionally, AAM is a plug‐and‐play architectural unit that can be directly used to replace standard convolutions in various convolutional neural networks. Extensive tests on CIFAR‐100, ImageNet‐1k, BDD100K, and MS COCO demonstrate that AAM improves the baseline network performance under various models and tasks, thereby validating its versatility.

A novel method for identifying key genes in macroevolution based on deep learning with attention mechanism

Macroevolution can be regarded as the result of evolutionary changes of synergistically acting genes. Unfortunately, the importance of these genes in macroevolution is difficult to assess and hence the identification of macroevolutionary key genes is a major challenge in evolutionary biology. In this study, we designed various word embedding libraries of natural language processing (NLP) considering the multiple mechanisms of evolutionary genomics. A novel method (IKGM) based on three types of attention mechanisms (domain attention, kmer attention and fused attention) were proposed to calculate the weights of different genes in macroevolution. Taking 34 species of diurnal butterflies and nocturnal moths in Lepidoptera as an example, we identified a few of key genes with high weights, which annotated to the functions of circadian rhythms, sensory organs, as well as behavioral habits etc. This study not only provides a novel method to identify the key genes of macroevolution at the genomic level, but also helps us to understand the microevolution mechanisms of diurnal butterflies and nocturnal moths in Lepidoptera.

RelTR: Relation Transformer for Scene Graph Generation.

Different objects in the same scene are more or less related to each other, but only a limited number of these relationships are noteworthy. Inspired by Detection Transformer, which excels in object detection, we view scene graph generation as a set prediction problem. In this paper, we propose an end-to-end scene graph generation model Relation Transformer (RelTR), which has an encoder-decoder architecture. The encoder reasons about the visual feature context while the decoder infers a fixed-size set of triplets subject-predicate-object using different types of attention mechanisms with coupled subject and object queries. We design a set prediction loss performing the matching between the ground truth and predicted triplets for the end-to-end training. In contrast to most existing scene graph generation methods, RelTR is a one-stage method that predicts sparse scene graphs directly only using visual appearance without combining entities and labeling all possible predicates. Extensive experiments on the Visual Genome, Open Images V6, and VRD datasets demonstrate the superior performance and fast inference of our model.

Violation Behavior Detection for Non-motor Vehicles

<p>Non-motor vehicles are widely used in the urban and rural transportation system for their portability, but the related violations also occur frequently and are difficult to be supervised intelligently, considering their colossal quantity, various styles, and small volumes. To solve this problem, this paper presents a non-motor vehicle violation detection algorithm with efficient target detection and deliberate logical calculation. A target detection network with high speed and accuracy is constructed firstly by fusing two different types of attention mechanism. Specifically, the Squeeze-and-Excitation Network is employed to optimize the extraction of local features, which can effectively reduce the error rate of target detection. Meanwhile, the Transformer Network is adopted to strengthen the extraction of global features, which can improve the target location performance for target tracking in high-density scenarios. As the global and local features are integrated with attention mechanism, the proposed network can accurately identify and locate the numerous small targets in real-time, and avoid identity switching caused by target occlusion. Finally, the violation of non-motor vehicles is recognized in real-time by constructing the logical calculation between the target features and their motion trajectories. Experiments on datasets show that the detection accuracy of the proposed algorithm is better than the current mainstream algorithms, especially the accuracy of small targets such as Head and Helmet is higher than 92.2%. And our ID Switch is dropped by more than 60% compared with the classical Deep SORT algorithm. In real-life scenarios, the proposed algorithm also shows excellent accuracy and real-time performance for non-motor vehicle violations.</p> <p> </p>

Joint Graph Learning and Matching for Semantic Feature Correspondence

In recent years, powered by the learned discriminative representation via graph neural network (GNN) models, deep graph matching methods have made great progresses in the task of matching semantic features. However, these methods usually rely on heuristically generated graph patterns, which may introduce unreliable relationships to hurt the matching performance. In this paper, we propose a joint graph learning and matching network, named GLAM, to explore reliable graph structures for boosting graph matching. GLAM adopts a pure attention-based framework for both graph learning and graph matching. Specifically, it employs two types of attention mechanisms, self-attention and cross-attention for the task. The self-attention discovers the relationships between features to further update feature representations over the learnt structures; and the cross-attention computes cross-graph correlations between the two feature sets to be matched for feature reconstruction. Moreover, the final matching solution is directly derived from the output of the cross-attention layer, without employing a specific matching decision module. The proposed method is evaluated on three popular visual matching benchmarks (Pascal VOC, Willow Object and SPair-71k), and it outperforms previous state-of-the-art graph matching methods on all benchmarks. Furthermore, the graph patterns learnt by our model are validated to be able to remarkably enhance previous deep graph matching methods by replacing their handcrafted graph structures with the learnt ones.

An Improved U-Net Model for Segmenting Wind Turbines From UAV-Taken Images

Unmanned aerial vehicle (UAV) based inspection of wind turbine surface conditions is a research hot spot in the wind energy industry. To tackle the fundamental problems in UAV-based inspection, this letter proposes an improved U-Net model for segmenting wind turbines from UAV-taken images. In the proposed segmentation model, ResNet is employed as the backbone for feature extraction while two types of attention mechanism, ECA-Net and PSA-Net, are used to capture important details in images. Therefore, multiscale features are effectively fused and cross-channel attention is obtained. The feasibility and efficiency of the proposed method are validated based on UAV-taken images collected from four commercial wind farms. Meanwhile, the proposed method is benchmarked with the generic U-Net, UNet++, Res-UNet, DenseUNet, and DeepLabv3. The segmentation results show the proposed model outperforms other methods, and, thus, the proposed model is applicable for real UAV-based inspection tasks.

MAM: A multipath attention mechanism for image recognition

Attention mechanism has shown excellent performance in many computer vision tasks, while the previous literature may not adequately consider different types of attention mechanisms or is individual elaborate designed for a certain network. In this paper, a general yet effective multipath attention mechanism (MAM) to explore the effect of visual attention for image recognition is proposed. In contrast with other attentions that leverage global pooling, the main advantage is that the MAM considers both the correlation of featuremaps and different scale structural information into account. The backbone representations are enhanced by adding MAM laterally along independent and separate dimensions, channel and spatial. Due to only a simple and unified calculation block is generated, MAM can be flexibly integrated into various CNNs within few parameters and trained together end-to-end. Furthermore, the topology structures of attention path arrangement are investigated using different connection schemes. Experimental results on several image recognition datasets show that the model outperforms various existing models. Finally, performance improvement through visualisation is intuitively discussed. The source code for the proposed attention module is publicly available.

Neural attention for image captioning: review of outstanding methods

Image captioning is the task of automatically generating sentences that describe an input image in the best way possible. The most successful techniques for automatically generating image captions have recently used attentive deep learning models. There are variations in the way deep learning models with attention are designed. In this survey, we provide a review of literature related to attentive deep learning models for image captioning. Instead of offering a comprehensive review of all prior work on deep image captioning models, we explain various types of attention mechanisms used for the task of image captioning in deep learning models. The most successful deep learning models used for image captioning follow the encoder-decoder architecture, although there are differences in the way these models employ attention mechanisms. Via analysis on performance results from different attentive deep models for image captioning, we aim at finding the most successful types of attention mechanisms in deep models for image captioning. Soft attention, bottom-up attention, and multi-head attention are the types of attention mechanism widely used in state-of-the-art attentive deep learning models for image captioning. At the current time, the best results are achieved from variants of multi-head attention with bottom-up attention.

Image-based thickener mud layer height prediction with attention mechanism-based CNN

Mud layer height of thickener is the key quality index of thickening process which is difficult to achieve real-time detection with existing methods in reality. While the need of developing a soft sensor model which can be used for real-time detection of mud layer height, we proposed an end-to-end mud layer height prediction method with attention mechanism-based convolutional neural network (CNN). The dynamic features are firstly extracted from the image samples based on CNN, and then two types of attention mechanism are embedded sequentially to contribute to more precise prediction results. Compared with the traditional spatial attention mechanism, the regional spatial attention mechanism we proposed selectively divides the spatial feature map into regions, while regions containing important features are assigned larger weights. Adding the channel and regional spatial attention mechanism in CNN not only effectively improve both the precision and calculation speed, but also affect the dimension of the output feature map, so as to avoid the loss of channel or spatial attention information of the feature map. To verify the validity of the proposed method, different attention mechanisms are embedded in the CNN, and the corresponding experiments are carried out on the dataset of the thickener mud layer. The experimental results demonstrate the feasibility and effectiveness of the mud layer height prediction method.

Double attention U-Net for brain tumor MR image segmentation

PurposeAutomatic segmentation of brain tumor from medical images is a challenging task because of tumor's uneven and irregular shapes. In this paper, the authors propose an attention-based nested segmentation network, named DAU-Net. In total, two types of attention mechanisms are introduced to make the U-Net network focus on the key feature regions. The proposed network has a deep supervised encoder–decoder architecture and a redesigned dense skip connection. DAU-Net introduces an attention mechanism between convolutional blocks so that the features extracted at different levels can be merged with a task-related selection.Design/methodology/approachIn the coding layer, the authors designed a channel attention module. It marks the importance of each feature graph in the segmentation task. In the decoding layer, the authors designed a spatial attention module. It marks the importance of different regional features. And by fusing features at different scales in the same coding layer, the network can fully extract the detailed information of the original image and learn more tumor boundary information.FindingsTo verify the effectiveness of the DAU-Net, experiments were carried out on the BRATS 2018 brain tumor magnetic resonance imaging (MRI) database. The segmentation results show that the proposed method has a high accuracy, with a Dice similarity coefficient (DSC) of 89% in the complete tumor, which is an improvement of 8.04 and 4.02%, compared with fully convolutional network (FCN) and U-Net, respectively.Originality/valueThe experimental results show that the proposed method has good performance in the segmentation of brain tumors. The proposed method has potential clinical applicability.

Deep learning-based semantic classification of EMF-related scientific literature

Semantic classification of scientific literature using machine learning approaches is challenging due to the difficulties in labeling data and the length of the texts [2, 7]. Most of the work has been done for keyword-based categorization tasks, which take care of occurrence of important terms, whereas semantic classification requires understanding of terms and the meaning of sentences in a context. In this study, we have evaluated neural network models on a semantic classification task using 1091 labeled EMF-related scientific papers listed in the Powerwatch study. The EMF-related papers are labeled into three categories: positive, null finding, and neither. We have conducted neural architecture and hyperparameter search to find the most suitable model for the task. In experiments, we compared the performance of several neural network models in terms of classification accuracy. In addition, we have tested two different types of attention mechanisms. First, a Fully Convolutional Neural Network (FCN) has been used to identify important sentences in the text for the semantic classification. Second, the Transformer, a self-attention-based model, has been tested on the dataset. The experimental result showed that the BiLSTM performed best on both unbalanced and balanced data and the FCN was able to identify important parts in input texts.

Research on Text Multi-Feature Fusion Algorithm Based on AM-CNN

How to know the changes of students’ emotions in real time in online education has always been one of the important issues concerned by education departments and teachers. Making use of the special functions of global, part of speech and positional attention mechanisms in text processing, a deep learning network model based on the combination of multiple attention mechanisms and convolution neural networks is proposed. Firstly, the blending characteristics between the types of attention mechanism and CNN are analyzed by using the standard ChnSentiCorp_htl_all data set to clarify the effectiveness of the combination of the three attention mechanisms and CNN. Then the model is applied to the analysis of the evaluation text of the course “big data Technology principles and applications” on the MOOC of Chinese universities, and it is verified that the evaluation indexes of this model are better than the existing conventional models.

Interactive Dual Attention Network for Text Sentiment Classification.

Text sentiment classification is an essential research field of natural language processing. Recently, numerous deep learning-based methods for sentiment classification have been proposed and achieved better performances compared with conventional machine learning methods. However, most of the proposed methods ignore the interactive relationship between contextual semantics and sentimental tendency while modeling their text representation. In this paper, we propose a novel Interactive Dual Attention Network (IDAN) model that aims to interactively learn the representation between contextual semantics and sentimental tendency information. Firstly, we design an algorithm that utilizes linguistic resources to obtain sentimental tendency information from text and then extract word embeddings from the BERT (Bidirectional Encoder Representations from Transformers) pretraining model as the embedding layer of IDAN. Next, we use two Bidirectional LSTM (BiLSTM) networks to learn the long-range dependencies of contextual semantics and sentimental tendency information, respectively. Finally, two types of attention mechanisms are implemented in IDAN. One is multihead attention, which is the next layer of BiLSTM and is used to learn the interactive relationship between contextual semantics and sentimental tendency information. The other is global attention that aims to make the model focus on the important parts of the sequence and generate the final representation for classification. These two attention mechanisms enable IDAN to interactively learn the relationship between semantics and sentimental tendency information and improve the classification performance. A large number of experiments on four benchmark datasets show that our IDAN model is superior to competitive methods. Moreover, both the result analysis and the attention weight visualization further demonstrate the effectiveness of our proposed method.

An Attention-based Deep Relevance Model for Few-shot Document Filtering

With the large quantity of textual information produced on the Internet, a critical necessity is to filter out the irrelevant information and organize the rest into categories of interest (e.g., an emerging event). However, supervised-learning document filtering methods heavily rely on a large number of labeled documents for model training. Manually identifying plenty of positive examples for each category is expensive and time-consuming. Also, it is unrealistic to cover all the categories from an evolving text source that covers diverse kinds of events, user opinions, and daily life activities. In this article, we propose a novel attention-based deep relevance model for few-shot document filtering (named ADRM), inspired by the relevance feedback methodology proposed for ad hoc retrieval. ADRM calculates the relevance score between a document and a category by taking a set of seed words and a few seed documents relevant to the category. It constructs the category-specific conceptual representation of the document based on the corresponding seed words and seed documents. Specifically, to filter irrelevant yet noisy information in the seed documents, ADRM employs two types of attention mechanisms (namely whole-match attention and max-match attention ) and generates category-specific representations for them. Then ADRM is devised to extract the relevance signals by modeling the hidden feature interactions in the word embedding space. The relevance signals are extracted through a gated convolutional process, a self-attention layer, and a relevance aggregation layer. Extensive experiments on three real-world datasets show that ADRM consistently outperforms the existing technical alternatives, including the conventional classification and retrieval baselines, and the state-of-the-art deep relevance ranking models for few-shot document filtering. We also perform an ablation study to demonstrate that each component in ADRM is effective for enhancing filtering performance. Further analysis shows that ADRM is robust under varying parameter settings.

AGLNet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network

The extensive computational burden limits the usage of convolutional neural networks (CNNs) in edge devices for image semantic segmentation, which plays a significant role in many real-world applications, such as augmented reality, robotics, and self-driving. To address this problem, this paper presents an attention-guided lightweight network, namely AGLNet, which employs an encoder–decoder architecture for real-time semantic segmentation. Specifically, the encoder adopts a novel residual module to abstract feature representations, where two new operations, channel split and shuffle, are utilized to greatly reduce computation cost while maintaining higher segmentation accuracy. On the other hand, instead of using complicated dilated convolution and artificially designed architecture, two types of attention mechanism are subsequently employed in the decoder to upsample features to match input resolution. Specifically, a factorized attention pyramid module (FAPM) is used to explore hierarchical spatial attention from high-level output, still remaining fewer model parameters. To delineate object shapes and boundaries, a global attention upsample module (GAUM) is adopted as global guidance for high-level features. The comprehensive experiments demonstrate that our approach achieves state-of-the-art results in terms of speed and accuracy on three self-driving datasets: CityScapes, CamVid, and Mapillary Vistas. AGLNet achieves 71.3%, 69.4%, and 30.7% mean IoU on these datasets with only 1.12M model parameters. Our method also achieves 52 FPS, 90 FPS, and 53 FPS inference speed, respectively, using a single GTX 1080Ti GPU. Our code is open-source and available at https://github.com/xiaoyufenfei/Efficient-Segmentation-Networks.

FRF-Net: Land Cover Classification From Large-Scale VHR Optical Remote Sensing Images

Deep learning (DL) technique is widely applied in remote sensing (RS) applications because of its outstanding nonlinear feature extraction ability. However, with regard to the issues of large-scale and very high-resolution (VHR) land cover classification, multi-object distributions and clear appearance with large intraclass difference become challenges for refined pixelwise land cover mapping. Focusing on these problems, the letter proposed a novel encoding-to-decoding method called the full receptive field (RF) network (FRF-Net) based on two types of attention mechanism. In the FRF-Net, ResNet-101 is used as the basic backbone. Then, the ensemble feature is generated by encoding the high-level features based on the self-attention mechanism which could achieve full RF to capture long-range semantic. Next, the encoding result is decoded by the fusion attention mechanism combined with the low-level feature to produce a fusion feature which contains a refined semantic description for accurate land cover mapping. Extensive experiments based on the GID and ISPRS data sets proved that the proposed network outperforms the state-of-the-art methods. The FRF-Net achieved 66.71% and 64.17% of the mean of classwise Intersection over Union (mIOU) with smaller computation cost on ISPRS and GID, respectively.

Combining attention-based bidirectional gated recurrent neural network and two-dimensional convolutional neural network for document-level sentiment classification

Neural networks lately have achieved a great success on sentiment classification due to their ability of feature extraction. However, it remains as an enormous challenge to model long texts in document-level sentiment classification as well as to explore semantics between sentences and dependencies between features. Moreover, most existing methods can hardly differentiate the importance of different contents when constructing a document representation. To tackle these problems, we propose a novel neural network model: AttDR-2DCNN, which mainly consists of two parts. The first part utilizes a two-layer compositional bidirectional Gated Recurrent Unit (GRU) to obtain the compositional semantics of the document, where the first layer learns the feature vector of the sentence, and the second layer learns the document matrix representation consisting of two dimensions of the time-step dimension and feature dimension, from the sentence vectors. The second part applies a two-dimensional convolution operation and two-dimensional max pooling to capture more dependencies between sentences features. We further utilize different types of attention mechanism in these two parts to distinguish the importance of words, sentences and features in the document. Experiments are conducted on four publicly available document-level review datasets and the result shows that the proposed model outperforms some existing models.