Traditional Attention Mechanism Research Articles

Efficient agricultural pest classification using vision transformer with hybrid pooled multihead attention

Accurate pest classification plays a pivotal role in modern agriculture for effective pest management, ensuring crop health and productivity. While Convolutional Neural Networks (CNNs) have been widely used for classification, their limited ability to capture both local and global information hinders precise pest identification. In contrast, vision transformers have shown promise in capturing global dependencies and enhancing classification performance. However, the traditional attention mechanism employed in vision transformers, which uses the same query (Q), key (K), and value (V), overlooks spatial relationships between patches, limiting the model's capacity to capture fine-grained details and long-range dependencies in the image. To address these limitations, this study presents a novel approach, termed Hybrid Pooled Multihead Attention (HPMA), for superior pest classification that outperforms both CNN models and vision transformers. The HPMA model integrates hybrid pooling techniques and modifies the attention mechanism to effectively capture local and global features within images. By emphasizing discriminative features and suppressing irrelevant information, the HPMA model achieves heightened robustness and generalization capabilities. The model is trained and tested on a newly built dataset consisting of 10 pest classes, achieving a remarkable accuracy of 98 %. Furthermore, the proposed HPMA model is validated on two benchmark datasets and achieves accuracies of 98 % and 95 %, demonstrating its effectiveness across diverse pest datasets. The results and ablation study of the proposed model contribute to exceptional performance in accurate pest classification. This tackles agricultural pest challenges and enables prompt pest control to reduce crop losses.

Spatial–temporal combination and multi-head flow-attention network for traffic flow prediction

Traffic flow prediction based on spatial–temporal data plays a vital role in traffic management. However, it still faces serious challenges due to the complex spatial–temporal correlation in nonlinear spatial–temporal data. Some previous methods have limited ability to capture spatial–temporal correlation, and ignore the quadratic complexity problem in the traditional attention mechanism. To this end, we propose a novel spatial–temporal combination and multi-head flow-attention network (STCMFA) to model the spatial–temporal correlation in road networks. Firstly, we design a temporal sequence multi-head flow attention (TS-MFA), in which the unique source competition mechanism and sink allocation mechanism make the model avoid attention degradation without being affected by inductive biases. Secondly, we use GRU instead of the linear layer in traditional attention to map the input sequence, which further enhances the temporal modeling ability of the model. Finally, we combine the GCN with the TS-MFA module to capture the spatial–temporal correlation, and introduce residual mechanism and feature aggregation strategy to further improve the performance of STCMFA. Extensive experiments on four real-world traffic datasets show that our model has excellent performance and is always significantly better than other baselines.

Dual-Path Graph Neural Network with Adaptive Auxiliary Module for Link Prediction.

Link prediction, which has important applications in many fields, predicts the possibility of the link between two nodes in a graph. Link prediction based on Graph Neural Network (GNN) obtains node representation and graph structure through GNN, which has attracted a growing amount of attention recently. However, the existing GNN-based link prediction approaches possess some shortcomings. On the one hand, because a graph contains different types of nodes, it leads to a great challenge for aggregating information and learning node representation from its neighbor nodes. On the other hand, the attention mechanism has been an effect instrument for enhancing the link prediction performance. However, the traditional attention mechanism is always monotonic for query nodes, which limits its influence on link prediction. To address these two problems, a Dual-Path Graph Neural Network (DPGNN) for link prediction is proposed in this study. First, we propose a novel Local Random Features Augmentation for Graph Convolution Network as a baseline of one path. Meanwhile, Graph Attention Network version 2 based on dynamic attention mechanism is adopted as a baseline of the other path. And then, we capture more meaningful node representation and more accurate link features by concatenating the information of these two paths. In addition, we propose an adaptive auxiliary module for better balancing the weight of auxiliary tasks, which brings more benefit to link prediction. Finally, extensive experiments verify the effectiveness and superiority of our proposed DPGNN for link prediction.

Enhanced Chinese named entity recognition with multi-granularity BERT adapter and efficient global pointer

AbstractNamed Entity Recognition (NER) plays a crucial role in the field of Natural Language Processing, holding significant value in applications such as information extraction, knowledge graphs, and question–answering systems. However, Chinese NER faces challenges such as semantic complexity, uncertain entity boundaries, and nested structures. To address these issues, this study proposes an innovative approach, namely Multi-Granularity BERT Adapter and Efficient Global Pointer (MGBERT-Pointer). The semantic encoding layer adopts Multi-Granularity Adapter (MGA), while the decoding layer employs Efficient Global Pointer (EGP) network, ensuring collaborative functionality. The MGA, incorporating Character Adapter, Entity Adapter, and Lexicon Adapter through interactive mechanisms, are deeply integrated into the BERT base, significantly enhancing the model’s ability to handle complex contexts and ambiguities. The EGP, utilizing Rotary Position Embedding, resolves the issue of insufficient boundary information in traditional attention mechanisms, thereby improving the model’s understanding and recognition of nested entity structures. Experimental results on four public datasets demonstrate a significant enhancement in Chinese NER performance achieved by the MGBERT-Pointer model.

Topological charge identification of superimposed orbital angular momentum beams under turbulence using an attention mechanism.

Due to the unique features, orbital angular momentum (OAM) beams have been widely explored for different applications. Accurate determination of the topological charge (TC) of these beams is crucial for their optimal utilization. In this paper, we propose a method that combines adaptive image processing techniques with a simple, parameter-free attention module (SimAM) based convolutional neural network to accurately identify the TC of high-order superimposed OAM beams. Experimental results demonstrate that under the combined influence of non-extreme light intensity and turbulence, it can achieve >95% identification accuracy of TCs ranging from ±1 to ±40. Moreover, even under partial-pattern-missing conditions, our method maintains an accuracy rate of over 80%. Compared with traditional attention mechanisms, SimAM does not require additional network design, significantly reducing the computational costs. Our approach showcases remarkable efficiency, robustness, and cost-effectiveness, making it adaptable to challenging factors such as non-uniform lighting and partially occluded light paths. This research provides a new direction for recognizing OAM modes with valuable implications for the future of communication systems.

Modeling the air pollution process using a novel multi-site and multi-scale method with adaptive utilization of spatio-temporal information

This study focuses on modeling air quality with an adaptive utilization of spatio-temporal information from multiple air quality monitoring stations under a multi-scale framework. To this end, it is necessary to consider different strategies to combine different methods to decompose the given series and to fuse multi-site information. Based on a systematic comparative analysis, we propose a novel multi-scale and multi-site modeling method named the multivariate empirical mode decomposition and spatial cosine-attention-based long short-term memory (MEMD-SCA-LSTM). The MEMD-SCA-LSTM first employs MEMD to decompose the multi-site air quality series into the scale-aligned components and then models the components at different scales. The high-frequency components are modeled by a novel SCA-LSTM, which employs LSTM with residual blocks to extract the temporal information and utilizes an attention mechanism based on the cosine similarity to adaptively extract interactions among different sites. Other relatively regular components are modeled by the LSTM. Empirical study on PM2.5 in Hong Kong has demonstrated the effectiveness of fusing multi-site information using the spatial attention (SA) mechanism under the multi-scale framework with MEMD. The proposed MEMD-SCA-LSTM can improve the one-day ahead modeling performance with the mean absolute error and the root mean square error reduced over 10%, compared to the baseline modeling methods. For the two-day and three-day ahead performance, the MEMD-SCA-LSTM is still the best one. Furthermore, by visualizing the attention weights, we illustrate that our proposed SCA-LSTM can overcome some limitations of the traditional attention mechanisms and that the attention weights exhibit more informative patterns which could be used to analysis the transport of air pollutant between sites. The proposed modeling method is a general method, which is feasible and applicable to other pollutants in other cities or regions.

Attention Block Based on Binary Pooling

Image classification has become highly significant in the field of computer vision due to its wide array of applications. In recent years, Convolutional Neural Networks (CNN) have emerged as potent tools for addressing this task. Attention mechanisms offer an effective approach to enhance the accuracy of image classification. Despite Global Average Pooling (GAP) being a crucial component of traditional attention mechanisms, it only computes the average of spatial elements in each channel, failing to capture the complete range of feature information, resulting in fewer and less expressive features. To address this limitation, we propose a novel pooling operation named “Binary Pooling” and integrate it into the attention block. Binary pooling combines both GAP and Global Max Pooling (GMP), obtaining a more comprehensive feature vector by extracting average and maximum values, thereby enriching the diversity of extracted image features. Furthermore, to further enhance the extraction of image features, dilation operations and pointwise convolutions are applied on the channel-wise. The proposed attention block is simple yet highly effective. Upon integration into ResNet18/50 models, it leads to accuracy improvements of 2.02%/0.63% on ImageNet.

A novel approach to attention mechanism using kernel functions: Kerformer.

Artificial Intelligence (AI) is driving advancements across various fields by simulating and enhancing human intelligence. In Natural Language Processing (NLP), transformer models like the Kerformer, a linear transformer based on a kernel approach, have garnered success. However, traditional attention mechanisms in these models have quadratic calculation costs linked to input sequence lengths, hampering efficiency in tasks with extended orders. To tackle this, Kerformer introduces a nonlinear reweighting mechanism, transforming maximum attention into feature-based dot product attention. By exploiting the non-negativity and non-linear weighting traits of softmax computation, separate non-negativity operations for Query(Q) and Key(K) computations are performed. The inclusion of the SE Block further enhances model performance. Kerformer significantly reduces attention matrix time complexity from O(N2) to O(N), with N representing sequence length. This transformation results in remarkable efficiency and scalability gains, especially for prolonged tasks. Experimental results demonstrate Kerformer's superiority in terms of time and memory consumption, yielding higher average accuracy (83.39%) in NLP and vision tasks. In tasks with long sequences, Kerformer achieves an average accuracy of 58.94% and exhibits superior efficiency and convergence speed in visual tasks. This model thus offers a promising solution to the limitations posed by conventional attention mechanisms in handling lengthy tasks.

DB-TASNet for disease diagnosis and lesion segmentation in medical images

Deep learning algorithms have been successfully used in the field of medical image analysis and have greatly improved application of intelligent algorithms to medical diagnosis. However, existing deep-learning-based diagnostic methods still suffer from several drawbacks: (1) In most medical image multi-tasking methods, focus segmentation and disease classification are often performed linearly, resulting in excessive reliance on the final results of focus segmentation. (2) The computational cost of the traditional attention mechanism for performing the segmentation task is very high and the convolutional architecture cannot be used to model long-distance dependencies, which in turn affects the segmentation accuracy. To address these issues, we propose a disease diagnosis and lesion segmentation model, Dual-Branch with Transformer Axial-attention Segmentation Net (DB-TASNet). DB-TASNet is built by the DenseNet-121 classification network and U-Net segmentation network improved using an axial-attention transformer model. Moreover, DB-TASNet also includes a lesion integration module to integrate segmentation results with the classification network in order to increase its attention to lesions and improve the diagnosis results. Experimental results on the Pneumothorax dataset provided by the Society for Imaging Informatics in Medicine (SIIM) show that the average AUC of the DB-TASNet classification task reaches 0.939, and the DICE coefficient of the segmentation task reaches 0.886. Such performance suggests that the proposed model may provide an efficient and effective diagnosis tool for medical personnel.

Replay Attention and Data Augmentation Network for 3-D Face and Object Reconstruction

3D face reconstruction from single-view images plays an important role in the field of biometrics, which is a long-standing challenging problem in the wild. Traditional 3DMM-based methods directly regressed parameters, which probably caused that the network learned the discriminative informative features insufficiently. In this paper, we propose a replay attention and data augmentation network (RADAN) for 3D dense alignment and face reconstruction. Unlike the traditional attention mechanism, our replay attention module aims to increase the sensitivity of the network to informative features by adaptively recalibrating the weight response in the attention, which typically reinforces the distinguishability of the learned feature representation. In this way, the network can further improve the accuracy of face reconstruction and dense alignment in unconstrained environments. Moreover, to improve the generalization performance of the model and the ability of the network to capture local details, we present a data augmentation strategy to preprocess the sample data, which generates the images that contain more local details and occluded face in cropping and pasting manner. Furthermore, we also apply the replay attention to 3D object reconstruction task to verify the commonality of this mechanism. Extensive experimental results on widely-evaluated datasets demonstrate that our approach achieves competitive performance compared to state-of-the-art methods. Code is available at https://github.com/zhouzhiyuan1/RADANet.

Chinese Text Sentiment Analysis Based on BERT-BiGRU Fusion Gated Attention

To address the problem that Word2vec static encoding cannot give accurate word vectors about contextual semantics and cannot solve the problem of multiple meanings of words, we propose to use the BERT pre-training model as a word embedding layer to obtain word vectors dynamically; we introduce the gating idea to improve on the traditional attention mechanism and propose BERT-BiGRU-GANet model. The model firstly uses the BERT pre-training model as the word vector layer to vectorize the input text by dynamic encoding; secondly, uses the bi-directional gated recursive unit model (BiGRU) to capture the dependencies between long discourse and further analyze the contextual semantics; finally, before output classification, adds the attention mechanism of fusion gating to ignore the features with little relevance and highlight the key features with weight ratio features. We conducted several comparison experiments on the Jingdong public product review dataset, and the model achieved an F1 value of 93.06%, which is 3.41%, 2.55%, and 1.12% more accurate than the BiLSTM, BiLSTM-Att, and BERT-BiGRU models, respectively. It indicates that the use of the BERT-BiGRU-GANet model has some improvement on Chinese text sentiment analysis, which is helpful in the analysis of goods and service reviews, for consumers to select goods, and for merchants to improve their goods or service reviews.

Identification of depression state based on multi‐scale acoustic features in interrogation environment

AbstractDepression diagnosis based on speech signals has the advantages of non‐invasiveness, low cost, and few restrictions on portability. The research on the recognition of the depression state is carried out based on the acoustic information in the speech signal. Aiming at the interview dialogue speech in the consultation environment, a hierarchical attention temporal convolutional network (HATCN) acoustic depression recognition model is proposed. For sentence acoustic feature learning, a regional attention mechanism is introduced to extract multi‐scale sentence features; for segment acoustic feature extraction, the traditional attention mechanism is used to calculate, which is in line with human cognitive mechanism. In addition, a periodic focal loss function is introduced to address the imbalance of positive and negative samples in depression diagnosis. Experiments show that the proposed acoustic depression recognition model has a certain improvement in recognition performance compared with other methods. At the same time, the influence of noise on the recognition of acoustic depression in the real consultation environment is analysed through experiments, and the data enhancement is carried out utilising speech noise, which proves the effectiveness of the data expansion of speech noise.

Full-Memory Transformer for Image Captioning

The Transformer-based approach represents the state-of-the-art in image captioning. However, existing studies have shown Transformer has a problem that irrelevant tokens with overlapping neighbors incorrectly attend to each other with relatively large attention scores. We believe that this limitation is due to the incompleteness of the Self-Attention Network (SAN) and Feed-Forward Network (FFN). To solve this problem, we present the Full-Memory Transformer method for image captioning. The method improves the performance of both image encoding and language decoding. In the image encoding step, we propose the Full-LN symmetric structure, which enables stable training and better model generalization performance by symmetrically embedding Layer Normalization on both sides of the SAN and FFN. In the language decoding step, we propose the Memory Attention Network (MAN), which extends the traditional attention mechanism to determine the correlation between attention results and input sequences, guiding the model to focus on the words that need to be attended to. Our method is evaluated on the MS COCO dataset and achieves good performance, improving the result in terms of BLEU-4 from 38.4 to 39.3.

Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity

Multimodal biometric system has received increasing interest as it offers a more secure and accurate authentication solution than unimodal systems. However, existing biometric fusion methods are still inadequate in dealing with correlations and redundancy of multimodal features simultaneously, causing bottlenecks in performance improvement. To overcome the above problem, this paper proposes an end-to-end multimodal finger recognition model that incorporates attention mechanisms into a similarity-aware encoder to achieve accurate recognition results for accurate identification results. Firstly, due to the different distribution of fingerprint and finger vein images, we propose a finger asymmetric backbone network (FAB-Net) for extracting discriminative intra-modal features, which reduces the network width by efficient utilization of feature maps. Then, a novel attention-based encoder fusion network (AEF-Net) with fused similarity performs dimensionality reduction-based fusion on multimodal multilevel features to alleviate performance degradation due to information redundancy. We also introduce channel attention in AEF-Net, which differs from the traditional attention mechanism by considering interdependencies between modalities to further improve performance. Extensive recognition experiments are conducted on three multimodal finger databases to verify the effectiveness of our method compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrated that encoder-based reconstruction of redundant information can improve recognition performance.

A Novel Hyperspectral Image Classification Model Using Bole Convolution With Three-Direction Attention Mechanism: Small Sample and Unbalanced Learning

Currently, the use of rich spectral and spatial information of hyperspectral images (HSIs) to classify ground objects is a research hotspot. However, the classification ability of existing models is significantly affected by its high data dimensionality and massive information redundancy. Therefore, we focus on the elimination of redundant information and the mining of promising features and propose a novel Bole convolution (BC) neural network with a tandem three-direction attention (TDA) mechanism (BTA-Net) for the classification of HSI. A new BC is proposed for the first time in this algorithm, whose core idea is to enhance effective features and eliminate redundant features through feature punishment and reward strategies. Considering that traditional attention mechanisms often assign weights in a one-direction manner, leading to a loss of the relationship between the spectra, a novel three-direction (horizontal, vertical, and spatial directions) attention mechanism is proposed, and an addition strategy and a maximization strategy are used to jointly assign weights to improve the context sensitivity of spatial–spectral features. In addition, we also designed a tandem TDA mechanism module and combined it with a multiscale BC output to improve classification accuracy and stability even when training samples are small and unbalanced. We conducted scene classification experiments on four commonly used hyperspectral datasets to demonstrate the superiority of the proposed model. The proposed algorithm achieves competitive performance on small samples and unbalanced data, according to the results of comparison and ablation experiments. The source code for BTA-Net can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/vivitsai/BTA-Net</uri> .

A Video Description Model with Improved Attention Mechanism

Video description generation refers to the automatic generation of text descriptions of videos by computers, which belongs to the intersection of computer vision and natural language processing. Aiming at the problem that the traditional attention mechanism has insufficient ability to extract video features, the model is complex and the description quality is not high, this paper proposes a video description model with an improved attention mechanism. The model is based on the encoder-decoder structure, uses inception-v4 as the encoder to extract features, and introduces a lightweight coordinate attention module (CA) into the attention mechanism, which improves the feature extraction effect and reduces the model complexity, and sends the extracted important feature information into the decoder long short-term memory network (LSTM) to generate the description sentence corresponding to the video. The model is validated on the MSVD dataset using various evaluation metrics (BLEU, ROUGEL, CIDEr, METEOR). The experimental results show that the improved attention mechanism of the video description model proposed in this paper has better accuracy in different performance metrics and can further improve the performance of video description.

Improved One-Stage Detectors with Neck Attention Block for Object Detection in Remote Sensing

Object detection in remote sensing is becoming a conspicuous challenge with the rapidly increasing quantity and quality of remote sensing images. Although the application of Deep Learning has obtained remarkable performance in Computer Vision, detecting multi-scale targets in remote sensing images is still an unsolved problem, especially for small instances which possess limited features and intricate backgrounds. In this work, we managed to cope with this problem by designing a neck attention block (NAB), a simple and flexible module which combines the convolutional bottleneck structure and the attention mechanism, different from traditional attention mechanisms that focus on designing complicated attention branches. In addition, Vehicle in High-Resolution Aerial Imagery (VHRAI), a diverse, dense, and challenging dataset, was proposed for studying small object detection. To validate the effectiveness and generalization of NAB, we conducted experiments on a variety of datasets with the improved YOLOv3, YOLOv4-Tiny, and SSD. On VHRAI, the improved YOLOv3 and YOLOv4-Tiny surpassed the original models by 1.98% and 1.89% mAP, respectively. Similarly, they exceeded the original models by 1.12% and 3.72% mAP on TGRS-HRRSD, a large multi-scale dataset. Including SSD, these three models also showed excellent generalizability on PASCAL VOC.

Adoption of Wireless Network and Artificial Intelligence Algorithm in Chinese-English Tense Translation

In order to solve the problem of tense consistency in Chinese-English neural machine translation (NMT) system, a Chinese verb tense annotation model is proposed. Firstly, a neural network is used to build a Chinese tense annotation model. During the translation process, the source tense is passed to the target side through the alignment matrix of the traditional Attention mechanism. The probability of the candidate words inconsistent with the corresponding tense of source words in the candidate translation word set is also reduced. Then, the Chinese-English temporal annotation algorithm is integrated into the MT model, so as to build a Chinese-English translation system with temporal processing function. The essence of the system is that, in the process of translation, Chinese-English temporal annotation algorithm is used to obtain temporal information from Chinese sentences and transfer it to the corresponding English sentences, so as to realize the temporal processing of English sentences and obtain the English sentences corresponding to the tenses of the original Chinese sentences. The experimental results show that the Chinese tense annotation model of bidirectional long short-term memory (LSTM) is more accurate for the prediction of Chinese verb tense, so the improvement effect of NMT model is also the most obvious, especially on the NIST06 test set, where the BLEU value is increased by 1.07%. As the mainstream translation model, the transformer model contains multihead Attention mechanism, which can pay attention to some temporal information and has a certain processing ability for temporal translation. It solves the tense problems encountered in the process of MT and improves the credibility of Chinese-English machine translation (MT).

Application of Attention Mechanism with Prior Information in Natural Language Processing

When using deep learning methods to model natural language, a recurrent neural network that can map input sequences to output sequences is usually used. Considering that natural language contains more complicated syntactic structures, and the performance of cyclic neural networks in long sentence processing will decrease, scholars have introduced an attention mechanism into the model, which has improved the above problems to a certain extent. The existing attention mechanism still has some shortcomings, such as the inability to explicitly obtain the known syntactic structure information in the sentence, and the poor interpretability of the output probability. In response to the above problems, this article will improve the attention mechanism in the recurrent neural network model. Firstly, the prior information in the natural language sequence is constructed as a graph model through syntactic analysis and other means, and then the graph structure regularization term is introduced into the sparse mapping. A new function netmax is constructed to replace the softmax function in the traditional attention mechanism, thereby improving the performance of the model and making the degree of association. The input values corresponding to larger input samples are closer, making the output of the attention mechanism easier to understand. The innovation of this paper mainly lies in that the weight calculation method which can be widely used in the attention mechanism is proposed by combining the deep learning model with statistical knowledge, which opens a channel to introduce the prior information for the deep learning model in natural language processing tasks.

Identification of Apple Disease Grades Based on the Attention Mechanism of Lesion Location and Improved Data Enhancement Method

The output and quality of apples were greatly threatened by plant diseases. Identifying the types and grades of diseases in time was helpful to the management of diseases. When the disease occurs on a leaf, there was little change in the leaf except for the affected area. The traditional attention mechanism changed the weight of the network to all pixels in the image, which affected the ability of the network to extract the features of the lesion area. And the traditional method of data enhancement was easy to cause local similarity and local discontinuity of all sample features in the same disease grade. In this paper, the improved metric matrix for kernel regression (IMMKR) was used to reduce the influence of local similarity and local discontinuity of all sample features in the same class. Then, a new attention mechanism by fusing the lesion location based on visual features was proposed, and the attention of the model to the lesion area was strengthened. The experiments were carried out on three different diseases of apple named black rot, scab, and rust. The accuracy rate and recall rate of the new method on the combined dataset of PlantVillage and PlantDoc were 91.55% and 92.06%, respectively, which was superior to existing methods. This algorithm has important reference significance for the identification and promotion of crop diseases.