Attention Weights Research Articles

Human attention during goal-directed reading comprehension relies on task optimization.

The computational principles underlying attention allocation in complex goal-directed tasks remain elusive. Goal-directed reading, that is, reading a passage to answer a question in mind, is a common real-world task that strongly engages attention. Here, we investigate what computational models can explain attention distribution in this complex task. We show that the reading time on each word is predicted by the attention weights in transformer-based deep neural networks (DNNs) optimized to perform the same reading task. Eye tracking further reveals that readers separately attend to basic text features and question-relevant information during first-pass reading and rereading, respectively. Similarly, text features and question relevance separately modulate attention weights in shallow and deep DNN layers. Furthermore, when readers scan a passage without a question in mind, their reading time is predicted by DNNs optimized for a word prediction task. Therefore, we offer a computational account of how task optimization modulates attention distribution during real-world reading.

Human attention during goal-directed reading comprehension relies on task optimization

The computational principles underlying attention allocation in complex goal-directed tasks remain elusive. Goal-directed reading, that is, reading a passage to answer a question in mind, is a common real-world task that strongly engages attention. Here, we investigate what computational models can explain attention distribution in this complex task. We show that the reading time on each word is predicted by the attention weights in transformer-based deep neural networks (DNNs) optimized to perform the same reading task. Eye tracking further reveals that readers separately attend to basic text features and question-relevant information during first-pass reading and rereading, respectively. Similarly, text features and question relevance separately modulate attention weights in shallow and deep DNN layers. Furthermore, when readers scan a passage without a question in mind, their reading time is predicted by DNNs optimized for a word prediction task. Therefore, we offer a computational account of how task optimization modulates attention distribution during real-world reading.

DM-CNN: Dynamic Multi-scale Convolutional Neural Network with uncertainty quantification for medical image classification

Convolutional neural network (CNN) has promoted the development of diagnosis technology of medical images. However, the performance of CNN is limited by insufficient feature information and inaccurate attention weight. Previous works have improved the accuracy and speed of CNN but ignored the uncertainty of the prediction, that is to say, uncertainty of CNN has not received enough attention. Therefore, it is still a great challenge for extracting effective features and uncertainty quantification of medical deep learning models In order to solve the above problems, this paper proposes a novel convolutional neural network model named DM-CNN, which mainly contains the four proposed sub-modules : dynamic multi-scale feature fusion module (DMFF), hierarchical dynamic uncertainty quantifies attention (HDUQ-Attention) and multi-scale fusion pooling method (MF Pooling) and multi-objective loss (MO loss). DMFF select different convolution kernels according to the feature maps at different levels, extract different-scale feature information, and make the feature information of each layer have stronger representation ability for information fusion HDUQ-Attention includes a tuning block that adjust the attention weight according to the different information of each layer, and a Monte-Carlo (MC) dropout structure for quantifying uncertainty MF Pooling is a pooling method designed for multi-scale models, which can speed up the calculation and prevent overfitting while retaining the main important information Because the number of parameters in the backbone part of DM-CNN is different from other modules, MO loss is proposed, which has a fast optimization speed and good classification effect DM-CNN conducts experiments on publicly available datasets in four areas of medicine (Dermatology, Histopathology, Respirology, Ophthalmology), achieving state-of-the-art classification performance on all datasets. DM-CNN can not only maintain excellent performance, but also solve the problem of quantification of uncertainty, which is a very important task for the medical field. The code is available: https://github.com/QIANXIN22/DM-CNN.

Planetary gearbox fault diagnosis based on FDKNN-DGAT with few labeled data

Although data-driven methods have been widely used in planetary gearbox fault diagnosis, the difficulty and high cost of manual labeling leads to little labeled training data, which limits the classification performance of traditional data-driven methods. Therefore, the semi-supervised fault diagnosis method with few labeled samples becomes one of the main research directions. Graph attention network (GAT) is distinguished from traditional classification network by using graph structure for fault node information aggregation and feature extraction, which is an effective semi-supervised learning algorithm. This paper uses fast Fourier transform to process the original vibration signal of gearbox and use it as graph nodes, and propose a KNN graph construction method using pooling for fuzzy distance calculation. In addition, this paper improves the distribution of attention weights by introducing dynamic graph attention networks to correct the problem that classical static GATs cannot clearly distinguish the weights of different categories of nodes. Experiments show that the method proposed in this paper can better extract fault features in complex gearbox vibration signals with an accuracy of more than 99% with very few labeled samples, and has better diagnostic performance compared with other graph neural network architectures and traditional classification networks.

Data-to-text generation using conditional generative adversarial with enhanced transformer

Abstract In this paper, we propose an enhanced version of the vanilla transformer for data-to-text generation and then use it as the generator of a conditional generative adversarial model to improve the semantic quality and diversity of output sentences. Specifically, by adding a diagonal mask matrix to the attention scores of the encoder and using the history of the attention weights in the decoder, this enhanced version of the vanilla transformer prevents semantic defects in the output text. Also, using this enhanced transformer along with a triplet network, respectively, as the generator and discriminator of conditional generative adversarial network, diversity and semantic quality of sentences are guaranteed. To prove the effectiveness of the proposed model, called conditional generative adversarial with enhanced transformer (CGA-ET), we performed experiments on three different datasets and observed that our proposed model is able to achieve better results than the baselines models in terms of BLEU, METEOR, NIST, ROUGE-L, CIDEr, BERTScore, and SER automatic evaluation metrics as well as human evaluation.

Metal surface defect detection based on improved YOLOv5

During the production of metal material, various complex defects may come into being on the surface, together with large amount of background texture information, causing false or missing detection in the process of small defect detection. To resolve those problems, this paper introduces a new model which combines the advantages of CSPlayer module and Global Attention Enhancement Mechanism based on the YOLOv5s model. First of all, we replace C3 module with CSPlayer module to augment the neural network model, so as to improve its flexibility and adaptability. Then, we introduce the Global Attention Mechanism (GAM) and build the generalized additive model. In the meanwhile, the attention weights of all dimensions are weighted and averaged as output to promote the detection speed and accuracy. The results of the experiment in which the GC10-DET augmented dataset is involved, show that the improved algorithm model performs better than YOLOv5s in precision, mAP@0.5 and mAP@0.5: 0.95 by 5.3%, 1.4% and 1.7% respectively, and it also has a higher reasoning speed.

EEG-TransMTL: A transformer-based multi-task learning network for thermal comfort evaluation of railway passenger from EEG

The evaluation of thermal comfort for railway passengers holds considerable importance, not only in reducing energy consumption but also in enhancing the passengers' experience. This paper presents a Transformer-based multi-task learning network (TransMTL) designed for railway passenger thermal comfort evaluation using EEG. We utilized manual features to extract temporal and frequency information, while a Transformer encoder distilled spatial information. The multi-task learning structure enhances model robustness by leveraging thermal comfort task correlations. We conducted experiments during winter and summer with high-speed railway passengers, establishing a comprehensive EEG dataset. The results demonstrated that our proposed EEG-TransMTL model outperformed classical machine learning and deep learning models in all four thermal comfort evaluation tasks, achieving accuracy rates of 65.00%, 66.70%, 80.38%, and 71.01%, respectively. We enhanced model interpretability by visualizing attention weights from the Transformer encoder, identifying key EEG channels. A simplified model utilizing only eight crucial channels also delivered notable performance. This research provides a practical and neuro-mechanism interpretable solution for thermal comfort evaluation.

Bayesian variational transformer: A generalizable model for rotating machinery fault diagnosis

Transformer has been widely applied in the research of rotating machinery fault diagnosis due to its ability to explore the internal correlation of vibration signals. However, challenges still exist despite the countless efforts. Generally, Transformer is more prone to overfitting than CNN on small-scale datasets. In practical engineering, collecting sufficient fault samples for training is difficult, resulting in poor generalization of Transformer. In addition, the measured signals are often accompanied with severe noise, further reducing the generalization performance of the model. Meanwhile, the collected signals often follow different distributions due to the changing operating conditions, which places higher demands on the generalizability of Transformer. This paper proposes a Bayesian variational Transformer (Bayesformer) to cope with the abovementioned problems. In Bayesformer, all the attention weights are treated as latent random variables, rather than determined values as the previous studies. This allows to train an ensemble of networks, instead of a single one, enhancing the generalizability of the model. Three experimental studies are conducted to illustrate the developed model and superior diagnostic performance is showed throughout the experiments.

Consistent Weighted Correlation-Based Attention for Transformer Tracking

Attention mechanism takes a crucial role among the key technologies in transformer-based visual tracking. However, the current methods for attention computing neglect the correlation between the query and the key, which results in erroneous correlations. To address this issue, a CWCTrack framework is proposed in this study for transformer visual tracking. To balance the weights of the attention module and enhance the feature extraction of the search region and template region, a consistent weighted correlation (CWC) module is introduced into the cross-attention block. The CWC module computes the correlation score between each query and all keys. Then, the correlation multiplies the consistent weights of the other query–key pairs to acquire the final attention weights. The weights of consistency are computed by the relevance of the query–key pairs. The correlation is enhanced for the relevant query–key pair and suppressed for the irrelevant query–key pair. Experimental results conducted on four prevalent benchmarks demonstrate that the proposed CWCTrack yields preferable performances.

On-line defect recognition of MIG lap welding for stainless steel sheet based on weld image and CMT voltage: Feature fusion and attention weights visualization

At present, sensor recognition technology has become a research hotspot in welding process monitoring. However, most of the existing defect recognition methods are based on single-sensor information, which hinders the further improvement of recognition accuracy and weakens the anti-interference ability under complex conditions. To overcome the shortcomings of single-sensor recognition technology, multi-source information fusion technology can exploit the advantages of different types of sensors to achieve more comprehensive and accurate monitoring of the welding process. However, given the process of fusion of different information, the feature mining mechanism of the deep learning model is still under-explained, and the direct complementarity of the different information remains unclear. In this paper, we propose a hybrid model that uses the fusion of molten pool image and voltage signals to quickly and accurately identify three types of defects in lap MIG welding: burn-through, weld bias, and uneven width. Firstly, a synchronized trigger device is used to acquire molten pool images and weld voltage information, and the voltage data is sliced and matched with the molten pool image data. Secondly, the Short-Time Fourier Transform (STFT) is applied to explore the frequency variation components in the voltage data, and the resulting time-frequency matrix is transformed into a time spectrogram. Finally, the molten pool image and the time spectrogram are simultaneously fed into the hybrid CNN model for online monitoring of welding defect states. The test results show that the average accuracy of the hybrid CNN model in detecting five different welding states reaches 98.87 %, which is significantly higher than three other models based on single sensor information. The visualization of the confusion matrix and Grad-CAM shows that the hybrid CNN model based on multi-source heterogeneous information has a higher accuracy for detecting the transition states that are difficult to detect in welding, and has a more reasonable mechanism for allocating attention, so it can achieve a more accurate and balanced recognition effect in the whole welding manufacturing process. These studies can provide guidance for quality monitoring of welding processes based on multi-source sensor technology, which can be extended to the field of arc additive monitoring.

An attention-based multi-modal MRI fusion model for major depressive disorder diagnosis

Objective. Major depressive disorder (MDD) is one of the biggest threats to human mental health. MDD is characterized by aberrant changes in both structure and function of the brain. Although recent studies have developed some deep learning models based on multi-modal magnetic resonance imaging (MRI) for MDD diagnosis, the latent associations between deep features derived from different modalities were largely unexplored by previous studies, which we hypothesized may have potential benefits in improving the diagnostic accuracy of MDD. Approach. In this study, we proposed a novel deep learning model that fused both structural MRI (sMRI) and resting-state MRI (rs-fMRI) data to enhance the diagnosis of MDD by capturing the interactions between deep features extracted from different modalities. Specifically, we first employed a brain function encoder (BFE) and a brain structure encoder (BSE) to extract the deep features from fMRI and sMRI, respectively. Then, we designed a function and structure co-attention fusion (FSCF) module that captured inter-modal interactions and adaptively fused multi-modal deep features for MDD diagnosis. Main results. This model was evaluated on a large cohort and achieved a high classification accuracy of 75.2% for MDD diagnosis. Moreover, the attention distribution of the FSCF module assigned higher attention weights to structural features than functional features for diagnosing MDD. Significance. The high classification accuracy highlights the effectiveness and potential clinical of the proposed model.

Method for Predicting Transverse Wave Velocity Using a Gated Recurrent Unit Based on Spatiotemporal Attention Mechanism

Abstract Transverse wave velocity plays an important role in seismic exploration and reservoir assessment in the oil and gas industry. Due to the lack of transverse wave velocity data from actual production activities, it is necessary to predict transverse wave velocity based on longitudinal wave velocity and other reservoir parameters. This paper proposes a fusion network based on spatiotemporal attention mechanism and gated recurrent unit (STAGRU) due to the significant correlation between the transverse wave velocity and reservoir parameters in the spatiotemporal domain. In the case of tight sandstone reservoirs in the Junggar Basin, the intersection plot technique is used to select four well logging parameters that are sensitive to transverse wave velocity: longitudinal wave velocity, density, natural gamma, and neutron porosity. The autocorrelation technique is employed to analyze the depth-related correlation of well logging curves. The relationship between the spatiotemporal characteristics of these well logging data and the network attention weights is also examined to validate the rationale behind incorporating the spatiotemporal attention mechanism. Finally, the actual measurement data from multiple wells are utilized to analyze the performance of the training set and test set separately. The results indicate that the predictive accuracy and generalization ability of the proposed STAGRU method are superior to the single-parameter fitting method, multiparameter fitting method, Xu-White model method, GRU network, and 2DCNN-GRU hybrid network. This demonstrates the feasibility of the transverse wave velocity prediction method based on the spatiotemporal attention mechanism in the study of rock physics modeling for tight sandstone reservoirs.

STCA-SNN: self-attention-based temporal-channel joint attention for spiking neural networks.

Spiking Neural Networks (SNNs) have shown great promise in processing spatio-temporal information compared to Artificial Neural Networks (ANNs). However, there remains a performance gap between SNNs and ANNs, which impedes the practical application of SNNs. With intrinsic event-triggered property and temporal dynamics, SNNs have the potential to effectively extract spatio-temporal features from event streams. To leverage the temporal potential of SNNs, we propose a self-attention-based temporal-channel joint attention SNN (STCA-SNN) with end-to-end training, which infers attention weights along both temporal and channel dimensions concurrently. It models global temporal and channel information correlations with self-attention, enabling the network to learn 'what' and 'when' to attend simultaneously. Our experimental results show that STCA-SNNs achieve better performance on N-MNIST (99.67%), CIFAR10-DVS (81.6%), and N-Caltech 101 (80.88%) compared with the state-of-the-art SNNs. Meanwhile, our ablation study demonstrates that STCA-SNNs improve the accuracy of event stream classification tasks.

Predicting drug-drug interactions based on multi-view and multichannel attention deep learning.

Predicting drug-drug interactions (DDIs) has become a major concern in the drug research field because it helps explore the pharmacological function of drugs and enables the development of new therapeutic drugs. Existing prediction methods simply integrate multiple drug attributes or perform tasks on a biomedical knowledge graph (KG). Though effective, few methods can fully utilize multi-source drug data information. In this paper, a multi-view and multichannel attention deep learning (MMADL) model is proposed, which not only extracts rich drug features containing both drug attributes and drug-related entity information from multi-source databases, but also considers the consistency and complementarity of different drug feature representation learning approaches to improve the effectiveness and accuracy of DDI prediction. A single-layer perceptron encoder is applied to encode multi-source drug information to obtain multi-view drug representation vectors in the same linear space. Then, the multichannel attention mechanism is introduced to obtain the attention weight by adaptively learning the importance of drug features according to their contributions to DDI prediction. Further, the representation vectors of multi-view drug pairs with attention weights are used as inputs of the deep neural network to predict potential DDI. The accuracy and precision-recall curves of MMADL are 93.05 and 95.94, respectively. The results indicate that the proposed method outperforms other state-of-the-art methods.

Region-Attention Prompt Learning for CLIP

Pre-trained Visual Language Models (VLMs) like CLIP have shown great potential in the multimodal domain. Among this, using different modal contexts and interaction features to construct prompt can stimulate the model’s prior knowledge circuit more accurately, thus generating better outputs. However, in CLIP, the formal mismatch of textual descriptions between the pre-training and inference phases results in a suboptimal representation ability of prompt, which is detrimental to model alignment learning. Therefore, Region-Attention Prompt (RAP) is proposed, which introduces region features to enrich the semantic representation of prompt. RAP is acquired by the Cross-Attention mechanism between images and texts, and it is essentially a region-level prompt with category-sensitive properties. For each category, RAP adaptively assigns greater attention weight to image regions that are more semantically relevant to the category. Besides, CLIP is equipped with RAP (called RA-CLIP) to improve image classification performance in generalization scenarios. Extensive experiments demonstrate that RA-CLIP outperforms the current SOTA CoCoOp 0.4% - 4.16% on base classes and 0.25% - 11.34% on new classes, across 7 datasets. In addition, we show that focusing on category-related regions to construct prompt can further improve the model’s alignment ability.

DualGGAN: A New Facial Style Transfer Network

In response to the background penetration problem of unsupervised style transfer algorithms in most cases, a Transformer style transfer network DualGGAN based on dual generators and fusion of relative position encoding is proposed. The network is trained using the least squares generative adversarial network, and the neural network is used as the image feature extractor to generate feature maps to obtain facial image features with attention weights from the feature maps, utilizing relative position encoding and mask loss to jointly constrain feature region style transfer. The experimental results show that the DualGGAN network effectively reduces artifact generation when implementing facial style transfer, maintains good background consistency, and has good generalization ability. Experiments have shown that the FID and KID indicators on the cat2dog and older2adult datasets are significantly improved compared to other algorithms.

Functional diversity of visual cortex improves constraint-free natural image reconstruction from human brain activity

Previous brain decoding studies using functional magnetic resonance imaging (fMRI) have greatly advanced our understanding of human visual coding and non-invasive brain-machine interfaces. However, most of these studies focus on classifying a limited number of image categories or reconstructing visual images with additional information, e.g., semantic categories and textual cues. Constraint-free visual reconstruction remains scarce. Here, we propose a generative network based on the functional diversity of the human visual cortex (FDGen) that takes multivariate brain activity as input and directly reconstructs natural images perceived by observers without any additional cues (semantic categories or textual description). Our FDGen is augmented by two bio-inspired computational modules. Based on the functional specializations of the human visual cortex, we propose a new function-based input module (FIM) that projects responses from different brain regions into separate feature spaces. Second, inspired by human attention, we construct a computational module to derive attentive feature weights at the function level to refine the feature map. These function-selection modules (FSMs) allow the network to dynamically select multiscale visual information during the generation process. We test FDGen on the popular fMRI datasets of natural images and achieve highly robust performance. Our work represents an important step forward in the development of fMRI-based brain decoding algorithms and highlights the utility of neuroscience theories in the design of deep learning models.

Neural-Free Attention for Monaural Speech Enhancement Toward Voice User Interface for Consumer Electronics

The traditional graphic user interface in healthcare-oriented consumer electronics faced challenges such as high operational complexity, time-consuming operations, and a high risk of infection. The adoption of voice user interface (VUI) could promote network automation with enhanced efficiency, reduced simplicity and operating expense in various applications. Given noisy operational environments, speech enhancement acts as an indispensable component for VUIs towards consumer devices. Recently, attention mechanism is studied for speech enhancement and exhibits promising potential. In this paper, we propose a novel and effective attention module for speech enhancement, called neural-free attention (NFA), which is a lightweight and plug-and-play module that enables the backbone network to capture the energy distribution information of speech signals along frequency-wise channels. Particularly, NFA adopts a learnable Gaussian function to perform the excitation operation and produce the attention weights for each frequency channel. The NFA is comprehensively evaluated as part of the residual temporal convolution network (ResTCN) backbone network on two commonly used training targets. Experimental results show NFA substantially improves the ResTCN backbone in speech quality and intelligibility, with extremely low parameter overhead. Also, the ResTCN+NFA shows superiority over several recent baseline models, indicating the strong potential for VUIs toward consumer devices.

T-MGCL: Molecule Graph Contrastive Learning Based on Transformer for Molecular Property Prediction.

In recent years, machine learning has gained increasing traction in the study of molecules, enabling researchers to tackle challenging tasks including molecular property prediction and drug design.Consequently, there remains an open challenge to develop a neural network architecture that can make use of extensive amounts of unlabeled data for training while still providing competitive results in various molecular property prediction tasks. To address this challenge, we propose a Molecule Graph Contrastive Learning approach based on the Transformer framework (T-MGCL). Our approach involves expanding numerous unsupervised molecular graphs and using a contrast estimator to ensure consistency among various graph expansions of the same molecule. Transformer framework is employed to consider the distance between atoms and molecular graph attributes, thereby accounting for structural information that may be overlooked by traditional graph neural networks. Our experimental results demonstrate that the T-MGCL model outperforms other models in several molecular property prediction tasks. Additionally, we observe that the attention weight learned by T-MGCL can be easily explained from a chemical perspective.

A Dense Small Object Detection Algorithm Based on a Global Normalization Attention Mechanism

To address the challenges of detecting a large number of objects and a high proportion of small objects in aerial drone imagery, we proposed an aerial dense small object detection algorithm called Global Normalization Attention Mechanism You Only Look Once (GNYL) based on the Global Normalization Attention Mechanism. In the backbone network of GNYL, we embedded a GNAM (Global Normalization Attention Mechanism) that explores channel attention features and spatial attention features from input features in a concatenated manner. It utilizes batch normalization’s scale factors to suppress irrelevant channels or pixels. Furthermore, the spatial attention sub-module introduces a three-dimensional arrangement with a multi-layer perceptron to reduce information loss and amplify global interaction representation. Finally, the computed attention weights are weighted to form the global normalized attention weights, which increases the utilization of effective information in input feature channels and spatial dimensions. We have optimized the backbone network, feature enhancement network, and detection heads to improve detection accuracy while ensuring a lightweight detection network. Specifically, we have added a small object detection layer to enhance the localization accuracy for the abundant small objects in aerial imagery. The algorithm’s performance was evaluated using the publicly available VisDrone2019 dataset. Compared to the baseline network YOLOv8l, GNYL achieved a 7.2% improvement in mAP0.5 and a 5.0% improvement in mAP0.95. Compared to CDNet, GNYL showed a 14.5% improvement in mAP0.5 and a 9.1% improvement in mAP0.95. These experimental results demonstrate the strong practicality of the GNYL object detection network for detecting dense small objects in the aerial imagery captured by unmanned aerial vehicles.