Attention Weights Research Articles

A Dual-Branch Spatio-Temporal-Spectral Transformer Feature Fusion Network for EEG-Based Visual Recognition

Recognizing visual objects from single-trial electroencephalograph (EEG) signals is a promising brain-computer interface (BCI) technology. However, due to the redundant features from noisy multi-channel EEG signals, it is still a challenging task to achieve high precision recognition. Recent deep learning approaches commonly extract spatio-temporal features of EEG signals, which neglect important spectral-temporal features and may degrade the EEG recognition performance. To address the deficiency, we propose a novel channel attention weighting and multi-level adaptive spectral aggregation based dual-branch spatio-temporal-spectral Transformer feature fusion network (CAW-MASA-STST) for EEG-based visual recognition. Specially, we first develop a channel attention weighting (CAW) to automatically learn the channel weights of EEG signals. Then, a graph convolution-based multi-level adaptive spectral aggregation (MASA) is employed to aggregate spectral-temporal features of different sub-bands. Finally, a spatio-temporal-spectral Transformer (STST) is designed to fuse spatio-temporal and spectral-temporal features, which enhances the comprehensive learning ability by modeling the temporal dependencies of the fused features. Competitive experimental results on two public datasets demonstrate that the proposed method is able to achieve superior recognition performance compared with the state-of-the-art methods, indicating a feasible solution for visual recognition-based BCI technology. The code of our proposed method will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ljbuaa/VisualDecoding</uri> .

AttentiveSkin: To Predict Skin Corrosion/Irritation Potentials of Chemicals via Explainable Machine Learning Methods.

Skin Corrosion/Irritation (Corr./Irrit.) has long been a health hazard in the Globally Harmonized System (GHS). Several in silico models have been built to predict Skin Corr./Irrit. as an alternative to the increasingly restricted animal testing. However, current studies are limited by data amount/quality and model availability. To address these issues, we compiled a traceable consensus GHS data set comprising 731 Corr., 1283 Irrit., and 1205 negative (Neg.) samples from 6 governmental databases and 2 external data sets. Then, a series of binary classifiers were developed with five machine learning (ML) algorithms and six molecular representations. For 10-fold cross-validation, the best Corr. vs Neg. classifier achieved an Area Under the Receiver Operating Characteristic Curve (AUC) of 97.1%, while the best Irrit. vs Neg. classifier achieved an AUC of 84.7%. Compared with existing in silico tools on external validation, our Attentive FP classifiers showed the highest metrics on Corr. vs Neg. and the second highest accuracy on Irrit. vs Neg. The SHapley Additive exPlanation approach was further applied to figure out important molecular features, and the attention weights were visualized to perform interpretable prediction. Structural alerts associated with Skin Corr./Irrit. were also identified. The interpretable Attentive FP classifiers were integrated into the software AttentiveSkin at https://github.com/BeeBeeWong/AttentiveSkin. The conventional ML classifiers are also provided on our platform admetSAR at http://lmmd.ecust.edu.cn/admetsar2/. Considering the data deficiency and the limited model availability of Skin Corr./Irrit., we believe that our data set and models could facilitate chemical safety assessment and relevant studies.

Modeling Magnitude Discrimination: Effects of Internal Precision and Attentional Weighting of Feature Dimensions.

Given a rich environment, how do we decide on what information to use? A view of a single entity (e.g., a group of birds) affords many distinct interpretations, including their number, average size, and spatial extent. An enduring challenge for cognition, therefore, is to focus resources on the most relevant evidence for any particular decision. In the present study, subjects completed three tasks-number discrimination, surface area discrimination, and convex hull discrimination-with the same stimulus set, where these three features were orthogonalized. Therefore, only the relevant feature provided consistent evidence for decisions in each task. This allowed us to determine how well humans discriminate each feature dimension and what evidence they relied on to do so. We introduce a novel computational approach that fits both feature precision and feature use. We found that the most relevant feature for each decision is extracted and relied on, with minor contributions from competing features. These results suggest that multiple feature dimensions are separately represented for each attended ensemble of many items and that cognition is efficient at selecting the appropriate evidence for a decision.

Interpretable spatial-temporal attention convolutional network for rainfall forecasting

The interpretability of rainfall forecasting models is a major challenge in the field of artificial intelligence. Its importance is equal to the evaluation of model accuracy. Owing to the uncertainty and nonlinearity of the rainfall forecasting process, existing hydrological rainfall forecasting models often have low prediction robustness, and the machine learning method ignores the influence of physical factors in the surrounding areas; thus, the physical interpretability of the model is inferior. This study proposes an interpretable spatial-temporal attention convolution network for rainfall interval forecasting (IDSTA-TCN). To analyze and mine the uncertainty of input data more effectively, we enhanced a variational mode decomposition (EBO-VMD) algorithm based on Bayesian optimization to decompose the input data. To improve the performance of the original temporal convolution network (TCN) unit in rainfall forecasting and to effectively use the spatial-temporal information of rainfall forecasting data, we improved the spatial-temporal attention mechanism, dynamically combined the geographical information of the meteorological stations in the surrounding area to assist the network unit in better mining the nonlinear impact of historical input information on current rainfall forecasting, and improved the prediction performance of the IDSTA-TCN model. Finally, we designed a more reasonable error factor adjustment interval output strategy to better quantify the impact of prediction errors on the rainfall interval prediction results. Moreover, we conducted a series of experiments to compare and analyze the accuracy of each model according to the most commonly used evaluation indicators in the field of rainfall forecasting. The results demonstrate that the proposed IDSTA-TCN model is superior to the baseline models in most cases. We preliminarily analyzed the spatial-temporal attention weights, which explained the physical rationality of our model.

OrangeStereo: A navel orange stereo matching network for 3D surface reconstruction

Three-dimensional (3D) surface reconstruction is crucial to accurately detecting navel orange external qualities such as size and shape, and binocular stereo vision is promising for practical sorting scenarios due to its high frame rates and ease of deployment. For objects with texture-less surface, such as navel oranges, it is very challenging to match the corresponding points using traditional stereo matching algorithms because the detected feature points have high similarity in color as well as structure. Most of the existing deep learning-based stereo matching algorithms focus on the stereo matching performance of the whole image and lack the attention to the matching performance of the main region. Meanwhile, simple background with less texture will bring more uncertainty to the matching of corresponding points, which affects the robustness of the model. The OrangeStereo was proposed based on the GwcNet structure to achieve a better corresponding points matching effect of navel oranges, which can provide more accurate 3D surface information. The Structural Features Extraction (SFE) module was introduced after the initial feature extraction module to extract stable structural features and avoid the limitations of using single convolutional features. The Attention Weights Generation (AWG) module was utilized at matching cost calculation stage to help suppress redundant information and make the network pay more attention to the matching performance of specific regions. The improved loss function assigned a greater weight to the loss of the navel orange region using semantic information from left image to make the network more focused on the matching effect of the navel orange region. An ablation study was conducted to confirm the efficiency of the SFE module, AWG module, and the improved loss function of the algorithm. The ablation study contained five sub-experiments, including the baseline model, adding the SFE and the AWG modules respectively, and whether to add the improved loss function based on the addition of previous two modules. Additionally, PSMNet, GwcNet, RealtimeStereo, and ACVNet, four typical stereo matching methods, were contrasted with the OrangeStereo. The proposed OrangeStereo achieved better performance with EPE, RMSE, bad-1, bad-3, bad-5, and inference time of 0.60 pixels, 3.53 pixels, 2.37 %, 1.83 %, 1.62 %, and 33 ms in the orange region, respectively. The R2 and RMSE for the depth of orange utilizing our method were 0.982 and 0.81 mm, respectively. The proposed algorithm can be employed to obtain accurate depth information of fruits quickly, and it is expected to be used in the 3D reconstruction of fruits in the actual commercial fruit sorting lines to achieve more accurate external quality assessment.

LSAB: User Behavioral Pattern Modeling in Sequential Recommendation by Learning Self-Attention Bias

Since the weight of a self-attention model is not affected by the sequence interval, it can more accurately and completely describe the user interests, so it is widely used in processing sequential recommendation. However, the mainstream self-attention model focuses on the similarity between items when calculating the attention weight of user behavioral patterns but fails to reflect the impact of user sudden drift decisions on the model in time. In this article, we introduce a bias strategy in the self-attention module, referred to as Learning Self-Attention Bias (LSAB) to more accurately learn the fast-changing user behavioral patterns. The introduction of LSAB allows for the adjustment of bias resulting from self-attention weights, leading to enhanced prediction performance in sequential recommendation. In addition, this article designs four attention-weight bias types catering to diverse user behavior preferences. After testing on the benchmark datasets, each bias strategy in LSAB is useful for state-of-the-art and can improve the performance of the models by nearly 5% on average. The source code listing is publicly available at https://gitee.com/kyle-liao/lsab .

Computer-aided diagnosis of distal metastasis in non-small cell lung cancer by low-dose CT based radiomics and deep learning signatures.

This study aimed to develop and validate radiomics and deep learning (DL) signatures for predicting distal metastasis (DM) of non-small cell lung cancer (NSCLC) in low-dose computed tomography (LDCT). Images and clinical data were retrospectively collected for 381 NSCLC patients and prospectively collected for 114 patients at the Fifth Affiliated Hospital of Sun Yat-Sen University. Additionally, we enrolled 179 patients from the Jiangmen Central Hospital to externally validate the signatures. Machine-learning algorithms were employed to develop radiomics signature while the DL signature was developed using neural architecture search. The diagnostic efficiency was primarily quantified with the area under receiver operating characteristic curve (AUC). We interpreted the reasoning process of the radiomics signature and DL signature by radiomics voxel mapping and attention weight tracking. A total of 674 patients with pathologically-confirmed NSCLC were included from two institutions, with 143 of them having DM. The radiomics signature achieved AUCs of 0.885, 0.854, and 0.733 in the internal validation, prospective validation, and external validation while those for DL signature were 0.893, 0.786, and 0.780. The proposed signatures achieved a promising performance in predicting the DM of NSCLC and outperformed the approaches proposed in previous studies. Interpretability analysis revealed that both radiomics and DL signatures could detect the variations among voxels inside tumors, which helped in identifying the DM of NSCLC. Our study demonstrates the potential of LDCT-based radiomics and DL signatures for predicting DM in NSCLC. These signatures could help improve lung cancer screening regarding further diagnostic tests and treatment strategies.

Adaptively Enhancing Facial Expression Crucial Regions via a Local Non-local Joint Network

AbstractFacial expression recognition (FER) is still challenging due to the small interclass discrepancy in facial expression data. In view of the significance of facial crucial regions for FER, many existing studies utilize the prior information from some annotated crucial points to improve the performance of FER. However, it is complicated and time-consuming to manually annotate facial crucial points, especially for vast wild expression images. Based on this, a local non-local joint network is proposed to adaptively enhance the facial crucial regions in feature learning of FER in this paper. In the proposed method, two parts are constructed based on facial local and non-local information, where an ensemble of multiple local networks is proposed to extract local features corresponding to multiple facial local regions and a non-local attention network is addressed to explore the significance of each local region. In particular, the attention weights obtained by the non-local network are fed into the local part to achieve interactive feedback between the facial global and local information. Interestingly, the non-local weights corresponding to local regions are gradually updated and higher weights are given to more crucial regions. Moreover, U-Net is employed to extract the integrated features of deep semantic information and low hierarchical detail information of expression images. Finally, experimental results illustrate that the proposed method achieves more competitive performance than several state-of-the-art methods on five benchmark datasets.

Seismic vulnerability signal analysis of low tower cable-stayed bridges method based on convolutional attention network

Abstract Due to the particularity and complexity of sedimentary environments, the wave impedance differences between different reflection interfaces in underground media may vary greatly. Therefore, an encoder–decoder neural network is proposed to enhance erroneous seismic weak reflection signals. The convolutional neural network (CNN) has the problem of difficulty in parallel computing, resulting in slow network training and computational efficiency. Considering that attention has an innate global self-attention mechanism, can compensate for long-term dependency deficiencies, and has the ability to perform parallel computing, which greatly compensates for the shortcomings of CNNs and recurrent neural networks, a seismic impedance inversion method based on convolutional attention networks is proposed. To improve the ability to extract noise, residual structure and convolutional attention module (CBAM) were introduced. The residual structure utilizes residual jump to weaken network degradation and reduce the difficulty of feature mapping. The CBAM uses a mixed attention weight of channel and space, which can enhance features with high correlation and suppress features with low correlation. In the decoder, in order to improve the dimension recovery ability of feature fusion, bilinear interpolation is selected for upsampling. The application results of the model and actual data indicate that this method can effectively enhance the weak reflection signals caused by the formation itself and improve the reservoir identification ability of seismic data.

DSAMR: Dual-Stream Attention Multi-hop Reasoning for knowledge-based visual question answering

Knowledge-based visual question answering aims to associate external knowledge facts for answering questions about images. Most existing methods emphasize high-order associations between knowledge facts and questions, and fail to consider the negative effects of unnecessary knowledge facts in multi-hop reasoning. In this paper, we propose a Dual-Stream Attention Multi-hop Reasoning (DSAMR) architecture that constructs two different attention streams to mitigate unnecessary knowledge facts. This dual-stream mechanism enables the model to reduce the attention weights on unnecessary knowledge while gathering essential knowledge by learning the implicit correlations between knowledge facts and questions. In addition, we designed a hypergraph knowledge extraction module in the architecture to extract optimal knowledge facts by evaluating the relevance of each knowledge fact to the question. The experimental results demonstrate the effectiveness of our method not only on the knowledge-based visual question answering dataset KVQA, but also on the multi-hop question answering dataset PathQuestion.

Interpretable Detection of Diabetic Retinopathy, Retinal Vein Occlusion, Age-Related Macular Degeneration, and Other Fundus Conditions.

Diabetic retinopathy (DR), retinal vein occlusion (RVO), and age-related macular degeneration (AMD) pose significant global health challenges, often resulting in vision impairment and blindness. Automatic detection of these conditions is crucial, particularly in underserved rural areas with limited access to ophthalmic services. Despite remarkable advancements in artificial intelligence, especially convolutional neural networks (CNNs), their complexity can make interpretation difficult. In this study, we curated a dataset consisting of 15,089 color fundus photographs (CFPs) obtained from 8110 patients who underwent fundus fluorescein angiography (FFA) examination. The primary objective was to construct integrated models that merge CNNs with an attention mechanism. These models were designed for a hierarchical multilabel classification task, focusing on the detection of DR, RVO, AMD, and other fundus conditions. Furthermore, our approach extended to the detailed classification of DR, RVO, and AMD according to their respective subclasses. We employed a methodology that entails the translation of diagnostic information obtained from FFA results into CFPs. Our investigation focused on evaluating the models' ability to achieve precise diagnoses solely based on CFPs. Remarkably, our models showcased improvements across diverse fundus conditions, with the ConvNeXt-base + attention model standing out for its exceptional performance. The ConvNeXt-base + attention model achieved remarkable metrics, including an area under the receiver operating characteristic curve (AUC) of 0.943, a referable F1 score of 0.870, and a Cohen's kappa of 0.778 for DR detection. For RVO, it attained an AUC of 0.960, a referable F1 score of 0.854, and a Cohen's kappa of 0.819. Furthermore, in AMD detection, the model achieved an AUC of 0.959, an F1 score of 0.727, and a Cohen's kappa of 0.686. Impressively, the model demonstrated proficiency in subclassifying RVO and AMD, showcasing commendable sensitivity and specificity. Moreover, our models enhanced interpretability by visualizing attention weights on fundus images, aiding in the identification of disease findings. These outcomes underscore the substantial impact of our models in advancing the detection of DR, RVO, and AMD, offering the potential for improved patient outcomes and positively influencing the healthcare landscape.

A heterogeneous attention fusion mechanism for the cross-environment scene classification of the home service robot

There have been many methods to improve the capacity of scene classification of service robots. However, most of them are proposed from a technical standpoint but without reference to any cognitive principle of the brain, and furthermore, from design to evaluation, the particularity of the robot task is still not fully considered, such as cross-environment generalization, explicit semantic preservation and interpretation. Thus, the scene cognitive behavior of robots is far from humans, and their environmental adaptability is still poor. It is difficult to complete learning place concepts from discrete fragments and then continuously perceiving them with a limited view in unvisited spaces. Inspired by the recent findings from neuroscience, an attention-based global and object attribute fusion mechanism (AGOFM for short) constructed by three parts is proposed to overcome these deficiencies. In the global attribute part, a global feature extractor and a sequence context extractor are used to generate the holistic feature. The involved context integrates limited views to form an overall impression of a scene for guiding attention. In the object attribute part, a novel object vector is proposed. It simultaneously involves the detected object quantity, category and confidence information, which are all related to the vector index and high-level semantics. In the attention generation part, two sorted top-X characteristics deriving from the above two parts are fed into a fully connected (FC) network with batch normalization to generate effective attention. The attention weights are then applied to the batch normalized global and object vectors respectively, and subsequently, the two heterogeneous information are directly fused by another FC network to achieve scene classification. The policies for multi-learner fusion and frame rejection are also provided. Finally, a novel evaluation paradigm is proposed that the model is trained on a discrete prior dataset, and then the inference is tested on a traditional dataset and two robot view datasets. This simulates the cross-environment situation. Under such severe conditions, the results demonstrate that the proposed method outperforms several popular methods.

Enhancing heart disease prediction using a self-attention-based transformer model

Cardiovascular diseases (CVDs) continue to be the leading cause of more than 17 million mortalities worldwide. The early detection of heart failure with high accuracy is crucial for clinical trials and therapy. Patients will be categorized into various types of heart disease based on characteristics like blood pressure, cholesterol levels, heart rate, and other characteristics. With the use of an automatic system, we can provide early diagnoses for those who are prone to heart failure by analyzing their characteristics. In this work, we deploy a novel self-attention-based transformer model, that combines self-attention mechanisms and transformer networks to predict CVD risk. The self-attention layers capture contextual information and generate representations that effectively model complex patterns in the data. Self-attention mechanisms provide interpretability by giving each component of the input sequence a certain amount of attention weight. This includes adjusting the input and output layers, incorporating more layers, and modifying the attention processes to collect relevant information. This also makes it possible for physicians to comprehend which features of the data contributed to the model's predictions. The proposed model is tested on the Cleveland dataset, a benchmark dataset of the University of California Irvine (UCI) machine learning (ML) repository. Comparing the proposed model to several baseline approaches, we achieved the highest accuracy of 96.51%. Furthermore, the outcomes of our experiments demonstrate that the prediction rate of our model is higher than that of other cutting-edge approaches used for heart disease prediction.

ML-FGAT: Identification of multi-label protein subcellular localization by interpretable graph attention networks and feature-generative adversarial networks

The prediction of multi-label protein subcellular localization (SCL) is a pivotal area in bioinformatics research. Recent advancements in protein structure research have facilitated the application of graph neural networks. This paper introduces a novel approach termed ML-FGAT. The approach begins by extracting node information of proteins from sequence data, physical-chemical properties, evolutionary insights, and structural details. Subsequently, various evolutionary techniques are integrated to consolidate multi-view information. A linear discriminant analysis framework, grounded on entropy weight, is then employed to reduce the dimensionality of the merged features. To enhance the robustness of the model, the training dataset is augmented using feature-generative adversarial networks. For the primary prediction step, graph attention networks are employed to determine multi-label protein SCL, leveraging both node and neighboring information. The interpretability is enhanced by analyzing the attention weight parameters. The training is based on the Gram-positive bacteria dataset, while validation employs newly constructed datasets: human, virus, Gram-negative bacteria, plant, and SARS-CoV-2. Following a leave-one-out cross-validation procedure, ML-FGAT demonstrates noteworthy superiority in this domain.

Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments.

In both plant breeding and crop management, interpretability plays a crucial role in instilling trust in AI-driven approaches and enabling the provision of actionable insights. The primary objective of this research is to explore and evaluate the potential contributions of deep learning network architectures that employ stacked LSTM for end-of-season maize grain yield prediction. A secondary aim is to expand the capabilities of these networks by adapting them to better accommodate and leverage the multi-modality properties of remote sensing data. In this study, a multi-modal deep learning architecture that assimilates inputs from heterogeneous data streams, including high-resolution hyperspectral imagery, LiDAR point clouds, and environmental data, is proposed to forecast maize crop yields. The architecture includes attention mechanisms that assign varying levels of importance to different modalities and temporal features that, reflect the dynamics of plant growth and environmental interactions. The interpretability of the attention weights is investigated in multi-modal networks that seek to both improve predictions and attribute crop yield outcomes to genetic and environmental variables. This approach also contributes to increased interpretability of the model's predictions. The temporal attention weight distributions highlighted relevant factors and critical growth stages that contribute to the predictions. The results of this study affirm that the attention weights are consistent with recognized biological growth stages, thereby substantiating the network's capability to learn biologically interpretable features. Accuracies of the model's predictions of yield ranged from 0.82-0.93 R2 ref in this genetics-focused study, further highlighting the potential of attention-based models. Further, this research facilitates understanding of how multi-modality remote sensing aligns with the physiological stages of maize. The proposed architecture shows promise in improving predictions and offering interpretable insights into the factors affecting maize crop yields, while demonstrating the impact of data collection by different modalities through the growing season. By identifying relevant factors and critical growth stages, the model's attention weights provide valuable information that can be used in both plant breeding and crop management. The consistency of attention weights with biological growth stages reinforces the potential of deep learning networks in agricultural applications, particularly in leveraging remote sensing data for yield prediction. To the best of our knowledge, this is the first study that investigates the use of hyperspectral and LiDAR UAV time series data for explaining/interpreting plant growth stages within deep learning networks and forecasting plot-level maize grain yield using late fusion modalities with attention mechanisms.

Adaptive Prototype Learning for Weakly-supervised Temporal Action Localization.

Weakly-supervised Temporal Action Localization (WTAL) aims to localize action instances with only video-level labels during training, where two primary issues are localization incompleteness and background interference. To relieve these two issues, recent methods adopt an attention mechanism to activate action instances and simultaneously suppress background ones, which have achieved remarkable progress. Nevertheless, we argue that these two issues have not been well resolved yet. On the one hand, the attention mechanism adopts fixed weights for different videos, which are incapable of handling the diversity of different videos, thus deficient in addressing the problem of localization incompleteness. On the other hand, previous methods only focus on learning the foreground attention and the attention weights usually suffer from ambiguity, resulting in difficulty of suppressing background interference. To deal with the above issues, in this paper we propose an Adaptive Prototype Learning (APL) method for WTAL, which includes two key designs: (1) an Adaptive Transformer Network (ATN) to explicitly model background and learn video-adaptive prototypes for each specific video, (2) an OT-based Collaborative (OTC) training strategy to guide the learning of prototypes and remove the ambiguity of the foreground-background separation by introducing an Optimal Transport (OT) algorithm into the collaborative training scheme between RGB and FLOW streams. These two key designs can work together to learn video-adaptive prototypes and solve the above two issues, achieving robust localization. Extensive experimental results on two standard benchmarks (THUMOS14 and ActivityNet) demonstrate that our proposed APL performs favorably against state-of-the-art methods.

Predictive analysis algorithm in educational technology: student behavior prediction and intervention strategy design

Abstract In this paper, we use random forest feature extraction to classify and rank the importance of the behavioral features in the student behavior dataset and obtain the behavioral features with top importance. In the knowledge tracking model, the multidimensional feature strategy is integrated, and the attention weight is introduced in the prediction stage, respectively, so as to predict the results of students’ spatiotemporal behavioral prediction behavioral prediction. The results show that the dormitory area activity has the highest percentage of 30.27%, followed by the teaching area and dining hall area activities. Rest &gt; Study &gt; Eat reflects the regularity of students’ behavior. Behaviors vary at different times of the day. From 0:00 to 7:00, most behaviors are related to rest, while from 8:00 to 11:00, behaviors related to class and eating are predominant. Attending classes abnormally only happened in the second week (3%) and the third week (5%). In the prediction of consumption behavior, the sixth type of students, the average monthly consumption is shallow (541.34) and less frequent (249), and teachers should pay more attention to the life of these students and intervene in the education of mental and physical health.

Rethinking Attention Weights as Bidirectional Coefficients

Attention mechanism has become an almost ubiquitous model architecture in deep learning. One of its distinctive features is to compute non-negative probabilistic distribution to re-weight input representations. This work reconsiders attention weights as bidirectional coefficients instead of probabilistic measures for potential benefits in interpretability and representational capacity. After analyzing the iteration process of attention scores through backwards gradient propagation, we proposed a novel activation function, TanhMax, which possesses several favorable properties to satisfy the requirements of bidirectional attention. We conduct a battery of experiments to validate our analyses and advantages of proposed method on both text and image datasets. The results show that bidirectional attention is effective in revealing input unit’s semantics, presenting more interpretable explanations and increasing the expressive power of attention-based model.

Multi-modal feature fusion with multi-head self-attention for epileptic EEG signals.

It is important to classify electroencephalography (EEG) signals automatically for the diagnosis and treatment of epilepsy. Currently, the dominant single-modal feature extraction methods cannot cover the information of different modalities, resulting in poor classification performance of existing methods, especially the multi-classification problem. We proposed a multi-modal feature fusion (MMFF) method for epileptic EEG signals. First, the time domain features were extracted by kernel principal component analysis, the frequency domain features were extracted by short-time Fourier extracted transform, and the nonlinear dynamic features were extracted by calculating sample entropy. On this basis, the features of these three modalities were interactively learned through the multi-head self-attention mechanism, and the attention weights were trained simultaneously. The fused features were obtained by combining the value vectors of feature representations, while the time, frequency, and nonlinear dynamics information were retained to screen out more representative epileptic features and improve the accuracy of feature extraction. Finally, the feature fusion method was applied to epileptic EEG signal classifications. The experimental results demonstrated that the proposed method achieves a classification accuracy of 92.76 ± 1.64% across the five-category classification task for epileptic EEG signals. The multi-head self-attention mechanism promotes the fusion of multi-modal features and offers an efficient and novel approach for diagnosing and treating epilepsy.

Unsupervised Abundance Matrix Reconstruction Transformer-Guided Fractional Attention Mechanism for Hyperspectral Anomaly Detection.

Hyperspectral anomaly detection (HAD), a challenging inverse problem, has found numerous scientific applications. Although extant HAD algorithms have achieved remarkable results, there are still several issues remained unresolved: 1) low spatial resolution (and spectral redundancy) in typical hyperspectral images prevents effectively distinguishing the abnormal pixels from those normal ones and 2) the reconstruction from existing residual-based frameworks would not completely remove anomaly effects, making the detection solely from the residual impractical. In this article, we propose a novel HAD method, termed transformer-guided fractional attention within the abundance domain (TGFA-AD), which substitutes raw input image with the abundance matrix obtained via blind source separation (BSS). First, the proposed abundance spatial-channel reconstruction transformer (ASCR-Former) is customized for rebuilding the abundance matrix. According to the image self-similarity, the abundance is patch-wisely encoded with class (CLS) tokens. The transformer encoders intensify the spatial and channel characteristics between tokens for reconstructing the abundance, followed by deriving the initial detection from the abundance residual matrix. Second, a novel fractional abundance attention (FAA) mechanism is proposed, where the attention weights coming from a specific linear combination of abundances are guided by the initial detection with convex [Formula: see text] -quadratic norm. Finally, the fractional convolution is incorporated to fuse the abundance and residual into the fractional feature for yielding the final detection result. Real data experiments quantitatively and qualitatively exhibit the state-of-the-art performance of TGFA-AD.