Attention-based Deep Learning Model Research Articles

Interpretable multi-step hybrid deep learning model for karst spring discharge prediction: Integrating temporal fusion transformers with ensemble empirical mode decomposition

Karst groundwater is a critical freshwater resource for numerous regions worldwide. Monitoring and predicting karst spring discharge is essential for effective groundwater management and the preservation of karst ecosystems. However, the high heterogeneity and karstification pose significant challenges to physics-based models in providing robust predictions of karst spring discharge. In this study, an interpretable multi-step hybrid deep learning model called selective EEMD-TFT is proposed, which adaptively integrates temporal fusion transformers (TFT) with ensemble empirical mode decomposition (EEMD) for predicting karst spring discharge. The selective EEMD-TFT hybrid model leverages the strengths of both EEMD and TFT techniques to learn inherent patterns and temporal dynamics from nonlinear and nonstationary signals, eliminate redundant components, and emphasize useful characteristics of input variables, leading to the improvement of prediction performance and efficiency. It consists of two stages: in the first stage, the daily precipitation data is decomposed into multiple intrinsic mode functions using EEMD to extract valuable information from nonlinear and nonstationary signals. All decomposed components, temperature and categorical date features are then fed into the TFT model, which is an attention-based deep learning model that combines high-performance multi-horizon prediction and interpretable insights into temporal dynamics. The importance of input variables will be quantified and ranked. In the second stage, the decomposed precipitation components with high importance are selected to serve as the TFT model’s input features along with temperature and categorical date variables for the final prediction. Results indicate that the selective EEMD-TFT model outperforms other sequence-to-sequence deep learning models, such as LSTM and single TFT models, delivering reliable and robust prediction performance. Notably, it maintains more consistent prediction performance at longer forecast horizons compared to other sequence-to-sequence models, highlighting its capacity to learn complex patterns from the input data and efficiently extract valuable information for karst spring prediction. An interpretable analysis of the selective EEMD-TFT model is conducted to gain insights into relationships among various hydrological processes and analyze temporal patterns.

A novel energy management of public charging stations using attention-based deep learning model

Electricity grids are complex systems that must balance the supply and demand of electricity in real-time. However, with the increasing adoption of electric vehicles (EVs), managing the grid’s stability has become more challenging. EV charging can cause spikes in electricity demand, leading to peak demand periods that strain the power grid’s infrastructure. With the help of load forecasting, this effect on the grid can be mitigated by predicting the charging demand of electric vehicles in advance. This will help utilities adjust their energy supply in real-time, ensuring enough energy is available to meet demand, and preventing overloads or under utilization of the grid. Moreover, the EV charging demand is influenced by a wide range of factors, including charging station locations, weather, and time of day. Therefore, advanced deep learning models are required to learn these complex relationships and identify patterns in EV charging demand, enabling utilities to make more informed decisions. In this research, an attention-based deep learning approach is proposed for more accurate prediction of EV load demand. This novel approach integrates attention mechanisms with traditional deep learning models like LSTM and GRU, allowing the model to dynamically weight the importance of different features and focus on the most relevant information. The outcomes are compared to conventional deep learning and machine learning algorithms. To test the efficacy of the proposed framework, an actual ACN dataset for public EV charging stations is utilized.

DMSS: An Attention-Based Deep Learning Model for High-Quality Mass Spectrometry Prediction

DMSS: An Attention-Based Deep Learning Model for High-Quality Mass Spectrometry Prediction

An Attention-Based Deep Learning Model for Phase-Resolved Wave Prediction

Abstract Phase-resolved wave prediction capability, even if only over two wave periods in advance, is of value for optimal control of wave energy converters, resulting in a dramatic increase in power generation efficiency. Previous studies on wave-by-wave predictions have shown that an artificial neural network (ANN) model can outperform the traditional linear wave theory-based model in terms of both prediction accuracy and prediction horizon when using synthetic wave data. However, the prediction performance of ANN models is significantly reduced by the varying wave conditions and buoy positions that occur in the field. To overcome these limitations, a novel wave prediction method is developed based on the neural network with an attention mechanism. This study validates the new model using wave data measured at sea. The model utilizes past time histories of three Sofar Spotter wave buoys at upwave locations to predict the vertical motion of a Datawell Waverider-4 at a downwave location. The results show that the attention-based neural network model is capable of capturing the slow variation in the displacement of the buoys, which reduces the prediction error compared to a standard ANN and long short-term memory model.

Enhancing smart grid load forecasting: An attention-based deep learning model integrated with federated learning and XAI for security and interpretability

Smart grid is a transformative advancement that modernized the traditional power system for effective electricity management, and involves optimized energy distribution by load forecasting. Precise load forecasting provides the best utilization of energy resources and increases sustainability. Dynamic changes of several connected factors, such as temporal and geographical variability, pose challenges to accurate load prediction. Integrating Artificial Intelligence (AI) in the smart grid can enhance the performance of the forecasting process by capturing these changes. This study investigated load forecasting tasks on four different datasets. Several preprocessing and augmentation techniques are applied to increase the data quality. An attention-based 1D-CNN-GRU model is proposed to capture the temporal patterns from the time-series data, and the hyperparameters of the model are optimized using a particle swarm optimization (PSO) algorithm that also accelerates the convergence and results in an efficient training session. Empirical evaluations highlight that the proposed model substantially minimizes the loss, reflecting the ability to make accurate predictions. It obtains MAE values of 0.12, 0.8, 16.48, and 82.64 for the four datasets. Moreover, the explainable AI (XAI) technique is applied using Shapley Additive explanations (SHAP) to interpret the model prediction, providing the feature ranking based on their prediction score. Moreover, this study utilizes federated learning, enables collaborative training, maintains the privacy of the grid data, and secures the process comprehensively. The aggregation mechanism in federated learning is modified using pruning-based methods that reduce the parameters and computational cost, resulting in a more efficient framework. Integrating all these approaches provides valuable insights for developing a load forecasting model and outlines potential contributions in the smart grid domain.

Reliable multi-horizon water demand forecasting model: A temporal deep learning approach

Accurate water demand forecasting can help understand water usage dynamics, which has a potential application in water saving and demand management. Despite extensive research, most exiting methods cannot capture long-range dependency of water demand and maintain high-accuracy in multi-horizon forecasting continuously. To address these issues, the focus of this research is primarily on the temporal model for multi-horizon water demand forecasting using deep learning. This research proposes a signal enhancement strategy and develops an attention-based deep learning model. Firstly, the Fourier transform is used for sparse approximation of water demand data, which helps represent the signal more compactly. To enhance the performance of multi-horizon forecasting model, the complex water demand time series is decomposed into trend and seasonal components. Then, an attention mechanism is utilized to learn temporal dependencies within the data, providing assists for multi-horizon water demand forecasting. Furthermore, a comparative study is carried out between the proposed model and the state-of-the-art methods using the water demand data from four usage scenarios. Experimental results suggest that the present model has the ability to produce accurate and robust forecasting, especially for multi-horizon water demand. Therefore, this research has the potential to enhance water resource management in a socioecological context.

DeepCTF: transcription factor binding specificity prediction using DNA sequence plus shape in an attention-based deep learning model

DeepCTF: transcription factor binding specificity prediction using DNA sequence plus shape in an attention-based deep learning model

Assessing robustness to adversarial attacks in attention-based networks: Case of EEG-based motor imagery classification

The classification of motor imagery (MI) using Electroencephalography (EEG) plays a pivotal role in facilitating communication for individuals with physical limitations through Brain-Computer Interface (BCI) systems. Recent strides in Attention-Based Networks (ATN) have showcased remarkable performance in EEG signal classification, presenting a promising alternative to conventional Convolutional Neural Networks (CNNs). However, while CNNs have been extensively analyzed for their resilience against adversarial attacks, the susceptibility of ATNs in comparable scenarios remains largely unexplored. This paper aims to fill this gap by investigating the robustness of ATNs in adversarial contexts. We propose a high-performing attention-based deep learning model specifically designed for classifying Motor Imagery (MI) brain signals extracted from EEG data. Subsequently, we conduct a thorough series of experiments to assess various attack strategies targeting ATNs employed in EEG-based BCI tasks. Our analysis utilizes the widely recognized BCI Competition 2a dataset to demonstrate the effectiveness of attention mechanisms in BCI endeavors. Despite achieving commendable classification results in terms of accuracy (87.15%) and kappa score (0.8287), our findings reveal the vulnerability of attention-based models to adversarial manipulations (accuracy: 9.07%, kappa score: -0.21), highlighting the imperative for bolstering the robustness of attention architectures for EEG classification tasks.

Triplet attention-based deep learning model for hierarchical image classification of household items for robotic applications

Triplet attention-based deep learning model for hierarchical image classification of household items for robotic applications

Abstract 2584: Decoding colon cancer recurrence: Unveiling accurate predictions with attention-guided deep neural networks on histopathological whole slide images

Abstract Background: Up to 40% of colon cancer patients are at high risk of cancer recurrence, yet accurate and timely prediction tools are lacking. Leveraging whole slide images (WSIs) and deep learning models, we aimed to develop precise algorithm for prediction of colon cancer recurrence. Thus, enables risk stratification for optimized therapeutic interventions and improved health outcomes. Design: We designed an attention-based deep learning model for predicting colon cancer recurrence on paraffin-embedded, hematoxylin and eosin-stained colon tissue biopsies of digital slides. The WSIs were downloaded from the TCGA dataset, and preprocessed for color normalization, tissue segmentation, tilling, and histopathological features extraction. The entire dataset was then labeled based on cancer recurrence status, and divided into training (70%), validation (15%), and testing sets (15%). The model's performance was then evaluated by standard evaluation metrics, including receiver operating characteristic Area Under the Curve (AUC) and accuracy, on the training, validation, and testing datasets, where interpretability attention heatmaps were applied to gain insights into specific histological features and patterns involved in the model's decision-making process. The study is supported by the NIH-NCI-T32 for Next Generation Pathologists Program at our institution. Results: A total of 350 WSIs were included and labeled into two classes: post-therapeutic colon cancer recurrence and no recurrence. The tissue segmentation process involved converting RGB images to HSV color space, applying a median blur filter (kernel size: 7), and performing thresholding using Otsu's method. The extracted features were utilized to train and construct a Clustering-constrained Attention Multiple Instance Learning (CLAM) model. The model demonstrated consistent performance across the validation and testing datasets, achieving an AUC of 0.85 with an accuracy of 0.83 on the validation set, and an identical AUC of 0.85 with an accuracy of 0.83 on the testing set, indicating the model's robust ability to identify patients at risk of cancer recurrence. Conclusion: The study demonstrates the strong performance of our deep learning model in accurately identifying patients at risk for colon cancer recurrence based on H&E WSIs. This capability paves the way for optimizing therapeutic interventions and implementing effective surveillance strategies. Consequently, highlighting the crucial role of pathologists in collaborating with the oncologist for optimizing the management of colon cancer care in the era of personalized medicine. Citation Format: Mohammad K. Alexanderani, Mohamed Omar, Matthew Greenblatt, Ethel Cesarman, Maria T. Diaz-Meco, Jorge Moscat, Luigi Marchionni. Decoding colon cancer recurrence: Unveiling accurate predictions with attention-guided deep neural networks on histopathological whole slide images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2584.

MOB-CBAM: A dual-channel attention-based deep learning generalizable model for breast cancer molecular subtypes prediction using mammograms

Background and objectiveDeep Learning models have emerged as a significant tool in generating efficient solutions for complex problems including cancer detection, as they can analyze large amounts of data with high efficiency and performance. Recent medical studies highlight the significance of molecular subtype detection in breast cancer, aiding the development of personalized treatment plans as different subtypes of cancer respond better to different therapies. MethodsIn this work, we propose a novel lightweight dual-channel attention-based deep learning model MOB-CBAM that utilizes the backbone of MobileNet-V3 architecture with a Convolutional Block Attention Module to make highly accurate and precise predictions about breast cancer. We used the CMMD mammogram dataset to evaluate the proposed model in our study. Nine distinct data subsets were created from the original dataset to perform coarse and fine-grained predictions, enabling it to identify masses, calcifications, benign, malignant tumors and molecular subtypes of cancer, including Luminal A, Luminal B, HER-2 Positive, and Triple Negative. The pipeline incorporates several image pre-processing techniques, including filtering, enhancement, and normalization, for enhancing the model's generalization ability. ResultsWhile identifying benign versus malignant tumors, i.e., coarse-grained classification, the MOB-CBAM model produced exceptional results with 99 % accuracy, precision, recall, and F1-score values of 0.99 and MCC of 0.98. In terms of fine-grained classification, the MOB-CBAM model has proven to be highly efficient in accurately identifying mass with (benign/malignant) and calcification with (benign/malignant) classification tasks with an impressive accuracy rate of 98 %. We have also cross-validated the efficiency of the proposed MOB-CBAM deep learning architecture on two datasets: MIAS and CBIS-DDSM. On the MIAS dataset, an accuracy of 97 % was reported for the task of classifying benign, malignant, and normal images, while on the CBIS-DDSM dataset, an accuracy of 98 % was achieved for the classification of mass with either benign or malignant, and calcification with benign and malignant tumors. ConclusionThis study presents lightweight MOB-CBAM, a novel deep learning framework, to address breast cancer diagnosis and subtype prediction. The model's innovative incorporation of the CBAM enhances precise predictions. The extensive evaluation of the CMMD dataset and cross-validation on other datasets affirm the model's efficacy.

Attention-Based Deep Learning Model for Prediction of Major Adverse Cardiovascular Events in Peritoneal Dialysis Patients.

Major adverse cardiovascular events (MACE) encompass pivotal cardiovascular outcomes such as myocardial infarction, unstable angina, and cardiovascular-related mortality. Patients undergoing peritoneal dialysis (PD) exhibit specific cardiovascular risk factors during the treatment, which can escalate the likelihood of cardiovascular events. Hence, the prediction and key factor analysis of MACE have assumed paramount significance for peritoneal dialysis patients. Current pathological methodologies for prognosis prediction are not only costly but also cumbersome in effectively processing electronic health records (EHRs) data with high dimensionality, heterogeneity, and time series. Therefore in this study, we propose the CVEformer, an attention-based neural network designed to predict MACE and analyze risk factors. CVEformer leverages the self-attention mechanism to capture temporal correlations among time series variables, allowing for weighted integration of variables and estimation of the probability of MACE. CVEformer first captures the correlations among heterogeneous variables through attention scores. Then, it analyzes the correlations within the time series data to identify key risk variables and predict the probability of MACE. When trained and evaluated on data from a large cohort of peritoneal dialysis patients across multiple centers, CVEformer outperforms existing models in terms of predictive performance.

Identification of surface defects on solar PV panels and wind turbine blades using attention based deep learning model

The global generation of renewable energy has rapidly increased, primarily due to the installation of large-scale renewable energy power plants. However, monitoring renewable energy assets in these large plants remains challenging due to environmental factors that could result in reduced power generation, malfunctioning, and degradation of asset life. Therefore, the detection of surface defects on renewable energy assets is crucial for maintaining the performance and efficiency of these plants. This paper proposes an innovative detection framework to achieve an economical surface monitoring system for renewable energy assets. High-resolution images of the assets are captured regularly and inspected to identify surface or structural damages on solar panels and wind turbine blades. Vision transformer (ViT), one of the latest attention-based deep learning (DL) models in computer vision, is proposed in this work to classify surface defects. The ViT model outperforms other DL models, including MobileNet, VGG16, Xception, EfficientNetB7, and ResNet50, achieving high accuracy scores above 97% for both wind and solar plant assets. From the results, our proposed model demonstrates its potential for monitoring and detecting damages in renewable energy assets for efficient and reliable operation of renewable power plants.

Improved detection of aortic dissection in non-contrast-enhanced chest CT using an attention-based deep learning model

Rationale and objectivesThis study investigated the effects of implementing an attention-based deep learning model for the detection of aortic dissection (AD) using non-contrast-enhanced chest computed tomography (CT). Materials and methodsWe analysed the records of 1300 patients who underwent contrast-enhanced chest CT at 2 medical centres between January 2015 and February 2023. We considered an internal cohort of 200 patients with AD and 200 patients without AD and an external test cohort of 40 patients with AD and 40 patients without AD. The internal cohort was divided into training and test sets, and a deep learning model was trained using 9600 CT images. A convolutional block attention module (CBAM) and a traditional deep learning architecture (namely, You Only Look Once version 5 [YOLOv5]) were combined into an attention-based model (i.e., YOLOv5-CBAM). Its performance was measured against the unmodified YOLOv5 model, and the accuracy, sensitivity, and specificity of the algorithm were evaluated by two independent radiologists. ResultsThe CBAM-based model outperformed the traditional deep learning model. In the external testing set, YOLOv5-CBAM achieved an area under the curve (AUC) of 0.938, accuracy of 91.5 %, sensitivity of 90.0 %, and specificity of 92.9 %, whereas the unmodified model achieved an AUC of 0.844, accuracy of 83.6 %, sensitivity of 71.2 %, and specificity of 96.0 %. The sensitivity results of the unmodified algorithms were not significantly different from those of the radiologists; however, the proposed YOLOv5-CBAM algorithm outperformed the unmodified algorithms in terms of detection. ConclusionsIncorporating the CBAM attention mechanism into a deep learning model can significantly improve AD detection in non-contrast-enhanced chest CT. This approach may aid radiologists in the timely and accurate diagnosis of AD, which is important for improving patient outcomes.

Pentad-mean air temperature prediction using spatial autocorrelation and attention-based deep learning model

Pentad-mean air temperature prediction using spatial autocorrelation and attention-based deep learning model

GAABind: a geometry-aware attention-based network for accurate protein-ligand binding pose and binding affinity prediction.

Protein-ligand interactions are increasingly profiled at high-throughput, playing a vital role in lead compound discovery and drug optimization. Accurate prediction of binding pose and binding affinity constitutes a pivotal challenge in advancing our computational understanding of protein-ligand interactions. However, inherent limitations still exist, including high computational cost for conformational search sampling in traditional molecular docking tools, and the unsatisfactory molecular representation learning and intermolecular interaction modeling in deep learning-based methods. Here we propose a geometry-aware attention-based deep learning model, GAABind, which effectively predicts the pocket-ligand binding pose and binding affinity within a multi-task learning framework. Specifically, GAABind comprehensively captures the geometric and topological properties of both binding pockets and ligands, and employs expressive molecular representation learning to model intramolecular interactions. Moreover, GAABind proficiently learns the intermolecular many-body interactions and simulates the dynamic conformational adaptations of the ligand during its interaction with the protein through meticulously designed networks. We trained GAABind on the PDBbindv2020 and evaluated it on the CASF2016 dataset; the results indicate that GAABind achieves state-of-the-art performance in binding pose prediction and shows comparable binding affinity prediction performance. Notably, GAABind achieves a success rate of 82.8% in binding pose prediction, and the Pearson correlation between predicted and experimental binding affinities reaches up to 0.803. Additionally, we assessed GAABind's performance on the severe acute respiratory syndrome coronavirus 2 main protease cross-docking dataset. In this evaluation, GAABind demonstrates a notable success rate of 76.5% in binding pose prediction and achieves the highest Pearson correlation coefficient in binding affinity prediction compared with all baseline methods.

Explainable predictions of multi-component oxides enabled by attention-based neural networks

Multicomponent oxides (MCOs) have attracted considerable attention due to their wide range of applications. However, the extensive search space of MCO components and the scarcity of MCO crystal structures in existing literature have promoted the use of deep machine learning methods for predicting MCO properties. Despite these advances, accurately predicting the thermal expansion of MCOs across wide temperature and composition ranges remains a complex task. An innovative attention-based deep learning model was introduced in this study. The two proposed self-attention modules greatly improved the model's performance, achieving an 86.88% reduction in root mean square error for predicting thermal expansion coefficients of multicomponent oxides. Additionally, the model demonstrates impressive adaptability and interpretability. Its training results can further aid in comprehending the thermal expansion coefficient variations of multicomponent oxide materials. In summary, judiciously crafted self-attention models overcome tradeoffs between performance and interpretability for materials discovery.

An attention-based deep learning model for multi-horizon time series forecasting by considering periodic characteristic

Recently, transformer-based models have exhibited great performance in multi-horizon time series forecasting tasks. However, the core module of these models, the self-attention mechanism, is insensitive to the temporal order and suffers from attention dispersion over long time sequences. These limitations hinder the models from fully leveraging the features of time series data, particularly the periodicity. Furthermore, the lack of consideration for temporal order also hinders the identification of important temporal variables in transformers. To resolve these problems, this article develops an attention based deep learning model that can better utilize periodicity to improve prediction accuracy and enhance interpretability. We design a parallel skip LSTM module and a periodicity information utilization module to reinforce the connection between corresponding time steps within different periods and solve the problem of excessively sparse attention. An improved variable selection mechanism is embedded to the parallel skip LSTM such that temporal information can be considered when analyzing interpretability. The experimental findings on different types of real datasets show that the proposed model outperforms numerous baseline models in terms of prediction accuracy while obtaining certain interpretability.

Attention-based deep learning model to improving multi-criteria decision-making for customer loyalty

Understanding the factors that influence product loyalty is crucial for businesses to effectively attract and retain customers. This study suggests a novel approach to assess the importance and weight of criteria that lead to product loyalty by considering the Halo effect in customer decision-making. The suggested method utilizes an attention-based deep learning model to analyze customer feedback collected through the Net Promoter Score (NPS) scale, incorporating the insights of a large number of customers. The proposed method overcomes the limitations of traditional methods that rely on expert judgments or data mining, providing a more comprehensive and customer-centric perspective. By considering the Halo effect, which can lead to biased perceptions of product features, the method offers a more accurate assessment of criteria weights and their impact on product loyalty. A case study focusing on mobile phone selection and loyalty is conducted to explain the applicability and efficiency of the suggested method. The outcomes are compared with the NPS index and several common multi-criteria decision-making (MCDM) techniques. The findings highlight the superiority of the suggested method in capturing the complex relationships between criteria and product loyalty, surpassing the limitations of expert-based approaches and outperforming traditional MCDM methods. The suggested technique provides valuable insights for companies seeking to enhance customer loyalty and optimize product development strategies. However, it is important to acknowledge limitations related to the reliance on customer feedback and the contextual specificity of the results.

IChrom-Deep: An Attention-Based Deep Learning Model for Identifying Chromatin Interactions.

Identification of chromatin interactions is crucial for advancing our knowledge of gene regulation. However, due to the limitations of high-throughput experimental techniques, there is an urgent need to develop computational methods for predicting chromatin interactions. In this study, we propose a novel attention-based deep learning model, termed IChrom-Deep, to identify chromatin interactions using sequence features and genomic features. The experimental results based on the datasets of three cell lines demonstrate that the IChrom-Deep achieves satisfactory performance and is superior to the previous methods. We also investigate the effect of DNA sequence and associated features and genomic features on chromatin interactions, and highlight the applicable scenarios of some features, such as sequence conservation and distance. Moreover, we identify a few genomic features that are extremely important across different cell lines, and IChrom-Deep achieves comparable performance with only these significant genomic features versus using all genomic features. It is believed that IChrom-Deep can serve as a useful tool for future studies that seek to identify chromatin interactions.