Attention-based Model Research Articles

Development of Attention-based Prediction Models for All-cause Mortality, Home Care Need, and Nursing Home Admission in Ageing Adults in Spain Using Longitudinal Electronic Health Record Data

Predicting health-related outcomes can help with proactive healthcare planning and resource management. This is especially important on the older population, an age group growing in the coming decades. Considering longitudinal rather than cross-sectional information from primary care electronic health records (EHRs) can contribute to more informed predictions. In this work, we developed prediction models using longitudinal EHRs to inform resource allocation. In this study, we developed deep-learning-based prognostic models to predict 1-year and 5-year all-cause mortality, nursing home admission, and home care need in people over 65 years old using all the longitudinal information from EHRs. The models included attention mechanisms to increase their transparency. EHRs were drawn from SIDIAP (primary care, Catalonia (Spain)) from 2010-2019. Performance on the test set was compared to that from baseline models using cross-sectional one-year history only. Data from 1,456,052 individuals over 65 years old were considered. Cohen’s kappa obtained using longitudinal data was 3.4-fold (1-year all-cause mortality), 10.3-fold (5-year all-cause mortality), 1.1-fold (5-year nursing home admission), and 1.2-fold (5-year home care need) higher than that obtained by the one-year history baseline models. Our models performed better than those not considering longitudinal data, especially when predicting further into the future. However, nursing home admission and home care need in the long term were harder to predict, suggesting their dependence on more abrupt changes. The attention maps helped to understand the predictions, enhancing model transparency. These prediction models can contribute to improve resource allocation in the general population of aging adults.

XAttentionHAR Ensemble: Leveraging Cross-Modal Attention for Enhanced Activity Recognition

Human Activity Recognition (HAR) is pivotal in ubiquitous computing, offering benefits to human-centric services such as health monitoring, smart homes, and eldercare systems. HAR leverages smartphones, smartwatches, and other wearable devices to collect sensory data annotated with activity labels, which are then used to train machine learning or deep learning models for automatic activity recognition. Effective HAR systems must integrate information from multiple modalities to accurately assist users. This paper introduces the XAttentionHAR model, an innovative cross-modality attention-based ensemble model for HAR. Our approach utilizes a self-attention module to extract features within each modality and an inter-domain cross-attention module to capture and integrate long-term dependencies across domains. The cross-modality attention mechanism enhances the fusion of diverse modalities, enriching the semantic information. We conducted extensive experiments on the WISDM public dataset, which includes accelerometer and gyroscope data from smartwatches and smartphones. Our results demonstrate that XAttentionHAR outperforms other state-of-the-art methods in activity recognition, achieving 98.48% accuracy for smartphone-based HAR and 98.73% accuracy for smartwatch-based HAR, paving the way for improved human-centric services.

Enhancing intra-aural disease classification with attention-based deep learning models

Enhancing intra-aural disease classification with attention-based deep learning models

Control of wastewater treatment plants using economic-oriented MPC and attention-based RNN disturbance prediction models

Control of wastewater treatment plants using economic-oriented MPC and attention-based RNN disturbance prediction models

Attention-based Deep learning Models for Predicting Anomalous Shock of Wastewater Treatment Plants

Attention-based Deep learning Models for Predicting Anomalous Shock of Wastewater Treatment Plants

Identification of tomato diseases effectively using an attention-based EfficientNet model

Identification of tomato diseases effectively using an attention-based EfficientNet model

Discovering Time-Aware Hidden Dependencies with Personalized Graphical Structure in Electronic Health Records

Over the past decade, significant advancements in mining electronic health records (EHRs) have enabled a broad range of decision-support applications, and offered an unprecedented capacity for predicting critical events such as disease prognosis and mortality in healthcare. Despite the availability of comprehensive coding systems in EHRs (e.g., ICD-9), which are designed to record diverse information on diseases, procedures, and medications over time, the complex and dynamic dependencies among the recorded data are usually not captured. This limitation often hinders the contextual understanding of medical observations for effective EHR representation learning. Therefore, there is a compelling need to discover a hidden “EHR graph” that represents the medical relationship between the observed features according to a patient’s history. These hidden graphs consisting of the medical codes from the same visits can offer a comprehensive insight derived from disease-to-disease, disease-to-drug, and drug-to-drug dependencies. However, it is still unclear how to address the challenge that the dependencies may vary from patient to patient, and they can dynamically evolve from one visit to another. To this end, we propose Timeaware Personalized Graph Transformer (TPGT), a novel attention-based time-aware hidden graph model, that captures the personalized graphical structures among observed medical codes and summarizes the temporal code dependencies over time to improve patient representation for outcome prediction. Built upon an intra-visit and an inter-visit dual-attention mechanism to model patients’ EHR graphs, our model offers an interpretability of what diagnosis or medication in a patient’s history can interact, and how those interactions may change over time. We conduct extensive experiments on two real-world EHR datasets for different healthcare predictive tasks: acute kidney injury (AKI) prediction and ICU mortality prediction. The experimental results demonstrate a significant performance improvement of the proposed model over baselines through multi-aspect quantitative evaluation. Furthermore, we perform various qualitative studies to validate the interpretability of the model which highlights the application of the proposed method in the context of personalized medicine.

Deep learning-based intratumoral and peritumoral features for differentiating ocular adnexal lymphoma and idiopathic orbital inflammation.

To evaluate the value of deep-learning-based intratumoral and peritumoral features for differentiating ocular adnexal lymphoma (OAL) and idiopathic orbital inflammation (IOI). Nighty-seven patients with histopathologically confirmed OAL (n = 43) and IOI (n = 54) were randomly divided into training (n = 79) and test (n = 18) groups. DL-based intratumoral and peritumoral features were extracted to characterize the differences in heterogeneity and tissue invasion between different lesions, respectively. Subsequently, an attention-based fusion model was employed to fuse the features extracted from intra- and peritumoral regions and multiple MR sequences. A comprehensive comparison was conducted among different methods for extracting intratumoral, peritumoral, and fused features. Area under the curve (AUC) was used to evaluate the performance under a 10-fold cross-validation and independent test. Chi-square and student's t-test were used to compare discrete and continuous variables, respectively. Fused intra-peritumoral features achieved AUC values of 0.870-0.930 and 0.849-0.924 on individual MR sequences in the validation and test sets, respectively. This was significantly higher than those using intratumoral features (p < 0.05), but not significantly different than those using peritumoral features (p > 0.05). By combining multiple MR sequences, AUC values of the intra-peritumoral features were boosted to 0.943 and 0.940, higher than those obtained from each sequence alone. Moreover, intra-peritumoral features yielded higher AUC values compared to entire orbital cone features extracted by either the intra- or the peritumoral DL model, although no significant difference was found from the latter (p > 0.05). DL-based intratumoral, peritumoral, and especially fused intra-peritumoral features may help differentiate between OAL and IOI. Question What is the diagnostic value of the peritumoral region and its combination with the intratumoral region for radiomic analysis of orbital lymphoproliferative disorders? Findings Fused intra- and peritumoral features achieved significantly higher performance than intratumoral features, but had no significant difference to the peritumoral features. Clinical relevance Peritumoral contextual features, which characterize the invasion patterns of orbital lesions within the surrounding areas of the entire orbital cone, might serve as an independent imaging marker for differentiating between OAL and IOI.

PALM: Personalized Attention-based Language Model for Long-tail Query Understanding in Enterprise Search Systems

Enterprise search systems face significant challenges in handling long-tail queries, which constitute a substantial portion of search traffic but often receive inadequate attention in traditional systems. This paper introduces PALM (Personalized Attention-based Language Model), a novel framework designed to enhance long-tail query understanding in enterprise search environments. PALM integrates personalization capabilities with an advanced attention mechanism to improve search accuracy for infrequent queries while maintaining high performance on common queries. The framework employs a unique hierarchical architecture that combines user context, query semantics, and organizational knowledge through a sophisticated attention mechanism. The system features an innovative query embedding approach that adapts to individual user contexts while leveraging collective organizational knowledge. Extensive experiments on a large-scale enterprise dataset, comprising over 5 million queries from 50,000 users, demonstrate PALM's superior performance compared to state-of-the-art baselines. The results show significant improvements across multiple metrics, with a 17.5% increase in MAP for ultra-rare queries and a 10.4% overall improvement in NDCG@10. The framework exhibits robust performance across different organizational units and query types, making it particularly valuable for enterprise environments where query patterns are highly diverse and context-dependent. Our ablation studies confirm the effectiveness of each component in the PALM architecture, while case analyses provide insights into the framework's practical applications.

Attention-Based Malware Detection Model by Visualizing Latent Features Through Dynamic Residual Kernel Network.

In recent years, significant research has been directed towards the taxonomy of malware variants. Nevertheless, certain challenges persist, including the inadequate accuracy of sample classification within similar malware families, elevated false-negative rates, and significant processing time and resource consumption. Malware developers have effectively evaded signature-based detection methods. The predominant static analysis methodologies employ algorithms to convert the files. The analytic process is contingent upon the tool's functionality; if the tool malfunctions, the entire process is obstructed. Most dynamic analysis methods necessitate the execution of a binary file within a sandboxed environment to examine its behavior. When executed within a virtual environment, the detrimental actions of the file might be easily concealed. This research examined a novel method for depicting malware as images. Subsequently, we trained a classifier to categorize new malware files into their respective classifications utilizing established neural network methodologies for detecting malware images. Through the process of transforming the file into an image representation, we have made our analytical procedure independent of any software, and it has also become more effective. To counter such adversaries, we employ a recognized technique called involution to extract location-specific and channel-agnostic features of malware data, utilizing a deep residual block. The proposed approach achieved remarkable accuracy of 99.5%, representing an absolute improvement of 95.65% over the equal probability benchmark.

Enhanced Conformer-Based Speech Recognition via Model Fusion and Adaptive Decoding with Dynamic Rescoring

Speech recognition is widely applied in fields like security, education, and healthcare. While its development drives global information infrastructure and AI strategies, current models still face challenges such as overfitting, local optima, and inefficiencies in decoding accuracy and computational cost. These issues cause instability and long response times, hindering AI’s competitiveness. Therefore, addressing these technical bottlenecks is critical for advancing national scientific progress and global information infrastructure. In this paper, we propose improvements to the model structure fusion and decoding algorithms. First, based on the Conformer network and its variants, we introduce a weighted fusion method using training loss as an indicator, adjusting the weights, thresholds, and other related parameters of the fused models to balance the contributions of different model structures, thereby creating a more robust and generalized model that alleviates overfitting and local optima. Second, for the decoding phase, we design a dynamic adaptive decoding method that combines traditional decoding algorithms such as connectionist temporal classification and attention-based models. This ensemble approach enables the system to adapt to different acoustic environments, improving its robustness and overall performance. Additionally, to further optimize the decoding process, we introduce a penalty function mechanism as a regularization technique to reduce the model’s dependence on a single decoding approach. The penalty function limits the weights of decoding strategies to prevent over-reliance on any single decoder, thus enhancing the model’s generalization. Finally, we validate our model on the Librispeech dataset, a large-scale English speech corpus containing approximately 1000 h of audio data. Experimental results demonstrate that the proposed method achieves word error rates (WERs) of 3.92% and 4.07% on the development and test sets, respectively, significantly improving over single-model and traditional decoding methods. Notably, the method reduces WER by approximately 0.4% on complex datasets compared to several advanced mainstream models, underscoring its superior robustness and adaptability in challenging acoustic environments. The effectiveness of the proposed method in addressing overfitting and improving accuracy and efficiency during the decoding phase was validated, highlighting its significance in advancing speech recognition technology.

High-accuracy slope stability analysis using data-driven and attention-based deep learning model

High-accuracy slope stability analysis using data-driven and attention-based deep learning model

Predicting gene expression from histone marks using chromatin deep learning models depends on histone mark function, regulatory distance and cellular states.

To understand the complex relationship between histone mark activity and gene expression, recent advances have used in silico predictions based on large-scale machine learning models. However, these approaches have omitted key contributing factors like cell state, histone mark function or distal effects, which impact the relationship, limiting their findings. Moreover, downstream use of these models for new biological insight is lacking. Here, we present the most comprehensive study of this relationship to date - investigating seven histone marks in eleven cell types across a diverse range of cell states. We used convolutional and attention-based models to predict transcription from histone mark activity at promoters and distal regulatory elements. Our work shows that histone mark function, genomic distance and cellular states collectively influence a histone mark's relationship with transcription. We found that no individual histone mark is consistently the strongest predictor of gene expression across all genomic and cellular contexts. This highlights the need to consider all three factors when determining the effect of histone mark activity on transcriptional state. Furthermore, we conducted in silico histone mark perturbation assays, uncovering functional and disease related loci and highlighting frameworks for the use of chromatin deep learning models to uncover new biological insight.

ColonNeXt: Fully Convolutional Attention for Polyp Segmentation.

This study introduces ColonNeXt, a novel fully convolutional attention-based model for polyp segmentation from colonoscopy images, aimed at the enhancing early detection of colorectal cancer. Utilizing a purely convolutional neural network (CNN), ColonNeXt integrates an encoder-decoder structure with a hierarchical multi-scale context-aware network (MSCAN) in the encoder and a convolutional block attention module (CBAM) in the decoder. The decoder further includes a proposed CNN-based feature attention mechanism for selective feature enhancement, ensuring precise segmentation. A new refinement module effectively improves boundary accuracy, addressing challenges such as variable polyp size, complex textures, and inconsistent illumination. Evaluations on standard datasets show that ColonNeXt achieves high accuracy and efficiency, significantly outperforming competing methods. These results confirm its robustness and precision, establishing ColonNeXt as a state-of-the-art model for polyp segmentation. The code is available at: https://github.com/long-nguyen12/colonnext-pytorch .

Mapping Lithology with Hybrid Attention Mechanism–Long Short-Term Memory: A Hybrid Neural Network Approach Using Remote Sensing and Geophysical Data

Remote sensing provides an efficient roadmap in geological analysis and interpretation. However, some challenges arise when remote sensing techniques are integrated with machine learning in geological surveys. Factors including irregular spatial distribution, sample imbalance, interclass resemblances, regolith, and geochemical similarities impede geological feature diagnosis, interpretation, and identification across varied remote sensing datasets. To address these limitations, a hybrid-attention-integrated long short-term memory (LSTM) network is employed to diagnose, interpret, and identify lithological feature representations in a remote sensing-based geological analysis using multisource data fusion. The experimental design integrates varied datasets including Sentinel-2A, Landsat-9, ASTER, ALOS PALSAR DEM, and Bouguer anomaly gravity data. The proposed model incorporates a hybrid attention mechanism (HAM) comprising channel and spatial attention submodules. HAM utilizes an adaptive technique that merges global-average-pooled features with max-pooled features, enhancing the model’s accuracy in identifying lithological units. Additionally, a channel separation operation is employed to allot refined channel features into clusters based on channel attention maps along the channel dimension. The comprehensive analysis of results from comparative extensive experiments demonstrates HAM-LSTM’s state-of-the-art performance, outperforming existing attention modules and attention-based models (ViT, SE-LSTM, and CBAM-LSTM). Comparing HAM-LSTM to baseline LSTM, the HAM module’s integrated configurations equip the proposed model to better diagnose and identify lithological units, thereby increasing the accuracy by 3.69%.

Cloud Removal in the Tibetan Plateau Region Based on Self-Attention and Local-Attention Models.

Optical remote sensing images have a wide range of applications but are often affected by cloud cover, which interferes with subsequent analysis. Therefore, cloud removal has become indispensable in remote sensing data processing. The Tibetan Plateau, as a sensitive region to climate change, plays a crucial role in the East Asian water cycle and regional climate due to its snow cover. However, the rich ice and snow resources, rapid snow condition changes, and active atmospheric convection in the plateau as well as its surrounding mountainous areas, make optical remote sensing prone to cloud interference. This is particularly significant when monitoring snow cover changes, where cloud removal becomes essential considering the complex terrain and unique snow characteristics of the Tibetan Plateau. This paper proposes a novel Multi-Scale Attention-based Cloud Removal Model (MATT). The model integrates global and local information by incorporating multi-scale attention mechanisms and local interaction modules, enhancing the contextual semantic relationships and improving the robustness of feature representation. To improve the segmentation accuracy of cloud- and snow-covered regions, a cloud mask is introduced in the local-attention module, combined with the local interaction module to modulate and reconstruct fine-grained details. This enables the simultaneous representation of both fine-grained and coarse-grained features at the same level. With the help of multi-scale fusion modules and selective attention modules, MATT demonstrates excellent performance on both the Sen2_MTC_New and XZ_Sen2_Dataset datasets. Particularly on the XZ_Sen2_Dataset, it achieves outstanding results: PSNR = 29.095, SSIM = 0.897, FID = 125.328, and LPIPS = 0.356. The model shows strong cloud removal capabilities in cloud- and snow-covered areas in mountainous regions while effectively preserving snow information, and providing significant support for snow cover change studies.

Enhanced Solar Power Prediction Using Attention-Based DiPLS-BiLSTM Model

The data for solar power generation contain a huge amount of data with a large number of features which are difficult to extract effectively. It is important for the grid management and operational efficiency of the solar farm to accurately predict the solar power. The existing prediction models utilize historical data but often fail to capture critical latent features. This limitation leads to overlooked complex dependencies or temporal relationships, reducing prediction accuracy, especially in load and generation forecasting. An attention-based dynamic inner partial least squares (DiPLS) model and a bidirectional long short-term memory (BiLSTM) model were used for solar power prediction. First, DiPLS is used to dynamically extract features, and then, an attention process is used to predict the importance of these features. Finally, the raw sets are input to the BiLSTM model to make predictions of solar power in the future. The proposed method improves prediction accuracy, achieving an R2 value of 0.965 for training and 0.961 for testing, compared to conventional models. Additionally, the method demonstrated lower root mean squared error (RMSE), indicating enhanced stability and accuracy for solar power forecasting.

Channel Attention-Based Conditional Diffusion Model Applied to Fault Diagnosis Under Imbalanced Data

Issues such as data scarcity and data imbalance have long posed significant difficulties in the field of intelligent fault diagnosis. They lead to reduced diagnostic accuracy and endanger the safety and reliability of industrial equipment. To address these challenges, this study introduces a novel channel attention-based conditional diffusion model (CAC-DM) that recalibrates features through a squeeze-and-excitation process. This enhancement boosts the model’s ability to focus on critical features while suppressing irrelevant information, thereby improving the UNet network’s discrimination capability in handling small-sample faults that are highly similar in nature. Experimental validation demonstrates that CAC-DM performs exceptionally well in scenarios with high class similarity, effectively distinguishing among categories with similar distributions in limited data and generating high-quality samples. Compared to existing generative methods, the CAC-DM exhibits significant advantages in producing distinguishable fault samples, particularly in cases of sample imbalance. This approach offers an effective new solution for fault diagnosis.

Enhancing Human Activity Detection and Classification Using Fine Tuned Attention-Based Transformer Models

Enhancing Human Activity Detection and Classification Using Fine Tuned Attention-Based Transformer Models

Interpretable prediction of drug-drug interactions via text embedding in biomedical literature

Polypharmacy is a promising approach for treating diseases, especially those with complex symptoms. However, it can lead to unexpected drug-drug interactions (DDIs), potentially reducing efficacy and triggering adverse drug reactions (ADRs). Predicting the risk of DDIs is crucial for ensuring safe drug use, particularly by identifying the types of DDIs and the mechanisms involved. Therefore, this study used biomedical literature to proposed hierarchical attention-based deep learning models to predict DDIs and their types. The proposed model consists of two components: drug embedding and DDI prediction. The drug embedding module extracts representation vectors that effectively capture drug properties using sentence and sequence embedding methods. For sentence embedding, a pre-trained biomedical language model is used to map drug-related sentences into vector space. For sequence embedding, sentence embedding vectors are sequentially fed into bidirectional long short-term memory with a hierarchical attention network, enabling the analysis of sentences relevant to DDI prediction while accounting for the order of the sentences. Finally, DDI prediction is performed using a deep neural network based on the sequence embedding vectors of a drug pair. Our model achieved high performances in the accuracy (0.85–0.90), AUROC (0.98–0.99), and AUPR (0.63–0.95) performance across 164 DDI types. Additionally, the proposed model showed improvements in up to 11 % in AUROC, and 8 % in AUPR. Furthermore, model interprets predictions by leveraging attention mechanisms and drug similarity. The results indicated that the model considered various factors beyond similarity to predict DDIs. These findings may help prevent unforeseen medical accidents and reduce healthcare costs by predicting detailed drug interaction types.