Attention-based Network Research Articles

An attention-based deep network for plant disease classification

Plant disease classification using machine learning in a real agricultural field environment is a difficult task. Often, an automated plant disease diagnosis method might fail to capture and interpret discriminatory information due to small variations among leaf sub-categories. Yet, modern Convolutional Neural Networks (CNNs) have achieved decent success in discriminating various plant diseases using leave images. A few existing methods have applied additional pre-processing modules or sub-networks to tackle this challenge. Sometimes, the feature maps ignore partial information for holistic description by part-mining. A deep CNN that emphasizes integration of partial descriptiveness of leaf regions is proposed in this work. The efficacious attention mechanism is integrated with high-level feature map of a base CNN for enhancing feature representation. The proposed method focuses on important diseased areas in leaves, and employs an attention weighting scheme for utilizing useful neighborhood information. The proposed Attention-based network for Plant Disease Classification (APDC) method has achieved state-of-the-art performances on four public plant datasets containing visual/thermal images. The best top-1 accuracies attained by the proposed APDC are: PlantPathology 97.74%, PaddyCrop 99.62%, PaddyDoctor 99.65%, and PlantVillage 99.97%. These results justify the suitability of proposed method.

CytoNet: an efficient dual attention based automatic prediction of cancer sub types in cytology studies

Computer-assisted diagnosis (CAD) plays a key role in cancer diagnosis or screening. Whereas, current CAD performs poorly on whole slide image (WSI) analysis, and thus fails to generalize well. This research aims to develop an automatic classification system to distinguish between different types of carcinomas. Obtaining rich deep features in multi-class classification while achieving high accuracy is still a challenging problem. The detection and classification of cancerous cells in WSI are quite challenging due to the misclassification of normal lumps and cancerous cells. This is due to cluttering, occlusion, and irregular cell distribution. Researchers in the past mostly obtained the hand-crafted features while neglecting the above-mentioned challenges which led to a reduction of the classification accuracy. To mitigate this problem we proposed an efficient dual attention-based network (CytoNet). The proposed network is composed of two main modules (i) Efficient-Net and (ii) Dual Attention Module (DAM). Efficient-Net is capable of obtaining higher accuracy and enhancing efficiency as compared to existing Convolutional Neural Networks (CNNs). It is also useful to obtain the most generic features as it has been trained on ImageNet. Whereas DAM is very robust in obtaining attention and targeted features while negating the background. In this way, the combination of an efficient and attention module is useful to obtain the robust, and intrinsic features to obtain comparable performance. Further, we evaluated the proposed network on two well-known datasets (i) Our generated thyroid dataset (ii) Mendeley Cervical dataset (Hussain in Data Brief, 2019) with enhanced performance compared to their counterparts. CytoNet demonstrated a 99% accuracy rate on the thyroid dataset in comparison to its counterpart. The precision, recall, and F1-score values achieved on the Mendeley Cervical dataset are 0.992, 0.985, and 0.977, respectively. The code implementation is available on GitHub. https://github.com/naveedilyas/CytoNet-An-Efficient-Dual-Attention-based-Automatic-Prediction-of-Cancer-Sub-types-in-Cytol.

CSA-Net: Channel and Spatial Attention-Based Network for Mammogram and Ultrasound Image Classification.

Breast cancer persists as a critical global health concern, emphasizing the advancement of reliable diagnostic strategies to improve patient survival rates. To address this challenge, a computer-aided diagnostic methodology for breast cancer classification is proposed. An architecture that incorporates a pre-trained EfficientNet-B0 model along with channel and spatial attention mechanisms is employed. The efficiency of leveraging attention mechanisms for breast cancer classification is investigated here. The proposed model demonstrates commendable performance in classification tasks, particularly showing significant improvements upon integrating attention mechanisms. Furthermore, this model demonstrates versatility across various imaging modalities, as demonstrated by its robust performance in classifying breast lesions, not only in mammograms but also in ultrasound images during cross-modality evaluation. It has achieved accuracy of 99.9% for binary classification using the mammogram dataset and 92.3% accuracy on the cross-modality multi-class dataset. The experimental results emphasize the superiority of our proposed method over the current state-of-the-art approaches for breast cancer classification.

Addressing the Generalizability of AI in Radiology Using a Novel Data Augmentation Framework with Synthetic Patient Image Data: Proof-of-Concept and External Validation for Classification Tasks in Multiple Sclerosis.

Artificial intelligence (AI) models often face performance drops after deployment to external datasets. This study evaluated the potential of a novel data augmentation framework based on generative adversarial networks (GANs) that creates synthetic patient image data for model training to improve model generalizability. Model development and external testing were performed for a given classification task, namely the detection of new fluid-attenuated inversion recovery lesions at MRI during longitudinal follow-up of patients with multiple sclerosis (MS). An internal dataset of 669 patients with MS (n = 3083 examinations) was used to develop an attention-based network, trained both with and without the inclusion of the GAN-based synthetic data augmentation framework. External testing was performed on 134 patients with MS from a different institution, with MR images acquired using different scanners and protocols than images used during training. Models trained using synthetic data augmentation showed a significant performance improvement when applied on external data (area under the receiver operating characteristic curve [AUC], 83.6% without synthetic data vs 93.3% with synthetic data augmentation; P = .03), achieving comparable results to the internal test set (AUC, 95.0%; P = .53), whereas models without synthetic data augmentation demonstrated a performance drop upon external testing (AUC, 93.8% on internal dataset vs 83.6% on external data; P = .03). Data augmentation with synthetic patient data substantially improved performance of AI models on unseen MRI data and may be extended to other clinical conditions or tasks to mitigate domain shift, limit class imbalance, and enhance the robustness of AI applications in medical imaging. Keywords: Brain, Brain Stem, Multiple Sclerosis, Synthetic Data Augmentation, Generative Adversarial Network Supplemental material is available for this article. © RSNA, 2024.

Abstract C065: PlastiNet: Understanding the Epithelial - Mesenchymal Transition Through Graphical Attention in Spatial Transcriptomics

Abstract Plasticity in tumor cells represents the ability of cancer cells to transition between different phenotypic states, including epithelial and mesenchymal states, known as the epithelial-mesenchymal transition (EMT). This transition is critical for processes such as invasion, metastasis, and resistance to therapy. Understanding the dynamics of plasticity and the extrinsic factors influencing state transitions may be crucial for improving therapeutic strategies and patient outcomes. However, deciphering the complex interactions and factors driving these state changes remains challenging. We introduce PlastiNet, a graphical attention-based network (GAT) designed to create spatially aware embeddings that enable the identification of cellular neighborhoods and the plasticity spectrum, despite gene panel limitations. PlastiNet incorporates both self-attention and neighbor-attention mechanisms in its GAT. This approach allows us to account for the influence of surrounding cells in determining the state of a target cell. We validated our model by constructing differentiation paths in healthy colon data, starting from stem cell signatures. The attention mechanisms highlighted known interactions, such as those between fibroblasts and epithelial cells. When applied to pancreatic ductal adenocarcinoma (PDAC) tumors, PlastiNet identified the SPP1-CD44 axis as a potential mediator in the classical-to-basal transition of tumor cells. To further explore the impact of CD44, particularly the isoform switch that occurs during EMT, we studied a patient-derived a PDAC cell line using single-cell RNA sequencing (scRNAseq) and Multiplexed Arrays Isoform Sequencing (Mas-Iso-Seq) to analyze cells spanning a range of the plasticity spectrum. Looping this analysis back to spatial transcriptomics data, we observed that the signature associated with CD44v8-10 or CD44s isoforms is co-localized with the SPP1 ligand, suggesting a spatially confined interaction. This finding underscores the potential of PlastiNet to generate hypotheses directly from patient-derived samples, offering new insights into the cellular mechanisms driving tumor plasticity. PlastiNet effectively constructs spatially aware embeddings that capture the dynamics of tumor cell plasticity. By identifying key interactions and extrinsic factors, this method opens new avenues for understanding and potentially targeting the mechanisms underlying tumor progression and treatment resistance. Citation Format: Izabella L Zamora, Milan Parikh, Samuel Wright, Lynn Bi, Bidish Patel, Dana Pe’er, David Ting, Nir Hacohen, Arnav Mehta. PlastiNet: Understanding the Epithelial - Mesenchymal Transition Through Graphical Attention in Spatial Transcriptomics [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Pancreatic Cancer Research; 2024 Sep 15-18; Boston, MA. Philadelphia (PA): AACR; Cancer Res 2024;84(17 Suppl_2):Abstract nr C065.

A Novel Network for Low-Dose CT Denoising Based on Dual-Branch Structure and Multi-Scale Residual Attention.

Deep learning-based denoising of low-dose medical CT images has received great attention both from academic researchers and physicians in recent years, and has shown important application value in clinical practice. In this work, a novel two-branch and multi-scale residual attention-based network for low-dose CT image denoising is proposed. It adopts a two-branch framework structure, to extract and fuse image features at shallow and deep levels respectively, to recover image texture and structure information as much as possible. We propose the adaptive dynamic convolution block (ADCB) in the local information extraction layer. It can effectively extract the detailed information of low-dose CT denoising and enables the network to better capture the local details and texture features of the image, thereby improving the denoising effect and image quality. Multi-scale edge enhancement attention block (MEAB) is proposed in the global information extraction layer, to perform feature fusion through dilated convolution and a multi-dimensional attention mechanism. A multi-scale residual convolution block (MRCB) is proposed to integrate feature information and improve the robustness and generalization of the network. To demonstrate the effectiveness of our method, extensive comparison experiments are conducted and the performances evaluated on two publicly available datasets. Our model achieves 29.3004 PSNR, 0.8659 SSIM, and 14.0284 RMSE on the AAPM-Mayo dataset. It is evaluated by adding four different noise levels σ = 15, 30, 45, and 60 on the Qin_LUNG_CT dataset and achieves the best results. Ablation studies show that the proposed ADCB, MEAB, and MRCB modules improve the denoising performances significantly. The source code is available at https://github.com/Ye111-cmd/LDMANet .

Advancing infrared and visible image fusion with an enhanced multiscale encoder and attention-based networks

Infrared and visible image fusion aims to produce images that highlight key targets and offer distinct textures, by merging the thermal radiation infrared images with the detailed texture visible images. Traditional auto encoder-decoder-based fusion methods often rely on manually designed fusion strategies, which lack flexibility across different scenarios. Addressing this limitation, we introduce EMAFusion, a fusion approach featuring an enhanced multiscale encoder and a learnable, lightweight fusion network. Our method incorporates skip connections, the convolutional block attention module (CBAM), and nest architecture within the auto encoder-decoder framework to adeptly extract and preserve multiscale features for fusion tasks. Furthermore, a fusion network driven by spatial and channel attention mechanisms is proposed, designed to precisely capture and integrate essential features from both image types. Comprehensive evaluations of the TNO image fusion dataset affirm the proposed method's superiority over existing state-of-the-art techniques, demonstrating its potential for advancing infrared and visible image fusion.

Efficient attention-based networks for fire and smoke detection

Efficient attention-based networks for fire and smoke detection

FA-Net: A hierarchical feature fusion and interactive attention-based network for dose prediction in liver cancer patients

Dose prediction is a crucial step in automated radiotherapy planning for liver cancer. Several deep learning-based approaches for dose prediction have been proposed to enhance the design efficiency and quality of radiotherapy plan. However, these approaches usually take CT images and contours of organs at risk (OARs) and planning target volume (PTV) as a multi-channel input and is thus difficult to extract sufficient feature information from each input, which results in unsatisfactory dose distribution. In this paper, we propose a novel dose prediction network for liver cancer based on hierarchical feature fusion and interactive attention. A feature extraction module is first constructed to extract multi-scale features from different inputs, and a hierarchical feature fusion module is then built to fuse these multi-scale features hierarchically. A decoder based on attention mechanism is designed to gradually reconstruct the fused features into dose distribution. Additionally, we design an autoencoder network to generate a perceptual loss during training stage, which is used to improve the accuracy of dose prediction. The proposed method is tested on private clinical dataset and obtains HI and CI of 0.31 and 0.87, respectively. The experimental results are better than those by several existing methods, indicating that the dose distribution generated by the proposed method is close to that approved in clinics. The codes are available at https://github.com/hired-ld/FA-Net.

A new attention-based CNN_GRU model for spatial-temporal PM2.5 prediction.

Accurately predicting thespatial-temporal distribution of PM2.5 is challenging due to missing data and selecting an appropriate modeling method. Effective imputation of missing data must consider the relationships between variables while preserving their inherent variability and uncertainty. In this study, we employed machine learning techniques to impute missing data by analyzing the relationships between meteorological variables and other pollutants. Subsequently, we introduced an innovative spatiotemporal hybrid model, AC_GRU, which integrates a one-dimensional convolutional neural network (CNN), GRU, and an attention-based network to predict PM2.5 concentrations in urban areas. The AC_GRU model utilizes meteorological variables, PM2.5 concentrations from nearby air quality monitoring stations, and concentrations of other pollutants as inputs. This approach allows the model to learn spatiotemporal correlations within the time-series data, enhancing the accuracy of PM2.5 predictions. Additionally, the attention mechanism improves prediction accuracy by automatically weighting the past input variables based on their importance for future PM2.5 predictions. The experimental results demonstrate that our AC_GRU model outperforms state-of-the-art methods, making it a valuable tool for urban air quality management and public health protection.

Knowledge Graph Enhanced Contextualized Attention-Based Network for Responsible User-Specific Recommendation

With ever-increasing dataset size and data storage capacity, there is a strong need to build systems that can effectively utilize these vast datasets to extract valuable information. Large datasets often exhibit sparsity and pose cold start problems, necessitating the development of responsible recommender systems. Knowledge graphs have utility in responsibly representing information related to recommendation scenarios. However, many studies overlook explicitly encoding contextual information, which is crucial for reducing the bias of multi-layer propagation. Additionally, existing methods stack multiple layers to encode high-order neighbor information while disregarding the relational information between items and entities. This oversight hampers their ability to capture the collaborative signal latent in user-item interactions. This is particularly important in health informatics, where knowledge graphs consist of various entities connected to items through different relations. Ignoring the relational information renders them insufficient for modeling user preferences. This work presents an end-to-end recommendation framework named KGCAN (Knowledge Graph Enhanced Contextualized Attention-Based Network), which explicitly encodes both relational and contextual information of entities to preserve the original entity information. Furthermore, a user-specific attention mechanism is employed to capture personalized recommendations. The proposed model is validated on three benchmark datasets through extensive experiments. The experimental results demonstrate that KGCAN outperforms existing knowledge graph based recommendation models. Additionally, a case study from the healthcare domain is discussed, highlighting the importance of attention mechanisms and high-order connectivity in the responsible recommendation system for health informatics.

DNACoder: a CNN-LSTM attention-based network for genomic sequence data compression

DNACoder: a CNN-LSTM attention-based network for genomic sequence data compression

A dense triple-level attention-based network for surgical instrument segmentation

A dense triple-level attention-based network for surgical instrument segmentation

MDEANet: modified detail-enhanced convolution and attention-based network for dehazing of remote sensing images

MDEANet: modified detail-enhanced convolution and attention-based network for dehazing of remote sensing images

A lightweight attention-based network for image dehazing

A lightweight attention-based network for image dehazing

A DRL-Based Reactive Scheduling Policy for Flexible Job Shops With Random Job Arrivals

In real-life production systems, arrivals of jobs are usually unpredictable, which makes it necessary to develop solid reactive scheduling policies to meet delivery requirements. Deep reinforcement learning (DRL) based scheduling methods are capable of quickly responding to dynamic events by learning from the training data. However, most of policy networks in DRL algorithms are trained to choose priority dispatching rules (PDR), thus, to some extent, the efficiency of obtained scheduling plans is limited by the performance of PDRs. This paper investigates a dynamic flexible job shop scheduling problem with random job arrivals for the total tardiness minimization. A DRL-based reactive scheduling method, proximal policy optimization with attention-based policy network (PPO-APN), is proposed to make real-time decisions for the dynamic scheduling environment, where the attention-based policy network (APN) is able to directly select pending jobs distinguished from the action space that consists of PDRs. Additionally, a global/local reward function (GLRF) is designed to address the reward sparsity issue during training processes. The proposed PPO-APN is tested on randomly generated instances with different production configurations, and is compared with frequently-used PDRs and DRL-based methods. Numerical experimental results indicate that APN and GLRF components significantly improve the training efficiency, and the PPO-APN shows better overall performance compared with other methods. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by a typical production scenario in discrete manufacturing systems, where orders randomly arrive at the shop floor and require to be scheduled in a short time to ensure the on-time delivery. Previous research work tends to apply DRL algorithms to choose suitable dispatching rules for the ease of implementation. Nevertheless, the jobs that can be selected by dispatching rules are rather limited, thus many possible high-quality scheduling plans are ignored. This work first sorts all the unscheduled jobs by a heuristic algorithm, and puts some of top-ranked jobs to a pool. When a machine becomes available, it will directly choose a job from the pool as the next processing task. The job selection policy is represented by a novel attention-based network, and is trained by a powerful DRL algorithm. The aforementioned process is repeatedly executed in a simulation environment to collect the training data. Therefore, after being trained for a certain period of time, the policy will become smarter and can be applied to make right decisions in real-time. The proposed reactive scheduling method has been proved to be more efficient than dispatching rules and DRL-based approaches, and is effective in the production scheduling for a wide variety of discrete manufacturing scenarios, such as automobile and electronics industries. Moreover, the proposed method can be further extended to address dynamic scheduling problems with some production characteristics via adding constraints for the job selection or re-defining calculations for the completion time of operations accordingly.

Domain composition and attention network trained with synthesized unlabeled images for generalizable medical image segmentation

Despite that deep learning models have achieved remarkable performance in medical image segmentation, their performance is often limited on testing images from new centers with a domain shift. To achieve Domain Generalization (DG) for medical image segmentation, we propose a Domain Composition and Attention-based Network (DCA-Net) combined with structure- and style-based data augmentation that generates unlabeled synthetic images for training. First, DCA-Net represents features in one certain domain by a linear combination of a set of basis representations that are learned by parallel domain preceptors with a divergence constraint. The linear combination is used to calibrate the feature maps of an input image, which enables the model to generalize to unseen domains. Second, considering the number of domains and images for training is limited, we employ generative models to synthesize images with a higher structure diversity, and to leverage the unlabeled synthetic images, we introduce a consistency constraint for their predictions under style augmentation based on frequency amplitude mixture. Additionally, a Test-Time Frequency Augmentation (TTFA) is proposed to neutralize the domain shift from the target to source domains. Experimental results on two multi-domain datasets for fundus structure and nasopharyngeal carcinoma segmentation showed that: (1) our method significantly outperformed several existing DG methods, and (2) the model’s generalizability was largely improved by domain composition and attention modules; (3) by leveraging the unlabeled synthetic images and the TTFA, the model could better deal with images from unseen domains.

Road Anomaly Detection with Unknown Scenes Using DifferNet-Based Automatic Labeling Segmentation

Obstacle avoidance is essential for the effective operation of autonomous mobile robots, enabling them to detect and navigate around obstacles in their environment. While deep learning provides significant benefits for autonomous navigation, it typically requires large, accurately labeled datasets, making the data’s preparation and processing time-consuming and labor-intensive. To address this challenge, this study introduces a transfer learning (TL)-based automatic labeling segmentation (ALS) framework. This framework utilizes a pretrained attention-based network, DifferNet, to efficiently perform semantic segmentation tasks on new, unlabeled datasets. DifferNet leverages prior knowledge from the Cityscapes dataset to identify high-entropy areas as road obstacles by analyzing differences between the input and resynthesized images. The resulting road anomaly map was refined using depth information to produce a robust drivable area and map of road anomalies. Several off-the-shelf RGB-D semantic segmentation neural networks were trained using pseudo-labels generated by the ALS framework, with validation conducted on the GMRPD dataset. Experimental results demonstrated that the proposed ALS framework achieved mean precision, mean recall, and mean intersection over union (IoU) rates of 80.31%, 84.42%, and 71.99%, respectively. The ALS framework, through the use of transfer learning and the DifferNet network, offers an efficient solution for semantic segmentation of new, unlabeled datasets, underscoring its potential for improving obstacle avoidance in autonomous mobile robots.

Learning Dynamic Multimodal Network Slot Concepts from the Web for Forecasting Environmental, Social and Governance Ratings

Dynamic multimodal networks are networks with node attributes from different modalities where the attributes and network relationships evolve across time, i.e., both networks and multimodal attributes are dynamic; for example, dynamic relationship networks between companies that evolve across time due to changes in business strategies and alliances, which are associated with dynamic company attributes from multiple modalities such as textual online news, categorical events, and numerical financial-related data. Such information can be useful in predictive tasks involving companies. Environmental, social, and governance (ESG) ratings of companies are important for assessing the sustainability risks of companies. The process of generating ESG ratings by expert analysts is, however, laborious and time-intensive. We thus explore the use of dynamic multimodal networks extracted from the web for forecasting ESG ratings. Learning such dynamic multimodal networks from the web for forecasting ESG ratings is, however, challenging due to its heterogeneity and the low signal-to-noise ratios and non-stationary distributions of web information. Human analysts cope with such issues by learning concepts from past experience through relational thinking and scanning for such concepts when analyzing new information about a company. In this article, we propose the Dynamic Multimodal Slot Concept Attention-based Network (DynScan) model. DynScan utilizes slot attention mechanisms together with slot concept alignment and disentanglement loss functions to learn latent slot concepts from dynamic multimodal networks to improve performance on ESG rating forecasting tasks. DynScan is evaluated on forecasting tasks on six datasets, comprising three ESG ratings across two sets of companies. Our experiments show that DynScan outperforms other state-of-the-art models on these forecasting tasks. We also visualize the slot concepts learned by DynScan on five synthetic datasets and three real-world datasets and observe distinct and meaningful slot concepts being learned by DynScan across both synthetic and real-world datasets.

Effectiveness of Adaptive Attention-Based Network for Situation Awareness Recognition

Effectiveness of Adaptive Attention-Based Network for Situation Awareness Recognition