Attention Heatmap Research Articles

Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability

ObjectiveThis study aims to develop an enhanced Transformer model for predicting mild cognitive impairment (MCI) using data from the China Health and Retirement Longitudinal Study (CHARLS), focusing on handling mixed data types and improving predictive accuracy.MethodsThe Transformer integrates categorical (integer-encoded) and continuous (floating-point) data, using multi-head attention with four heads to capture complex relationships. Preprocessing involved separate embedding layers for categorical data and feed-forward networks for continuous data. The model was compared with SVM and XGBoost, trained for 150 epochs with RMSProp and a cosine annealing scheduler. Key metrics included accuracy, Mean Absolute Error (MAE) tolerance, and training loss. An attention heatmap was generated to visualize feature importance.ResultsThe Transformer outperformed SVM and XGBoost, achieving over 90% accuracy at an MAE tolerance of 3.5. The model showed rapid convergence, with loss stabilizing within 20 epochs. The attention heatmap highlighted key features, confirming the effectiveness of the multi-head attention mechanism in identifying relevant variables.ConclusionThe enhanced Transformer model offers superior accuracy and efficiency in predicting cognitive decline compared to traditional models. Its capacity to process both continuous and categorical data and its interpretability through attention mechanisms make it a promising tool for early detection of neurodegenerative diseases, potentially improving clinical decision-making and interventions.

Diagnostic classification in toxicologic pathology using attention-guided weak supervision and whole slide image features: a pilot study in rat livers.

The diagnostic classification of digitized tissue images based on histopathologic lesions present in whole slide images (WSI) is a significant task that eludes modern image classification techniques. Even with advanced methods designed for digital histopathology, the domain of toxicologic pathology presents challenges in that histopathologic features may be at times complex, subtle, and/or rare. We propose an innovative weakly supervised learning method that leverages minimal annotations, a state-of-the-art self-supervised vision transformer for embedding extraction, and a novel guided attention mechanism that is better suited for heavily imbalanced datasets typical in toxicologic pathology. Our model demonstrates improvements in diagnostic classification and attention heatmap quality over the previously described clustering-constrained-attention multiple-instance learning method on several lesion classes in rat livers (38% improvement in AUC). We also demonstrate how an ensemble of binary classifiers improves interpretability and allows for multiclass classification and the classification of diagnostic regions of interest in each slide. The improved classification performance and higher contrast heatmaps better support toxicologic pathologists' histopathology analysis and will enable more efficient workflows as they are further refined and integrated into routine use.

ItpCtrl-AI: End-to-end interpretable and controllable artificial intelligence by modeling radiologists' intentions.

Using Deep Learning in computer-aided diagnosis systems has been of great interest due to its impressive performance in the general domain and medical domain. However, a notable challenge is the lack of explainability of many advanced models, which poses risks in critical applications such as diagnosing findings in CXR. To address this problem, we propose ItpCtrl-AI, a novel end-to-end interpretable and controllable framework that mirrors the decision-making process of the radiologist. By emulating the eye gaze patterns of radiologists, our framework initially determines the focal areas and assesses the significance of each pixel within those regions. As a result, the model generates an attention heatmap representing radiologists' attention, which is then used to extract attended visual information to diagnose the findings. By allowing the directional input, our framework is controllable by the user. Furthermore, by displaying the eye gaze heatmap which guides the diagnostic conclusion, the underlying rationale behind the model's decision is revealed, thereby making it interpretable. In addition to developing an interpretable and controllable framework, our work includes the creation of a dataset, named Diagnosed-Gaze++, which aligns medical findings with eye gaze data. Our extensive experimentation validates the effectiveness of our approach in generating accurate attention heatmaps and diagnoses. The experimental results show that our model not only accurately identifies medical findings but also precisely produces the eye gaze attention of radiologists. The dataset, models, and source code will be made publicly available upon acceptance.

Weakly supervised pathological differentiation of primary central nervous system lymphoma and glioblastoma on multi-site whole slide images.

Differentiating primary central nervous system lymphoma (PCNSL) and glioblastoma (GBM) is crucial because their prognosis and treatment differ substantially. Manual examination of their histological characteristics is considered the golden standard in clinical diagnosis. However, this process is tedious and time-consuming and might lead to misdiagnosis caused by morphological similarity between their histology and tumor heterogeneity. Existing research focuses on radiological differentiation, which mostly uses multi-parametric magnetic resonance imaging. By contrast, we investigate the pathological differentiation between the two types of tumors using whole slide images (WSIs) of postoperative formalin-fixed paraffin-embedded samples. To learn the specific and intrinsic histological feature representations from the WSI patches, a self-supervised feature extractor is trained. Then, the patch representations are fused by feeding into a weakly supervised multiple-instance learning model for the WSI classification. We validate our approach on 134 PCNSL and 526 GBM cases collected from three hospitals. We also investigate the effect of feature extraction on the final prediction by comparing the performance of applying the feature extractors trained on the PCNSL/GBM slides from specific institutions, multi-site PCNSL/GBM slides, and large-scale histopathological images. Different feature extractors perform comparably with the overall area under the receiver operating characteristic curve value exceeding 85% for each dataset and close to 95% for the combined multi-site dataset. Using the institution-specific feature extractors generally obtains the best overall prediction with both of the PCNSL and GBM classification accuracies reaching 80% for each dataset. The excellent classification performance suggests that our approach can be used as an assistant tool to reduce the pathologists' workload by providing an accurate and objective second diagnosis. Moreover, the discriminant regions indicated by the generated attention heatmap improve the model interpretability and provide additional diagnostic information.

Patch-based feature mapping with generative adversarial networks for auxiliary hip fracture detection.

Hip fractures are a significant public health issue, particularly among the elderly population. Pelvic radiographs (PXRs) play a crucial role in diagnosing hip fractures and are commonly used for their evaluation. Previous research has demonstrated promising performance in classification models for hip fracture detection. However, these models sometimes focus on the images' non-fracture regions, reducing their explainability. This study applies weakly supervised learning techniques to address this issue and improve the model's focus on the fracture region. Additionally, we introduce a method to quantitatively evaluate the model's focus on the region of interest (ROI). We propose a new auxiliary module called the patch-auxiliary generative adversarial network (PAGAN) for weakly supervised learning tasks. PAGAN can be integrated with any state-of-the-art (SOTA) classification model, such as EfficientNetB0, ResNet50, and DenseNet121, to enhance hip fracture detection. This training strategy incorporates global information (the entire PXR image) and local information (the hip region patch) for more effective learning. Furthermore, we employ GradCAM to generate attention heatmaps, highlighting the focus areas within the classification model. The intersection over union (IOU) and dice coefficient (Dise) are then computed between the attention heatmap and the fracture area, enabling a quantitative assessment of the model's explainability. Incorporating PAGAN improved the performance of the classification models. The accuracy of EfficientNetB0 increased from 93.61% to 95.97%, ResNet50 improved from 90.66% to 94.89%, and DenseNet121 saw an increase from 93.51% to 94.49%. Regarding model explainability, the integration of PAGAN into classification models led to a more pronounced attention to ROI. The average IOU improved from 0.32 to 0.54 for EfficientNetB0, from 0.28 to 0.40 for ResNet50, and from 0.37 to 0.51 for DenseNet121. These results indicate that PAGAN improves hip fracture classification performance and substantially enhances the model's focus on the fracture region, thereby increasing its explainability.

Transformers in Cybersecurity: Advancing Threat Detection and Response through Machine Learning Architectures

The increasing sophistication of cyber threats has outpaced the capabilities of traditional detection and response systems, necessitating the adoption of advanced machine learning architectures. This study investigates the application of Transformer-based models in cybersecurity, focusing on their ability to enhance threat detection and response. Leveraging publicly available datasets, including CICIDS 2017 and UNSW-NB15, the research employs a systematic methodology encompassing data preprocessing, model optimization, and comparative performance evaluation. The Transformer model, tailored for cybersecurity, integrates self-attention mechanisms and positional encoding to capture complex dependencies in network traffic data. The experimental results reveal that the proposed model achieves an accuracy of 97.8%, outperforming conventional methods such as Random Forest (92.3%) and deep learning approaches like CNN (94.1%) and LSTM (95.6%). Additionally, the Transformer demonstrates high detection rates across diverse attack types, with rates exceeding 98% for Denial of Service and Brute Force attacks. Attention heatmaps provide valuable insights into feature importance, enhancing the interpretability of the model’s decisions. Scalability tests confirm the model’s ability to handle large datasets efficiently, positioning it as a robust solution for dynamic cybersecurity environments. This research contributes to the field by demonstrating the feasibility and advantages of employing Transformer architectures for complex threat detection tasks. The findings have significant implications for developing scalable, interpretable, and adaptive cybersecurity systems. Future studies should explore lightweight Transformer variants and evaluate the model in operational environments to address practical deployment challenges.

PATH-39. AI-BASED IDENTIFICATION OF GLIOMA IDH MUTATIONAL STATUS FROM H&E-STAINED WHOLE SLIDE IMAGES

Abstract INTRODUCTION Isocitrate-dehydrogenase (IDH) mutational status is diagnostically critical in adult gliomas, with prognostic and therapeutic implications. IDH status cannot be determined by sole histologic assessment and requires molecular testing. We hypothesize that AI-based analysis can accurately predict IDH mutational status in glioma, by retrieving sub-visual cues in H&E-stained digitized tissue sections. MATERIALS AND METHODS A retrospective discovery cohort of 1,534 cases (756/778 IDH-wildtype/IDH-mutant, including all available grades) from the TCGA-LGG/TCGA-GBM collections was used for model development with 10-fold cross-validation (CV). A replication cohort of 114 cases (82/32 IDH-wildtype/IDH-mutant) from the University of Pennsylvania Health System (UPHS) was used for independent hold-out evaluation. Each 20X magnification whole slide image (WSI) underwent comprehensive curation for elimination of artifactual content, followed by partitioning into 256x256 patches, and feature extraction using pre-trained i) ImageNet weights (baseline), and ii) self-supervised vision transformer (ViT). A weakly-supervised attention-based multiple-instance-learning framework distinguished cases as IDH-wildtype or IDH-mutant, while generating visually interpretable attention heatmaps. RESULTS Evaluation on discovery and replication cohorts with the ViT yield accuracy of 90.84% (AUCDiscovery=97.16%) and 93.94% (AUCReplication=97.36%), respectively, with high sensitivity (i.e., high confidence in correctly predicting IDH-wildtype glioma). Comparison of the ViT with the baseline model on the replication cohort indicates the ViT superiority by improvements of 8% and 9.5% in accuracy and AUC, respectively. Histologic assessment of heatmaps indicates IDH-wildtype tumors exhibiting distinct regions of significant pleomorphism and microvascular proliferation, while IDH-mutant tumors exhibit dense nodular cell concentrations, microcystic architecture, and gemistocytic cells. CONCLUSION Our accurate H&E-based computational determination of glioma IDH status, with interpretations aligned with human-identifiable features, can contribute to obviating the need for molecular analyses and enable expedited diagnosis even in community healthcare settings. This study robustly identifies morpho-IDH correlates and could enable expedited patient prognostic stratification.

Attention Heat Map-Based Black-Box Local Adversarial Attack for Synthetic Aperture Radar Target Recognition

Synthetic aperture radar (SAR) automatic target recognition (ATR) models based on deep neural networks (DNNs) are susceptible to adversarial attacks. In this study, we proposed an SAR black-box local adversarial attack algorithm named attention heat map- based black-box local adversarial attack (AH-BLAA). First, we designed an attention heat map extraction module combined with the layer-wise relevance propagation (LRP) algorithm to obtain the high concerning areas of the SAR-ATR models. Then, to gener- ate SAR adversarial attack examples, we designed a perturbation generator module, introducing the structural dissimilarity (DSSIM) metric in the loss function to limit image distortion and the dif- ferential evolution (DE) algorithm to search for optimal perturba- tions. Experimental results on the MSTAR and FUSAR-Ship datasets showed that compared with existing adversarial attack algorithms, the attack success rate of the AH-BLAA algorithm increased by 0.63% to 33.59% and 1.05% to 17.65%, respectively. Moreover, the low- est perturbation ratios reached 0.23% and 0.13%, respectively.

Heatmaps: A Tool for Studying Book Selection Preferences on Discovery Platforms

ABSTRACTThis study evaluates the usefulness of heatmaps in understanding user preferences during book selection in online discovery platforms. It compares two heatmaps generated from user clicks and eye‐tracking data to identify key metadata influencing readers' choices. Initial results indicate that click heatmaps can be a useful tool for this type of investigation, especially considering the cost of eye‐tracking, which is typically used to generate attention heatmaps.

Enhancing early Parkinson’s disease detection through multimodal deep learning and explainable AI: insights from the PPMI database

Parkinson’s is the second most common neurodegenerative disease, affecting nearly 8.5M people and steadily increasing. In this research, Multimodal Deep Learning is investigated for the Prodromal stage detection of Parkinson's Disease (PD), combining different 3D architectures with the novel Excitation Network (EN) and supported by Explainable Artificial Intelligence (XAI) techniques. Utilizing data from the Parkinson’s Progression Markers Initiative, this study introduces a joint co-learning approach for multimodal fusion, enabling end-to-end training of deep neural networks and facilitating the learning of complementary information from both imaging and clinical modalities. DenseNet with EN outperformed other models, showing a substantial increase in accuracy when supplemented with clinical data. XAI methods, such as Integrated Gradients for ResNet and DenseNet, and Attention Heatmaps for Vision Transformer (ViT), revealed that DenseNet focused on brain regions believed to be critical to prodromal pathophysiology, including the right temporal and left pre-frontal areas. Similarly, ViT highlighted the lateral ventricles associated with cognitive decline, indicating their potential in the Prodromal stage. These findings underscore the potential of these regions as early-stage PD biomarkers and showcase the proposed framework's efficacy in predicting subtypes of PD and aiding in early diagnosis, paving the way for innovative diagnostic tools and precision medicine.

Deep Learning–Based Classification of Early-Stage Mycosis Fungoides and Benign Inflammatory Dermatoses on H&E-Stained Whole-Slide Images: A Retrospective, Proof-of-Concept Study

The diagnosis of early-stage mycosis fungoides (MF) is challenging due to shared clinical and histopathological features with benign inflammatory dermatoses (BIDs). Recent evidence has shown that deep learning (DL) can assist pathologists in cancer classification, but this field is largely unexplored for cutaneous lymphomas. This study evaluates DL in distinguishing early-stage MF from BIDs using a unique dataset of 924 hematoxylin and eosin-stained whole-slide images from skin biopsies, including 233 early-stage MF and 353 BID patients. All MF patients were diagnosed after clinicopathological correlation. The classification accuracy of weakly-supervised DL models was benchmarked against three expert pathologists. The highest performance on a temporal test set was at 200x magnification (0.25 μm per pixel resolution), with a mean area-under-the-curve of 0.827 ± 0.044 and a mean balanced accuracy of 76.2 ± 3.9%. This nearly matched the 77.7% mean balanced accuracy of the three expert-pathologists. Most (63.5%) attention heatmaps corresponded well with the pathologists’ region-of-interest. Considering the difficulty of the MF versus BID classification task, the results of this study show promise for future applications of weakly-supervised DL in diagnosing early-stage MF. Achieving clinical-grade performance will require larger multi-institutional datasets and improved methodologies, such as multimodal DL with incorporation of clinical data.

Color Fundus Photography and Deep Learning Applications in Alzheimer Disease

ObjectiveTo report the development and performance of 2 distinct deep learning models trained exclusively on retinal color fundus photographs to classify Alzheimer disease (AD). Patients and MethodsTwo independent datasets (UK Biobank and our tertiary academic institution) of good-quality retinal photographs derived from patients with AD and controls were used to build 2 deep learning models, between April 1, 2021, and January 30, 2024. ADVAS is a U-Net–based architecture that uses retinal vessel segmentation. ADRET is a bidirectional encoder representations from transformers style self-supervised learning convolutional neural network pretrained on a large data set of retinal color photographs from UK Biobank. The models’ performance to distinguish AD from non-AD was determined using mean accuracy, sensitivity, specificity, and receiving operating curves. The generated attention heatmaps were analyzed for distinctive features. ResultsThe self-supervised ADRET model had superior accuracy when compared with ADVAS, in both UK Biobank (98.27% vs 77.20%; P<.001) and our institutional testing data sets (98.90% vs 94.17%; P=.04). No major differences were noted between the original and binary vessel segmentation and between both eyes vs single-eye models. Attention heatmaps obtained from patients with AD highlighted regions surrounding small vascular branches as areas of highest relevance to the model decision making. ConclusionA bidirectional encoder representations from transformers style self-supervised convolutional neural network pretrained on a large data set of retinal color photographs alone can screen symptomatic AD with high accuracy, better than U-Net–pretrained models. To be translated in clinical practice, this methodology requires further validation in larger and diverse populations and integrated techniques to harmonize fundus photographs and attenuate the imaging-associated noise.

Predicting tumor mutation burden and VHL mutation from renal cancer pathology slides with self-supervised deep learning.

Tumor mutation burden (TMB) and VHL mutation play a crucial role in the management of patients with clear cell renal cell carcinoma (ccRCC), such as guiding adjuvant chemotherapy and improving clinical outcomes. However, the time-consuming and expensive high-throughput sequencing methods severely limit their clinical applicability. Predicting intratumoral heterogeneity poses significant challenges in biology and clinical settings. Our aimed to develop a self-supervised attention-based multiple instance learning (SSL-ABMIL) model to predict TMB and VHL mutation status from hematoxylin and eosin-stained histopathological images. We obtained whole slide images (WSIs) and somatic mutation data of 350 ccRCC patients from The Cancer Genome Atlas for developing SSL-ABMIL model. In parallel, 163 ccRCC patients from Clinical Proteomic Tumor Analysis Consortium cohort was used as independent external validation set. We systematically compared three different models (Wang-ABMIL, Ciga-ABMIL, and ImageNet-MIL) for their ability to predict TMB and VHL alterations. We first identified two groups of populations with high- and low-TMB (cut-off point = 0.9). In two independent cohorts, the Wang-ABMIL model achieved the highest performance with decent generalization performance (AUROC = 0.83 ± 0.02 and 0.8 ± 0.04 in predicting TMB and VHL, respectively). Attention heatmaps revealed that the Wang-ABMIL model paid the highest attention to tumor regions in high-TMB patients, while in VHL mutation prediction, non-tumor regions were also assigned high attention, particularly the stromal regions infiltrated by lymphocytes. Our results indicated that SSL-ABMIL can effectively extract histological features for predicting TMB and VHL mutation, demonstrating promising results in linking tumor morphology and molecular biology.

Alzheimer’s disease detection and stage identification from magnetic resonance brain images using vision transformer

Machine learning techniques applied in neuroimaging have prompted researchers to build models for early diagnosis of brain illnesses such as Alzheimer’s disease (AD). Although this task is difficult, advanced deep-learning (DL) approaches can be used. These DL models are effective, but difficult to interpret, time-consuming, and resource-intensive. Therefore, neuroscientists are interested in employing novel, less complex structures such as transformers that have superior pattern-extraction capabilities. In this study, an automated framework for accurate AD diagnosis and precise stage identification was developed by employing vision transformers (ViTs) with fewer computational resources. ViT, which captures the global context as opposed to convolutional neural networks (CNNs) with local receptive fields, is more efficient for brain image processing than CNN because the brain is a highly complex network with connected parts. The self-attention mechanism in the ViT helps to achieve this goal. Magnetic resonance brain images belonging to four stages were utilized to develop the proposed model, which achieved 99.83% detection accuracy, 99.69% sensitivity, 99.88% specificity, and 0.17% misclassification rate. Moreover, to prove the ability of the model to generalize, the mean distances of the transformer blocks and attention heat maps were visualized to understand what the model learned from the MRI input image.

Deep Learning-Based Eye-Tracking Analysis for Diagnosis of Alzheimer's Disease Using 3D Comprehensive Visual Stimuli.

Alzheimer's Disease (AD) is a neurodegenerative disorder that causes a continuous decline in cognitive functions and eventually results in death. An early AD diagnosis is important for taking active measures to slow its deterioration. Traditional diagnoses are usually based on clinical experience, which is limited by several realistic factors. In this paper, we focus on exploiting deep learning techniques to diagnose AD based on eye-tracking behaviors. Visual attention, as a typical eye-tracking behavior, is of great clinical value in detecting cognitive abnormalities in AD patients. To better analyze the differences in visual attention between AD patients and normals, we first conducted a 3D comprehensive visual task on a noninvasive eye-tracking system to collect visual attention heatmaps. Then a multilayered comparison convolutional neural network (MC-CNN) is proposed to distinguish the visual attention differences between AD patients and normals. In MC-CNN, the multilayered feature representations of heatmaps were obtained by hierarchical residual blocks to better encode eye movement behaviors, which were further integrated into a distance vector to benefit the comprehensive visual task. From evaluation, MC-CNN can distinguish AD patients from normals with 0.84 accuracy, 0.86 recall, 0.82 precision, 0.83 F1-score and 0.90 area under the curve (AUC). The above results demonstrate the effectiveness of the proposed MC-CNN in AD diagnosis based on the comprehensive 3D visual task.

Lung Cancer Diagnosis on Virtual Histologically Stained Tissue Using Weakly Supervised Learning

Lung adenocarcinoma (LUAD) is the most common primary lung cancer and accounts for 40% of all lung cancer cases. The current gold standard for lung cancer analysis is based on the pathologists’ interpretation of hematoxylin and eosin (H&E)-stained tissue slices viewed under a brightfield microscope or a digital slide scanner. Computational pathology using deep learning has been proposed to detect lung cancer on histology images. However, the histological staining workflow to acquire the H&E-stained images and the subsequent cancer diagnosis procedures are labor-intensive and time-consuming with tedious sample preparation steps and repetitive manual interpretation, respectively. In this work, we propose a weakly supervised learning method for LUAD classification on label-free tissue slices with virtual histological staining. The autofluorescence images of label-free tissue with histopathological information can be converted into virtual H&E-stained images by a weakly supervised deep generative model. For the downstream LUAD classification task, we trained the attention-based multiple-instance learning model with different settings on the open-source LUAD H&E-stained whole-slide images (WSIs) dataset from the Cancer Genome Atlas (TCGA). The model was validated on the 150 H&E-stained WSIs collected from patients in Queen Mary Hospital and Prince of Wales Hospital with an average area under the curve (AUC) of 0.961. The model also achieved an average AUC of 0.973 on 58 virtual H&E-stained WSIs, comparable to the results on 58 standard H&E-stained WSIs with an average AUC of 0.977. The attention heatmaps of virtual H&E-stained WSIs and ground-truth H&E-stained WSIs can indicate tumor regions of LUAD tissue slices. In conclusion, the proposed diagnostic workflow on virtual H&E-stained WSIs of label-free tissue is a rapid, cost effective, and interpretable approach to assist clinicians in postoperative pathological examinations. The method could serve as a blueprint for other label-free imaging modalities and disease contexts.

Abstract 2584: Decoding colon cancer recurrence: Unveiling accurate predictions with attention-guided deep neural networks on histopathological whole slide images

Abstract Background: Up to 40% of colon cancer patients are at high risk of cancer recurrence, yet accurate and timely prediction tools are lacking. Leveraging whole slide images (WSIs) and deep learning models, we aimed to develop precise algorithm for prediction of colon cancer recurrence. Thus, enables risk stratification for optimized therapeutic interventions and improved health outcomes. Design: We designed an attention-based deep learning model for predicting colon cancer recurrence on paraffin-embedded, hematoxylin and eosin-stained colon tissue biopsies of digital slides. The WSIs were downloaded from the TCGA dataset, and preprocessed for color normalization, tissue segmentation, tilling, and histopathological features extraction. The entire dataset was then labeled based on cancer recurrence status, and divided into training (70%), validation (15%), and testing sets (15%). The model's performance was then evaluated by standard evaluation metrics, including receiver operating characteristic Area Under the Curve (AUC) and accuracy, on the training, validation, and testing datasets, where interpretability attention heatmaps were applied to gain insights into specific histological features and patterns involved in the model's decision-making process. The study is supported by the NIH-NCI-T32 for Next Generation Pathologists Program at our institution. Results: A total of 350 WSIs were included and labeled into two classes: post-therapeutic colon cancer recurrence and no recurrence. The tissue segmentation process involved converting RGB images to HSV color space, applying a median blur filter (kernel size: 7), and performing thresholding using Otsu's method. The extracted features were utilized to train and construct a Clustering-constrained Attention Multiple Instance Learning (CLAM) model. The model demonstrated consistent performance across the validation and testing datasets, achieving an AUC of 0.85 with an accuracy of 0.83 on the validation set, and an identical AUC of 0.85 with an accuracy of 0.83 on the testing set, indicating the model's robust ability to identify patients at risk of cancer recurrence. Conclusion: The study demonstrates the strong performance of our deep learning model in accurately identifying patients at risk for colon cancer recurrence based on H&amp;E WSIs. This capability paves the way for optimizing therapeutic interventions and implementing effective surveillance strategies. Consequently, highlighting the crucial role of pathologists in collaborating with the oncologist for optimizing the management of colon cancer care in the era of personalized medicine. Citation Format: Mohammad K. Alexanderani, Mohamed Omar, Matthew Greenblatt, Ethel Cesarman, Maria T. Diaz-Meco, Jorge Moscat, Luigi Marchionni. Decoding colon cancer recurrence: Unveiling accurate predictions with attention-guided deep neural networks on histopathological whole slide images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2584.

Abstract 3521: Robust inference of PTEN deletion in prostate cancer from routine histopathology whole slide images using attention-based deep learning

Abstract The diagnostic workup of prostate cancer (PCa) requires a thorough manual histopathological inspection of prostate tissue specimens stained with hematoxylin and eosin (H&amp;E). While immunohistochemistry and advanced molecular tests like FISH could uncover key genetic alterations, they are not cost-effective and often require special training, necessitating the development of more effective tools that can accurately infer the status of such alterations directly from the tissue morphology depicted in routine H&amp;E-stained glass slides. PTEN deletion, which is present in 15-20% of PCa, is significantly associated with tumor aggressiveness and negative prognostic impact. Deep learning (DL) models can be trained to predict the presence of PTEN deletion on routine whole slide images (WSIs) and select patients that could benefit from further treatment. In this study, we trained and tested the performance of a weakly supervised, multiple-instance learning network on a cohort of 449 WSIs radical prostatectomy specimens stained with hematoxylin &amp; eosin. Our model demonstrated robust performance in predicting PTEN loss based on tumor morphology alone. Through the model, we also generated attention heatmaps of the most informative tiles of each slide, allowing us to quantify the different cellular components, thus inferring the morphological features associated with PTEN loss. This study shows that DL is a potential tool that can assist the pathologist in accurately inferring the molecular status of key genetic alterations using the tissue morphology depicted in histopathology slides. Moreover, the morphological information that can be obtained by attention-based models can be leveraged to close the gap between the pathologist’s morphological analysis and the interpretability of artificial intelligence models. Citation Format: Federico Scarfò, Mohamed Omar, Luigi Marchionni. Robust inference of PTEN deletion in prostate cancer from routine histopathology whole slide images using attention-based deep learning [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3521.

A multi-modal vision-language pipeline strategy for contour quality assurance and adaptive optimization

Objective. Accurate delineation of organs-at-risk (OARs) is a critical step in radiotherapy. The deep learning generated segmentations usually need to be reviewed and corrected by oncologists manually, which is time-consuming and operator-dependent. Therefore, an automated quality assurance (QA) and adaptive optimization correction strategy was proposed to identify and optimize ‘incorrect’ auto-segmentations. Approach. A total of 586 CT images and labels from nine institutions were used. The OARs included the brainstem, parotid, and mandible. The deep learning generated contours were compared with the manual ground truth delineations. In this study, we proposed a novel contour quality assurance and adaptive optimization (CQA-AO) strategy, which consists of the following three main components: (1) the contour QA module classified the deep learning generated contours as either accepted or unaccepted; (2) the unacceptable contour categories analysis module provided the potential error reasons (five unacceptable category) and locations (attention heatmaps); (3) the adaptive correction of unacceptable contours module integrate vision-language representations and utilize convex optimization algorithms to achieve adaptive correction of ‘incorrect’ contours. Main results. In the contour QA tasks, the sensitivity (accuracy, precision) of CQA-AO strategy reached 0.940 (0.945, 0.948), 0.962 (0.937, 0.913), and 0.967 (0.962, 0.957) for brainstem, parotid and mandible, respectively. The unacceptable contour category analysis, the (FI,AccI,Fmicro,Fmacro) of CQA-AO strategy reached (0.901, 0.763, 0.862, 0.822), (0.855, 0.737, 0.837, 0.784), and (0.907, 0.762, 0.858, 0.821) for brainstem, parotid and mandible, respectively. After adaptive optimization correction, the DSC values of brainstem, parotid and mandible have been improved by 9.4%, 25.9%, and 13.5%, and Hausdorff distance values decreased by 62%, 70.6%, and 81.6%, respectively. Significance. The proposed CQA-AO strategy, which combines QA of contour and adaptive optimization correction for OARs contouring, demonstrated superior performance compare to conventional methods. This method can be implemented in the clinical contouring procedures and improve the efficiency of delineating and reviewing workflow.

Development of a deep learning model to distinguish the cause of optic disc atrophy using retinal fundus photography

The differential diagnosis for optic atrophy can be challenging and requires expensive, time-consuming ancillary testing to determine the cause. While Leber's hereditary optic neuropathy (LHON) and optic neuritis (ON) are both clinically significant causes for optic atrophy, both relatively rare in the general population, contributing to limitations in obtaining large imaging datasets. This study therefore aims to develop a deep learning (DL) model based on small datasets that could distinguish the cause of optic disc atrophy using only fundus photography. We retrospectively reviewed fundus photographs of 120 normal eyes, 30 eyes (15 patients) with genetically-confirmed LHON, and 30 eyes (26 patients) with ON. Images were split into a training dataset and a test dataset and used for model training with ResNet-18. To visualize the critical regions in retinal photographs that are highly associated with disease prediction, Gradient-Weighted Class Activation Map (Grad-CAM) was used to generate image-level attention heat maps and to enhance the interpretability of the DL system. In the 3-class classification of normal, LHON, and ON, the area under the receiver operating characteristic curve (AUROC) was 1.0 for normal, 0.988 for LHON, and 0.990 for ON, clearly differentiating each class from the others with an overall total accuracy of 0.93. Specifically, when distinguishing between normal and disease cases, the precision, recall, and F1 scores were perfect at 1.0. Furthermore, in the differentiation of LHON from other conditions, ON from others, and between LHON and ON, we consistently observed precision, recall, and F1 scores of 0.8. The model performance was maintained until only 10% of the pixel values of the image, identified as important by Grad-CAM, were preserved and the rest were masked, followed by retraining and evaluation.