AI-derived prognostic biomarkers from melanoma whole slide image segmentation: an initial discovery and assessment.

The current melanoma staging system is predictive of 74% of the variance in survival, with prognostic biomarkers subject to high levels of inter-observer variation. The application of convolutional neural networks (CNNs) to whole slide images (WSIs), may reveal new insights into tumor morphology and therefore patient prognosis. Melanoma morphology appears to be of greater significance than in other solid tumors, with Breslow thickness remaining the strongest prognostic indicator. Other biomarkers based on tumor morphology have been generated and although some have been found to be prognostically superior to Breslow thickness, none have been integrated into clinical workflows. This may in part be explained by their demands on pathologist time. Therefore, this work outlines the development and evaluation of a CNN for invasive cutaneous melanoma detection in WSIs, which may be used for prognostic biomarker generation. 1,157 WSIs containing cutaneous melanoma from five datasets (three from the University of Leeds, one from the Melanoma Institute Australia, as well as The Cancer Genome Atlas) have been used in the initial development and evaluation of the CNN. A custom-designed 2-class tumor segmentation network with a fully convolutional architecture was trained using annotations. The CNN was evaluated using various methodologies, including comparison at per-pixel and per-tumor levels as compared to manual annotation, as well as variation across 3 scanning platforms (Leica Aperio AT2 (Milton Keynes, UK), Roche Ventana DP600 (Arizona, US) and Hamamatsu NanoZoomer S360 (Hamamatsu City, Japan). The CNN detected and located invasive melanoma tissue of no specific type with an average per-pixel sensitivity and specificity of 97.6% and 99.9% respectively across the 5 test sets. There were no statistical differences between tumor dimensions generated by the CNN as compared to manual annotation. Similarly, there were no statistically significant differences between CNN generated tumor dimensions across three scanning platforms. We have developed and performed initial evaluation of a CNN which appears to accurately detect invasive cutaneous melanoma of no specific type in WSIs for objective evaluation of tumor morphology. Future work should interrogate these data further for its propensity to predict survival outcomes. Citation Format: Emily L. Clarke, Derek Magee, Julia Newton-Bishop, William Merchant, Marlous Hall, Robert Insall, Nigel Maher, Richard Scolyer, Grace Farnworth, Anisah Ali, Sally O'Shea, Darren Treanor. The development and evaluation of a convolutional neural network for melanoma detection in whole slide images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6173.

The current melanoma staging system is predictive of 74% of the variance in survival, with prognostic biomarkers subject to high levels of interobserver and intraobserver variation. Melanoma morphology appears to be of greater significance than in other solid tumours, with Breslow thickness remaining the strongest prognostic indicator. The application of convolutional neural networks (CNNs) to whole-slide images (WSIs) may reveal new insights into tumour morphology and therefore patient prognosis. This work outlines the development and evaluation of a CNN for invasive cutaneous melanoma detection in WSIs, to enable the creation of objective prognostic biomarkers based on the tissue morphology. In total, 1157 WSIs containing cutaneous melanoma from five sources have been used in the initial development and evaluation of a custom-designed two-class tumour segmentation network with a fully convolutional architecture. The CNN detected and located invasive melanoma tissue of no specific type with an average per-pixel sensitivity and specificity of 97.6% and 99.9%, respectively across, the five test sets (three external). There were no statistical differences between tumour dimensions generated by the CNN compared with manual annotation. Similarly, there were no statistically significant differences between CNN-generated tumour dimensions across three scanning platforms. Furthermore, we have identified that this CNN can be used to calculate the ‘digital Breslow thickness’, which is a strong independent prognostic predictor of overall survival and melanoma-specific survival, across three test sets with follow-up data (hazard ratio 1.26, 95% confidence interval 1.19–1.34, P < 0.001). We have also shown that the ‘nodularity index’ determined by the tumour shape, independently of size, is predictive of survival; the rounder the tumour the worse the outcome (hazard ratio 0.71, 95% confidence interval 0.60–0.83, P < 0.001). We have developed and performed initial evaluation of a CNN that accurately detects invasive cutaneous melanoma in WSIs, enabling an objective evaluation of tumour morphology. This CNN has afforded the development of the first objective biomarkers based on the tumour’s architectural morphology. The utility of these biomarkers will be further evaluated on WSIs from additional institutions – this work is currently underway.

Semantic segmentation of whole slide images (WSIs) helps pathologists identify lesions and cancerous nests. However, training fully supervised segmentation networks usually requires plenty of pixel-level annotations, which consume lots of time and human efforts. Coming from tissues of different patients with large amounts of pixels, WSIs exhibit various patterns, resulting in intra-class heterogeneity and inter-class homogeneity. Meanwhile, most existing methods for WSIs focus on extracting a certain type of features, neglecting the relations between different features and their joint effect on segmentation. Therefore, we propose a novel weakly supervised network based on tensor graphs (WSNTG) for WSI segmentation. Using only sparse point annotations, it efficiently segments WSIs by superpixel-wise classification and credible node reweighting. To deal with the variability of WSIs, the proposed network represents multiple hand-crafted features and hierarchical features yielded by a pretrained Convolutional Neural Network (CNN). Particularly, it learns over the semi-labeled tensor graphs constructed on the hierarchical features to exploit nonlinear data structures and associations. It gains robustness via the tensor-graph Laplacian of the hand-crafted features superimposed on the segmentation loss. We evaluated WSNTG on two WSI datasets, DigestPath2019 and SICAPV2. Results show that it outperforms many fully supervised and weakly supervised methods with minimal point annotations in WSI segmentation. The codes are published at https://github.com/zqh369/WSNTG.KeywordsWeakly-supervised segmentationPathology image segmentationGraph convolutional networksNode reweighting

Colorectal cancer (CRC) is a prevalent gastrointestinal tumour with high incidence and mortality rates. Early screening for CRC can improve cure rates and reduce mortality. Recently, deep convolution neural network (CNN)-based pathological image diagnosis has been intensively studied to meet the challenge of time-consuming and labour-intense manual analysis of high-resolution whole slide images (WSIs). Despite the achievements made, deep CNN-based methods still suffer from some limitations, and the fundamental problem is that they cannot capture global features. To address this issue, we propose a hybrid deep learning framework (RGSB-UNet) for automatic tumour segmentation in WSIs. The framework adopts a UNet architecture that consists of the newly-designed residual ghost block with switchable normalization (RGS) and the bottleneck transformer (BoT) for downsampling to extract refined features, and the transposed convolution and 1 × 1 convolution with ReLU for upsampling to restore the feature map resolution to that of the original image. The proposed framework combines the advantages of the spatial-local correlation of CNNs and the long-distance feature dependencies of BoT, ensuring its capacity of extracting more refined features and robustness to varying batch sizes. Additionally, we consider a class-wise dice loss (CDL) function to train the segmentation network. The proposed network achieves state-of-the-art segmentation performance under small batch sizes. Experimental results on DigestPath2019 and GlaS datasets demonstrate that our proposed model produces superior evaluation scores and state-of-the-art segmentation results.

BackgroundImage-based machine learning tools hold great promise for clinical applications in pathology research. However, the ideal end-users of these computational tools (e.g., pathologists and biological scientists) often lack the programming experience required for the setup and use of these tools which often rely on the use of command line interfaces.MethodsWe have developed Histo-Cloud, a tool for segmentation of whole slide images (WSIs) that has an easy-to-use graphical user interface. This tool runs a state-of-the-art convolutional neural network (CNN) for segmentation of WSIs in the cloud and allows the extraction of features from segmented regions for further analysis.ResultsBy segmenting glomeruli, interstitial fibrosis and tubular atrophy, and vascular structures from renal and non-renal WSIs, we demonstrate the scalability, best practices for transfer learning, and effects of dataset variability. Finally, we demonstrate an application for animal model research, analyzing glomerular features in three murine models.ConclusionsHisto-Cloud is open source, accessible over the internet, and adaptable for segmentation of any histological structure regardless of stain.

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

The histological identification of papillary thyroid carcinoma (PTC) is straightforward for experienced endocrine pathologists. The increase in radical thyroidectomies led to a raise in the rate of postoperative incidental subcentimeter PTC foci and the recent introduction of the Non-Invasive Follicular Thyroid Neoplasm with Papillary-like Nuclear Features (NIFTP) as a less aggressive mimicker of PTC, which significantly complicated the histology screening of thyroid histology specimens. Artificial Intelligence (AI) applied to Whole Slide Images (WSI) can speed up these processes, aiding pathologists to improve diagnostic accuracy and turnaround times. Here we present a computational pathology pipeline for the identification of Microscopic foci of papillary Thyroid Carcinoma-like nuclear features using Artificial intelligence (MiThyCA). This algorithm relies on a tandem architecture consisting of a Convolutional Neural Network (CNN) designed to identify neoplastic areas within thyroid specimens, and a Vision Transformer (TinyViT) focused on detecting PTC-like areas within the neoplastic regions identified by the first model. The study was conducted on a multi-institutional cohort of 73 WSIs from 67 patients with normal thyroid tissue (n = 22 patients, 33%), NIFTP (n = 19, 28%), PTC (n = 23, 34%), and lymph nodes (n = 3, 5%). Cases were divided into training (n = 40 patients, 41 WSIs), validation (n = 13 patients, 14 WSIs) and test (n = 14 patients, 18 WSIs) sets. Each model singly demonstrated excellent performance at the tile-level on the validation set (accuracy = 0.95 and AUC-ROC = 0.95 for CNN, accuracy = 0.86 and AUC-ROC = 0.84 for TinyViT), with their tandem combination in MiThyCA showing accuracy = 0.85 and F1 score = 0.8 on the validation set at the whole WSI-level. The average total execution time of MiThyCA on the test set WSIs was 51 ± 27s on average on workstations not equipped with GPU, and up to 16 ± 6s and 11 ± 4s per WSI with Nvidia GPU and Apple's laptop chip, respectively. Worthy of note, WSIs dimension did not significantly impact the algorithm processing time. Given its speed and accessibility, MiThyCA is a promising AI-based computer-aided diagnostic tool for the detection of subcentimeter PTC foci in histology.

Invasive ductal carcinoma (IDC) is the most common type of breast cancer. Every year a numerous number of women in this world are diagnosed as having IDC. Accurately detecting IDC is a time consuming and challenging task as the pathologists need to focus on the specific regions of whole slide images (WSI) that contain IDC. Precise and early diagnosis of IDC is a must because it helps to estimate the subsequent tumor aggressiveness that can be caused by this type of breast cancer. The goal of this research is to create an automated system that will analyze the whole mount slide images of breast cancer specimens to indicate the exact positions of IDC inside of the slides and give a decision based on the results. A multilayered convolutional neural network is designed which is trained over a large number of whole slide images. The dataset consists of 162 cases of patients diagnosed with IDC. We found an accuracy of 89.34% in f1 score using convolutional neural network to achieve the state of the art result on IDC classification.

Morphologic information from bone marrow (BM) examinations remains critical for diagnosing leukemia and lymphoma, yet manual interpretation of BM smear slides is labor-intensive for hematologists and pathologists. Few automatic diagnostic tools have successfully classified whole slide images (WSIs) of hematologic malignancies. This study aimed to develop a deep convolutional neural network (DCNN) pipeline to classify leukemia subtypes and lymphoma using whole-slide BM aspirate images. The process involved two stages: first, a quality assessment model selected 200 regions of interest (ROIs) from each WSI to exclude non-informative areas. Next, eight DCNN models with different architectures were trained to classify each ROI into one of five hematologic malignancies, and tile-level predictions were aggregated to produce patient-level results. External evaluation and ancillary analyses were performed to demonstrate generalizability and robustness. In total, 1,022 WSIs were enrolled. The results showed average patient-level accuracy, balanced accuracy, F1-score, and area under the receiver operating characteristic curve (AUC) of 95.2%, 94.7%, 94.8%, and 0.993, respectively, with DenseNet121 achieving the highest balanced accuracy (97.6%). Our method outperformed clustering-constrained attention multiple instance learning (CLAM) in comparison study (accuracy 97.1% vs. 85.0%) and reached accuracies of 83.8% and 86.7% on external datasets. Visualization maps showed consistency between model salience and cell distribution. Our results demonstrate the ability of DCNNs to achieve accurate diagnosis in hematologic pathology, and the pipeline holds potential to assist in early diagnosis and workflow augmentation for hematologists and pathologists.

Nuclei segmentation in whole slide images (WSIs) stained with Hematoxylin and Eosin (H&E) dye, is a key step in computational pathology which aims to automate the laborious process of manual counting and segmentation. Nuclei segmentation is a challenging problem that involves challenges such as touching nuclei resolution, small-sized nuclei, size, and shape variations. With the advent of deep learning, convolution neural networks (CNNs) have shown a powerful ability to extract effective representations from microscopic H&E images. We propose a novel dual encoder Attention U-net (DEAU) deep learning architecture and pseudo hard attention gating mechanism, to enhance the attention to target instances. We added a new secondary encoder to the attention U-net to capture the best attention for a given input. Since H captures nuclei information, we propose a stain-separated H channel as input to the secondary encoder. The role of the secondary encoder is to transform attention prior to different spatial resolutions while learning significant attention information. The proposed DEAU performance was evaluated on three publicly available H&E data sets for nuclei segmentation from different research groups. Experimental results show that our approach outperforms other attention-based approaches for nuclei segmentation.

Digital pathology provides an excellent opportunity for applying fully convolutional networks (FCNs) to tasks, such as semantic segmentation of whole slide images (WSIs). However, standard FCNs face challenges with respect to multi-resolution, inherited from the pyramid arrangement of WSIs. As a result, networks specifically designed to learn and aggregate information at different levels are desired. In this paper, we propose two novel multi-resolution networks based on the popular `U-Net' architecture, which are evaluated on a benchmark dataset for binary semantic segmentation in WSIs. The proposed methods outperform the U-Net, demonstrating superior learning and generalization capabilities.

BackgroundOne challenge to train deep convolutional neural network (CNNs) models with whole slide images (WSIs) is providing the required large number of costly, manually annotated image regions. Strategies to alleviate the scarcity of annotated data include: using transfer learning, data augmentation and training the models with less expensive image-level annotations (weakly-supervised learning). However, it is not clear how to combine the use of transfer learning in a CNN model when different data sources are available for training or how to leverage from the combination of large amounts of weakly annotated images with a set of local region annotations. This paper aims to evaluate CNN training strategies based on transfer learning to leverage the combination of weak and strong annotations in heterogeneous data sources. The trade-off between classification performance and annotation effort is explored by evaluating a CNN that learns from strong labels (region annotations) and is later fine-tuned on a dataset with less expensive weak (image-level) labels.ResultsAs expected, the model performance on strongly annotated data steadily increases as the percentage of strong annotations that are used increases, reaching a performance comparable to pathologists (kappa = 0.691 pm 0.02). Nevertheless, the performance sharply decreases when applied for the WSI classification scenario with kappa = 0.307 pm 0.133. Moreover, it only provides a lower performance regardless of the number of annotations used. The model performance increases when fine-tuning the model for the task of Gleason scoring with the weak WSI labels kappa = 0.528 pm 0.05.ConclusionCombining weak and strong supervision improves strong supervision in classification of Gleason patterns using tissue microarrays (TMA) and WSI regions. Our results contribute very good strategies for training CNN models combining few annotated data and heterogeneous data sources. The performance increases in the controlled TMA scenario with the number of annotations used to train the model. Nevertheless, the performance is hindered when the trained TMA model is applied directly to the more challenging WSI classification problem. This demonstrates that a good pre-trained model for prostate cancer TMA image classification may lead to the best downstream model if fine-tuned on the WSI target dataset. We have made available the source code repository for reproducing the experiments in the paper: https://github.com/ilmaro8/Digital_Pathology_Transfer_Learning

Current whole slide image (WSI) segmentation aims at extracting tumor regions from the background. Unlike this, segmenting distinct tumor areas (instances) within a WSI driven by limited annotated data remains under-explored. In this paper, we formally propose semi-supervised instance segmentation (Semi-IS) in WSIs. We address a key challenge: learning intra-class similarity and inter-class dissimilarity driven by unlabeled data. Specifically, we generally perceive the patch as composed of tokens (together), not the patch alone. We employ contrastive learning to develop a segmentation framework. In the Semi-IS, we find that the boundaries of segmented instances are usually disturbed by noise. We jointly eliminate and preserve noise features to address this problem. We conduct extensive experiments to evaluate the effectiveness and generalizability of Semi-IS, including histopathology and cellular pathology. The results show that in clinical multi instance segmentation tasks, Semi-IS achieves almost full-supervised state-of-the-art results with only 30% annotated data. Semi-IS can improve segmentation accuracy by about 2% on public cell pathology datasets.

Convolutional neural networks have led to significant breakthroughs in the domain of medical image analysis. However, the task of breast cancer segmentation in whole-slide images (WSIs) is still underexplored. WSIs are large histopathological images with extremely high resolution. Constrained by the hardware and field of view, using high-magnification patches can slow down the inference process and using low-magnification patches can cause the loss of information. In this paper, we aim to achieve two seemingly conflicting goals for breast cancer segmentation: accurate and fast prediction. We propose a simple yet efficient framework Reinforced Auto-Zoom Net (RAZN) to tackle this task. Motivated by the zoom-in operation of a pathologist using a digital microscope, RAZN learns a policy network to decide whether zooming is required in a given region of interest. Because the zoom-in action is selective, RAZN is robust to unbalanced and noisy ground truth labels and can efficiently reduce overfitting. We evaluate our method on a public breast cancer dataset. RAZN outperforms both single-scale and multi-scale baseline approaches, achieving better accuracy at low inference cost.

Purpose: To compare deep learning (DL) approaches applied to hematoxylin and eosin (H&E)-stained whole slide images (WSIs) from women with benign breast disease (BBD) to predict risk of developing invasive breast cancer (BC). Method: Two deep convolutional neural networks (CNNs) based on a customized 16-layer CNN (known as VGG-16 by Visual Geometry Group, University of Oxford) and an automated CNN (Google’s AutoML) were trained using H&E-stained WSIs to identify distinct histological features on diagnostic BBD biopsies that characterize BBD patients who were (cases, n=347) and were not (controls, n=347) subsequently diagnosed with invasive BC. The CNNs consisted of multiple convolutions, max pooling, fully connected, etc., layers. To incorporate our data into the VGG network, we customized the network architecture and hyperparameters to enhance the classification performances. For AutoML, we used the system's default network with standard hyperparameters. The trained model was then tested on a held-out set of 140 patients (70 cases and 70 controls). The quantitative performance was evaluated using accuracy (ACC), sensitivity (SE), precision (PR), area under the receiver operating characteristic curve (AUROC), etc. For qualitative results, we generated heatmaps using weights and feature maps from the final convolution layer of our customized CNN. Heatmaps were superimposed onto original H&E images to highlight different unique features (such as pattern, texture, color, and morphology). Results: We found both deep learning methods to demonstrate remarkable ability in predicting case-control status in the held-out set (AUROC= 90% and 89% for customized CNN and AutoML, respectively). However, our customized CNN outperformed AutoML in terms of ACC (83.57% (95% confidence interval (CI): 76-89%) vs 77.86% (95%CI: 70-84%), respectively); SE (82.85% (95%CI: 72-91%) vs 77.86% (95%CI: 70-84%), respectively); PR (84.05% (95%CI: 73-92%) vs 81.97% (95%CI: 70-91%), respectively); F1 score (83.45% (95%CI: 76-89%) vs 76.34% (95%CI: 68-83%), respectively); as well as error rates (0.16% (95%CI: 0.11-0.24%) vs 0.22% (95%CI: 0.16-0.30%), respectively). Heatmaps revealed specific stromal and epithelial features that were distinct between case and control images. Conclusion: By using routinely available H&E-stained WSIs, we developed a customized CNN that outperformed AutoML in distinguishing future BC cases from controls in a BBD population. The qualitative results identified stromal and epithelial regions in the BBD biopsies that were highly predictive of being a case versus control and vice versa thereby providing etiologic clues into breast cancer development following BBD. Future research will focus on leveraging DL to better understand the histologic basis of BBD progression to invasive BC. Citation Format: Monjoy Saha, Mustapha Abubakar, Thomas E. Rohan, Ruth M. Pfeiffer, Máire A. Duggan, Kathryn Richert-Boe, Jonine D. Figueroa, Jonas S. Almeida, Gretchen L. Gierach. Comparison of deep learning approaches applied to hematoxylin and eosin-stained whole slide images from women with benign breast disease to predict risk of developing invasive breast cancer. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5393.