Weakly Supervised Segmentation by Tensor Graph Learning for Whole Slide Images

This paper presents a deep learning approach for automatic detection and visual analysis of invasive ductal carcinoma (IDC) tissue regions in whole slide images (WSI) of breast cancer (BCa). Deep learning approaches are learn-from-data methods involving computational modeling of the learning process. This approach is similar to how human brain works using different interpretation levels or layers of most representative and useful features resulting into a hierarchical learned representation. These methods have been shown to outpace traditional approaches of most challenging problems in several areas such as speech recognition and object detection. Invasive breast cancer detection is a time consuming and challenging task primarily because it involves a pathologist scanning large swathes of benign regions to ultimately identify the areas of malignancy. Precise delineation of IDC in WSI is crucial to the subsequent estimation of grading tumor aggressiveness and predicting patient outcome. DL approaches are particularly adept at handling these types of problems, especially if a large number of samples are available for training, which would also ensure the generalizability of the learned features and classifier. The DL framework in this paper extends a number of convolutional neural networks (CNN) for visual semantic analysis of tumor regions for diagnosis support. The CNN is trained over a large amount of image patches (tissue regions) from WSI to learn a hierarchical part-based representation. The method was evaluated over a WSI dataset from 162 patients diagnosed with IDC. 113 slides were selected for training and 49 slides were held out for independent testing. Ground truth for quantitative evaluation was provided via expert delineation of the region of cancer by an expert pathologist on the digitized slides. The experimental evaluation was designed to measure classifier accuracy in detecting IDC tissue regions in WSI. Our method yielded the best quantitative results for automatic detection of IDC regions in WSI in terms of F-measure and balanced accuracy (71.80%, 84.23%), in comparison with an approach using handcrafted image features (color, texture and edges, nuclear textural and architecture), and a machine learning classifier for invasive tumor classification using a Random Forest. The best performing handcrafted features were fuzzy color histogram (67.53%, 78.74%) and RGB histogram (66.64%, 77.24%). Our results also suggest that at least some of the tissue classification mistakes (false positives and false negatives) were less due to any fundamental problems associated with the approach, than the inherent limitations in obtaining a very highly granular annotation of the diseased area of interest by an expert pathologist.

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

The histological identification of papillary thyroid carcinoma (PTC) is straightforward for experienced endocrine pathologists. The increase in radical thyroidectomies led to a raise in the rate of postoperative incidental subcentimeter PTC foci and the recent introduction of the Non-Invasive Follicular Thyroid Neoplasm with Papillary-like Nuclear Features (NIFTP) as a less aggressive mimicker of PTC, which significantly complicated the histology screening of thyroid histology specimens. Artificial Intelligence (AI) applied to Whole Slide Images (WSI) can speed up these processes, aiding pathologists to improve diagnostic accuracy and turnaround times. Here we present a computational pathology pipeline for the identification of Microscopic foci of papillary Thyroid Carcinoma-like nuclear features using Artificial intelligence (MiThyCA). This algorithm relies on a tandem architecture consisting of a Convolutional Neural Network (CNN) designed to identify neoplastic areas within thyroid specimens, and a Vision Transformer (TinyViT) focused on detecting PTC-like areas within the neoplastic regions identified by the first model. The study was conducted on a multi-institutional cohort of 73 WSIs from 67 patients with normal thyroid tissue (n = 22 patients, 33%), NIFTP (n = 19, 28%), PTC (n = 23, 34%), and lymph nodes (n = 3, 5%). Cases were divided into training (n = 40 patients, 41 WSIs), validation (n = 13 patients, 14 WSIs) and test (n = 14 patients, 18 WSIs) sets. Each model singly demonstrated excellent performance at the tile-level on the validation set (accuracy = 0.95 and AUC-ROC = 0.95 for CNN, accuracy = 0.86 and AUC-ROC = 0.84 for TinyViT), with their tandem combination in MiThyCA showing accuracy = 0.85 and F1 score = 0.8 on the validation set at the whole WSI-level. The average total execution time of MiThyCA on the test set WSIs was 51 ± 27s on average on workstations not equipped with GPU, and up to 16 ± 6s and 11 ± 4s per WSI with Nvidia GPU and Apple's laptop chip, respectively. Worthy of note, WSIs dimension did not significantly impact the algorithm processing time. Given its speed and accessibility, MiThyCA is a promising AI-based computer-aided diagnostic tool for the detection of subcentimeter PTC foci in histology.

The current melanoma staging system predicts 74% of the variance in survival, with prognostic biomarkers subject to high levels of inter-observer variation. This work assesses whether a previously developed convolutional neural network (CNN) for invasive melanoma segmentation in whole slide images (WSIs) may reveal new insights into melanoma morphology and patient prognosis. This paper uses Cox proportional multivariate regression analyses to evaluate the ability of the CNN outputs to predict patient survival across 745 WSIs from 5 data sources. Five objective histomorphological parameters of tumour size and shape that are independently associated with overall and melanoma-specific survival were created from the CNN: tumour area(log) (HR 1.48 CI 1.30-1.68, p < 0.001), tumour perimeter(log) (HR 1.86 CI 1.48-2.32, p < 0.001), major axis length(log) (HR 1.88 CI 1.42-2.48, p < 0.001), Nodularity Index(log) (HR 1.77 CI 1.28-2.43, p < 0.001) and digital Breslow thickness(log) (HR 2.04, CI 1.63-2.54, p < 0.001). These results indicate that melanoma segmentation of the entire lesion within a WSI may be used to predict patient outcome. Moreover, this technology can be used to make new morphological discoveries to provide information not currently contained within our staging system (e.g. Nodularity Index), as well as provide objectivity and automation of current biomarkers (e.g. digital Breslow thickness). Further work is required to validate this initial discovery and evaluation.

Invasive ductal carcinoma (IDC) is the most common type of breast cancer. Every year a numerous number of women in this world are diagnosed as having IDC. Accurately detecting IDC is a time consuming and challenging task as the pathologists need to focus on the specific regions of whole slide images (WSI) that contain IDC. Precise and early diagnosis of IDC is a must because it helps to estimate the subsequent tumor aggressiveness that can be caused by this type of breast cancer. The goal of this research is to create an automated system that will analyze the whole mount slide images of breast cancer specimens to indicate the exact positions of IDC inside of the slides and give a decision based on the results. A multilayered convolutional neural network is designed which is trained over a large number of whole slide images. The dataset consists of 162 cases of patients diagnosed with IDC. We found an accuracy of 89.34% in f1 score using convolutional neural network to achieve the state of the art result on IDC classification.

Multiple instance learning (MIL) is a key algorithm for classification of whole slide images (WSI). Histology WSIs can have billions of pixels, which create enormous computational and annotation challenges. Typically, such images are divided into a set of patches (a bag of instances), where only bag-level class labels are provided. Deep learning based MIL methods calculate instance features using convolutional neural network (CNN). Our proposed approach is also deep learning based, with the following two contributions: Firstly, we propose to explicitly account for dependencies between instances during training by embedding self-attention Transformer blocks to capture dependencies between instances. For example, a tumor grade may depend on the presence of several particular patterns at different locations in WSI, which requires to account for dependencies between patches. Secondly, we propose an instance-wise loss function based on instance pseudo-labels. We compare the proposed algorithm to multiple baseline methods, evaluate it on the PANDA challenge dataset, the largest publicly available WSI dataset with over 11K images, and demonstrate state-of-the-art results.

BackgroundImage-based machine learning tools hold great promise for clinical applications in pathology research. However, the ideal end-users of these computational tools (e.g., pathologists and biological scientists) often lack the programming experience required for the setup and use of these tools which often rely on the use of command line interfaces.MethodsWe have developed Histo-Cloud, a tool for segmentation of whole slide images (WSIs) that has an easy-to-use graphical user interface. This tool runs a state-of-the-art convolutional neural network (CNN) for segmentation of WSIs in the cloud and allows the extraction of features from segmented regions for further analysis.ResultsBy segmenting glomeruli, interstitial fibrosis and tubular atrophy, and vascular structures from renal and non-renal WSIs, we demonstrate the scalability, best practices for transfer learning, and effects of dataset variability. Finally, we demonstrate an application for animal model research, analyzing glomerular features in three murine models.ConclusionsHisto-Cloud is open source, accessible over the internet, and adaptable for segmentation of any histological structure regardless of stain.

Colorectal cancer (CRC) is a prevalent gastrointestinal tumour with high incidence and mortality rates. Early screening for CRC can improve cure rates and reduce mortality. Recently, deep convolution neural network (CNN)-based pathological image diagnosis has been intensively studied to meet the challenge of time-consuming and labour-intense manual analysis of high-resolution whole slide images (WSIs). Despite the achievements made, deep CNN-based methods still suffer from some limitations, and the fundamental problem is that they cannot capture global features. To address this issue, we propose a hybrid deep learning framework (RGSB-UNet) for automatic tumour segmentation in WSIs. The framework adopts a UNet architecture that consists of the newly-designed residual ghost block with switchable normalization (RGS) and the bottleneck transformer (BoT) for downsampling to extract refined features, and the transposed convolution and 1 × 1 convolution with ReLU for upsampling to restore the feature map resolution to that of the original image. The proposed framework combines the advantages of the spatial-local correlation of CNNs and the long-distance feature dependencies of BoT, ensuring its capacity of extracting more refined features and robustness to varying batch sizes. Additionally, we consider a class-wise dice loss (CDL) function to train the segmentation network. The proposed network achieves state-of-the-art segmentation performance under small batch sizes. Experimental results on DigestPath2019 and GlaS datasets demonstrate that our proposed model produces superior evaluation scores and state-of-the-art segmentation results.

Accurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep learning framework that integrates three complementary approaches, YOLOv11, StarDist, and Segment Anything Model v2 (SAM2), to achieve robust and precise cell segmentation. The proposed pipeline utilizes YOLOv11 as an object detector to localize regions of interest, generating bounding boxes or preliminary masks that are subsequently used either as prompts to guide SAM2 or to filter segmentation outputs. StarDist is employed to model cell and nuclear boundaries with high geometric precision using star-convex polygon representations, which are particularly effective in densely packed cellular regions. The framework was evaluated on a unique WSI dataset comprising 256 × 256 image tiles annotated with high-resolution cell-level masks. Quantitative evaluations using the Dice coefficient, intersection over union (IoU), F1-score, precision, and recall demonstrated that the proposed method significantly outperformed individual baseline models. The integration of object detection and prompt-based segmentation led to enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types. This work contributes a scalable and modular solution for advancing automated histopathological image analysis.

The tissue sample analysis based on the visualization of the entire tissue sample represents an important methodology in clinical diagnostics, research, and education. This process that enables visualization of the entire tissue sample is named whole slide imaging. For clinical purposes, professional and very expensive scanners are used for tissue sample digitization and the creation of whole slide images. In education, it is possible to use whole slide images that do not necessarily follow strict medical standards and regulations. Therefore, the important research task is to propose a methodology for the manual creation of whole slide images and its adequate use in medical education. In this paper, the use of manually created whole slide images in histology classes education was analyzed. In the experimental part of this research, the methodology of manual creation and usage of whole slide images (WSI) in histology education is presented in more detail. The subjective assessment of the user's satisfaction with WSI usage in education was also performed. It was shown that the whole slide images were evaluated very positively in the histological analysis and education process. Using WSI can significantly improve histology teaching, especially when distance education principles are implemented. Therefore, in crises like the pandemic situation COVID-19 WSI usage becomes a very important educational tool for teaching microscope-related topics. The results of this research showed that the use of whole slide images in histology classes is in line with the process of digital transformation in education.

Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.

A joint Multi-decoder Dual-attention U-Net framework for tumor segmentation in Whole Slide Images

Digital pathology provides an excellent opportunity for applying fully convolutional networks (FCNs) to tasks, such as semantic segmentation of whole slide images (WSIs). However, standard FCNs face challenges with respect to multi-resolution, inherited from the pyramid arrangement of WSIs. As a result, networks specifically designed to learn and aggregate information at different levels are desired. In this paper, we propose two novel multi-resolution networks based on the popular `U-Net' architecture, which are evaluated on a benchmark dataset for binary semantic segmentation in WSIs. The proposed methods outperform the U-Net, demonstrating superior learning and generalization capabilities.

Although microscopic analysis of tissue slides has been the basis for disease diagnosis for decades, intra- and inter-observer variabilities remain issues to be resolved. The recent introduction of digital scanners has allowed for using deep learning in the analysis of tissue images because many whole slide images (WSIs) are accessible to researchers. In the present study, we investigated the possibility of a deep learning-based, fully automated, computer-aided diagnosis system with WSIs from a stomach adenocarcinoma dataset. Three different convolutional neural network architectures were tested to determine the better architecture for tissue classifier. Each network was trained to classify small tissue patches into normal or tumor. Based on the patch-level classification, tumor probability heatmaps can be overlaid on tissue images. We observed three different tissue patterns, including clear normal, clear tumor and ambiguous cases. We suggest that longer inspection time can be assigned to ambiguous cases compared to clear normal cases, increasing the accuracy and efficiency of histopathologic diagnosis by pre-evaluating the status of the WSIs. When the classifier was tested with completely different WSI dataset, the performance was not optimal because of the different tissue preparation quality. By including a small amount of data from the new dataset for training, the performance for the new dataset was much enhanced. These results indicated that WSI dataset should include tissues prepared from many different preparation conditions to construct a generalized tissue classifier. Thus, multi-national/multi-center dataset should be built for the application of deep learning in the real world medical practice.

Introduction: Artifacts are often introduced during tissue collection and processing, slide preparation, and/or when generating whole slide images (WSI). The presence of artifacts has a negative impact on the digital pathology workflow as artifacts may hinder diagnostic reporting and can lead to false positive and false negative results when using image analysis algorithms or computer-aided diagnosis systems. Manual quality control of WSI is a time-consuming procedure and therefore automated quality control tools, which report and exclude artifacts, are highly desirable to streamline digital pathology workflows. To automate the quality control step, we developed SlideQC, an AI-based quality control tool that automatically detects, reports, and outlines artifacts such as air bubbles, dust/debris, folds, out-of-focus,and pen marks, in both research and clinical workflows. Methods: SlideQC was trained with a DenseNet-based network using 1984 annotations for artifacts including air bubbles, dust/debris, folds, out-of-focus, and pen markers, across 254 Haematoxylin and Eosin (H&amp;E) stained WSI from more than 9 tissue types. A set of 2048 annotations from synthetically generated out-of-focus images was added to supplement the training data. The performance of the SlideQC was evaluated on an external test cohort of 49 WSI H&amp;E images sourced from the open-source database ‘HistoQCRepo’, across 375 annotations (tissue and artifact), and compared with the performance of HistoQC, an open-source quality control tool for digital pathology slides. Results: On the external test cohort, SlideQC showed high sensitivity, specificity, and F1-score with average values of 0.93, 0.99, and 0.93, across the five artifact types. In the same cohort, HistoQC attained an average sensitivity, specificity, and F1-score of 0.65, 0.79, and 0.54, respectively. Conclusions: SlideQC achieved high sensitivity, specificity, and F1-score on an external test cohort. SlideQC can add efficiency gains to a workflow by performing quality control on 100% of slides rather than the currently manually performed on only a subset of the slides in clinical pathology departments. SlideQC can allowthe triaging and alerting of slides containing a high level of artifact within a digital pathology workflow. The tool can also be used to exclude the artifact region from downstream analysis by subsequent image analysis algorithms. Citation Format: Daniela Rodrigues, Stefan Reinhard, Therese Waldburger, Daniel Martin, Suzana Couto, Inti Zlobec, Peter Caie, Erik Burlingame. SlideQC: An AI-based tool for automated quality control of whole-slide digital pathology images. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5442.