Whole-slide Histopathology Images Research Articles

Image-based consensus molecular subtyping in rectal cancer biopsies and response to neoadjuvant chemoradiotherapy

The development of deep learning (DL) models to predict the consensus molecular subtypes (CMS) from histopathology images (imCMS) is a promising and cost-effective strategy to support patient stratification. Here, we investigate whether imCMS calls generated from whole slide histopathology images (WSIs) of rectal cancer (RC) pre-treatment biopsies are associated with pathological complete response (pCR) to neoadjuvant long course chemoradiotherapy (LCRT) with single agent fluoropyrimidine. DL models were trained to classify WSIs of colorectal cancers stained with hematoxylin and eosin into one of the four CMS classes using a multi-centric dataset of resection and biopsy specimens (n = 1057 WSIs) with paired transcriptional data. Classifiers were tested on a held out RC biopsy cohort (ARISTOTLE) and correlated with pCR to LCRT in an independent dataset merging two RC cohorts (ARISTOTLE, n = 114 and SALZBURG, n = 55 patients). DL models predicted CMS with high classification performance in multiple comparative analyses. In the independent cohorts (ARISTOTLE, SALZBURG), cases with WSIs classified as imCMS1 had a significantly higher likelihood of achieving pCR (OR = 2.69, 95% CI 1.01–7.17, p = 0.048). Conversely, imCMS4 was associated with lack of pCR (OR = 0.25, 95% CI 0.07–0.88, p = 0.031). Classification maps demonstrated pathologist-interpretable associations with high stromal content in imCMS4 cases, associated with poor outcome. No significant association was found in imCMS2 or imCMS3. imCMS classification of pre-treatment biopsies is a fast and inexpensive solution to identify patient groups that could benefit from neoadjuvant LCRT. The significant associations between imCMS1/imCMS4 with pCR suggest the existence of predictive morphological features that could enhance standard pathological assessment.

Abstract 4938: Prediction of gene mutation from colorectal adenocarcinoma whole slide images via integrated deep learning pipeline

Abstract Colorectal cancer (CRC) stands as one of the deadliest cancer types worldwide. Identifying specific gene mutations crucial for targeted therapy is pivotal for CRC treatment. However, the current genetic testing method involves liquid biopsy, which is often cumbersome and cost-intensive. Histopathological whole-slide images (WSIs) are routinely generated for diagnostic purposes and can offer valuable insights into cellular heterogeneity. In this study, we aim to develop an integrated deep learning pipeline for predicting TP53 mutations in CRC, one of the most frequently mutated genes in CRC. We commenced by collecting 424 diagnostic WSIs from the TCGA-COAD cohorts. Each WSI was split as small patches at a resolution of 0.5 micrometers per pixel for following analyses. Quality control was implemented using Otsu thresholding and Gaussian blur filtering, both at the slide-level masking and patch-level filtering stages. Additionally, we applied the Macenko algorithm for color normalization to eliminate systematic errors. Following image preprocessing, we constructed a tissue-type classification model based on VGG19, which was trained and tested using the NCT-CRC-HE-100K and CRC-VAL-HE-7K datasets, both containing pixels with tissue types labeled by human experts. The trained classifier was then used to identify adenocarcinoma epithelium patches from TCGA-COAD. Subsequently, we applied multiple pre-trained Convolutional Neural Network (CNN) models, including ResNet-101, DenseNet-201, and Inception-ResNet-v2, to predict TP53 mutations for the identified adenocarcinoma epithelium patches from TCGA-COAD. A novel integration approach was employed to aggregate patch-level predictions into slide-level outcomes. We evaluated the trained models using various metrics, including accuracy, precision, recall, F1 score, AUC (Area Under the Curve), and Cohen's Kappa coefficient. The tissue-type classification model exhibited robust performance, achieving an overall accuracy of 94.7%, a Cohen's Kappa coefficient of 94%, a precision of 95.3%, a recall of 99.76%, and an F1 score of 97.48% for identifying adenocarcinoma epithelium patches. Regarding the TP53 mutation prediction models, ResNet-101, DenseNet-201, and Inception-ResNet-v2 exhibit the predictive ability to both patch level (AUC=65%) and slide level (AUC=70%). In summary, the proposed deep learning pipeline offers an efficient way to acquire gene mutation information from WSIs. Moreover, it demonstrates a capability to discover potential molecular aberration from the clinical images. Citation Format: Geng-Yun Tien, Liang-Chuan Lai, Tzu-Pin Lu, Mong-Hsun Tsai, Hsiang-Han Chen, Eric Y. Chuang. Prediction of gene mutation from colorectal adenocarcinoma whole slide images via integrated deep learning pipeline [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4938.

Deep learning-based accurate diagnosis and quantitative evaluation of microvascular invasion in hepatocellular carcinoma on whole-slide histopathology images.

Microvascular invasion (MVI) is an independent prognostic factor that is associated with early recurrence and poor survival after resection of hepatocellular carcinoma (HCC). However, the traditional pathology approach is relatively subjective, time-consuming, and heterogeneous in the diagnosis of MVI. The aim of this study was to develop a deep-learning model that could significantly improve the efficiency and accuracy of MVI diagnosis. We collected H&E-stained slides from 753 patients with HCC at the First Affiliated Hospital of Zhejiang University. An external validation set with 358 patients was selected from The Cancer Genome Atlas database. The deep-learning model was trained by simulating the method used by pathologists to diagnose MVI. Model performance was evaluated with accuracy, precision, recall, F1 score, and the area under the receiver operating characteristic curve. We successfully developed a MVI artificial intelligence diagnostic model (MVI-AIDM) which achieved an accuracy of 94.25% in the independent external validation set. The MVI positive detection rate of MVI-AIDM was significantly higher than the results of pathologists. Visualization results demonstrated the recognition of micro MVIs that were difficult to differentiate by the traditional pathology. Additionally, the model provided automatic quantification of the number of cancer cells and spatial information regarding MVI. We developed a deep learning diagnostic model, which performed well and improved the efficiency and accuracy of MVI diagnosis. The model provided spatial information of MVI that was essential to accurately predict HCC recurrence after surgery.

A multi-class brain tumor grading system based on histopathological images using a hybrid YOLO and RESNET networks

Gliomas are primary brain tumors caused by glial cells. These cancers’ classification and grading are crucial for prognosis and treatment planning. Deep learning (DL) can potentially improve the digital pathology investigation of brain tumors. In this paper, we developed a technique for visualizing a predictive tumor grading model on histopathology pictures to help guide doctors by emphasizing characteristics and heterogeneity in forecasts. The proposed technique is a hybrid model based on YOLOv5 and ResNet50. The function of YOLOv5 is to localize and classify the tumor in large histopathological whole slide images (WSIs). The suggested technique incorporates ResNet into the feature extraction of the YOLOv5 framework, and the detection results show that our hybrid network is effective for identifying brain tumors from histopathological images. Next, we estimate the glioma grades using the extreme gradient boosting classifier. The high-dimensional characteristics and nonlinear interactions present in histopathology images are well-handled by this classifier. DL techniques have been used in previous computer-aided diagnosis systems for brain tumor diagnosis. However, by combining the YOLOv5 and ResNet50 architectures into a hybrid model specifically designed for accurate tumor localization and predictive grading within histopathological WSIs, our study presents a new approach that advances the field. By utilizing the advantages of both models, this creative integration goes beyond traditional techniques to produce improved tumor localization accuracy and thorough feature extraction. Additionally, our method ensures stable training dynamics and strong model performance by integrating ResNet50 into the YOLOv5 framework, addressing concerns about gradient explosion. The proposed technique is tested using the cancer genome atlas dataset. During the experiments, our model outperforms the other standard ways on the same dataset. Our results indicate that the proposed hybrid model substantially impacts tumor subtype discrimination between low-grade glioma (LGG) II and LGG III. With 97.2% of accuracy, 97.8% of precision, 98.6% of sensitivity, and the Dice similarity coefficient of 97%, the proposed model performs well in classifying four grades. These results outperform current approaches for identifying LGG from high-grade glioma and provide competitive performance in classifying four categories of glioma in the literature.

Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics

The development of artificial intelligence (AI)-based melanoma classifiers typically calls for large, centralized datasets, requiring hospitals to give away their patient data, which raises serious privacy concerns. To address this concern, decentralized federated learning has been proposed, where classifier development is distributed across hospitals. To investigate whether a more privacy-preserving federated learning approach can achieve comparable diagnostic performance to a classical centralized (ie, single-model) and ensemble learning approach for AI-based melanoma diagnostics. This multicentric, single-arm diagnostic study developed a federated model for melanoma-nevus classification using histopathological whole-slide images prospectively acquired at 6 German university hospitals between April 2021 and February 2023 and benchmarked it using both a holdout and an external test dataset. Data analysis was performed from February to April 2023. All whole-slide images were retrospectively analyzed by an AI-based classifier without influencing routine clinical care. The area under the receiver operating characteristic curve (AUROC) served as the primary end point for evaluating the diagnostic performance. Secondary end points included balanced accuracy, sensitivity, and specificity. The study included 1025 whole-slide images of clinically melanoma-suspicious skin lesions from 923 patients, consisting of 388 histopathologically confirmed invasive melanomas and 637 nevi. The median (range) age at diagnosis was 58 (18-95) years for the training set, 57 (18-93) years for the holdout test dataset, and 61 (18-95) years for the external test dataset; the median (range) Breslow thickness was 0.70 (0.10-34.00) mm, 0.70 (0.20-14.40) mm, and 0.80 (0.30-20.00) mm, respectively. The federated approach (0.8579; 95% CI, 0.7693-0.9299) performed significantly worse than the classical centralized approach (0.9024; 95% CI, 0.8379-0.9565) in terms of AUROC on a holdout test dataset (pairwise Wilcoxon signed-rank, P < .001) but performed significantly better (0.9126; 95% CI, 0.8810-0.9412) than the classical centralized approach (0.9045; 95% CI, 0.8701-0.9331) on an external test dataset (pairwise Wilcoxon signed-rank, P < .001). Notably, the federated approach performed significantly worse than the ensemble approach on both the holdout (0.8867; 95% CI, 0.8103-0.9481) and external test dataset (0.9227; 95% CI, 0.8941-0.9479). The findings of this diagnostic study suggest that federated learning is a viable approach for the binary classification of invasive melanomas and nevi on a clinically representative distributed dataset. Federated learning can improve privacy protection in AI-based melanoma diagnostics while simultaneously promoting collaboration across institutions and countries. Moreover, it may have the potential to be extended to other image classification tasks in digital cancer histopathology and beyond.

Deep Multi-Dictionary Learning for Survival Prediction With Multi-Zoom Histopathological Whole Slide Images.

Survival prediction based on histopathological whole slide images (WSIs) is of great significance for risk-benefit assessment and clinical decision. However, complex microenvironments and heterogeneous tissue structures in WSIs bring challenges to learning informative prognosis-related representations. Additionally, previous studies mainly focus on modeling using mono-scale WSIs, which commonly ignore useful subtle differences existed in multi-zoom WSIs. To this end, we propose a deep multi-dictionary learning framework for cancer survival prediction with multi-zoom histopathological WSIs. The framework can recognize and learn discriminative clusters (i.e., microenvironments) based on multi-scale deep representations for survival analysis. Specifically, we learn multi-scale features based on multi-zoom tiles from WSIs via stacked deep autoencoders network followed by grouping different microenvironments by cluster algorithm. Based on multi-scale deep features of clusters, a multi-dictionary learning method with a post-pruning strategy is devised to learn discriminative representations from selected prognosis-related clusters in a task-driven manner. Finally, a survival model (i.e., EN-Cox) is constructed to estimate the risk index of an individual patient. The proposed model is evaluated on three datasets derived from The Cancer Genome Atlas (TCGA), and the experimental results demonstrate that it outperforms several state-of-the-art survival analysis approaches.

Machine Learning of Histopathological Images Predicts Recurrences of Resected Pancreatic Ductal Adenocarcinoma With Adjuvant Treatment.

Pancreatic ductal adenocarcinoma is an intractable disease with frequent recurrence after resection and adjuvant therapy. The present study aimed to clarify whether artificial intelligence-assisted analysis of histopathological images can predict recurrence in patients with pancreatic ductal adenocarcinoma who underwent resection and adjuvant chemotherapy with tegafur/5-chloro-2,4-dihydroxypyridine/potassium oxonate. Eighty-nine patients were enrolled in the study. Machine-learning algorithms were applied to 10-billion-scale pixel data of whole-slide histopathological images to generate key features using multiple deep autoencoders. Areas under the curve were calculated from receiver operating characteristic curves using a support vector machine with key features alone and by combining with clinical data (age and carbohydrate antigen 19-9 and carcinoembryonic antigen levels) for predicting recurrence. Supervised learning with pathological annotations was conducted to determine the significant features for predicting recurrence. Areas under the curves obtained were 0.73 (95% confidence interval, 0.59-0.87) by the histopathological data analysis and 0.84 (95% confidence interval, 0.73-0.94) by the combinatorial analysis of histopathological data and clinical data. Supervised learning model demonstrated that poor tumor differentiation was significantly associated with recurrence. Results indicate that machine learning with the integration of artificial intelligence-driven evaluation of histopathological images and conventional clinical data provides relevant prognostic information for patients with pancreatic ductal adenocarcinoma.

Detection and subtyping of basal cell carcinoma in whole-slide histopathology using weakly-supervised learning

The frequency of basal cell carcinoma (BCC) cases is putting an increasing strain on dermatopathologists. BCC is the most common type of skin cancer, and its incidence is increasing rapidly worldwide. AI can play a significant role in reducing the time and effort required for BCC diagnostics and thus improve the overall efficiency of the process. To train such an AI system in a fully-supervised fashion however, would require a large amount of pixel-level annotation by already strained dermatopathologists. Therefore, in this study, our primary objective was to develop a weakly-supervised for the identification of basal cell carcinoma (BCC) and the stratification of BCC into low-risk and high-risk categories within histopathology whole-slide images (WSI). We compared Clustering-constrained Attention Multiple instance learning (CLAM) with StreamingCLAM and hypothesized that the latter would be the superior approach. A total of 5147 images were used to train and validate the models, which were subsequently tested on an internal set of 949 images and an external set of 183 images. The labels for training were automatically extracted from free-text pathology reports using a rule-based approach. All data has been made available through the COBRA dataset. The results showed that both the CLAM and StreamingCLAM models achieved high performance for the detection of BCC, with an area under the ROC curve (AUC) of 0.994 and 0.997, respectively, on the internal test set and 0.983 and 0.993 on the external dataset. Furthermore, the models performed well on risk stratification, with AUC values of 0.912 and 0.931, respectively, on the internal set, and 0.851 and 0.883 on the external set. In every single metric the StreamingCLAM model outperformed the CLAM model or is on par. The performance of both models was comparable to that of two pathologists who scored 240 BCC positive slides. Additionally, in the public test set, StreamingCLAM demonstrated a comparable AUC of 0.958, markedly superior to CLAM’s 0.803. This difference was statistically significant and emphasized the strength and better adaptability of the StreamingCLAM approach.

Extracting structured information from unstructured histopathology reports using generative pre-trained transformer 4 (GPT-4).

Deep learning applied to whole-slide histopathology images (WSIs) has the potential to enhance precision oncology and alleviate the workload of experts. However, developing these models necessitates large amounts of data with ground truth labels, which can be both time-consuming and expensive to obtain. Pathology reports are typically unstructured or poorly structured texts, and efforts to implement structured reporting templates have been unsuccessful, as these effortslead to perceived extra workload. In this study, we hypothesised that large language models (LLMs), such as the generative pre-trained transformer 4 (GPT-4), can extract structured data from unstructured plain language reports using a zero-shot approach without requiring any re-training. We tested this hypothesis by utilising GPT-4 to extract information from histopathological reports, focusing on two extensive sets of pathology reports for colorectal cancer and glioblastoma. We found a high concordance between LLM-generated structured data and human-generated structured data. Consequently, LLMs could potentially be employed routinely to extract ground truth data for machine learning from unstructured pathology reports in the future. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Feed-forward networks using logistic regression and support vector machine for whole-slide breast cancer histopathology image classification

The performance of an image classification depends on the efficiency of the feature learning process. This process is a challenging task that traditionally requires prior knowledge from domain experts. Recently, representation learning was introduced to extract features directly from the raw images without any prior knowledge. Deep learning using a Convolutional Neural Network (CNN) has gained massive attention for performing image classification, as it achieves remarkable accuracy that sometimes exceeds human performance. But this type of network learns features by using a back-propagation approach. This approach requires a huge amount of training data and suffers from the vanishing gradient problem that deteriorates the feature learning. The forward-propagation approach uses predefined filters or filters learned outside the model and applied in a feed-forward manner. This approach is proven to achieve good results with small size labeled datasets. In this work, we investigate the suitability of using two feed-forward methods such as Convolutional Logistic Regression Network (CLR), and Convolutional Support Vector Machine Network for Histopathology Images (CSVM-H). The experiments we have conducted on two small breast cancer datasets (Sultan Qaboos University Hospital (SQUH) and BreaKHis dataset) demonstrate the advantage of using feed-forward approaches over the traditional back-propagation ones. On those datasets, the proposed models CLR and CSVM-H were faster to train and achieved better classification performance than the traditional back-propagation methods (VggNet-16 and ResNet-50) on the SQUH dataset. Importantly, our proposed approach CLR and CSVM-H efficiently learn representations from small amounts of breast cancer whole-slide images and achieve an AUC of 0.83 and 0.84, respectively, on the SQUH dataset. Moreover, the proposed models reduce memory footprint in the classification of Whole-Slide histopathology images since their training time is significantly reduced compared to the traditional CNN on the SQUH and BreaKHis datasets.

Graph-based multi-modality integration for prediction of cancer subtype and severity

Personalised cancer screening before therapy paves the way toward improving diagnostic accuracy and treatment outcomes. Most approaches are limited to a single data type and do not consider interactions between features, leaving aside the complementary insights that multimodality and systems biology can provide. In this project, we demonstrate the use of graph theory for data integration via individual networks where nodes and edges are individual-specific. We showcase the consequences of early, intermediate, and late graph-based fusion of RNA-Seq data and histopathology whole-slide images for predicting cancer subtypes and severity. The methodology developed is as follows: (1) we create individual networks; (2) we compute the similarity between individuals from these graphs; (3) we train our model on the similarity matrices; (4) we evaluate the performance using the macro F1 score. Pros and cons of elements of the pipeline are evaluated on publicly available real-life datasets. We find that graph-based methods can increase performance over methods that do not study interactions. Additionally, merging multiple data sources often improves classification compared to models based on single data, especially through intermediate fusion. The proposed workflow can easily be adapted to other disease contexts to accelerate and enhance personalized healthcare.

Shared-Specific Feature Learning With Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.

The fusion of multi-modal medical data is essential to assist medical experts to make treatment decisions for precision medicine. For example, combining the whole slide histopathological images (WSIs) and tabular clinical data can more accurately predict the lymph node metastasis (LNM) of papillary thyroid carcinoma before surgery to avoid unnecessary lymph node resection. However, the huge-sized WSI provides much more high-dimensional information than low-dimensional tabular clinical data, making the information alignment challenging in the multi-modal WSI analysis tasks. This paper presents a novel transformer-guided multi-modal multi-instance learning framework to predict lymph node metastasis from both WSIs and tabular clinical data. We first propose an effective multi-instance grouping scheme, named siamese attention-based feature grouping (SAG), to group high-dimensional WSIs into representative low-dimensional feature embeddings for fusion. We then design a novel bottleneck shared-specific feature transfer module (BSFT) to explore the shared and specific features between different modalities, where a few learnable bottleneck tokens are utilized for knowledge transfer between modalities. Moreover, a modal adaptation and orthogonal projection scheme were incorporated to further encourage BSFT to learn shared and specific features from multi-modal data. Finally, the shared and specific features are dynamically aggregated via an attention mechanism for slide-level prediction. Experimental results on our collected lymph node metastasis dataset demonstrate the efficiency of our proposed components and our framework achieves the best performance with AUC (area under the curve) of 97.34%, outperforming the state-of-the-art methods by over 1.27%.

14P Classification of molecular subtypes of breast cancer in whole-slide histopathological images using a novel deep learning algorithm

14P Classification of molecular subtypes of breast cancer in whole-slide histopathological images using a novel deep learning algorithm

Comparative Study of Deep Learning Algorithms for Classification of Lung Cancer Types from Histopathological Images

Abstract: Global cancer statistics shows that Lung cancer is the 2nd most common cancer and the leading cause of cancerrelated mortality worldwide in 2020.Histopathology image analysis act as the prominent technique for cancer diagnosis. The assessment of whole-slide histopathology images remains a limiting factor for timely treatments. The automation of histopathology analysis is a much-needed solution to alleviate the burden of workload and tackle the problem of medical personnel scarcity in underserved regions and populations. To help make the classification of cancer-type techniques less formidable to at least a certain extent, this report presents a comparative analysis of deep learning algorithms for the efficient classification of lung cancer types. We have compared six different pre-trained Convolutional neural networks (CNN) to find the one best suitable for the classification of lung cancer types. The models include ResNet50 , VGG-16 , EfficientNet-B0, InceptionV3 , DenseNet121, and NasNetLarge . The models were trained using histopathology images from dataset LC25000. 4 different evaluation metrics, which are accuracy, precision, recall, and F1-score were used. We could achieve the highest accuracy using EfficientNetB0, which was 99.77%, while ResNet50 yielded an accuracy of 99.66% and VGG-16 gave an accuracy of 93.50%. Overall, we can say that this research can aid in implementing models that can increase the efficiency and accuracy in the lung cancer type classification in biomedical fields.

Identification of lymph node metastasis in pre-operation cervical cancer patients by weakly supervised deep learning from histopathological whole-slide biopsy images.

Lymph node metastasis (LNM) significantly impacts the prognosis of individuals diagnosed with cervical cancer, as it is closely linked to disease recurrence and mortality, thereby impacting therapeutic schedule choices for patients. However, accurately predicting LNM prior to treatment remains challenging. Consequently, this study seeks to utilize digital pathological features extracted from histopathological slides of primary cervical cancer patients to preoperatively predict the presence of LNM. A deep learning (DL) model was trained using the Vision transformer (ViT) and recurrent neural network (RNN) frameworks to predict LNM. This prediction was based on the analysis of 554 histopathological whole-slide images (WSIs) obtained from Qilu Hospital of Shandong University. To validate the model's performance, an external test was conducted using 336 WSIs from four other hospitals. Additionally, the efficiency of the DL model was evaluated using 190 cervical biopsies WSIs in a prospective set. In the internal test set, our DL model achieved an area under the curve (AUC) of 0.919, with sensitivity and specificity values of 0.923 and 0.905, respectively, and an accuracy (ACC) of 0.909. The performance of the DL model remained strong in the external test set. In the prospective cohort, the AUC was 0.91, and the ACC was 0.895. Additionally, the DL model exhibited higher accuracy compared to imaging examination in the evaluation of LNM. By utilizing the transformer visualization method, we generated a heatmap that illustrates the local pathological features in primary lesions relevant to LNM. DL-based image analysis has demonstrated efficiency in predicting LNM in early operable cervical cancer through the utilization of biopsies WSI. This approach has the potential to enhance therapeutic decision-making for patients diagnosed with cervical cancer.

Transcriptional intra-tumour heterogeneity predicted by deep learning in routine breast histopathology slides provides independent prognostic information

BackgroundIntra-tumour heterogeneity (ITH) causes diagnostic challenges and increases the risk for disease recurrence. Quantification of ITH is challenging and has not been demonstrated in large studies. It has previously been shown that deep learning can enable spatially resolved prediction of molecular phenotypes from digital histopathology whole slide images (WSIs). Here we propose a novel method (Deep-ITH) to predict and measure ITH, and we evaluate its prognostic performance in breast cancer. MethodsDeep convolutional neural networks were used to spatially predict gene-expression (PAM50 set) from WSIs. For each predicted transcript, 12 measures of heterogeneity were extracted in the training data set (N = 931). A prognostic score to dichotomise patients into Deep-ITH low- and high-risk groups was established using an elastic-net regularised Cox proportional hazards model (recurrence-free survival). Prognostic performance was evaluated in two independent data sets: SöS-BC-1 (N = 1358) and SCAN-B-Lund (N = 1262). ResultsWe observed an increase in risk of recurrence in the high-risk group with hazard ratio (HR) 2.11 (95%CI:1.22–3.60; p = 0.007) using nested cross-validation. Subgroup analyses confirmed the prognostic performance in oestrogen receptor (ER)-positive, human epidermal growth factor receptor 2 (HER2)-negative, grade 3, and large tumour subgroups. The prognostic value was confirmed in the independent SöS-BC-1 cohort (HR=1.84; 95%CI:1.03–3.3; p = 3.99 ×10−2). In the other external cohort, significant HR was observed in the subgroup of histological grade 2 patients, as well as in the subgroup of patients with small tumours (<20 mm). ConclusionWe developed a novel method for an automated, scalable, and cost-efficient measure of ITH from WSIs that provides independent prognostic value for breast cancer. SignificanceTranscriptional ITH predicted by deep learning models enables prediction of patient survival from routine histopathology WSIs in breast cancer.

Deep learning for multi-class semantic segmentation enables colorectal cancer detection and classification in digital pathology images

In colorectal cancer (CRC), artificial intelligence (AI) can alleviate the laborious task of characterization and reporting on resected biopsies, including polyps, the numbers of which are increasing as a result of CRC population screening programs ongoing in many countries all around the globe. Here, we present an approach to address two major challenges in the automated assessment of CRC histopathology whole-slide images. We present an AI-based method to segment multiple (n=14) tissue compartments in the H &E-stained whole-slide image, which provides a different, more perceptible picture of tissue morphology and composition. We test and compare a panel of state-of-the-art loss functions available for segmentation models, and provide indications about their use in histopathology image segmentation, based on the analysis of (a) a multi-centric cohort of CRC cases from five medical centers in the Netherlands and Germany, and (b) two publicly available datasets on segmentation in CRC. We used the best performing AI model as the basis for a computer-aided diagnosis system that classifies colon biopsies into four main categories that are relevant pathologically. We report the performance of this system on an independent cohort of more than 1000 patients. The results show that with a good segmentation network as a base, a tool can be developed which can support pathologists in the risk stratification of colorectal cancer patients, among other possible uses. We have made the segmentation model available for research use on https://grand-challenge.org/algorithms/colon-tissue-segmentation/.

A correlation graph attention network for classifying chromosomal instabilities from histopathology whole-slide images

The chromosome instability (CIN) is one of the hallmarks of cancer and is closely related to tumor metastasis. However, the sheer size and resolution of histopathology whole-slide images (WSIs) already challenges the capabilities of computational pathology. In this study, we propose a correlation graph attention network (MLP-GAT) that can construct graphs for classifying multi-type CINs from the WSIs of breast cancer. We construct a WSIs dataset of breast cancer from the Cancer Genome Atlas Breast Invasive Carcinoma (TCGA-BRCA). Extensive experiments show that MLP-GAT far outperforms accepted state-of-the-art methods and demonstrate the advantages of the constructed graph networks for analyzing WSI data. The visualization shows the difference among the tiles in a WSI. Furthermore, the generalization performance of the proposed method was verified on the stomach cancer. This study provides guidance for studying the relationship between CIN and cancer from the perspective of image phenotype.

Weakly supervised histopathology image segmentation with self-attention.

Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.

Deep learning-based semantic segmentation of non-melanocytic skin tumors in whole-slide histopathological images.

Basal cell carcinoma (BCC) and squamous cell carcinoma (SCC) are the two most common skin cancer and impose a huge medical burden on society. Histopathological examination based on whole-slide images (WSIs) remains to be the confirmatory diagnostic method for skin tumors. Accurate segmentation of tumor tissue in WSIs by deep-learning (DL) models can reduce the workload of pathologists and help surgeons ensure the complete removal of tumors. To accurately segment the tumor areas in WSIs of BCC, SCC and squamous cell papilloma (SCP, homologous to SCC) with robust models. We established a data set (ZJU-NMSC) containing 151 WSIs of BCC, SCC and SCP in total. Seven models were utilized to segment WSIs, including the state-of-the-art model, models proposed by us and other models. Dice score, intersection over union, accuracy, sensitivity and specificity were used to evaluate and compare the performance of different models. Heatmaps and tumor tissue masks were generated to reflect the results of the segmentation. The processing times of models are also recorded and compared. While the dice score of most models is higher than 0.85, deeplab v3+ has the best performance and the corresponding tumor tissue mask is more consistent with the ground truth tumor areas even with complex and small lobular lesions. This study broadens the use of DL-based segmentation models in WSIs of skin tumors in terms of tumor types and computational approaches. Segmenting tumor areas can simplify the process of histopathological inspection and benefit the diagnosis and following management of the diseases in practice.