Whole-slide Images Research Articles

Real-time segmentation and classification of whole-slide images for tumor biomarker scoring

Histopathology image segmentation and classification are essential for diagnosing and treating breast cancer. This study introduced a highly accurate segmentation and classification for histopathology images using a single architecture. We utilized the famous segmentation architectures, SegNet and U-Net, and modified the decoder to attach ResNet, VGG and DenseNet to perform classification tasks. These hybrid models are integrated with Stardist as the backbone, and implemented in a real-time pathologist workflow with a graphical user interface. These models were trained and tested offline using the ER-IHC-stained private and H&E-stained public datasets (MoNuSeg). For real-time evaluation, the proposed model was evaluated using PR-IHC-stained glass slides. It achieved the highest segmentation pixel-based F1-score of 0.902 and 0.903 for private and public datasets respectively, and a classification-based F1-score of 0.833 for private dataset. The experiment shows the robustness of our method where a model trained on ER-IHC dataset able to perform well on real-time microscopy of PR-IHC slides on both 20x and 40x magnification. This will help the pathologists with a quick decision-making process.

CytoNet: an efficient dual attention based automatic prediction of cancer sub types in cytology studies

Computer-assisted diagnosis (CAD) plays a key role in cancer diagnosis or screening. Whereas, current CAD performs poorly on whole slide image (WSI) analysis, and thus fails to generalize well. This research aims to develop an automatic classification system to distinguish between different types of carcinomas. Obtaining rich deep features in multi-class classification while achieving high accuracy is still a challenging problem. The detection and classification of cancerous cells in WSI are quite challenging due to the misclassification of normal lumps and cancerous cells. This is due to cluttering, occlusion, and irregular cell distribution. Researchers in the past mostly obtained the hand-crafted features while neglecting the above-mentioned challenges which led to a reduction of the classification accuracy. To mitigate this problem we proposed an efficient dual attention-based network (CytoNet). The proposed network is composed of two main modules (i) Efficient-Net and (ii) Dual Attention Module (DAM). Efficient-Net is capable of obtaining higher accuracy and enhancing efficiency as compared to existing Convolutional Neural Networks (CNNs). It is also useful to obtain the most generic features as it has been trained on ImageNet. Whereas DAM is very robust in obtaining attention and targeted features while negating the background. In this way, the combination of an efficient and attention module is useful to obtain the robust, and intrinsic features to obtain comparable performance. Further, we evaluated the proposed network on two well-known datasets (i) Our generated thyroid dataset (ii) Mendeley Cervical dataset (Hussain in Data Brief, 2019) with enhanced performance compared to their counterparts. CytoNet demonstrated a 99% accuracy rate on the thyroid dataset in comparison to its counterpart. The precision, recall, and F1-score values achieved on the Mendeley Cervical dataset are 0.992, 0.985, and 0.977, respectively. The code implementation is available on GitHub. https://github.com/naveedilyas/CytoNet-An-Efficient-Dual-Attention-based-Automatic-Prediction-of-Cancer-Sub-types-in-Cytol.

Deep Convolutional Neural Network for Accurate Classification of Myofibroblastic Lesions on Patch-Based Images.

This study aimed to implement and evaluate a Deep Convolutional Neural Network for classifying myofibroblastic lesions into benign and malignant categories based on patch-based images. A Residual Neural Network (ResNet50) model, pre-trained with weights from ImageNet, was fine-tuned to classify a cohort of 20 patients (11 benign and 9 malignant cases). Following annotation of tumor regions, the whole-slide images (WSIs) were fragmented into smaller patches (224 × 224 pixels). These patches were non-randomly divided into training (308,843 patches), validation (43,268 patches), and test (42,061 patches) subsets, maintaining a 78:11:11 ratio. The CNN training was caried out for 75 epochs utilizing a batch size of 4, the Adam optimizer, and a learning rate of 0.00001. ResNet50 achieved an accuracy of 98.97%, precision of 99.91%, sensitivity of 97.98%, specificity of 99.91%, F1 score of 98.94%, and AUC of 0.99. The ResNet50 model developed exhibited high accuracy during training and robust generalization capabilities in unseen data, indicating nearly flawless performance in distinguishing between benign and malignant myofibroblastic tumors, despite the small sample size. The excellent performance of the AI model in separating such histologically similar classes could be attributed to its ability to identify hidden discriminative features, as well as to use a wide range of features and benefit from proper data preprocessing.

Artificial Intelligence in Head and Neck Cancer Diagnosis: A Comprehensive Review with Emphasis on Radiomics, Histopathological, and Molecular Applications

The present review discusses the transformative role of AI in the diagnosis and management of head and neck cancers (HNCs). Methods: It explores how AI technologies, including ML, DL, and CNNs, are applied in various diagnostic tasks, such as medical imaging, molecular profiling, and predictive modeling. Results: This review highlights AI’s ability to improve diagnostic accuracy and efficiency, particularly in analyzing medical images like CT, MRI, and PET scans, where AI sometimes outperforms human radiologists. This paper also emphasizes AI’s application in histopathology, where algorithms assist in whole-slide image (WSI) analysis, tumor-infiltrating lymphocytes (TILs) quantification, and tumor segmentation. AI shows promise in identifying subtle or rare histopathological patterns and enhancing the precision of tumor grading and treatment planning. Furthermore, the integration of AI with molecular and genomic data aids in mutation analysis, prognosis, and personalized treatment strategies. Conclusions: Despite these advancements, the review identifies challenges in AI adoption, such as data standardization and model interpretability, and calls for further research to fully integrate AI into clinical practice for improved patient outcomes.

Development and Evaluation of a Convolutional Neural Network for Microscopic Diagnosis Between Pleomorphic Adenoma and Carcinoma Ex-Pleomorphic Adenoma.

To develop a model capable of distinguishing carcinoma ex-pleomorphic adenoma from pleomorphic adenoma using a convolutional neural network architecture. A cohort of 83 Brazilian patients, divided into carcinoma ex-pleomorphic adenoma (n = 42) and pleomorphic adenoma (n = 41), was used for training a convolutional neural network. The whole-slide images were annotated and fragmented into 743 869 (carcinoma ex-pleomorphic adenomas) and 211 714 (pleomorphic adenomas) patches, measuring 224 × 224 pixels. Training (80%), validation (10%), and test (10%) subsets were established. The Residual Neural Network (ResNet)-50 was chosen for its recognition and classification capabilities. The training and validation graphs, and parameters derived from the confusion matrix, were evaluated. The loss curve recorded 0.63, and the accuracy reached 0.93. Evaluated parameters included specificity (0.88), sensitivity (0.94), precision (0.96), F1 score (0.95), and area under the curve (0.97). The study underscores the potential of ResNet-50 in the microscopic diagnosis of carcinoma ex-pleomorphic adenoma. The developed model demonstrated strong learning potential, but exhibited partial limitations in generalization, as indicated by the validation curve. In summary, the study established a promising baseline despite limitations in model generalization. This indicates the need to refine methodologies, investigate new models, incorporate larger datasets, and encourage inter-institutional collaboration for comprehensive studies in salivary gland tumors.

Artificial Intelligence-Based Sentinel Lymph Node Metastasis Detection in Cervical Cancer

Background/objectives: Pathological ultrastaging, an essential part of sentinel lymph node (SLN) mapping, involves serial sectioning and immunohistochemical (IHC) staining in order to reliably detect clinically relevant metastases. However, ultrastaging is labor-intensive, time-consuming, and costly. Deep learning algorithms offer a potential solution by assisting pathologists in efficiently assessing serial sections for metastases, reducing workload and costs while enhancing accuracy. This proof-of-principle study evaluated the effectiveness of a deep learning algorithm for SLN metastasis detection in early-stage cervical cancer. Methods: We retrospectively analyzed whole slide images (WSIs) of hematoxylin and eosin (H&E)-stained SLNs from early-stage cervical cancer patients diagnosed with an SLN metastasis with either H&E or IHC. A CE-IVD certified commercially available deep learning algorithm, initially developed for detection of breast and colon cancer lymph node metastases, was employed off-label to assess its sensitivity in cervical cancer. Results: This study included 21 patients with early-stage cervical cancer, comprising 15 with squamous cell carcinoma, five with adenocarcinoma, and one with clear cell carcinoma. Among these patients, 10 had macrometastases and 11 had micrometastases in at least one SLN. The algorithm was applied to evaluate H&E WSIs of 47 SLN specimens, including 22 that were negative for metastasis, 13 with macrometastases, and 12 with micrometastases in the H&E slides. The algorithm detected all H&E macro- and micrometastases with 100% sensitivity. Conclusions: This proof-of-principle study demonstrated high sensitivity of a deep learning algorithm for detection of clinically relevant SLN metastasis in early-stage cervical cancer, despite being originally developed for adenocarcinomas of the breast and colon. Our findings highlight the potential of leveraging an existing algorithm for use in cervical cancer, warranting further prospective validation in a larger population.

Computational approach for counting of SISH amplification signals for HER2 status assessment

The human epidermal growth factor receptor 2 (HER2) gene is a critical biomarker for determining amplification status and targeting clinical therapies in breast cancer treatment. This study introduces a computer-aided method that automatically measures and scores HER2 gene status from invasive tissue regions of breast cancer using whole slide images (WSI) through silver in situ hybridization (SISH) staining. Image processing and deep learning techniques are employed to isolate untruncated and non-overlapping single nuclei from cancer regions. The Stardist deep learning model is fine-tuned on our HER2-SISH data to identify nuclei regions, followed by post-processing based on identified HER2 and CEP17 signals. Conventional thresholding techniques are used to segment HER2 and CEP17 signals. HER2 amplification status is determined by calculating the HER2-to-CEP17 signal ratio, in accordance with ASCO/CAP 2018 standards. The proposed method significantly reduces the effort and time required for quantification. Experimental results demonstrate a 0.91% correlation coefficient between pathologists manual enumeration and the proposed automatic SISH quantification approach. A one-sided paired t-test confirmed that the differences between the outcomes of the proposed method and the reference standard are statistically insignificant, with p-values exceeding 0.05. This study illustrates how deep learning can effectively automate HER2 status determination, demonstrating improvements over current manual methods and offering a robust, reproducible alternative for clinical practice.

Whole slide cervical cancer classification via graph attention networks and contrastive learning

Cervical cancer is one of the most common cancers among women, which seriously threatens women’s health. Early screening can reduce the incidence rate and mortality. Thinprep cytologic test (TCT) is one of the important means of cytological screening, which has high sensitivity and specificity, and has been widely used in the early screening of cervical cancer. The automatic diagnosis of whole slide images (WSIs) by computers can effectively improve the efficiency and accuracy of doctors’ diagnoses. However, current methods ignore the intrinsic relationships between cervical cells in WSIs and neglect contextual information from the surrounding suspicious areas, and therefore limit their robustness and generalizability. In this paper, we propose a novel two-stage method to implement the automatic diagnosis of WSIs, which constructs Graph Attention Networks (GAT) based on local and global fields respectively to capture their contextual information in a hierarchical manner. In the first stage, we extract representative patches from each WSI through suspicious cell detection, and then employ a Local GAT to classify cervical cells by capturing correlations between suspicious cells in image tiles. This classification process provides the confidence and feature vectors for each suspicious cell. In the second stage, we perform WSI classification using a Global GAT model. We construct graphs corresponding to top-Kg and bottom-Kg cells for each WSI based on results from Local GAT, and introduce a supervised contrastive learning strategy to enhance the discriminative power of the extracted features. Experimental results demonstrate that our proposed method outperforms conventional approaches and effectively showcases the benefits of supervised contrastive learning. Our source code and example data are available at https://github.com/feimanman/Whole-Slide-Cervical-Cancer-Classification.

A multi-model approach integrating whole-slide imaging and clinicopathologic features to predict breast cancer recurrence risk.

Breast cancer is the most common malignancy affecting women worldwide and is notable for its morphologic and biologic diversity, with varying risks of recurrence following treatment. The Oncotype DX Breast Recurrence Score test is an important predictive and prognostic genomic assay for estrogen receptor positive/HER2 negative breast cancer that guides therapeutic strategies; however, such tests can be expensive, delay care, and are not widely available. The aim of this study was to develop a multi-model approach integrating the analysis of whole-slide images and clinicopathologic data to predict their associated breast cancer recurrence risks and categorize these patients into two risk groups according to the predicted score: low-risk and high-risk. The proposed novel methodology uses convolutional neural networks for feature extraction and vision transformers for contextual aggregation, complemented by a logistic regression model that analyzes clinicopathologic data for classification into two risk categories. This method was trained and tested on 956 hematoxylin and eosin-stained whole-slide images of 950 ER+/HER2- breast cancer patients with corresponding clinicopathological features that had prior Oncotype DX testing. The model's performance was evaluated using an internal test set of 192 patients from Dartmouth Health and an external test set of 405 patients from the University of Chicago. The multi-model approach achieved an AUC of 0.91 (95% CI: 0.87-0.95) on the internal set and an AUC of 0.84 (95% CI: 0.78-0.89) on the external cohort for predicting low- and high-breast cancer recurrence risk categories based on the Oncotype DX recurrence score. With further validation, the proposed methodology could provide an alternative to assist clinicians in personalizing treatment for breast cancer patients and potentially improving their outcomes.

Comparative analysis of tertiary lymphoid structures for predicting survival of colorectal cancer: A whole-slide images-based study

Abstract Background Tertiary lymphoid structures (TLS) are major component in immune microenvironment, correlating with a favorable prognosis in colorectal cancer. However, the methods used to define and characterize TLS were not united, hindering its's clinical application. This study aims to seek a more stable method to characterize TLS and clarify its prognostic value in larger multicenter cohorts. Methods A total of 1609 patients from four hospitals and the TCGA database were analyzed. We quantified the number and maximum length of TLS along the invasive margin of tumor using hematoxylin and eosin (HE)-stained whole-slide images (WSIs). Additionally, the length of the invasive margin was determined to calculate the TLS density. The prognostic value of the TLS for overall survival was evaluated. In addition, we examined the association between TLS density and immune cell infiltration using immunohistochemistry-stained WSIs. The performance for predicting overall survival was measured using hazard ratios (HR) with 95% confidence intervals (CI). Results Among the three TLS quantification methods, TLS density has the strongest discriminative performance. Survival analysis indicated that higher TLS density correlated with better overall survival (HR for high vs. low 0.57 [95% CI 0.42–0.78] in primary cohort; 0.49 [0.35–0.69] in validation cohort; 0.35 [0.18–0.67] in TCGA cohort). A high TLS density was associated with a high level of CD3+ T cells infiltration. Conclusions Based on this comparative multicenter analysis, TLS density was identified as a simple, robust, and effective immune prognostic index for colorectal cancer.

Artificial intelligence can be trained to predict c-KIT-11 mutational status of canine mast cell tumors from hematoxylin and eosin-stained histological slides.

Numerous prognostic factors are currently assessed histologically and immunohistochemically in canine mast cell tumors (MCTs) to evaluate clinical behavior. In addition, polymerase chain reaction (PCR) is often performed to detect internal tandem duplication (ITD) mutations in exon 11 of the c-KIT gene (c-KIT-11-ITD) to predict the therapeutic response to tyrosine kinase inhibitors. This project aimed at training deep learning models (DLMs) to identify MCTs with c-KIT-11-ITD solely based on morphology. Hematoxylin and eosin (HE) stained slides of 368 cutaneous, subcutaneous, and mucocutaneous MCTs (195 with ITD and 173 without) were stained consecutively in 2 different laboratories and scanned with 3 different slide scanners. This resulted in 6 data sets (stain-scanner variations representing diagnostic institutions) of whole-slide images. DLMs were trained with single and mixed data sets and their performances were assessed under stain-scanner variations (domain shifts). The DLM correctly classified HE slides according to their c-KIT-11-ITD status in up to 87% of cases with a 0.90 sensitivity and a 0.83 specificity. A relevant performance drop could be observed when the stain-scanner combination of training and test data set differed. Multi-institutional data sets improved the average accuracy but did not reach the maximum accuracy of algorithms trained and tested on the same stain-scanner variant (ie, intra-institutional). In summary, DLM-based morphological examination can predict c-KIT-11-ITD with high accuracy in canine MCTs in HE slides. However, staining protocol and scanner type influence accuracy. Larger data sets of scans from different laboratories and scanners may lead to more robust DLMs to identify c-KIT mutations in HE slides.

Application of a Modified SegFormer Architecture for Gleason Pattern Segmentation

A key element of morphological assessment in prostate cancer (PCa) diagnostics is the Gleason score, which is based on the architecture of tumor growth in biopsy or prostatectomy samples. During this process, biopsies from the prostate's pathological areas are evaluated to identify the most common and severe architectural patterns (referred to as Gleason patterns), each of which is assigned a score in the Gleason grading system. The resulting total score serves as a critical predictor for disease progression and metastasis. This research focuses on enhancing the efficiency of morphological examination in PCa by utilizing automatic segmentation of Gleason patterns in whole slide images (WSI) in *.svs format, with 40x magnification and an average spatial resolution of 0.258 microns per pixel. Segmentation was performed using a neural network with a SegFormer architecture, supplemented by an EdgeNeXt (small variant) subnet. The proposed solution successfully identified Gleason patterns of grades 3, 4, and 5. The best model performance for detecting combined Gleason patterns 3, 4, 5, and 4, 5 was achieved with input images sized 512x512 (tile size) and a batch size of 8, resulting in an F-score of 0.835 and 0.769, respectively.

Concordance in the estimation of tumor percentage in non-small cell lung cancer using digital pathology

The incorporation of digital pathology in clinical practice will require the training of pathologists in digital skills. Our study aimed to assess the reliability among pathologists in determining tumor percentage in whole slide images (WSI) of non-small cell lung cancer (NSCLC) using digital image analysis, and study how the results correlate with the molecular findings. Pathologists from nine centers were trained to quantify epithelial tumor cells, tumor-associated stromal cells, and non-neoplastic cells from NSCLC WSI using QuPath. Then, we conducted two consecutive ring trials. In the first trial, analyzing four WSI, reliability between pathologists in the assessment of tumor cell percentage was poor (intraclass correlation coefficient (ICC) 0.09). After performing the first ring trial pathologists received feedback. The second trial, comprising 10 WSI with paired next-generation sequencing results, also showed poor reliability (ICC 0.24). Cases near the recommended 20% visual threshold for molecular techniques exhibited higher values with digital analysis. In the second ring trial reliability slightly improved and human errors were reduced from 5.6% to 1.25%. Most discrepancies arose from subjective tasks, such as the annotation process, suggesting potential improvement with future artificial intelligence solutions.

Method for co-registration of high-resolution specimen PET-CT with histopathology to improve insight into radiotracer distributions

BackgroundAs the spatial resolution of positron emission tomography (PET) scanners improves, understanding of radiotracer distributions in tissues at high resolutions is important. Hence, we propose a method for co-registration of high-resolution ex vivo specimen PET images, combined with computed tomography (CT) images, and the corresponding specimen histopathology.MethodsWe applied our co-registration method to breast cancer (BCa) specimens of patients who were preoperatively injected with 0.8 MBq/kg [18\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{18}$$\\end{document}F]fluorodeoxyglucose ([18F]FDG). The method has two components. First, we used an image acquisition scheme that minimises and tracks tissue deformation: (1) We acquired sub-millimetre (micro)-PET-CT images of ±2 mm-thick lamellas of the fresh specimens, enclosed in tissue cassettes. (2) We acquired micro-CT images of the same lamellas after formalin fixation to visualise tissue deformation. (3) We obtained 1 hematoxylin and eosin (H&E) stained histopathology section per lamella of which we captured a digital whole slide image (WSI). Second, we developed an automatic co-registration algorithm to improve the alignment between the micro-PET-CT images and WSIs, guided by the micro-CT of the fixated lamellas. To estimate the spatial co-registration error, we calculated the distance between corresponding microcalcifications in the micro-CTs and WSIs. The co-registered images allowed to study standardised uptake values (SUVs) of different breast tissues, as identified on the WSIs by a pathologist.ResultsWe imaged 22 BCa specimens, 13 cases of invasive carcinoma of no special type (NST), 6 of invasive lobular carcinoma (ILC), and 3 of ductal carcinoma in situ (DCIS). While the cassette framework minimised tissue deformation, the best alignment between the micro-PET-CT images and WSIs was achieved after deformable co-registration. We found an overall average co-registration error of 0.74 ± 0.17 mm between the micro-PET images and WSIs. (Pre)malignant tissue (including NST, ILC, and DCIS) generally showed higher SUVs than healthy tissue (including healthy glandular, connective, and adipose tissue). As expected, inflamed tissue and skin also showed high uptake.ConclusionsWe developed a method to co-register micro-PET-CT images of surgical specimens and WSIs with an accuracy comparable to the spatial resolution of the micro-PET images. While currently, we only applied this method to BCa specimens, we believe this method is applicable to a wide range of specimens and radiotracers, providing insight into distributions of (new) radiotracers in human malignancies at a sub-millimetre resolution.

Artificial intelligence-based quantification of lymphocytes in feline small intestinal biopsies.

Feline chronic enteropathy is a poorly defined condition of older cats that encompasses chronic enteritis to low-grade intestinal lymphoma. The histological evaluation of lymphocyte numbers and distribution in small intestinal biopsies is crucial for classification and grading. However, conventional histological methods for lymphocyte quantification have low interobserver agreement, resulting in low diagnostic reliability. This study aimed to develop and validate an artificial intelligence (AI) model to detect intraepithelial and lamina propria lymphocytes in hematoxylin and eosin-stained small intestinal biopsies from cats. The median sensitivity, positive predictive value, and F1 score of the AI model compared with the majority opinion of 11 veterinary anatomic pathologists, were 100% (interquartile range [IQR] 67%-100%), 57% (IQR 38%-83%), and 67% (IQR 43%-80%) for intraepithelial lymphocytes, and 89% (IQR 71%-100%), 67% (IQR 50%-82%), and 70% (IQR 43%-80%) for lamina propria lymphocytes, respectively. Errors included false negatives in whole-slide images with faded stain and false positives in misidentifying enterocyte nuclei. Semiquantitative grading at the whole-slide level showed low interobserver agreement among pathologists, underscoring the need for a reproducible quantitative approach. While semiquantitative grade and AI-derived lymphocyte counts correlated positively, the AI-derived lymphocyte counts overlapped between different grades. Our AI model, when supervised by a pathologist, offers a reproducible, objective, and quantitative assessment of feline intestinal lymphocytes at the whole-slide level, and has the potential to enhance diagnostic accuracy and consistency for feline chronic enteropathy.

Application of artificial intelligence model in pathological staging and prognosis of clear cell renal cell carcinoma

This study aims to develop a deep learning (DL) model based on whole-slide images (WSIs) to predict the pathological stage of clear cell renal cell carcinoma (ccRCC). The histopathological images of 513 ccRCC patients were downloaded from The Cancer Genome Atlas (TCGA) database and randomly divided into training set and validation set according to the ratio of 8∶2. The CLAM algorithm was used to establish the DL model, and the stability of the model was evaluated in the external validation set. DL features were extracted from the model to construct a prognostic risk model, which was validated in an external dataset. The results showed that the DL model showed excellent prediction ability with an area under the curve (AUC) of 0.875 and an average accuracy score of 0.809, indicating that the model could reliably distinguish ccRCC patients at different stages from histopathological images. In addition, the prognostic risk model constructed by DL characteristics showed that the overall survival rate of patients in the high-risk group was significantly lower than that in the low-risk group (P = 0.003), and AUC values for predicting 1-, 3- and 5-year overall survival rates were 0.68, 0.69 and 0.69, respectively, indicating that the prediction model had high sensitivity and specificity. The results of the validation set are consistent with the above results. Therefore, DL model can accurately predict the pathological stage and prognosis of ccRCC patients, and provide certain reference value for clinical diagnosis.

Vision transformer introduces a new vitality to the classification of renal pathology

Recent advancements in computer vision within the field of artificial intelligence (AI) have made significant inroads into the medical domain. However, the application of AI for classifying renal pathology remains challenging due to the subtle variations in multiple renal pathological classifications. Vision Transformers (ViT), an adaptation of the Transformer model for image recognition, have demonstrated superior capabilities in capturing global features and providing greater explainability. In our study, we developed a ViT model using a diverse set of stained renal histopathology images to evaluate its effectiveness in classifying renal pathology. A total of 1861 whole slide images (WSI) stained with HE, MASSON, PAS, and PASM were collected from 635 patients. Renal tissue images were then extracted, tiled, and categorized into 14 classes on the basis of renal pathology. We employed the classic ViT model from the Timm library, utilizing images sized 384 × 384 pixels with 16 × 16 pixel patches, to train the classification model. A comparative analysis was conducted to evaluate the performance of the ViT model against traditional convolutional neural network (CNN) models. The results indicated that the ViT model demonstrated superior recognition ability (accuracy: 0.96–0.99). Furthermore, we visualized the identification process of the ViT models to investigate potentially significant pathological ultrastructures. Our study demonstrated that ViT models outperformed CNN models in accurately classifying renal pathology. Additionally, ViT models are able to focus on specific, significant structures within renal histopathology, which could be crucial for identifying novel and meaningful pathological features in the diagnosis and treatment of renal disease.

Pathologist-level diagnosis of ulcerative colitis inflammatory activity level using an automated histological grading method

Background and AimsInflammatory bowel disease (IBD) is a global disease that is evolving with increasing incidence. However, there are few works on computationally assisted diagnosis of IBD based on pathological images. Therefore, based on the UK and Chinese IBD diagnostic guidelines, our study established an artificial intelligence-assisted diagnostic system for histologic grading of inflammatory activity in ulcerative colitis (UC). MethodsWe proposed an efficient deep-learning (DL) method for grading inflammatory activity in whole-slide images (WSIs) of UC pathology. Our model was constructed using 603 UC WSIs from Nanjing Drum Tower Hospital for model train set and internal test set. We collected 212 UC WSIs from Zhujiang Hospital as an external test set. Initially, the pre-trained ResNet50 model on the ImageNet dataset was employed to extract image patch features from UC patients. Subsequently, a multi-instance learning (MIL) approach with embedded self-attention was utilized to aggregate tissue image patch features, representing the entire WSI. Finally, the model was trained based on the aggregated features and WSI annotations provided by senior gastrointestinal pathologists to predict the level of inflammatory activity in UC WSIs. ResultsIn the task of distinguishing the presence or absence of inflammatory activity, the Area Under Curve (AUC) value in the internal test set is 0.863 (95% confidence interval [CI] 0.829, 0.898), with a sensitivity of 0.913 (95% [CI] 0.866, 0.961), and specificity of 0.816 (95% [CI] 0.771, 0.861). The AUC in the external test set is 0.947 (95% confidence interval [CI] 0.939, 0.955), with a sensitivity of 0.889 (905% [CI] 0.837, 0.940), and specificity of 0.858 (95% [CI] 0.777, 0.939). For distinguishing different levels of inflammatory activity in UC, the average Macro-AUC in the internal test set and the external test set are 0.827 (95% [CI] 0.803, 0.850) and 0.908 (95% [CI] 0.882, 0.935). the average Micro-AUC in the internal test set and the external test set are 0.816 (95% [CI] 0.792, 0.840) and 0.898 (95% [CI] 0.869, 0.926). ConclusionsComparative analysis with diagnoses made by pathologists at different expertise levels revealed that the algorithm reached a proficiency comparable to the pathologist with 5 years of experience. Furthermore, our algorithm performed superior to other MIL algorithms.

Predicting Response to Intravesical BCG in High-Risk NMIBC Using an Artificial Intelligence-Powered Pathology Assay: Development and Validation in an International 12-Center Cohort.

There are few markers to identify those likely to recur or progress after treatment with intravesical bacillus Calmette-Guérin (BCG). We developed and validated artificial intelligence-based histologic assays that extract interpretable features from transurethral resection of bladder tumor digitized pathology images to predict risk of recurrence, progression, development of BCG-unresponsive disease, and cystectomy. Pre-BCG resection-derived whole-slide images and clinical data were obtained for high-risk NMIBC cases treated with BCG from 12 centers and were analyzed through a segmentation and feature extraction pipeline. Features associated with clinical outcomes were defined and tested on independent development and validation cohorts. Cases were classified into high or low risk for recurrence, progression, BCG-unresponsive disease, and cystectomy. Nine hundred forty-four cases (development: 303, validation: 641, median follow-up: 36 months) representative of the intended use population were included (high-grade Ta: 34.1%, high-grade T1: 54.8%; carcinoma in situ only: 11.1%, any carcinoma in situ: 31.4%). In the validation cohort, "high recurrence risk" cases had inferior high-grade recurrence-free survival vs "low recurrence risk" cases (HR, 2.08, P < .0001). "High progression risk" patients had poorer progression-free survival (HR, 3.87, P < .001) and higher risk of cystectomy (HR, 3.35, P < .001) than "low progression risk" patients. Cases harboring the BCG-unresponsive disease signature had a shorter time to development of BCG-unresponsive disease than cases without the signature (HR, 2.31, P < .0001). AI assays provided predictive information beyond clinicopathologic factors. We developed and validated AI-based histologic assays that identify high-risk NMIBC cases at higher risk of recurrence, progression, BCG-unresponsive disease, and cystectomy, potentially aiding clinical decision making.

Equipping computational pathology systems with artifact processing pipelines: a showcase for computation and performance trade-offs

BackgroundHistopathology is a gold standard for cancer diagnosis. It involves extracting tissue specimens from suspicious areas to prepare a glass slide for a microscopic examination. However, histological tissue processing procedures result in the introduction of artifacts, which are ultimately transferred to the digitized version of glass slides, known as whole slide images (WSIs). Artifacts are diagnostically irrelevant areas and may result in wrong predictions from deep learning (DL) algorithms. Therefore, detecting and excluding artifacts in the computational pathology (CPATH) system is essential for reliable automated diagnosis.MethodsIn this paper, we propose a mixture of experts (MoE) scheme for detecting five notable artifacts, including damaged tissue, blur, folded tissue, air bubbles, and histologically irrelevant blood from WSIs. First, we train independent binary DL models as experts to capture particular artifact morphology. Then, we ensemble their predictions using a fusion mechanism. We apply probabilistic thresholding over the final probability distribution to improve the sensitivity of the MoE. We developed four DL pipelines to evaluate computational and performance trade-offs. These include two MoEs and two multiclass models of state-of-the-art deep convolutional neural networks (DCNNs) and vision transformers (ViTs). These DL pipelines are quantitatively and qualitatively evaluated on external and out-of-distribution (OoD) data to assess generalizability and robustness for artifact detection application.ResultsWe extensively evaluated the proposed MoE and multiclass models. DCNNs-based MoE and ViTs-based MoE schemes outperformed simpler multiclass models and were tested on datasets from different hospitals and cancer types, where MoE using (MobileNet) DCNNs yielded the best results. The proposed MoE yields 86.15 % F1 and 97.93% sensitivity scores on unseen data, retaining less computational cost for inference than MoE using ViTs. This best performance of MoEs comes with relatively higher computational trade-offs than multiclass models. Furthermore, we apply post-processing to create an artifact segmentation mask, a potential artifact-free RoI map, a quality report, and an artifact-refined WSI for further computational analysis. During the qualitative evaluation, field experts assessed the predictive performance of MoEs over OoD WSIs. They rated artifact detection and artifact-free area preservation, where the highest agreement translated to a Cohen Kappa of 0.82, indicating substantial agreement for the overall diagnostic usability of the DCNN-based MoE scheme.ConclusionsThe proposed artifact detection pipeline will not only ensure reliable CPATH predictions but may also provide quality control. In this work, the best-performing pipeline for artifact detection is MoE with DCNNs. Our detailed experiments show that there is always a trade-off between performance and computational complexity, and no straightforward DL solution equally suits all types of data and applications. The code and HistoArtifacts dataset can be found online at Github and Zenodo, respectively.