MuCoSA: Multi-contextual similarity assessment for histopathology image search

Introduction: In digital pathology, large amounts of time and money are spent annotating whole slide images (WSIs). By using an AI algorithm trained on thousands of WSIs, it is possible to remove the process of annotating from the workflow whilst simultaneously finding potential imaging features that correlate with drug response, molecular phenotypes, histologic features (e.g. Tertiary Lymphoid Structures), inflammation at the tumor-stroma interface, artifacts, etc. Method: A pipeline was constructed for ingesting WSI from the set of diagnostic slides available in the cancer genome archive (TCGA). The WSIs were masked for tissue areas and divided into patches extracted at multiple magnification levels. These patches were used to train a U-Net (initialised with a pre-trained ResNet34) to complete a self-supervised in-painting task and evaluated with perceptual loss using VGG16. After training, embeddings were extracted from the bottleneck layer of the U-Net, and these were evaluated for their capacity to cluster according to tissue type and magnification level. Results: The trained embeddings showed strong clustering by tissue type, magnification level, and separation of artefacts. The results were qualitatively evaluated by pathologist with consensus and exceeded the performance of a baseline pre-trained ImageNet model. Conclusion: These results highlight a novel methodology for AI algorithm development that removes the need for numerous pathologists to annotate pathology images by hand and converts their role to reviewing and quality-checking images. AI algorithms based on this approach can have a significant impact in the pathologic evaluation of tissue for oncology clinical trials by extracting a richer set of features than glass microscopy alone would permit. Furthermore, self-supervised methods, such as in-painting allow for the training of embeddings that encapsulate a richer understanding of the data and are consequently more repurposable than supervised/labelled approaches. Citation Format: Jason Hipp, Mona Xu, Lucas Bordeaux, Feng Gu, Carlos Pedrinaci, Khan Baykaner. Unsupervised learning of image embeddings enables new opportunities to extract novel information from digital pathology H&E images [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-076.

e16023 Background: Immune checkpoint inhibitors (ICIs) are standard-of-care for patients with ESCC, but the association between PD-L1 and clinical outcomes of patients with ESCC receiving ICIs treatment is unclear. Currently, different PD-L1 antibody clones, scoring, and cutoffs were used for different ICIs development in ESCC. Thus, we develop a digital pathology (DP) algorithm to generate multiple scores at the same time with high reproducibility. We evaluate DP algorithm by: (1) comparing the digital and manual scores of SP263 and 22C3 assays; (2) evaluating the association with OS. Methods: Tumor tissue samples were collected from ESCC patients treated at National Taiwan University Hospital between 2017 and 2020. Slides were tested for PD-L1 using SP263 and 22C3 assays at a College of American Pathologists accredited central laboratory. H&E and PD-L1 slides were scanned for whole slide images (WSIs) using DP200 scanner at 20x. HALO (v3.5) by indica labs was used for DP development by deep learning neural network algorithms. WSIs of SP263 were annotated for region of interest (ROI), tumor/stroma area, and tumor, immune and fibroblast cells by board certified pathologists. After the algorithm is developed and run on SP263 WSIs, a board-certified pathologist reviewed the images and provided an agreement score of DP markups. The images were excluded if an agreement was less than 80%. This algorithm was then applied to 22C3 WSIs. Data of areas and cells were exported for subsequent statistical analysis using R. We calculated the DP scores based on the definition of manual readouts of TAP, CPS and s/iTILs in ROI and tumor and stroma regions in both clones’ WSIs. These DP scores were compared with manual scores using Spearman’s rank-order correlation. Association with OS were evaluated using cox proportional hazard regression. Results: Of 235 evaluable images, twelve (5.1%) were excluded due to the low agreement with the pathologist's judgment. DP generated scores showed a strong correlation with pathologists scoring (rho = 0.80 for SP263 and rho = 0.74 for 22C3). DP generated CPS-like and TAP-like scores in ROI, and tumor and stroma regions from SP263 and 22C3 also demonstrated a strong correlation between each other (rho: 0.77 - 0.86). However, s/iTILs correlation between DP and pathologist score is negligible (rho: 0.002 - 0.34). While examining the association of PD-L1 scores and s/iTILs with OS, no clear associations were found with either DP or pathologists’ scores, after adjusting for stage, ECOG, age and gender. Conclusions: DP algorithms developed in this study demonstrated the efficiency of generating multiple DP scores of CPS, TAP, and s/iTILs from one image at the same time, which performed comparable to manual scores. DP algorithm developed based on SP263 is generalizable to 22C3 and showed a good performance. Further algorithm adjustment will be made to overcome the misclassification in the 12 failed cases.

Introduction: Artifacts are often introduced during tissue collection and processing, slide preparation, and/or when generating whole slide images (WSI). The presence of artifacts has a negative impact on the digital pathology workflow as artifacts may hinder diagnostic reporting and can lead to false positive and false negative results when using image analysis algorithms or computer-aided diagnosis systems. Manual quality control of WSI is a time-consuming procedure and therefore automated quality control tools, which report and exclude artifacts, are highly desirable to streamline digital pathology workflows. To automate the quality control step, we developed SlideQC, an AI-based quality control tool that automatically detects, reports, and outlines artifacts such as air bubbles, dust/debris, folds, out-of-focus,and pen marks, in both research and clinical workflows. Methods: SlideQC was trained with a DenseNet-based network using 1984 annotations for artifacts including air bubbles, dust/debris, folds, out-of-focus, and pen markers, across 254 Haematoxylin and Eosin (H&E) stained WSI from more than 9 tissue types. A set of 2048 annotations from synthetically generated out-of-focus images was added to supplement the training data. The performance of the SlideQC was evaluated on an external test cohort of 49 WSI H&E images sourced from the open-source database ‘HistoQCRepo’, across 375 annotations (tissue and artifact), and compared with the performance of HistoQC, an open-source quality control tool for digital pathology slides. Results: On the external test cohort, SlideQC showed high sensitivity, specificity, and F1-score with average values of 0.93, 0.99, and 0.93, across the five artifact types. In the same cohort, HistoQC attained an average sensitivity, specificity, and F1-score of 0.65, 0.79, and 0.54, respectively. Conclusions: SlideQC achieved high sensitivity, specificity, and F1-score on an external test cohort. SlideQC can add efficiency gains to a workflow by performing quality control on 100% of slides rather than the currently manually performed on only a subset of the slides in clinical pathology departments. SlideQC can allowthe triaging and alerting of slides containing a high level of artifact within a digital pathology workflow. The tool can also be used to exclude the artifact region from downstream analysis by subsequent image analysis algorithms. Citation Format: Daniela Rodrigues, Stefan Reinhard, Therese Waldburger, Daniel Martin, Suzana Couto, Inti Zlobec, Peter Caie, Erik Burlingame. SlideQC: An AI-based tool for automated quality control of whole-slide digital pathology images. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5442.

Whole slide images (WSIs) are digitized histopathology images. WSIs are stored in a pyramidal data structure that contains the same images at multiple magnification levels. In digital pathology, most algorithmic approaches to analyze WSIs use a single magnification level. However, images at different magnification levels may reveal relevant and distinct properties in the image, such as global context or detailed spatial arrangement. Given their high resolution, WSIs cannot be processed as a whole and are broken down into smaller pieces called tiles. Then, a prediction at the tile-level is made for each tile in the larger image. As many classification problems require a prediction at a slide-level, there exist common strategies to integrate the tile-level insights into a slide-level prediction. We explore two approaches to tackle this problem, namely a multiple instance learning framework and a representation learning algorithm (the so-called “barcode approach”) based on clustering. In this work, we apply both approaches in a single- and multi-scale setting and compare the results in a multi-label histopathology classification task to show the promises and pitfalls of multi-scale analysis. Our work shows a consistent improvement in performance of the multi-scale models over single-scale ones. Using multiple instance learning and the barcode approach we achieved a 0.06 and 0.06 improvement in F1 score, respectively, highlighting the importance of combining multiple scales to integrate contextual and detailed information.

This paper investigates the effect of magnification on content-based image search in digital pathology archives and proposes to use multi-magnification image representation. Image search in large archives of digital pathology slides provides researchers and medical professionals with an opportunity to match records of current and past patients and learn from evidently diagnosed and treated cases. When working with microscopes, pathologists switch between different magnification levels while examining tissue specimens to find and evaluate various morphological features. Inspired by the conventional pathology workflow, we have investigated several magnification levels in digital pathology and their combinations to minimize the gap between AI-enabled image search methods and clinical settings. The proposed searching framework does not rely on any regional annotation and potentially applies to millions of unlabelled (raw) whole slide images. This paper suggests two approaches for combining magnification levels and compares their performance. The first approach obtains a single-vector deep feature representation for a digital slide, whereas the second approach works with a multi-vector deep feature representation. We report the search results of 20×, 10×, and 5× magnifications and their combinations on a subset of The Cancer Genome Atlas (TCGA) repository. The experiments verify that cell-level information at the highest magnification is essential for searching for diagnostic purposes. In contrast, low-magnification information may improve this assessment depending on the tumor type. Our multi-magnification approach achieved up to 11% F1-score improvement in searching among the urinary tract and brain tumor subtypes compared to the single-magnification image search.

For molecular classification of endometrial carcinoma, testing for mismatch repair (MMR) status is becoming a routine process. Mismatch repair deficiency (MMR-D) is caused by loss of expression in one or more of the 4 major MMR proteins: MLH1, MSH2, MSH6, PHS2. Over 30% of patients with endometrial cancer have MMR-D. Determining the MMR status holds significance as individuals with MMR-D are potential candidates for immunotherapy. Pathological whole slide image (WSI) of endometrial cancer with immunohistochemistry results of MMR proteins were gathered. Color normalization was applied to the tiles using a CycleGAN-based network. The WSI was divided into tiles at three different magnifications (2.5 × , 5 × , and 10 ×). Three distinct networks of the same architecture were employed to include features from all three magnification levels and were stacked for ensemble learning. Three architectures, InceptionResNetV2, EfficientNetB2, and EfficientNetB3 were employed and subjected to comparison. The per-tile results were gathered to classify MMR status in the WSI, and prediction accuracy was evaluated using the following performance metrics: AUC, accuracy, sensitivity, and specificity. The EfficientNetB2 was able to make predictions with an AUC of 0.821, highest among the three architectures, and an overall AUC range of 0.767 - 0.821 was reported across the three architectures. In summary, our study successfully predicted MMR classification from pathological WSIs in endometrial cancer through a multi-resolution ensemble learning approach, which holds the potential to facilitate swift decisions on tailored treatment, such as immunotherapy, in clinical settings.

From slides to AI-ready maps: Standardized multi-layer tissue maps as metadata for artificial intelligence in digital pathology.

The presence of artifacts in whole slide images (WSIs), such as tissue folds, air bubbles, and out‐of‐focus regions, can significantly impact WSI digitization, pathologists' evaluation, and the accuracy of downstream analyses. We present SlideInspect, a novel AI‐based framework for comprehensive artifact detection and quality control in digital pathology. Our system leverages deep learning techniques to segment multiple artifact types across diverse tissue types and staining methods. SlideInspect provides a hierarchical output: a color‐coded slide quality indicator (green, yellow, red) with recommended actions (no action, re‐scan, re‐mount, re‐cut) based on artifact type and extent, and pixel‐level segmentation masks for detailed analysis. The system operates at multiple magnifications (1.25× for tissue segmentation, 5× for artifact detection) and also incorporates stain quality assessment for histological stain evaluation. We validated SlideInspect on a large, multi‐centric, multi‐scanner dataset of over 3000 WSIs, demonstrating robust performance across different tissue types, staining methods, and scanning platforms. The system achieves high segmentation accuracy for various artifacts while maintaining computational efficiency (average processing time: 72.7 s per WSI). Pathologist evaluations confirmed the clinical relevance and accuracy of SlideInspect's quality assessments. By providing actionable insights at multiple levels of granularity, SlideInspect significantly improves the efficiency and standardization of digital pathology workflows. Its vendor‐agnostic design and multi‐stain capability make it suitable for integration into diverse clinical and research settings.

Toward Routine Use of 3D Histopathology as a Research Tool

Pathological examination of nasopharyngeal carcinoma (NPC) is an indispensable factor for diagnosis, guiding clinical treatment and judging prognosis. Traditional and fully supervised NPC diagnosis algorithms require manual delineation of regions of interest on the gigapixel of whole slide images (WSIs), which however is laborious and often biased. In this paper, we propose a weakly supervised framework based on Tokens-to-Token Vision Transformer (WS-T2T-ViT) for accurate NPC classification with only a slide-level label. The label of tile images is inherited from their slide-level label. Specifically, WS-T2T-ViT is composed of the multi-resolution pyramid, T2T-ViT and multi-scale attention module. The multi-resolution pyramid is designed for imitating the coarse-to-fine process of manual pathological analysis to learn features from different magnification levels. The T2T module captures the local and global features to overcome the lack of global information. The multi-scale attention module improves classification performance by weighting the contributions of different granularity levels. Extensive experiments are performed on the 802-patient NPC and CAMELYON16 dataset. WS-T2T-ViT achieves an area under the receiver operating characteristic curve (AUC) of 0.989 for NPC classification on the NPC dataset. The experiment results of CAMELYON16 dataset demonstrate the robustness and generalizability of WS-T2T-ViT in WSI-level classification.

Recent advances in medical imaging have created many possibilities for the exploitation of both microscopic images in digital form and the whole slide images (WSIs) for multiple tasks such as classification, prediction, and retrieval. This is mainly due to annotated datasets available through various research organizations. Magnification level is an important factor as pathologist views the biopsy samples at various magnifications to reach a diagnosis. Whereas WSIs generally do contain the magnification information, microscopic snapshots are often captured without attaching the magnification information. In this paper, we introduce a new dataset, Kimia-5MAG, consisting of 33,345 patches at 5 different magnification classes created from WSIs made publicly available by The Cancer Genome Atlas (TCGA). There exists a large number of microscopic snapshots captured from camera-mounted microscopes but are of little use for automatic processing due to lack of magnification information. One direction to make use of these datasets is learning the magnification level from high resolutions captured WSIs and transferring the knowledge to microscopic snapshots. We investigate combinations of several deep networks and classifiers to predict different magnification levels. The proposed framework achieves 93% classification accuracy. We also analyze the effect of rotation on magnification prediction.

Introduction/Objective Digital pathology (DP) applications are rapidly rising in scope. However, DP implementation costs are non-trivial and are a major hurdle for widespread adoption. Furthermore, translational DP uses commercial, closed-source instruments and technical solutions. Translational research using manual histopathology approaches (i.e., glass slides) creates barriers to research due to limited pathologist expertise and availability. Lastly, manual slide reviews lack the data analytics and algorithmic evaluation potential that DP solutions can provide. In the current project, we have developed an end-to-end, home-built, open-source DP solution to advance translational research. Methods/Case Report We leveraged an existing microscope (Olympus BX41) and a full-frame camera solution (Canon D6) within the lab to establish a digital pathology imaging and storage solution. An open-source Linux server (running Ubuntu 24.04 LTS) served as the operating system for the whole slide image (WSIs) repository. The server was implemented as a headless node from which the WSIs were actively managed using the open-source image management solution (OMERO open microscopy environment (OME), v.5.6.0). A four-terabyte network-attached storage (NAS) served as the image backup. OMERO client nodes were installed remotely on pathologist computers to evaluate the performance of the DP WSI workflow. Results We assembled a customizable WSI setup using readily available components and software in a translational pathology lab. Imaging at full-frame resolution (5472 x 3648 pixels) was achieved with a low-cost camera solution (Canon 6D). Individual image acquisition was followed by the creation of WSI image mosaics using open-source image-stitching software (Hugin) for the DP workflow. Usability assessments were conducted through survey questionnaires provided to multiple pathologists (n = 4). The pathologists reviewed the ease of usability of the DP WSI workflow. The image quality and feasibility of the approach for translational DP research activities were evaluated. Conclusion Customizable digital pathology (DP) workflows can be extremely beneficial for low-bandwidth digital pathology initiatives, such as translational pathology research projects. We have developed a low-cost, custom-built whole slide imaging (WSI) workflow solution in our lab that will be utilized for future collaborative translational DP research projects at UNM.

There are definite lessons to be learned from the digital radiology experience as pathology transitions toward more widespread use of digital images for diagnostic purposes. While there are parallels between digital radiology and digital pathology in terms of work flow gains and losses, there are also important differences like pathology’s need for color images and the large files that are created when slides are digitized as whole slide images (WSI). The paper by Patterson et al,[1] points out that the adoption of digital imaging by pathologists has been slower than that encountered with filmless radiology. To explore this issue deeper, the authors conducted semi-structured interviews with radiologists and pathologists and looked at the adoption of other health information technologies. Their results indicated that pathologists have a lower opinion of the overall performance of digital systems than their radiologist counterparts. Specific issues of concern included: differences in magnification and image scale as compared to light microscopy, large file sizes and data management, longer time to review individual slides and an inability to focus on folded or uneven areas of tissue. In addition, hardware and software costs, information technology (IT) support, LIS integration, regulatory issues and lack of standards or best practice guidelines also represent key obstacles to the more widespread adoption of digital pathology. Even if the issues around regulations, standards, cost and infrastructure were to suddenly vanish, the perception of inferior performance of WSI systems by pathologists will prevent more rapid adoption. The issue of inability to adjust focus on digital images on folded or uneven areas of tissue highlights the dependence of current-state digital pathology systems on good quality histology. It also raises obvious concerns about diagnostic accuracy. As pointed out by one of the pathologists in the study by Patterson et al, inferior performance (assuming this refers to diagnostic accuracy) on even a small percentage of cases could have major implications for high risk diagnoses. The importance of this point as a barrier to adoption cannot be overemphasized. As pointed out in a recent Scientific American article,[2] the practice of pathology has been based on glass slides and light microscopes for over 100 years and digital pathology systems represent disruptive technology. The prospect of such a major change will naturally cause pathologists to be reluctant about adopting a technology that could both slow them down and introduce the possibility of diagnostic error. Having said this, it must also be acknowledged that WSI technology is steadily improving and vendors are acutely aware of the need for outstanding image quality and faster scanning speeds. Validation studies performed in a variety of institutions and settings using these improved systems will play a critical role in determining the rate of adoption. Whether these studies are based on histologic feature recognition, diagnostic concordance or a combination of the two, the results must demonstrate equivalence between WSI systems and light microscopy if a sense of confidence is to develop across the pathology community as a whole. The survey by Patterson et al, identified many facilitators for the adoption of digital pathology. These included the benefits of using digital pathology platforms for medical student and pathology resident training, continuing medical education and tumor boards. All of these activities share a theme of increasing the exposure of the pathologists to this technology using optimized images and presenting them in an environment that is essentially free of worry over diagnostic accuracy. Such exposure should only be beneficial in terms of building a comfort level among pathologists, especially as the technology continues to improve and new generations of pathologists encounter WSI technology throughout their residency training. The paper by Patterson et al, also includes a comprehensive list of survey questions that was pilot tested at a recent pathology informatics conference. The list of questions explores issues related to work environment in digital pathology as compared to light microscopy. I feel that this is an important and relatively underexplored aspect of digital pathology. It may be that pathologists will need to adjust their work environment so as to minimize the chance of being distracted by incoming e-mail or being interrupted by others while trying to sort out a live frozen section on their computer screen. As a pathologist who currently uses WSI to read frozen sections at my institution,[3] it has been my experience that I am significantly (dare I say without implying rigorous data collection with supporting statistics!) more likely to be interrupted when looking at cases on my office computer screen as opposed to using my microscope. While a lot of work remains to be done before pathology reaches the level of adoption seen in digital radiology, I agree with the comment by Patterson et al, the performance barriers are tractable. The big question is how long the process will take. Once the performance problems have been overcome, the full potential of WSI systems as a basis for diagnostic telepathology networks can be realized in terms of improving access to sub-specialty diagnostic opinions and providing pathology services to remote or underserviced locations.

Multiple Instance Learning (MIL) methods have become increasingly popular for classifying gigapixel-sized Whole-Slide Images (WSIs) in digital pathology. Most MIL methods operate at a single WSI magnification, by processing all the tissue patches. Such a formulation induces high computational requirements and constrains the contextualization of the WSI-level representation to a single scale. Certain MIL methods extend to multiple scales, but they are computationally more demanding. In this paper, inspired by the pathological diagnostic process, we propose ZoomMIL, a method that learns to perform multi-level zooming in an end-to-end manner. ZoomMIL builds WSI representations by aggregating tissue-context information from multiple magnifications. The proposed method outperforms the state-of-the-art MIL methods in WSI classification on two large datasets, while significantly reducing computational demands with regard to Floating-Point Operations (FLOPs) and processing time by 40–50\(\times \). Our code is available at: https://github.com/histocartography/zoommil.KeywordsWhole-slide image classificationMultiple instance learningMulti-scale zoomingEfficient computational pathology

Algorithms proposed in computational pathology can allow to automatically analyze digitized tissue samples of histopathological images to help diagnosing diseases. Tissue samples are scanned at a high-resolution and usually saved as images with several magnification levels, namely whole slide images (WSIs). Convolutional neural networks (CNNs) represent the state-of-the-art computer vision methods targeting the analysis of histopathology images, aiming for detection, classification and segmentation. However, the development of CNNs that work with multi-scale images such as WSIs is still an open challenge. The image characteristics and the CNN properties impose architecture designs that are not trivial. Therefore, single scale CNN architectures are still often used. This paper presents Multi_Scale_Tools, a library aiming to facilitate exploiting the multi-scale structure of WSIs. Multi_Scale_Tools currently include four components: a pre-processing component, a scale detector, a multi-scale CNN for classification and a multi-scale CNN for segmentation of the images. The pre-processing component includes methods to extract patches at several magnification levels. The scale detector allows to identify the magnification level of images that do not contain this information, such as images from the scientific literature. The multi-scale CNNs are trained combining features and predictions that originate from different magnification levels. The components are developed using private datasets, including colon and breast cancer tissue samples. They are tested on private and public external data sources, such as The Cancer Genome Atlas (TCGA). The results of the library demonstrate its effectiveness and applicability. The scale detector accurately predicts multiple levels of image magnification and generalizes well to independent external data. The multi-scale CNNs outperform the single-magnification CNN for both classification and segmentation tasks. The code is developed in Python and it will be made publicly available upon publication. It aims to be easy to use and easy to be improved with additional functions.