Multiple Instance Learning Model Research Articles

Weakly supervised multiple instance learning model with generalization ability for clinical adenocarcinoma screening on serous cavity effusion pathology

Weakly supervised multiple instance learning model with generalization ability for clinical adenocarcinoma screening on serous cavity effusion pathology

MYC Rearrangement Prediction From LYSA Whole Slide Images in Large B-Cell Lymphoma: A Multicentric Validation of Self-supervised Deep Learning Models

Large B-cell lymphoma (LBCL) is a heterogeneous lymphoid malignancy in which MYC gene rearrangement (MYC-R) is associated with a poor prognosis, prompting the recommendation for more intensive treatment. MYC-R detection relies on fluorescence in situ hybridization method which is time consuming, expensive, and not available in all laboratories. Automating MYC-R detection on hematoxylin-and-eosin–stained whole slide images of LBCL would decrease the need for costly molecular testing and improve pathologists’ productivity. We developed an interpretable deep learning algorithm to detect MYC-R considering recent advances in self-supervised learning and providing an extensive comparison of 7 feature extractors and 6 multiple instance learning models, themselves. Four different multicentric cohorts, including 1247 patients with LBCL, were used for training and validation. The best deep learning model reached an average area under the receiver operating characteristic curve score of 81.9% during crossvalidation on the largest LBCL cohort, and area under the receiver operating characteristic curve scores ranging from 62.2% to 74.5% when evaluated on other unseen cohorts. In addition, we demonstrated that using this model as a prescreening tool (with a false-negative rate of 0%), fluorescence in situ hybridization testing would be avoided in 35% of cases. This work demonstrates the feasibility of developing a medical device to efficiently detect MYC gene rearrangement on hematoxylin-and-eosin–stained whole slide images in daily practice.

Unveiling the Power of Model-Agnostic Multiscale Analysis for Enhancing Artificial Intelligence Models in Breast Cancer Histopathology Images.

Developing AI models for digital pathology has traditionally relied on single-scale analysis of histopathology slides. However, a whole slide image is a rich digital representation of the tissue, captured at various magnification levels. Limiting our analysis to a single scale overlooks critical information, spanning from intricate high-resolution cellular details to broad low-resolution tissue structures. In this study, we propose a model-agnostic multiresolution feature aggregation framework tailored for the analysis of histopathology slides in the context of breast cancer, on a multicohort dataset of 2038 patient samples. We have adapted 9 state-of-the-art multiple instance learning models on our multi-scale methodology and evaluated their performance on grade prediction, TP53 mutation status prediction and survival prediction. The results prove the dominance of the multiresolution methodology, and specifically, concatenating or linearly transforming via a learnable layer the feature vectors of image patches from a high (20x) and low (10x) magnification factors achieve improved performance for all prediction tasks across domain-specific and imagenet-based features. On the contrary, the performance of uniresolution baseline models was not consistent across domain-specific and imagenet-based features. Moreover, we shed light on the inherent inconsistencies observed in models trained on whole-tissue-sections when validated against biopsy-based datasets. Despite these challenges, our findings underscore the superiority of multiresolution analysis over uniresolution methods. Finally, cross-scale analysis also benefits the explainability aspects of attention-based architectures, since one can extract attention maps at the tissue- and cell-levels, improving the interpretation of the model's decision.

Weakly supervised deep learning image analysis can differentiate melanoma from naevi on haematoxylin and eosin-stained histopathology slides.

The broad histomorphological spectrum of melanocytic pathologies requires large data sets to develop accurate and generalisable deep learning (DL)-based diagnostic pathology classifiers. Weakly supervised DL promotes utilisation of larger training data sets compared to fully supervised (patch annotation) approaches. To evaluate weakly supervised DL image classifiers for discriminating melanomas from naevi on haematoxylin and eosin (H&E)-stained pathology slides. A representative H&E slide for 260 naevi and 260 melanomas from mucocutaneous sites at one tertiary institution was digitized. Clinicopathological features were recorded for each case including thickness and histological subtype. Whole-slide or whole-tissue section labels were applied. The ground truth was established by consensus diagnosis from two pathologists. Multiple-instance learning models, Trans-MIL, CLAM and DTFD-MIL were evaluated at 10×, 20× and 40× magnifications using stratified fivefold Monte Carlo cross-validation, with 80/10/10 splits for training/validation/test groups, to predict melanoma from naevus. Heatmaps were generated to understand model performance. Naevi cases were younger (median age: 51 years; melanoma median age: 71.5 years), with more balanced sex distribution (males: 48.8%, melanoma male subgroup: 64.2%). The most frequent histological subtypes of naevi and melanomas were dysplastic compound (n = 99, 38.1%) and superficial spreading (n = 124, 47.7%), respectively. Average AUC (±1 SD) for Trans-MIL, CLAM and DTFD-MIL across test groups were 0.9952 ± 0.006, 0.9925 ± 0.0052 and 0.9708 ± 0.0328, at 20× magnification, respectively. Performance of the models varied according to the magnification used. Heatmaps from the two best performing models, Trans-MIL and CLAM, generally indicated attention on appropriate tissue regions for interpretation. Weakly supervised DL on pathological slides of common mucocutaneous melanocytic tumours provides highly accurate diagnostic value for discrimination of melanomas and naevi. External validation and further assessment on less frequently occurring histologic subtypes and borderline cases using this method is required.

Multiple Instance Pathology Image Diagnosis Model based on Channel Attention and Data Augmentation

The application of machine learning in the medical field has resulted in significant advancements in computer-aided pathological diagnosis. Multiple instance learning (MIL) has emerged as a promising approach for pathological image classification, particularly in scenarios where local annotations are lacking. However, current MIL models often overlook the importance of feature weights in the channel dimension and struggle with imbalanced positive and negative data. To address these limitations, an integration of a channel attention (CA) module and an augmented data (AUG) mechanism into the MIL model is proposed, resulting in improved performance. The CA module dynamically assigns weights to example features in the channel dimension, enhancing or suppressing features adaptively. Additionally, the AUG mechanism effectively balances the distribution of positive and negative data, significantly reducing false negatives. Through ablation experiments, the contributions of the CA module and AUG mechanism in enhancing the overall model performance are analyzed. Experimental validations on the CAMELYON16/17 public pathological image datasets demonstrate that the proposed model and method outperform existing approaches, with particular emphasis on reducing false negatives.

Interpretable artificial intelligence-based analysis for morphologic classification of neuroblastic tumors.

10023 Background: Rapid and accurate identification of morphologic features of neuroblastic tumors (NTs) is critical for risk stratification and therapeutic decision making. The prognostic value of features like neuroblast differentiation, mitosis-karyorrhexis index (MKI), and Schwannian stromal presence is well established. Deep learning permits objective histopathological analysis, streamlining workflows for pathologists, notably in rare cancers. In rare cancers, our method minimizes bias and optimizes limited data using transfer and self-supervised learning (SSL) for feature extraction, with improved explainability. Here, we used an artificial intelligence-based model to morphologically classify NT tumors and MYCN-amplification. Methods: Annotated H&E-stained slides of diagnostic NT tumor biopsies from the University of Chicago and the Children’s Oncology Group were digitalized. Pathologists defined three binarized measures including diagnostic category (ganglioneuroblastoma/neuroblastoma), grade (differentiating/poorly differentiating), and MKI (low and intermediate/high). MYCN status was abstracted from patient records (amplified/non-amplified). Using Slideflow, our open-source pipeline, we developed an attention-based multiple instance learning model with features extracted by CTransPath, a SSL model pretrained on pan-cancer images from The Cancer Genome Atlas. For each measure, model performance was evaluated using 5-fold cross validation by aggregating k-fold model predictions across multiple metrics. Patients were excluded from a model if the measure of interest was unknown. Feature significance was assessed visually using Class Activation Mapping (Grad-CAM). Results: The mean age of the study cohort (n = 172) was 3.66 years. Of patients with clinical information, 84 of 138 (60.2%) had metastatic disease and 94 of 133 (70.7%) were high-risk. Of the 148 tumors with a diagnostic category of neuroblastoma, 93.2% were poorly differentiated and 25% had high MKI. Of the 135 tumors with known MYCN status, 40 were amplified (29.6%). The final models excelled across all outcomes, performing best for diagnostic category, grade, and MYCN status (Table 1). Physician review of the attention-based heatmaps for all measures highlighted biologically relevant regions such as neuropil. Conclusions: We created a deep learning pipeline for auto-characterization of digitized H&E-stained NT pathology slides. Our approach may also aid in identifying molecular features including MYCN-amplification. Review of heatmaps showed pertinent biological tissue, boosting model reliability.[Table: see text]

Abstract PO3-07-04: Prediction of PAM50 molecular subtypes from H&E-stained breast cancer specimens using tumor microenvironment features and additive multiple instance learning models

Abstract Background: PAM50, a 50-gene signature, classifies breast cancers into one of five subtypes (basal, luminal A, luminal B, HER2-enriched, and normal-like), revealing information about underlying tumor biology, and has emerged as a key prognostic indicator influencing treatment decisions. There is growing interest in bridging the gap between expression-based metrics and histopathology, where immunohistochemistry (IHC) and sequencing-based approaches have been proposed for this purpose. However, hematoxylin and eosin (H&E)-stained slides are ubiquitously utilized by pathologists for cancer diagnosis, while IHC and sequencing-based approaches require additional tissue and specialized processing and/or analysis. Here, we describe a computer vision-based approach to predict PAM50 classification using H&E-stained whole slide images (WSIs). Methods: We obtained expression-based PAM50 subtype labels and corresponding H&E-stained WSIs for 961 breast carcinomas from the TCGA BRCA cohort. We used two separate machine learning (ML) approaches to predict PAM50 subtypes from WSIs. In the first approach, we deployed previously trained PathExplore models to extract quantitative human-interpretable features (HIFs) that summarize the TME. We subsequently trained random forest classification models on these HIFs to predict PAM50 subtypes. For the second approach, we developed additive multiple instance learning (aMIL) models. Additionally, we explored the effects of PAM50 subtype labeling and aggregation strategies beyond the 5-class approach. Our 3-class approach combines Luminal A and B, as seen in IHC efforts to increase agreement with PAM50 assays, while excluding Normal, a category containing few and heterogeneous samples. We also performed binary classification for each subtype in the 3-class model (e.g. luminal vs. other). Slides were split into training (60%), validation (20%), and test (20%) sets, stratified by PAM50 labels, and model performance was assessed using the area under the receiver operator curve (AUROC) metric on the held-out test set, using a one vs. rest approach for multi-class models. To establish a baseline for PAM50 prediction, we developed random forest classification models using only clinical covariates (tumor stage, histologic grade, histological subtype, and BRCA1/2 status). Results: We compared the performance of our two ML models (HIF and aMIL) to that of the baseline model, and we report the AUROC values in Table 1. These models both performed well in predicting Basal, Luminal A, Luminal B, and Luminal (A+B), while the model performance was less strong for predictions of the HER2 and Normal classifications. The three-class model showed improved performance of predicting Luminal classifications relative to the five-class model that separates Luminal A and B. Although simplifying classification problems to a binary use case typically provides improved performance, this phenomenon was not observed for any of the PAM50 subtypes. Conclusions: These results demonstrate that AI-powered digital pathology can accurately and reproducibly perform molecular-based classification tasks, such as predicting PAM50 classifications, using WSIs, suggesting a more efficient path toward clinically relevant breast cancer characterization. Table 1. Performance of all models in predicting PAM50 molecular subtypes. AUROC values are shown. Shaded cells represent the best test-set performance for each class (row). Citation Format: Maria Guramare, Syed Ashar Javed, Christian Kirkup, Dinkar Juyal, Jacqueline Brosnan-Cashman, Victoria Mountain, Ryan Leung, Bahar Rahsepar, John Abel, Amaro Taylor-Weiner, Jake Conway. Prediction of PAM50 molecular subtypes from H&E-stained breast cancer specimens using tumor microenvironment features and additive multiple instance learning models [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO3-07-04.

Pseudo-Bag Mixup Augmentation for Multiple Instance Learning-Based Whole Slide Image Classification.

Given the special situation of modeling gigapixel images, multiple instance learning (MIL) has become one of the most important frameworks for Whole Slide Image (WSI) classification. In current practice, most MIL networks often face two unavoidable problems in training: i) insufficient WSI data and ii) the sample memorization inclination inherent in neural networks. These problems may hinder MIL models from adequate and efficient training, suppressing the continuous performance promotion of classification models on WSIs. Inspired by the basic idea of Mixup, this paper proposes a new Pseudo-bag Mixup (PseMix) data augmentation scheme to improve the training of MIL models. This scheme generalizes the Mixup strategy for general images to special WSIs via pseudo-bags so as to be applied in MIL-based WSI classification. Cooperated by pseudo-bags, our PseMix fulfills the critical size alignment and semantic alignment in Mixup strategy. Moreover, it is designed as an efficient and decoupled method, neither involving time-consuming operations nor relying on MIL model predictions. Comparative experiments and ablation studies are specially designed to evaluate the effectiveness and advantages of our PseMix. Experimental results show that PseMix could often assist state-of-the-art MIL networks to refresh their classification performance on WSIs. Besides, it could also boost the generalization performance of MIL models in special test scenarios, and promote their robustness to patch occlusion and label noise. Our source code is available at https://github.com/liupei101/PseMix.

Development and validation of a deep learning-based microsatellite instability predictor from prostate cancer whole-slide images

Microsatellite instability-high (MSI-H) is a tumor-agnostic biomarker for immune checkpoint inhibitor therapy. However, MSI status is not routinely tested in prostate cancer, in part due to low prevalence and assay cost. As such, prediction of MSI status from hematoxylin and eosin (H&E) stained whole-slide images (WSIs) could identify prostate cancer patients most likely to benefit from confirmatory testing to evaluate their eligibility for immunotherapy and need for Lynch syndrome testing. Prostate biopsies and surgical resections from prostate cancer patients referred to our institution were analyzed. MSI status was determined by next-generation sequencing. Patients sequenced before a cutoff date formed an algorithm development set (n = 4015, MSI-H 1.8%) and a paired validation set (n = 173, MSI-H 19.7%) that consisted of two serial sections from each sample, one stained and scanned internally and the other at an external site. Patients sequenced after the cutoff date formed a temporally independent validation set (n = 1350, MSI-H 2.3%). Attention-based multiple instance learning models were trained to predict MSI-H from H&E WSIs. The predictor achieved area under the receiver operating characteristic curve values of 0.78 (95% CI [0.69–0.86]), 0.72 (95% CI [0.63–0.81]), and 0.72 (95% CI [0.62–0.82]) on the internally prepared, externally prepared, and temporal validation sets, respectively, showing effective predictability and generalization to both external staining/scanning processes and temporally independent samples. While MSI-H status is significantly correlated with Gleason score, the model remained predictive within each Gleason score subgroup.

ProDiv: Prototype-driven consistent pseudo-bag division for whole-slide image classification

Background and ObjectivePathology image classification is one of the most essential auxiliary processes in cancer diagnosis. To overcome the problem of inadequate Whole-Slide Image (WSI) samples with weak labels, pseudo-bag-based multiple instance learning (MIL) methods have attracted wide attention in pathology image classification. In this type of method, the division scheme of pseudo-bags is usually a primary factor affecting classification performance. In order to improve the division of WSI pseudo-bags on existing random/clustering approaches, this paper proposes a new Prototype-driven Division (ProDiv) scheme for the pseudo-bag-based MIL classification framework on pathology images. MethodsThis scheme first designs an attention-based method to generate a bag prototype for each slide. On this basis, it further groups WSI patch instances into a series of instance clusters according to the feature similarities between the prototype and patches. Finally, pseudo-bags are obtained by randomly combining the non-overlapping patch instances of different instance clusters. Moreover, the design scheme of our ProDiv considers practicality, and it could be smoothly assembled with almost all the MIL-based WSI classification methods in recent years. ResultsEmpirical results show that our ProDiv, when integrated with several existing methods, can deliver classification AUC improvements of up to 7.3% and 10.3%, respectively on two public WSI datasets. ConclusionsProDiv could almost always bring obvious performance improvements to compared MIL models on typical metrics, which suggests the effectiveness of our scheme. Experimental visualization also visually interprets the correctness of the proposed ProDiv.

Boosting Multiple Instance Learning Models for Whole Slide Image Classification: A Model-Agnostic Framework Based on Counterfactual Inference

Multiple instance learning is an effective paradigm for whole slide image (WSI) classification, where labels are only provided at the bag level. However, instance-level prediction is also crucial as it offers insights into fine-grained regions of interest. Existing multiple instance learning methods either solely focus on training a bag classifier or have the insufficient capability of exploring instance prediction. In this work, we propose a novel model-agnostic framework to boost existing multiple instance learning models, to improve the WSI classification performance in both bag and instance levels. Specifically, we propose a counterfactual inference-based sub-bag assessment method and a hierarchical instance searching strategy to help to search reliable instances and obtain their accurate pseudo labels. Furthermore, an instance classifier is well-trained to produce accurate predictions. The instance embedding it generates is treated as a prompt to refine the instance feature for bag prediction. This framework is model-agnostic, capable of adapting to existing multiple instance learning models, including those without specific mechanisms like attention. Extensive experiments on three datasets demonstrate the competitive performance of our method. Code will be available at https://github.com/centurion-crawler/CIMIL.

Shapley Values-enabled Progressive Pseudo Bag Augmentation for Whole-Slide Image Classification.

In computational pathology, whole-slide image (WSI) classification presents a formidable challenge due to its gigapixel resolution and limited fine-grained annotations. Multiple-instance learning (MIL) offers a weakly supervised solution, yet refining instance-level information from bag-level labels remains challenging. While most of the conventional MIL methods use attention scores to estimate instance importance scores (IIS) which contribute to the prediction of the slide labels, these often lead to skewed attention distributions and inaccuracies in identifying crucial instances. To address these issues, we propose a new approach inspired by cooperative game theory: employing Shapley values to assess each instance's contribution, thereby improving IIS estimation. The computation of the Shapley value is then accelerated using attention, meanwhile retaining the enhanced instance identification and prioritization. We further introduce a framework for the progressive assignment of pseudo bags based on estimated IIS, encouraging more balanced attention distributions in MIL models. Our extensive experiments on CAMELYON-16, BRACS, TCGA-LUNG, and TCGA-BRCA datasets show our method's superiority over existing state-of-the-art approaches, offering enhanced interpretability and class-wise insights. We will release the code upon acceptance.

Abstract B010: Spatially-resolved prediction of gene expression signatures in H&E whole slide images using additive multiple instance learning models

Abstract Background: Newly developed molecular technologies, such as spatial multiplexed assays and single-cell sequencing, have provided increased resolution and output for tumor analysis. However, these assays are often cost-prohibitive, making them inadequate ways to detect clinical biomarkers. In contrast, hematoxylin and eosin (H&E) staining is routine for cancer diagnostics but does not provide molecular information, potentially limiting its utility in the targeted therapy era. Machine learning models could augment the information revealed by H&E, potentially allowing molecular information to be inferred. Here, we describe a novel approach to predict gene expression signatures (GES) in H&E-stained whole slide images (WSI) using an additive multiple instance learning (aMIL) end-to-end model (1). We present results in breast cancer predicting spatially resolved levels of a TGFb GES, a proposed biomarker for TGFb antagonists and immunotherapy. Methods: H&E-stained WSI from the TCGA BRCA cohort (N=1090) were split into training (60%), validation (20%), and test (20%) sets. TGFb-CAF GES (2) were computed, and median expression cut-off on training data was used to define “high” and “low” TGFb-CAF levels. aMIL models were optimized in training data for the binary classification of TGFb-CAF levels. Top-performing model iterations were compared on the validation set, and the optimal model was deployed on the held-out test set. aMIL heatmaps were merged with PathExplore tumor microenvironment (TME) model heatmaps to characterize cell, tissue, and nuclear spatial distributions and morphology in terms of human interpretable features (HIFs). HIFs were extracted from high-importance patches (top 25% of aMIL scores) for both TGFb-CAF-high and -low. Results: Our model accurately predicted TGFb-CAF-high vs. -low BRCA samples (test AUROC=0.80). Also, model deployment on WSI provided interpretable heatmaps depicting TGFb-CAF predictions in tissue, providing spatial resolution to TGFb-CAF expression. Patches contributing most to TGFb-CAF-high prediction were enriched for cancer stroma, as well as cancer-infiltrating and stromal fibroblasts. Furthermore, significant differences in HIFs relating to fibroblast nucleus size and lymphocyte nucleus shape were observed between patches contributing most to TGFb-CAF-high and -low predictions. Conclusions: We have developed a method to predict GES with spatial resolution in H&E-stained WSI. aMIL models provide exact marginal contributions of each patch towards every class prediction, allowing downstream analysis of tissue, cell, and nuclear features and providing biological interpretability not found in typical black-box models. The ability of our method to detect GES in H&E-stained WSI allows complex molecular information to be detected in routine clinical specimens with spatial specificity, providing a means for GES to potentially be realized as clinical biomarkers.

Abstract B016: AI analysis of histological images accurately identifies luminal subtype urothelial carcinomas characterized by high PPARG expression

Abstract Background PPARG is a cell lineage determining transcription factor in muscle invasive urothelial carcinoma (MIUC), where high expression is associated with the luminal subtype (1, 2). As FX-909(2), a PPARG inverse agonist, enters the clinic, biomarkers that reflect the luminal subtype will reveal patients with the potential to respond to PPARG inhibition. The determination of luminal status is generally performed via RNA-seq and/or multiple immunohistochemistry stains, which are costly and time-consuming. However, MIUC biopsies are routinely stained with hematoxylin and eosin (H&E). Machine learning (ML)-driven analyses of H&E-stained tissue may enable the identification of patients with luminal MIUC and have advantages over the current molecular approach. Methods H&E-stained slides from 367 unique primary MIUC cases from the TCGA BLCA dataset were split into training (70%), validation (15%), and held-out test (15%) sets by preserving the data distribution of patient metadata. A curated retrospective cohort of 42 localized, stage III-IV primary MIUCs was used as an independent test set. Molecular classification as luminal (luminal papillary, luminal, and luminal infiltrated subtypes) or non-luminal (basal-squamous and neuronal subtypes) was performed and used as ground truth(3). Pretrained artifact and tissue segmentation models were deployed on all images to identify artifact-free areas of cancer and cancer-associated stroma. An end-to-end (E2E) additive multiple instance learning model was trained to identify luminal cases using the training set. Top performing model iterations were compared on the validation set, and the optimal iteration was deployed on both test sets. Results We assessed the performance of our E2E model in predicting luminal status using the molecular subtypes derived from Robertson et al. as ground truth(3). The E2E model showed excellent performance when predicting luminal status in the TCGA validation, TCGA test, and independent test sets (AUROC = 0.96, 0.95, and 0.97, respectively). The accuracy in all three cohorts was 89-90%, with a sensitivity of 0.86-0.96, a specificity of 0.82-0.94, and an F1 score of 0.88-0.9. Conclusions We generated a robust ML model that accurately predicts luminal MIUC using H&E-stained slides. Luminal MIUC is dependent on PPARG, and PPARG inverse agonism represents a promising therapeutic approach for MIUC. Coupled with the first-in-class FX-909 therapeutic entering the clinic, the strong performance of our model highlights the potential for its application as a precision biomarker to identify patients with advanced urothelial carcinoma likely to respond to PPARG inhibition.

An end-to-end approach to combine attention feature extraction and Gaussian Process models for deep multiple instance learning in CT hemorrhage detection

Intracranial hemorrhage (ICH) is a serious life-threatening emergency caused by blood leakage inside the brain. Radiologists usually confirm the presence of ICH by analyzing computed tomography (CT) scans, so, developing an automated diagnosis system that can process this type of images has become an important research problem. One of the main challenges to apply AI algorithms in this setting is the lack of labeled data. To mitigate the labeling burden, Multiple Instance Learning (MIL) algorithms group instances into bags, relying solely on bag-level labels for model training. Due to their capacity to handle uncertainty and deliver accurate predictions, Gaussian Processes (GPs) stand out as promising classifiers for MIL problems. Recent research has also demonstrated the effectiveness of combining attention mechanisms with GPs for ICH detection. Nonetheless, existing methods have a notable limitation: they train the attention mechanism and the GP separately, resulting in suboptimal feature extraction for GP-based classification. In this study, we introduce an innovative end-to-end MIL model that concurrently trains the CNN backbone and attention mechanism along with the GP classifier. Our approach enhances the robustness and accuracy of bag predictions by optimizing feature extraction for GP-based classification. We validate our method experimentally by focusing on two ICH detection datasets. Our results reveal a significant performance advantage in terms of accuracy, F1-score, precision, and ROC-AUC score over existing MIL approaches, especially two-stage GP approaches. Additionally, we offer empirical insights into the functionality and effectiveness of our novel model.

Attention2Minority: A salient instance inference-based multiple instance learning for classifying small lesions in whole slide images

Multiple instance learning (MIL) models have achieved remarkable success in analyzing whole slide images (WSIs) for disease classification problems. However, with regard to giga-pixel WSI classification problems, current MIL models are often incapable of differentiating a WSI with extremely small tumor lesions. This minute tumor-to-normal area ratio in a MIL bag inhibits the attention mechanism from properly weighting the areas corresponding to minor tumor lesions. To overcome this challenge, we propose salient instance inference MIL (SiiMIL), a weakly-supervised MIL model for WSI classification. We introduce a novel representation learning for histopathology images to identify representative normal keys. These keys facilitate the selection of salient instances within WSIs, forming bags with high tumor-to-normal ratios. Finally, an attention mechanism is employed for slide-level classification based on formed bags. Our results show that salient instance inference can improve the tumor-to-normal area ratio in the tumor WSIs. As a result, SiiMIL achieves 0.9225 AUC and 0.7551 recall on the Camelyon16 dataset, which outperforms the existing MIL models. In addition, SiiMIL can generate tumor-sensitive attention heatmaps that is more interpretable to pathologists than the widely used attention-based MIL method. Our experiments imply that SiiMIL can accurately identify tumor instances, which could only take up less than 1% of a WSI, so that the ratio of tumor to normal instances within a bag can increase by two to four times.

HistoMIL: A Python package for training multiple instance learning models on histopathology slides

Hematoxylin and eosin (H&E) stained slides are widely used in disease diagnosis. Remarkable advances in deep learning have made it possible to detect complex molecular patterns in these histopathology slides, suggesting automated approaches could help inform pathologists' decisions. Multiple instance learning (MIL) algorithms have shown promise in this context, outperforming transfer learning (TL) methods for various tasks, but their implementation and usage remains complex. We introduce HistoMIL, a Python package designed to streamline the implementation, training and inference process of MIL-based algorithms for computational pathologists and biomedical researchers. It integrates a self-supervised learning module for feature encoding, and a full pipeline encompassing TL and three MIL algorithms: ABMIL, DSMIL, and TransMIL. The PyTorch Lightning framework enables effortless customization and algorithm implementation. We illustrate HistoMIL's capabilities by building predictive models for 2,487 cancer hallmark genes on breast cancer histology slides, achieving AUROC performances of up to 85%.

Data-Efficient Computational Pathology Platform for Faster and Cheaper Breast Cancer Subtype Identifications: Development of a Deep Learning Model.

Breast cancer subtyping is a crucial step in determining therapeutic options, but the molecular examination based on immunohistochemical staining is expensive and time-consuming. Deep learning opens up the possibility to predict the subtypes based on the morphological information from hematoxylin and eosin staining, a much cheaper and faster alternative. However, training the predictive model conventionally requires a large number of histology images, which is challenging to collect by a single institute. We aimed to develop a data-efficient computational pathology platform, 3DHistoNet, which is capable of learning from z-stacked histology images to accurately predict breast cancer subtypes with a small sample size. We retrospectively examined 401 cases of patients with primary breast carcinoma diagnosed between 2018 and 2020 at the Department of Pathology, National Cancer Center, South Korea. Pathology slides of the patients with breast carcinoma were prepared according to the standard protocols. Age, gender, histologic grade, hormone receptor (estrogen receptor [ER], progesterone receptor [PR], and androgen receptor [AR]) status, erb-B2 receptor tyrosine kinase 2 (HER2) status, and Ki-67 index were evaluated by reviewing medical charts and pathological records. The area under the receiver operating characteristic curve and decision curve were analyzed to evaluate the performance of our 3DHistoNet platform for predicting the ER, PR, AR, HER2, and Ki67 subtype biomarkers with 5-fold cross-validation. We demonstrated that 3DHistoNet can predict all clinically important biomarkers (ER, PR, AR, HER2, and Ki67) with performance exceeding the conventional multiple instance learning models by a considerable margin (area under the receiver operating characteristic curve: 0.75-0.91 vs 0.67-0.8). We further showed that our z-stack histology scanning method can make up for insufficient training data sets without any additional cost incurred. Finally, 3DHistoNet offered an additional capability to generate attention maps that reveal correlations between Ki67 and histomorphological features, which renders the hematoxylin and eosin image in higher fidelity to the pathologist. Our stand-alone, data-efficient pathology platform that can both generate z-stacked images and predict key biomarkers is an appealing tool for breast cancer diagnosis. Its development would encourage morphology-based diagnosis, which is faster, cheaper, and less error-prone compared to the protein quantification method based on immunohistochemical staining.

Multiple Instance Learning with Trainable Soft Decision Tree Ensembles

A new random forest-based model for solving the Multiple Instance Learning problem under small tabular data, called the Soft Tree Ensemble Multiple Instance Learning, is proposed. A new type of soft decision trees is considered, which is similar to the well-known soft oblique trees, but with a smaller number of trainable parameters. In order to train the trees, it is proposed to convert them into neural networks of a specific form, which approximate the tree functions. It is also proposed to aggregate the instance and bag embeddings (output vectors) by using the attention mechanism. The whole Soft Tree Ensemble Multiple Instance Learning model, including soft decision trees, neural networks, the attention mechanism and a classifier, is trained in an end-to-end manner. Numerical experiments with well-known real tabular datasets show that the proposed model can outperform many existing multiple instance learning models. A code implementing the model is publicly available.

Domain-Specific Pre-training Improves Confidence in Whole Slide Image Classification.

Whole Slide Images (WSIs) or histopathology images are used in digital pathology. WSIs pose great challenges to deep learning models for clinical diagnosis, owing to their size and lack of pixel-level annotations. With the recent advancements in computational pathology, newer multiple-instance learning-based models have been proposed. Multiple-instance learning for WSIs necessitates creating patches and uses the encoding of these patches for diagnosis. These models use generic pre-trained models (ResNet-50 pre-trained on ImageNet) for patch encoding. The recently proposed KimiaNet, a DenseNet121 model pre-trained on TCGA slides, is a domain-specific pre-trained model. This paper shows the effect of domain-specific pre-training on WSI classification. To investigate the effect of domain-specific pre-training, we considered the current state-of-the-art multiple-instance learning models, 1) CLAM, an attention-based model, and 2) TransMIL, a self-attention-based model, and evaluated the models' confidence and predictive performance in detecting primary brain tumors - gliomas. Domain-specific pre-training improves the confidence of the models and also achieves a new state-of-the-art performance of WSI-based glioma subtype classification, showing a high clinical applicability in assisting glioma diagnosis. We will publicly share our code and experimental results at https://github.com/soham-chitnis10/WSI-domain-specific.