Patch-level Features Research Articles

P0340 Endo-Histo Foundational Fusion Model: A Novel Artificial Intelligence Approach for Predicting Histologic Remission and Early Response to Therapy in a Phase 2 Ulcerative Colitis Clinical Trial

Abstract Background Artificial Intelligence (AI)- enabled endoscopy and histology offer accurate, objective, and rapid assessment of disease activity in Ulcerative Colitis (UC). Emerging multi-source AI models integrating diverse datasets may enhance standardised disease evaluation and outcome prediction. This study aimed to develop a novel AI model fusing endoscopic and histological findings to improve the assessment of disease remission and predict early response to therapy in UC clinical trials. Methods A novel multimodal AI fusion algorithm was developed by integrating paired endoscopic videos and histological whole-slide images (WSIs) from the phase 2 clinical trial of Mirikizumab in UC (NCT02589665). The endoscopy branch of the model was trained with 291 white-light videos, using a convolutional neural network to select informative frames and the BioMedCLIP foundational model to extract features. The histology branch utilised the CONCH foundational model and was fine-tuned on 291 WSIs to obtain patch-level features. Features from both modalities were aggregated and fused using multi-head self-attention. The model’s ability to assess histological remission and response to therapy at weeks 12 and 52 was evaluated. Histological remission was defined as Geboes ≤2B.0, while a response to therapy at different time points was based on histological remission or improvement (Geboes <3.1). Results The fusion model outperformed single-modality assessments for histological remission, achieving a sensitivity of 89.72% (95% CI: 82.35–94.76), specificity of 89.67% (95% CI: 84.34–93.67), and accuracy of 89.69% (95% CI: 85.61–92.94). It demonstrated remarkable performance in assessing response to therapy at 12 and 52 weeks, with sensitivity of 97.96% (95% CI: 89.15–99.95), specificity of 86.84% (95% CI: 71.91–95.59) and accuracy of 93.10% (95% CI 85.59 – 97.43) for histological remission at week 52. Substantial agreement was observed between the AI fusion model and central readout. Conclusion This innovative multimodal fusion AI model enhances the assessment of histological remission and accurately predicts response to therapy. By potentially standardising central readouts and enabling automated disease assessment, this novel tool marks a significant advancement towards precision medicine in clinical trials.

When multiple instance learning meets foundation models: Advancing histological whole slide image analysis.

Deep multiple instance learning (MIL) pipelines are the mainstream weakly supervised learning methodologies for whole slide image (WSI) classification. However, it remains unclear how these widely used approaches compare to each other, given the recent proliferation of foundation models (FMs) for patch-level embedding and the diversity of slide-level aggregations. This paper implemented and systematically compared six FMs and six recent MIL methods by organizing different feature extractions and aggregations across seven clinically relevant end-to-end prediction tasks using WSIs from 4044 patients with four different cancer types. We tested state-of-the-art (SOTA) FMs in computational pathology, including CTransPath, PathoDuet, PLIP, CONCH, and UNI, as patch-level feature extractors. Feature aggregators, such as attention-based pooling, transformers, and dynamic graphs were thoroughly tested. Our experiments on cancer grading, biomarker status prediction, and microsatellite instability (MSI) prediction suggest that (1) FMs like UNI, trained with more diverse histological images, outperform generic models with smaller training datasets in patch embeddings, significantly enhancing downstream MIL classification accuracy and model training convergence speed, (2) instance feature fine-tuning, known as online feature re-embedding, to capture both fine-grained details and spatial interactions can often further improve WSI classification performance, (3) FMs advance MIL models by enabling promising grading classifications, biomarker status, and MSI predictions without requiring pixel- or patch-level annotations. These findings encourage the development of advanced, domain-specific FMs, aimed at more universally applicable diagnostic tasks, aligning with the evolving needs of clinical AI in pathology.

Cross-attention guided loss-based deep dual-branch fusion network for liver tumor classification

Recently, convolutional neural networks (CNNs) and multiple instance learning (MIL) methods have been successfully applied to MRI images. However, CNNs directly utilize the whole image as the model input and the downsampling strategy (like max or mean pooling) to reduce the size of the feature map, thereby possibly neglecting some local details. And MIL methods learn instance-level or local features without considering spatial information. To overcome these issues, in this paper, we propose a novel cross-attention guided loss-based dual-branch framework (LCA-DB) to leverage spatial and local image information simultaneously, which is composed of an image-based attention network (IA-Net), a patch-based attention network (PA-Net) and a cross-attention module (CA). Specifically, IA-Net directly learns image features with loss-based attention to mine significant regions, meanwhile, PA-Net captures patch-specific representations to extract crucial patches related to the tumor. Additionally, the cross-attention module is designed to integrate patch-level features by using attention weights generated from each other, thereby assisting them in mining supplement region information and enhancing the interactive collaboration of the two branches. Moreover, we employ an attention similarity loss to further reduce the semantic inconsistency of attention weights obtained from the two branches. Finally, extensive experiments on three liver tumor classification tasks demonstrate the effectiveness of the proposed framework, e.g., on the LLD-MMRI–7, our method achieves 69.2%, 65.9% and 88.5% on the seven-class liver tumor classification tasks in terms of accuracy, F1 score and AUC, with the superior classification and interpretation performance over recent state-of-the-art methods. The source code of LCA-DB is available at https://github.com/Wangrui-berry/Cross-attention.

Task-dependent Slicing with Convolutional Network Model for Histopathological Image Analysis

Due to its high rates of both mortality and morbidity, cancer stands as the primary cause of death globally. The examination of histopathological images plays a crucial role in the early prediction of cancer, relying on manual practices for each individual affected. However, this phase is prone to errors, time-consuming, and doesn’t facilitate early-stage decision-making for pathologists. Despite significant advancements in computer-aided image processing, the analysis of histopathological morphology remains challenging due to the intricate nature of these images. Moreover, limited annotations restrict sample analysis. This work focuses on developing efficient deep-learning methods to analyze histopathological images by considering both global and local features. It involves the analysis of image patches and features to address issues related to image annotations. A unique task-specific slicing model, integrated with the convolutional network model (ts-CNN) is proposed. This framework aims to conduct patch-level feature examination and combine multiple samples to achieve the necessary outcomes for classification. The envisioned model aims to rectify inefficiencies present in various existing approaches. Using MATLAB 2020a, the proposed model was employed to classify cancer-based histopathological images, demonstrating superior predictive performance. It assisted pathologists in early cancer prediction with an accuracy of 98.2 % and aided in decision-making processes, offering promising results.

Distillation of multi-class cervical lesion cell detection via synthesis-aided pre-training and patch-level feature alignment

Automated detection of cervical abnormal cells from Thin-prep cytologic test (TCT) images is crucial for efficient cervical abnormal screening using computer-aided diagnosis systems. However, the construction of the detection model is hindered by the preparation of the training images, which usually suffers from issues of class imbalance and incomplete annotations. Additionally, existing methods often overlook the visual feature correlations among cells, which are crucial in cervical lesion cell detection as pathologists commonly rely on surrounding cells for identification. In this paper, we propose a distillation framework that utilizes a patch-level pre-training network to guide the training of an image-level detection network, which can be applied to various detectors without changing their architectures during inference. The main contribution is three-fold: (1) We propose the Balanced Pre-training Model (BPM) as the patch-level cervical cell classification model, which employs an image synthesis model to construct a class-balanced patch dataset for pre-training. (2) We design the Score Correction Loss (SCL) to enable the detection network to distill knowledge from the BPM model, thereby mitigating the impact of incomplete annotations. (3) We design the Patch Correlation Consistency (PCC) strategy to exploit the correlation information of extracted cells, consistent with the behavior of cytopathologists. Experiments on public and private datasets demonstrate the superior performance of the proposed distillation method, as well as its adaptability to various detection architectures.

Abstract PO4-07-04: Breast Cancer HER2 Status Prediction from Hematoxylin-Eosin Stained Images Using Point Cloud Transformer

Abstract Background: Human epidermal growth factor receptor-2 (HER2) status is a key factor in determining the treatment strategy for breast cancer patients. Patients with HER2-positive status are more likely to benefit from HER2-targeted therapy, leading to improved prognosis. In current routine diagnostic practice, pathologists utilize Hematoxylin-Eosin (HE) stained tumor tissues for histopathological assessment. Subsequently, IHC assessment and/or FISH test are performed to evaluate the HER2 status. However, manual assessment results may be affected by tissue usability and observer-subjectivity. Therefore, there is a necessity to predict the HER2 status directly from HE images to minimize time and cost while ensuring enhanced consistency. Methods: We identified 608 HE diagnostic slides with HER2 status from The Cancer Genome Atlas in breast cancer (TCGA-BRCA). It contains 474 HER2-positive slides (IHC 3+, IHC 2+ and FISH positive) and 134 HER2-negative slides (IHC 0, IHC 1+, IHC 2+ and FISH negative). To analyze these slides, we first tiled the HE images into patches with a fixed size of 256 × 256 at 20 × magnification. Then the patch-level feature was derived from a self-supervised pretrained transformer. Meanwhile, artificial intelligence (AI) methods are adopted to predict HER2 status from HE images. To capture the long-distance patch relations within a slide, we represented the patches as distinct points and utilized the Point Cloud Transformer model for HER2 status prediction. Specifically, 1024 patches (points) in each slide were randomly selected and input into the Point Cloud Transformer. This process yielded a slide-level prediction result for the HER2 status. Furthermore, Graph Attention Network (GAT), Graph Sample and Aggregate Network (Graph SAGE), and hierarchical Point Cloud Network (PointNet++) were adopted to compare the effectiveness of HER2 status prediction using Point Cloud Transformer. Of note that Point Cloud Transformer incorporated attention mechanisms for point aggregation compared to PointNet++. Results: The Point Cloud Transformer was trained with 5-fold cross-validation, and the area under the curve (AUC) of the receiver operating characteristic (ROC) curve was reported. The performance of point-based models outperformed the graph-based models. And the Point Cloud Transformer achieved the highest AUC of 0.7496 among all AI models. The detailed AUC for each AI model was shown in Table 1. Conclusion: Our study revealed that the HER2 status can be predicted directly from HE images without using IHC images. Furthermore, point-based models have demonstrated the ability to capture long-distance relations among patches, surpassing graph-based models in terms of prediction performance. To further enhance performance, we adopted a better point aggregation method, such as the Point Cloud Transformer, which held promise for further improving the accuracy of predictions in the future. Table 1: AUC of HER2 status prediction from HE images in the TCGA-BRCA dataset Method GAT Graph SAGE PointNet++ Point Cloud Transformer AUC 0.6640 0.6737 0.7201 0.7496 Citation Format: Bao Li, Zhenyu Liu, Yang Du, Jie Tian. Breast Cancer HER2 Status Prediction from Hematoxylin-Eosin Stained Images Using Point Cloud Transformer [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO4-07-04.

Patch-Level Feature Selection for Thoracic Disease Classification by Chest X-ray Images Using Information Bottleneck.

Chest X-ray (CXR) examination serves as a widely employed clinical test in medical diagnostics. Many studied have tried to apply artificial intelligence (AI) programs to analyze CXR images. Despite numerous positive outcomes, assessing the applicability of AI models for comprehensive diagnostic support remains a formidable challenge. We observed that, even when AI models exhibit high accuracy on one dataset, their performance may deteriorate when tested on another. To address this issue, we propose incorporating a variational information bottleneck (VIB) at the patch level to enhance the generalizability of diagnostic support models. The VIB introduces a probabilistic model aimed at approximating the posterior distribution of latent variables given input data, thereby enhancing the model's generalization capabilities on unseen data. Unlike the conventional VIB approaches that flatten features and use a re-parameterization trick to sample a new latent feature, our method applies the trick to 2D feature maps. This design allows only important pixels to respond, and the model will select important patches in an image. Moreover, the proposed patch-level VIB seamlessly integrates with various convolutional neural networks, offering a versatile solution to improve performance. Experimental results illustrate enhanced accuracy in standard experiment settings. In addition, the method shows robust improvement when training and testing on different datasets.

CaMIL: Causal Multiple Instance Learning for Whole Slide Image Classification

Whole slide image (WSI) classification is a crucial component in automated pathology analysis. Due to the inherent challenges of high-resolution WSIs and the absence of patch-level labels, most of the proposed methods follow the multiple instance learning (MIL) formulation. While MIL has been equipped with excellent instance feature extractors and aggregators, it is prone to learn spurious associations that undermine the performance of the model. For example, relying solely on color features may lead to erroneous diagnoses due to spurious associations between the disease and the color of patches. To address this issue, we develop a causal MIL framework for WSI classification, effectively distinguishing between causal and spurious associations. Specifically, we use the expectation of the intervention P(Y | do(X)) for bag prediction rather than the traditional likelihood P(Y | X). By applying the front-door adjustment, the spurious association is effectively blocked, where the intervened mediator is aggregated from patch-level features. We evaluate our proposed method on two publicly available WSI datasets, Camelyon16 and TCGA-NSCLC. Our causal MIL framework shows outstanding performance and is plug-and-play, seamlessly integrating with various feature extractors and aggregators.

HEAP: Unsupervised Object Discovery and Localization with Contrastive Grouping

Unsupervised object discovery and localization aims to detect or segment objects in an image without any supervision. Recent efforts have demonstrated a notable potential to identify salient foreground objects by utilizing self-supervised transformer features. However, their scopes only build upon patch-level features within an image, neglecting region/image-level and cross-image relationships at a broader scale. Moreover, these methods cannot differentiate various semantics from multiple instances. To address these problems, we introduce Hierarchical mErging framework via contrAstive grouPing (HEAP). Specifically, a novel lightweight head with cross-attention mechanism is designed to adaptively group intra-image patches into semantically coherent regions based on correlation among self-supervised features. Further, to ensure the distinguishability among various regions, we introduce a region-level contrastive clustering loss to pull closer similar regions across images. Also, an image-level contrastive loss is present to push foreground and background representations apart, with which foreground objects and background are accordingly discovered. HEAP facilitates efficient hierarchical image decomposition, which contributes to more accurate object discovery while also enabling differentiation among objects of various classes. Extensive experimental results on semantic segmentation retrieval, unsupervised object discovery, and saliency detection tasks demonstrate that HEAP achieves state-of-the-art performance.

Rethinking the Paradigm of Content Constraints in Unpaired Image-to-Image Translation

In an unpaired setting, lacking sufficient content constraints for image-to-image translation (I2I) tasks, GAN-based approaches are usually prone to model collapse. Current solutions can be divided into two categories, reconstruction-based and Siamese network-based. The former requires that the transformed or transforming image can be perfectly converted back to the original image, which is sometimes too strict and limits the generative performance. The latter involves feeding the original and generated images into a feature extractor and then matching their outputs. This is not efficient enough, and a universal feature extractor is not easily available. In this paper, we propose EnCo, a simple but efficient way to maintain the content by constraining the representational similarity in the latent space of patch-level features from the same stage of the encoder and decoder of the generator. For the similarity function, we use a simple MSE loss instead of contrastive loss, which is currently widely used in I2I tasks. Benefits from the design, EnCo training is extremely efficient, while the features from the encoder produce a more positive effect on the decoding, leading to more satisfying generations. In addition, we rethink the role played by discriminators in sampling patches and propose a discriminative attention-guided (DAG) patch sampling strategy to replace random sampling. DAG is parameter-free and only requires negligible computational overhead, while significantly improving the performance of the model. Extensive experiments on multiple datasets demonstrate the effectiveness and advantages of EnCo, and we achieve multiple state-of-the-art compared to previous methods.

P995 Development and validation of a deep learning model based on histological characteristics to predict primary non-response of infliximab in Crohn's disease

Abstract Background Predicting primary non-response to infliximab (IFX) is important to select the optimal therapy and utilize personalized management in patients of Crohn’s disease (CD). This study aimed to develop and validate a deep learning (DL) model based on histological characteristics for predicting primary non-response to IFX in CD. Methods A training cohort and a validation cohort were recruited from the First Affiliated Hospital of Sun Yat-sen University. Clinical features, serological biomarkers, imaging features and histological characteristics of colonoscopy biopsy at baseline of IFX therapy were obtained as candidate predictors. Primary non-response was assessed at week 14. We developed a model using a multi-instance learning framework. This model combines patch-level features from whole slide images (WSI) with multi-modal information, together with pre-treatment image features and clinical features, to predict the response to infliximab (IFX) therapy. Finally, the model was validated for area under the curve (AUC) in a reference validation cohort. Results A total of 131 patients were included in the derivation cohort. The DL model based on pathologic features only had an under-the-curve receiver operating characteristic (AUROC) value of 0.87 in the 10-fold cross-validation cohort. The mean AUROC for the model combining pathologic and radiographic features was 0.88. The DL model achieved the highest AUROC value of 0.91 when combining pathological, radiomics and clinical features (Figure 1). When the pathological film was visualized by attention thermography, it was found that the ulcerated tissue was focused on, especially in areas dominated by neutrophils, lymphocyte aggregation and abnormal crypt structure. Conclusion The study developed several accurate DL models for predicting primary non-response to IFX by WSI. The DL model combining pathological, radiomics and clinical features achieving best performance. This model may help clinicians select appropriate therapeutic strategy and avoid unnecessary exposure to IFX in CD patients.

Contrastive learning-based histopathological features infer molecular subtypes and clinical outcomes of breast cancer from unannotated whole slide images

The artificial intelligence-powered computational pathology has led to significant improvements in the speed and precision of tumor diagnosis, while also exhibiting substantial potential to infer genetic mutations and gene expression levels. However, current studies remain limited in predicting molecular subtypes and clinical outcomes in breast cancer. In this paper, we proposed a weakly supervised contrastive learning framework to address this challenge. Our framework first performed contrastive learning pretraining on a large number of unlabeled patches tiled from whole slide images (WSIs) to extract patch-level features. The gated attention mechanism was leveraged to aggregate patch-level features to produce slide feature that was then applied to various downstream tasks. To confirm the effectiveness of the proposed method, three public cohorts and one external independent cohort of breast cancer have been used to conducted evaluation experiments. The predictive powers of our model to infer gene expression, molecular subtypes, recurrence events and drug responses were validated across cohorts. In addition, the learned patch-level attention scores enabled us to generate heatmaps that were highly consistent with pathologist annotations and spatial transcriptomic data. These findings demonstrated that our model effectively established the high-order genotype-phenotype associations, thereby potentially extend the application of digital pathology in clinical practice.

Federated Unsupervised Cluster-Contrastive learning for person Re-identification: A coarse-to-fine approach

Person Re-identification (ReID) has attracted considerable interests in recent years, largely driven by the escalating demand for public safety measures. However, the acquisition and handling of sensitive personal data can trigger significant privacy concerns. Federated learning has been introduced as a potential solution to this problem, with the goal of limiting the exposure of sensitive data across different participating entities (clients). Existing methods often depend on labour-intensive data annotations and face difficulties in maintaining cross-domain uniformity. To tackle these challenges, we propose a Federated Unsupervised Cluster-Contrastive (FedUCC) method based on deep learning for Person ReID that follows a generic-to-specific learning strategy. First, FedUCC procures generic knowledge from a conventional federated learning scheme to aggregate and distribute parameters across local clients. Second, specialized knowledge is explored to facilitate client personalization by disentangling client-specific knowledge from generic knowledge through parameter localization. Third, to further enhance effective fine-grained patterns instead of overfitting on specialized client knowledge, we investigate two key aspects: patch-level feature alignment and camera-invariant learning. Comprehensive experiments on eight public benchmark datasets demonstrate the state-of-the-art performance of our proposed method.

Coarse-Refined Consistency Learning Using Pixel-Level Features for Semi-Supervised Medical Image Segmentation.

Pixel-level annotations are extremely expensive for medical image segmentation tasks as both expertise and time are needed to generate accurate annotations. Semi-supervised learning (SSL) for medical image segmentation has recently attracted growing attention because it can alleviate the exhausting manual annotations for clinicians by leveraging unlabeled data. However, most of the existing SSL methods do not take pixel-level information (e.g., pixel-level features) of labeled data into account, i.e., the labeled data are underutilized. Hence, in this work, an innovative Coarse-Refined Network with pixel-wise Intra-patch ranked loss and patch-wise Inter-patch ranked loss (CRII-Net) is proposed. It provides three advantages: i) it can produce stable targets for unlabeled data, as a simple yet effective coarse-refined consistency constraint is designed; ii) it is very effective for the extreme case where very scarce labeled data are available, as the pixel-level and patch-level features are extracted by our CRII-Net; and iii) it can output fine-grained segmentation results for hard regions (e.g., blurred object boundaries and low-contrast lesions), as the proposed Intra-Patch Ranked Loss (Intra-PRL) focuses on object boundaries and Inter-Patch Ranked loss (Inter-PRL) mitigates the adverse impact of low-contrast lesions. Experimental results on two common SSL tasks for medical image segmentation demonstrate the superiority of our CRII-Net. Specifically, when there are only 4% labeled data, our CRII-Net improves the Dice similarity coefficient (DSC) score by at least 7.49% when compared to five classical or state-of-the-art (SOTA) SSL methods. For hard samples/regions, our CRII-Net also significantly outperforms other compared methods in both quantitative and visualization results.

Deep joint learning of pathological region localization and Alzheimer’s disease diagnosis

The identification of Alzheimer’s disease (AD) using structural magnetic resonance imaging (sMRI) has been studied based on the subtle morphological changes in the brain. One of the typical approaches is a deep learning-based patch-level feature representation. For this approach, however, the predetermined patches before learning the diagnostic model can limit classification performance. To mitigate this problem, we propose the BrainBagNet with a position-based gate (PG), which applies position information of brain images represented through the 3D coordinates. Our proposed method represents the patch-level class evidence based on both MR scan and position information for image-level prediction. To validate the effectiveness of our proposed framework, we conducted comprehensive experiments comparing it with state-of-the-art methods, utilizing two publicly available datasets: the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and the Australian Imaging, Biomarkers and Lifestyle (AIBL) dataset. Furthermore, our experimental results demonstrate that our proposed method outperforms the existing competing methods in terms of classification performance for both AD diagnosis and mild cognitive impairment conversion prediction tasks. In addition, we performed various analyses of the results from diverse perspectives to obtain further insights into the underlying mechanisms and strengths of our proposed framework. Based on the results of our experiments, we demonstrate that our proposed framework has the potential to advance deep-learning-based patch-level feature representation studies for AD diagnosis and MCI conversion prediction. In addition, our method provides valuable insights, such as interpretability, and the ability to capture subtle changes, into the underlying pathological processes of AD and MCI, benefiting both researchers and clinicians.

Visual transformer with stable prior and patch-level attention for single image dehazing

Single-image dehazing aims to recover blurred image details and improve image quality, which is a challenging ill-posed problem due to severe information degradation. In the image dehazing task, extracting local features from adjacent regions is particularly important. However, Transformer-based methods lack relative awareness of patch-level features. Furthermore, due to the sensitivity of self-attention to textcolorreddata distribution, the model suffers severe performance degradation when migrating from synthetic domain to real domain. To alleviate the above problems, we propose visual transformer with stable prior and patch-level attention (VSPPA) for image dehazing. Firstly, we propose a region-aware patch-level attention module to obtain the positional correlation between local patches and contexts, which can enhance the concentration of local patch-related features. Next, due to the instability problem caused by distribution shifts, we introduce dataset-independent prior to guide the transformer model, thereby preventing feature drift thus to improve the robustness of the model. Finally, domain-drift leads to insufficient dehazing when the model trained on synthetic data while migrates to the real environment, we come up with a introduce a patch filling strategy (PFS) for fuzzy data to narrow the domain gap and realize the generalization in real scenes. Extensive experiments show that the model achieves State-of-the-Art on the SOTS synthetic dataset and effective generalization to real-world scenarios.

RLogist: Fast Observation Strategy on Whole-Slide Images with Deep Reinforcement Learning

Whole-slide images (WSI) in computational pathology have high resolution with gigapixel size, but are generally with sparse regions of interest, which leads to weak diagnostic relevance and data inefficiency for each area in the slide. Most of the existing methods rely on a multiple instance learning framework that requires densely sampling local patches at high magnification. The limitation is evident in the application stage as the heavy computation for extracting patch-level features is inevitable. In this paper, we develop RLogist, a benchmarking deep reinforcement learning (DRL) method for fast observation strategy on WSIs. Imitating the diagnostic logic of human pathologists, our RL agent learns how to find regions of observation value and obtain representative features across multiple resolution levels, without having to analyze each part of the WSI at the high magnification. We benchmark our method on two whole-slide level classification tasks, including detection of metastases in WSIs of lymph node sections, and subtyping of lung cancer. Experimental results demonstrate that RLogist achieves competitive classification performance compared to typical multiple instance learning algorithms, while having a significantly short observation path. In addition, the observation path given by RLogist provides good decision-making interpretability, and its ability of reading path navigation can potentially be used by pathologists for educational/assistive purposes. Our code is available at: https://github.com/tencent-ailab/RLogist.

MuRCL: Multi-Instance Reinforcement Contrastive Learning for Whole Slide Image Classification.

Multi-instance learning (MIL) is widely adop- ted for automatic whole slide image (WSI) analysis and it usually consists of two stages, i.e., instance feature extraction and feature aggregation. However, due to the "weak supervision" of slide-level labels, the feature aggregation stage would suffer from severe over-fitting in training an effective MIL model. In this case, mining more information from limited slide-level data is pivotal to WSI analysis. Different from previous works on improving instance feature extraction, this paper investigates how to exploit the latent relationship of different instances (patches) to combat overfitting in MIL for more generalizable WSI classification. In particular, we propose a novel Multi-instance Rein- forcement Contrastive Learning framework (MuRCL) to deeply mine the inherent semantic relationships of different patches to advance WSI classification. Specifically, the proposed framework is first trained in a self-supervised manner and then finetuned with WSI slide-level labels. We formulate the first stage as a contrastive learning (CL) process, where positive/negative discriminative feature sets are constructed from the same patch-level feature bags of WSIs. To facilitate the CL training, we design a novel reinforcement learning-based agent to progressively update the selection of discriminative feature sets according to an online reward for slide-level feature aggregation. Then, we further update the model with labeled WSI data to regularize the learned features for the final WSI classification. Experimental results on three public WSI classification datasets (Camelyon16, TCGA-Lung and TCGA-Kidney) demonstrate that the proposed MuRCL outperforms state-of-the-art MIL models. In addition, MuRCL can achieve comparable performance to other state-of-the-art MIL models on TCGA-Esca dataset.

Cancer Survival Prediction From Whole Slide Images With Self-Supervised Learning and Slide Consistency

Histopathological Whole Slide Images (WSIs) at giga-pixel resolution are the gold standard for cancer analysis and prognosis. Due to the scarcity of pixel- or patch-level annotations of WSIs, many existing methods attempt to predict survival outcomes based on a three-stage strategy that includes patch selection, patch-level feature extraction and aggregation. However, the patch features are usually extracted by using truncated models (e.g. ResNet) pretrained on ImageNet without fine-tuning on WSI tasks, and the aggregation stage does not consider the many-to-one relationship between multiple WSIs and the patient. In this paper, we propose a novel survival prediction framework that consists of patch sampling, feature extraction and patient-level survival prediction. Specifically, we employ two kinds of self-supervised learning methods, i.e. colorization and cross-channel, as pretext tasks to train convnet-based models that are tailored for extracting features from WSIs. Then, at the patient-level survival prediction we explicitly aggregate features from multiple WSIs, using consistency and contrastive losses to normalize slide-level features at the patient level. We conduct extensive experiments on three large-scale datasets: TCGA-GBM, TCGA-LUSC and NLST. Experimental results demonstrate the effectiveness of our proposed framework, as it achieves state-of-the-art performance in comparison with previous studies, with concordance index of 0.670, 0.679 and 0.711 on TCGA-GBM, TCGA-LUSC and NLST, respectively.

Multi-level fusion network for mild cognitive impairment identification using multi-modal neuroimages

Objective. Mild cognitive impairment (MCI) is a precursor to Alzheimer’s disease (AD) which is an irreversible progressive neurodegenerative disease and its early diagnosis and intervention are of great significance. Recently, many deep learning methods have demonstrated the advantages of multi-modal neuroimages in MCI identification task. However, previous studies frequently simply concatenate patch-level features for prediction without modeling the dependencies among local features. Also, many methods only focus on modality-sharable information or modality-specific features and ignore their incorporation. This work aims to address above-mentioned issues and construct a model for accurate MCI identification. Approach. In this paper, we propose a multi-level fusion network for MCI identification using multi-modal neuroimages, which consists of local representation learning and dependency-aware global representation learning stages. Specifically, for each patient, we first extract multi-pair of patches from multiple same position in multi-modal neuroimages. After that, in the local representation learning stage, multiple dual-channel sub-networks, each of which consists of two modality-specific feature extraction branches and three sine-cosine fusion modules, are constructed to learn local features that preserve modality-sharable and modality specific representations simultaneously. In the dependency-aware global representation learning stage, we further capture long-range dependencies among local representations and integrate them into global ones for MCI identification. Main results. Experiments on ADNI-1/ADNI-2 datasets demonstrate the superior performance of the proposed method in MCI identification tasks (Accuracy: 0.802, sensitivity: 0.821, specificity: 0.767 in MCI diagnosis task; accuracy: 0.849, sensitivity: 0.841, specificity: 0.856 in MCI conversion task) when compared with state-of-the-art methods. The proposed classification model has demonstrated a promising potential to predict MCI conversion and identify the disease-related regions in the brain. Significance. We propose a multi-level fusion network for MCI identification using multi-modal neuroimage. The results on ADNI datasets have demonstrated its feasibility and superiority.