Multiple Instance Learning Approach Research Articles

A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach for Immune Repertoire Classification

One individual human’s immune repertoire consists of a huge set of adaptive immune receptors at a certain time point, representing the individual's adaptive immune state. Immune repertoire classification and associated receptor identification have the potential to make a transformative contribution to the development of novel vaccines and therapies. The vast number of instances and exceedingly low witness rate pose a great challenge to the immune repertoire classification, which can be formulated as a Massive Multiple Instance Learning (MMIL) problem. Traditional MIL methods, at both bag-level and instance-level, confront the issues of substantial computational burden or supervision ambiguity when handling massive instances. To address these issues, we propose a novel label disambiguation-based multimodal massive multiple instance learning approach (LaDM³IL) for immune repertoire classification. LaDM³IL adapts the instance-level MIL paradigm to deal with the issue of high computational cost and employs a specially-designed label disambiguation module for label correction, mitigating the impact of misleading supervision. To achieve a more comprehensive representation of each receptor, LaDM³IL leverages a multimodal fusion module with gating-based attention and tensor-fusion to integrate the information from gene segments and amino acid (AA) sequences of each immune receptor. Extensive experiments on the Cytomegalovirus (CMV) and Cancer datasets demonstrate the superior performance of the proposed LaDM³IL for both immune repertoire classification and associated receptor identification tasks. The code is publicly available at https://github.com/Josie-xufan/LaDM3IL.

Multiple instance learning framework can facilitate explainability in murmur detection.

Cardiovascular diseases (CVDs) account for a high fatality rate worldwide. Heart murmurs can be detected from phonocardiograms (PCGs) and may indicate CVDs. Still, they are often overlooked as their detection and correct clinical interpretation require expert skills. In this work, we aim to predict the presence of murmurs and clinical outcomes from multiple PCG recordings employing an explainable multitask model. Our approach consists of a two-stage multitask model. In the first stage, we predict the murmur presence in single PCGs using a multiple instance learning (MIL) framework. MIL also allows us to derive sample-wise classifications (i.e. murmur locations) while only needing one annotation per recording ("weak label") during training. In the second stage, we fuse explainable hand-crafted features with features from a pooling-based artificial neural network (PANN) derived from the MIL framework. Finally, we predict the presence of murmurs and the clinical outcome for a single patient based on multiple recordings using a simple feed-forward neural network. We show qualitatively and quantitatively that the MIL approach yields useful features and can be used to detect murmurs on multiple time instances and may thus guide a practitioner through PCGs. We analyze the second stage of the model in terms of murmur classification and clinical outcome. We achieved a weighted accuracy of 0.714 and an outcome cost of 13612 when using the PANN model and demographic features on the CirCor dataset (hidden test set of the George B. Moody PhysioNet challenge 2022, team "Heart2Beat", rank 12 / 40). To the best of our knowledge, we are the first to demonstrate the usefulness of MIL in PCG classification. Also, we showcase how the explainability of the model can be analyzed quantitatively, thus avoiding confirmation bias inherent to many post-hoc methods. Finally, our overall results demonstrate the merit of employing MIL combined with handcrafted features for the generation of explainable features as well as for a competitive classification performance.

Deep Multi-Instance Conv-Transformer Frameworks for Landmark-Based Brain MRI Classification

For brain diseases, e.g., autism spectrum disorder (ASD), with unclear biological characteristics, the detection of imaging-based biomarkers is a critical task for diagnosis. Several landmark-based categorization approaches have been developed for the computer-aided diagnosis of brain diseases, such as Alzheimer’s disease (AD), utilizing structural magnetic resonance imaging (sMRI). With the automatic detection of the landmarks of brain disease, more detailed brain features were identified for clinical diagnosis. Multi-instance learning is an effective technique for classifying brain diseases based on landmarks. The multiple-instance learning approach relies on the assumption of independent distribution hypotheses and is mostly focused on local information, thus the correlation among different brain regions may be ignored. However, according to previous research on ASD and AD, the abnormal development of different brain regions is highly correlated. Vision Transformers, with self-attention modules to capture the relationship between embedded patches from a whole image, have recently demonstrated superior performances in many computer vision tasks. Nevertheless, the utilization of 3D brain MRIs imposes a substantial computational load, especially while training with Vision Transformer. To address the challenges mentioned above, in this research, we proposed a landmark-based multi-instance Conv-Transformer (LD-MILCT) framework as a solution to the aforementioned issues in brain disease diagnosis. In this network, a two-stage multi-instance learning strategy was proposed to explore both spatial and morphological information between different brain regions; the Vision Transformer utilizes a multi-instance learning head (MIL head) to fully utilize the features that are not involved in the ultimate classification. We assessed our proposed framework using T1-weighted MRI images from both AD and ASD databases. Our method outperformed existing deep learning and landmark-based methods in terms of brain MRI classification tasks.

Simultaneous instance pooling and bag representation selection approach for multiple-instance learning (MIL) using vision transformer

In multiple-instance learning (MIL), the existing bag encoding and attention-based pooling approaches assume that the instances in the bag have no relationship among them. This assumption is unsuited, as the instances in the bags are rarely independent in diverse MIL applications. In contrast, the instance relationship assumption-based techniques incorporate the instance relationship information in the classification process. However, in MIL, the bag composition process is complicated, and it may be possible that instances in one bag are related and instances in another bag are not. In present MIL algorithms, this relationship assumption is not explicitly modeled. The learning algorithm is trained based on one of two relationship assumptions (whether instances in all bags have a relationship or not). Hence, it is essential to model the assumption of instance relationships in the bag classification process. This paper proposes a robust approach that generates vector representation for the bag for both assumptions and the representation selection process to determine whether to consider the instances related or unrelated in the bag classification process. This process helps to determine the essential bag representation vector for every individual bag. The proposed method utilizes attention pooling and vision transformer approaches to generate bag representation vectors. Later, the representation selection subnetwork determines the vector representation essential for bag classification in an end-to-end trainable manner. The generalization abilities of the proposed framework are demonstrated through extensive experiments on several benchmark datasets. The experiments demonstrate that the proposed approach outperforms other state-of-the-art MIL approaches in bag classification.

Markedly Enhanced Analysis of Mass Spectrometry Images Using Weakly Supervised Machine Learning.

Supervised and unsupervised machine learning algorithms are routinelyapplied to time-of-flight secondary ion mass spectrometry (ToF-SIMS) imaging data and, more broadly, to mass spectrometry imaging (MSI). These algorithms have accelerated large-scale, single-pixel analysis, classification, and regression. However, there is relatively little research on methods suited for so-called weakly supervised problems, where ground-truth class labels exist at the image level, but not at the individual pixel level. Unsupervised learning methods are usually applied to these problems. However, these methods cannot make use of available labels. Here a novel method specifically designed for weakly supervised MSI data is presented. A dual-stream multiple instance learning (MIL) approach is adapted from computational pathology that reveals the spatial-spectral characteristics distinguishing different classes of MSI images. The method uses an information entropy-regularized attention mechanism to identify characteristic class pixels that are then used to extract characteristic mass spectra. This work provides a proof-of-concept exemplification using printed ink samples imaged by ToF-SIMS. A second application-oriented study is also presented, focusing on the analysis of a mixed powder sample type. Results demonstrate the potential of the MIL method for broader application in MSI, with implications for understanding subtle spatial-spectral characteristics in various applications and contexts.

Probabilistic Attention Based on Gaussian Processes for Deep Multiple Instance Learning.

Multiple instance learning (MIL) is a weakly supervised learning paradigm that is becoming increasingly popular because it requires less labeling effort than fully supervised methods. This is especially interesting for areas where the creation of large annotated datasets remains challenging, as in medicine. Although recent deep learning MIL approaches have obtained state-of-the-art results, they are fully deterministic and do not provide uncertainty estimations for the predictions. In this work, we introduce the attention Gaussian process (AGP) model, a novel probabilistic attention mechanism based on Gaussian processes (GPs) for deep MIL. AGP provides accurate bag-level predictions as well as instance-level explainability and can be trained end-to-end. Moreover, its probabilistic nature guarantees robustness to overfit on small datasets and uncertainty estimations for the predictions. The latter is especially important in medical applications, where decisions have a direct impact on the patient's health. The proposed model is validated experimentally as follows. First, its behavior is illustrated in two synthetic MIL experiments based on the well-known MNIST and CIFAR-10 datasets, respectively. Then, it is evaluated in three different real-world cancer detection experiments. AGP outperforms state-of-the-art MIL approaches, including deterministic deep learning ones. It shows a strong performance even on a small dataset with less than 100 labels and generalizes better than competing methods on an external test set. Moreover, we experimentally show that predictive uncertainty correlates with the risk of wrong predictions, and therefore it is a good indicator of reliability in practice. Our code is publicly available.

An end-to-end approach to combine attention feature extraction and Gaussian Process models for deep multiple instance learning in CT hemorrhage detection

Intracranial hemorrhage (ICH) is a serious life-threatening emergency caused by blood leakage inside the brain. Radiologists usually confirm the presence of ICH by analyzing computed tomography (CT) scans, so, developing an automated diagnosis system that can process this type of images has become an important research problem. One of the main challenges to apply AI algorithms in this setting is the lack of labeled data. To mitigate the labeling burden, Multiple Instance Learning (MIL) algorithms group instances into bags, relying solely on bag-level labels for model training. Due to their capacity to handle uncertainty and deliver accurate predictions, Gaussian Processes (GPs) stand out as promising classifiers for MIL problems. Recent research has also demonstrated the effectiveness of combining attention mechanisms with GPs for ICH detection. Nonetheless, existing methods have a notable limitation: they train the attention mechanism and the GP separately, resulting in suboptimal feature extraction for GP-based classification. In this study, we introduce an innovative end-to-end MIL model that concurrently trains the CNN backbone and attention mechanism along with the GP classifier. Our approach enhances the robustness and accuracy of bag predictions by optimizing feature extraction for GP-based classification. We validate our method experimentally by focusing on two ICH detection datasets. Our results reveal a significant performance advantage in terms of accuracy, F1-score, precision, and ROC-AUC score over existing MIL approaches, especially two-stage GP approaches. Additionally, we offer empirical insights into the functionality and effectiveness of our novel model.

Textural Analysis Supports Prostate MR Diagnosis in PIRADS Protocol

Prostate cancer is one of the most common cancers in the world. Due to the ageing of society and the extended life of the population, early diagnosis is a great challenge for healthcare. Unfortunately, the currently available diagnostic methods, in which magnetic resonance imaging (MRI) using the PIRADS protocol plays an increasingly important role, are imperfect, mostly in the inability to visualise small cancer foci and misinterpretation of the imagery data. Therefore, there is a great need to improve the methods currently applied and look for even better ones for the early detection of prostate cancer. In the presented research, anonymised MRI scans of 92 patients with evaluation in the PIRADS protocol were selected from the data routinely scanned for prostate cancer. Suspicious tissues were depicted manually under medical supervision. The texture features in the marked regions were calculated using the qMaZda software. The multiple-instance learning approach based on the SVM classifier allowed recognising between healthy and ill prostate tissue. The best F1 score equal to 0.77 with a very high recall equal to 0.70 and precision equal to 0.85 was recorded for the texture features describing the central zone. The research showed that the use of texture analysis in prostate MRI may allow for automation of the assessment of PIRADS scores.

Multiple Instance Learning Improves Ames Mutagenicity Prediction for Problematic Molecular Species.

The prediction of Ames mutagenicity continues to be a concern in both regulatory and pharmacological toxicology. Traditional quantitative structure-activity relationship (QSAR) models of mutagenicity make predictions based on molecular descriptors calculated on a chemical data set used in their training. However, it is known that molecules such as aromatic amines can be non-mutagenic themselves but metabolically activated by S9 rodent liver enzyme in Ames tests forming molecules such as iminoquinones or amine substituents that better stabilize mutagenic nitrenium ions in known pathways of mutagenicity. Modern in silico modeling methods can implicitly model these metabolites through consideration of the structural elements relevant to their formation but do not include explicit modeling of these metabolites' potential activity. These metabolites do not have a known individual mutagenicity label and, in their current state, cannot be fitted into a traditional QSAR model. Multiple instance learning (MIL) however can be applied to a group of metabolites and their parent under a single mutagenicity label. Here we trained MIL models on Ames data, first with an aromatic amines data set (n = 457), a class known to require metabolic activation, and subsequently on a larger data set (n = 6505) incorporating multiple molecular species. MIL was shown to be able to predict Ames mutagenicity with performance in line with previously established models (balanced accuracy = 0.778), suggesting its potential utility in Ames prediction applications. Furthermore, the MIL model predicted well on identified hard-to-predict molecule groups relative to the models in which these molecule groups were identified. These results are presumably due to the increased consideration of the metabolic contribution to the mutagenic outcome. Further exploration of MIL as a supplement to existing models could aid in the prediction of chemicals where implicit modeling of metabolites cannot fully grasp their characteristics. This paper demonstrates the potential of an MIL approach to modeling Ames tests with S9 and is particularly relevant to metabolically activated xenobiotic mutagens.

Malignant Mesothelioma subtyping via sampling driven multiple instance prediction on tissue image and cell morphology data

Malignant Mesothelioma is a difficult to diagnose and highly lethal cancer usually associated with asbestos exposure. It can be broadly classified into three subtypes: Epithelioid, Sarcomatoid, and a hybrid Biphasic subtype in which significant components of both of the previous subtypes are present. Early diagnosis and identification of the subtype informs treatment and can help improve patient outcome. However, the subtyping of malignant mesothelioma, and specifically the recognition of transitional features from routine histology slides has a high level of inter-observer variability.In this work, we propose an end-to-end multiple instance learning (MIL) approach for malignant mesothelioma subtyping. This uses an adaptive instance-based sampling scheme for training deep convolutional neural networks on bags of image patches that allows learning on a wider range of relevant instances compared to max or top-N based MIL approaches. We also investigate augmenting the instance representation to include aggregate cellular morphology features from cell segmentation. The proposed MIL approach enables identification of malignant mesothelial subtypes of specific tissue regions. From this a continuous characterisation of a sample according to predominance of sarcomatoid vs epithelioid regions is possible, thus avoiding the arbitrary and highly subjective categorisation by currently used subtypes. Instance scoring also enables studying tumor heterogeneity and identifying patterns associated with different subtypes. We have evaluated the proposed method on a dataset of 234 tissue micro-array cores with an AUROC of 0.89±0.05 for this task. The dataset and developed methodology is available for the community at: https://github.com/measty/PINS.

Point-Teaching: Weakly Semi-supervised Object Detection with Point Annotations

Point annotations are considerably more time-efficient than bounding box annotations. However, how to use cheap point annotations to boost the performance of semi-supervised object detection is still an open question. In this work, we present Point-Teaching, a weakly- and semi-supervised object detection framework to fully utilize the point annotations. Specifically, we propose a Hungarian-based point-matching method to generate pseudo labels for point-annotated images. We further propose multiple instance learning (MIL) approaches at the level of images and points to supervise the object detector with point annotations. Finally, we propose a simple data augmentation, named Point-Guided Copy-Paste, to reduce the impact of those unmatched points. Experiments demonstrate the effectiveness of our method on a few datasets and various data regimes. In particular, Point-Teaching outperforms the previous best method Group R-CNN by 3.1 AP with 5% fully labeled data and 2.3 AP with 30% fully labeled data on the MS COCO dataset. We believe that our proposed framework can largely lower the bar of learning accurate object detectors and pave the way for its broader applications. The code is available at https://github.com/YongtaoGe/Point-Teaching.

MHAttnSurv: Multi-head attention for survival prediction using whole-slide pathology images

Whole slide images (WSI) based survival prediction has attracted increasing interest in pathology. Despite this, extracting prognostic information from WSIs remains a challenging task due to their enormous size and the scarcity of pathologist annotations. Previous studies have utilized multiple instance learning approach to combine information from several randomly sampled patches, but this approach may not be adequate as different visual patterns may contribute unequally to prognosis prediction. In this study, we introduce a multi-head attention mechanism that allows each attention head to independently explore the utility of various visual patterns on a tumor slide, thereby enabling more comprehensive information extraction from WSIs. We evaluated our approach on four cancer types from The Cancer Genome Atlas database. Our model achieved an average c-index of 0.640, outperforming three existing state-of-the-art approaches for WSI-based survival prediction on these datasets. Visualization of attention maps reveals that the attention heads synergistically focus on different morphological patterns, providing additional evidence for the effectiveness of multi-head attention in survival prediction.

Abstract 5357: Estrogen receptor gene expression prediction from H&E-stained whole slide images

Abstract Gene expression profiling (GEP) represents an important approach to inform breast cancer treatment. However, access to GEP involves challenges associated with cost, tissue transportation, and turn around time. In this work, we explore the prediction of estrogen receptor gene (ESR1) expression directly from images of hematoxylin and eosin (H&E) stained, formalin-fixed paraffin-embedded (FFPE) breast cancer tissue. Since H&E staining is a fast and inexpensive component of the standard tissue preparation in pathology, this approach is tissue preserving and requires no additional tissue processing. Our method uses a deep multiple instance learning approach to process cropped image patches from whole-slide images (WSI) into a numeric embedding vector summarizing the information in each patch. A gated attention mechanism then aggregates these embeddings into a single prediction for the WSI. We train and tune the model on a site-based split of The Cancer Genome Atlas (TCGA) BRCA dataset, and evaluate it on both a heldout split of TCGA (independent sites) and a separate dataset from a tertiary teaching hospital (TTH). All splits of TCGA have ESR1 value, immunohistochemistry (IHC) estrogen receptor (ER) status, and limited clinical outcome data. The TTH dataset has IHC-based ER status and clinical outcome, but not ESR1 expression. On the TCGA heldout test split, our model’s root mean square error (RMSE) for predicting normalized gene expression counts (TPM) was 2.90 [95% CI: 2.57, 3.23], and the Pearson correlation was 0.57 [95%CI; 0.46, 0.67]. For predicting IHC-based ER status on the same TCGA split, this weakly-supervised ESR1-predicting model had an area under the receiver-operator curve (AUROC) of 0.81 [0.74, 0.87]. This was comparable to a strongly-supervised method directly predicting ER status (AUROC: 0.85 [0.77, 0.92]). Lastly, when evaluated for association with patient outcomes (progression-free interval; PFI) using the independent TTH dataset, this ESR1-predicting model had a concordance index (c-index) of 0.59 [0.52, 0.65]. For comparison, the c-index for PFI using the IHC-based ER status for these cases was 0.61 [0.54, 0.66]. This work further demonstrates the potential to infer gene expression from H&E stained images in a manner that shows meaningful associations with clinical variables. Because obtaining H&E stained images is substantially easier and faster than genetic testing, the capability to derive molecular genetic information from these images may increase access to this type of information for patient risk stratification and provide research insights into molecular-morphological associations. Future work incorporating more comprehensive sets of genes remains a valuable next step. Citation Format: Anvita A. Srinivas, Ronnachai Jaroensri, Ellery Wulczyn, James H. Wren, Elaine E. Thompson, Niels Olson, Fabien Beckers, Melissa Miao, Yun Liu, Cameron Chen, David F. Steiner. Estrogen receptor gene expression prediction from H&E-stained whole slide images. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5357.

Triple-kernel gated attention-based multiple instance learning with contrastive learning for medical image analysis.

In machine learning, multiple instance learning is a method evolved from supervised learning algorithms, which defines a "bag" as a collection of multiple examples with a wide range of applications. In this paper, we propose a novel deep multiple instance learning model for medical image analysis, called triple-kernel gated attention-based multiple instance learning with contrastive learning. It can be used to overcome the limitations of the existing multiple instance learning approaches to medical image analysis. Our model consists of four steps. i) Extracting the representations by a simple convolutional neural network using contrastive learning for training. ii) Using three different kernel functions to obtain the importance of each instance from the entire image and forming an attention map. iii) Based on the attention map, aggregating the entire image together by attention-based MIL pooling. iv) Feeding the results into the classifier for prediction. The results on different datasets demonstrate that the proposed model outperforms state-of-the-art methods on binary and weakly supervised classification tasks. It can provide more efficient classification results for various disease models and additional explanatory information.

Deep Multiple Instance Learning Approach for Classification in Clinical Decision Support Systems

To get around the drawbacks of conventional classification algorithms that required manual feature extraction and the high computational cost of neural networks, this paper introduces a deep convolutional neural network with multiple instance learning approaches, namely dynamic max pooling and sparse representation. For the categorization of tuberculosis lung illness, this model combines deep convolutional neural networks and multiple instance learning. The design was composed of four phases: pre-processing, instance production, feature extraction, and classification. To perform feature extraction, a model based on a customized version of the VGG16 architecture was trained from scratch. Multiple instance learning techniques such as Diverse Density (DD) and the Maximum pattern bag formulation of the Support Vector Machine were used to evaluate how well the proposed classification algorithm performed in comparison (SVM).The numerical findings demonstrated that the new method offered a higher level of accuracy than the methods that had been used in the past. When evaluating the efficacy of the current method, accuracy, specificity, sensitivity, and error rate were all taken into consideration. The accuracy of the max-pooling based framework and the sparse representation framework was found to be greater than that of the other multiple instance strategies, coming in at 91.51% and 89.84%, respectively, when compared to that of the other methods. The improved accuracy of the present system that makes use of deep neural networks is mostly attributable to the contributions made by features such as transfer learning and automatic feature extraction.

Multiple instance neural networks based on sparse attention for cancer detection using T-cell receptor sequences

Early detection of cancers has been much explored due to its paramount importance in biomedical fields. Among different types of data used to answer this biological question, studies based on T cell receptors (TCRs) are under recent spotlight due to the growing appreciation of the roles of the host immunity system in tumor biology. However, the one-to-many correspondence between a patient and multiple TCR sequences hinders researchers from simply adopting classical statistical/machine learning methods. There were recent attempts to model this type of data in the context of multiple instance learning (MIL). Despite the novel application of MIL to cancer detection using TCR sequences and the demonstrated adequate performance in several tumor types, there is still room for improvement, especially for certain cancer types. Furthermore, explainable neural network models are not fully investigated for this application. In this article, we propose multiple instance neural networks based on sparse attention (MINN-SA) to enhance the performance in cancer detection and explainability. The sparse attention structure drops out uninformative instances in each bag, achieving both interpretability and better predictive performance in combination with the skip connection. Our experiments show that MINN-SA yields the highest area under the ROC curve scores on average measured across 10 different types of cancers, compared to existing MIL approaches. Moreover, we observe from the estimated attentions that MINN-SA can identify the TCRs that are specific for tumor antigens in the same T cell repertoire.

1648P Deep learning identifies oncogenic genetic alterations in BRAF and NTRK in H&E whole slide images from thyroid carcinomas

Patient eligibility for matched oncogene-targeted therapies relies on detection by immunohistochemistry, Next Generation Sequencing (NGS) or PCR, but often is limited by clinical centers’ practice, lab capabilities and tissue-availability. Here, we tested the ability to predict oncogenic drivers (BRAF mutations and NTRK gene fusions) from thyroid tumors H&E whole slide images (WSIs) as a method to enrich for likely-positive patients that would merit NGS testing and benefit from targeted therapy. Using a set (N=833, of which BRAF+= 424, NTRK+= 43) of four cohorts (TCGA, Larotrectinib and Sorafenib trials and a biobank) of thyroid (mainly papillary and some anaplastic) carcinoma H&E stained digitized images we trained deep learning models to detect two known molecular oncogenic drivers: BRAF (mainly V600E) mutation (BRAF+) and NTRK gene fusions (NTRK+). We implemented an attention-based multiple instance learning (MIL) classifier and assessed its generalization in a leave-one-cohort-out paradigm. Different preanalytical variables (staining/section thickness/scanning) were evaluated. All slides and MIL heatmaps were reviewed by a pathologist to confirm areas of interest of the algorithm. Our MIL approach model detected both oncogenic drivers, with AUC up to 0.82 for BRAF+ and 0.90 for NTRK+ respectively. The model displayed highest attention in tumor areas (even in presence of immune infiltrate/normal glands), while being able to deal with various lab protocols in unseen cohorts. Our work shows that widely available H&E WSIs contain features that can predict the molecular status of BRAF+ and NTRK+, even when using a relatively small set of data (due to the rarity of NTRK+). Further refinement with additional datasets is ongoing. This deep learning method could offer a rapid and tissue-sparing alternative molecular diagnostic pathway supporting further detection and enrichment of NTRK+ patients. This would catalyze pathology workflows to bring additional efficiencies in testing of patients with the highest probability of having this rare molecular alteration, and provide them with opportunities to benefit from an NTRK targeted therapy.

Development and Evaluation of a Novel Deep-Learning-Based Framework for the Classification of Renal Histopathology Images.

Kidney cancer has several types, with renal cell carcinoma (RCC) being the most prevalent and severe type, accounting for more than 85% of adult patients. The manual analysis of whole slide images (WSI) of renal tissues is the primary tool for RCC diagnosis and prognosis. However, the manual identification of RCC is time-consuming and prone to inter-subject variability. In this paper, we aim to distinguish between benign tissue and malignant RCC tumors and identify the tumor subtypes to support medical therapy management. We propose a novel multiscale weakly-supervised deep learning approach for RCC subtyping. Our system starts by applying the RGB-histogram specification stain normalization on the whole slide images to eliminate the effect of the color variations on the system performance. Then, we follow the multiple instance learning approach by dividing the input data into multiple overlapping patches to maintain the tissue connectivity. Finally, we train three multiscale convolutional neural networks (CNNs) and apply decision fusion to their predicted results to obtain the final classification decision. Our dataset comprises four classes of renal tissues: non-RCC renal parenchyma, non-RCC fat tissues, clear cell RCC (ccRCC), and clear cell papillary RCC (ccpRCC). The developed system demonstrates a high classification accuracy and sensitivity on the RCC biopsy samples at the slide level. Following a leave-one-subject-out cross-validation approach, the developed RCC subtype classification system achieves an overall classification accuracy of 93.0% ± 4.9%, a sensitivity of 91.3% ± 10.7%, and a high classification specificity of 95.6% ± 5.2%, in distinguishing ccRCC from ccpRCC or non-RCC tissues. Furthermore, our method outperformed the state-of-the-art Resnet-50 model.

A multiple instance learning approach for detecting COVID-19 in peripheral blood smears.

A wide variety of diseases are commonly diagnosed via the visual examination of cell morphology within a peripheral blood smear. For certain diseases, such as COVID-19, morphological impact across the multitude of blood cell types is still poorly understood. In this paper, we present a multiple instance learning-based approach to aggregate high-resolution morphological information across many blood cells and cell types to automatically diagnose disease at a per-patient level. We integrated image and diagnostic information from across 236 patients to demonstrate not only that there is a significant link between blood and a patient's COVID-19 infection status, but also that novel machine learning approaches offer a powerful and scalable means to analyze peripheral blood smears. Our results both backup and enhance hematological findings relating blood cell morphology to COVID-19, and offer a high diagnostic efficacy; with a 79% accuracy and a ROC-AUC of 0.90.

GssMILP for anomaly classification in surveillance videos

Multiple Instance Learning (MIL) is widely used to locate abnormal video frames in surveillance videos due to its ability to work with weakly-labeled data. On the other hand, graph-based semi-supervised approaches are employed to handle label sparsity-related issues such as Low Degree-of-Supervision (LDoS) and High Class-Imbalance (HCI). However, the application of the MIL paradigm in anomaly classification of surveillance videos faces significant challenges such as (i) LDoS, (ii) HCI, and (iii) A multitude of types of abnormal events and actions instead of a unique pattern. The current work proposes a novel anomaly classifier to address these issues with a new objective function (GssMILP) by leveraging the benefits from graph-based semi-supervised and multiple-instance learning approaches. In this regard, our contributions are six-fold. First, a deep hierarchical architecture (C3D+3DCNN BICLSTM) to extract generic video descriptors. These descriptors represent the multitude of types of abnormal events and actions by incorporating the local–global spatial information and long-short-term temporal information related to the object motion, human actions, and behavioral features. Second, we develop a learning objective (GssMILP) that extends the MIL-based instance-level labeler to graph-based semi-supervising. Third, we use three different regularizer terms, namely graph-based regularizer, non-convex L2-regularizer, and temporal-smoothness regularizer. Fourth, we present an elaborate experimental study comparing the performance of the proposed approach with six similar baseline approaches. The results indicate that at degree-of-supervision 1.0%, the proposed method improves the f1-score by 45.26%, 18.19%, 45.28%, and 9.22% in UCSD-ped1, UCSD-ped2, MED, and RWAD datasets, respectively. Fifth, we present an ablation study on the impact of different features on the performance of an anomaly classifier. Sixth, we provide temporal annotations for a label resource of 1900 videos in the RWAD dataset.