Multiple Instance Learning Algorithm Research Articles

An interpretable deep learning model for detecting BRCA pathogenic variants of breast cancer from hematoxylin and eosin-stained pathological images.

Determining the status of breast cancer susceptibility genes (BRCA) is crucial for guiding breast cancer treatment. Nevertheless, the need for BRCA genetic testing among breast cancer patients remains unmet due to high costs and limited resources. This study aimed to develop a Bi-directional Self-Attention Multiple Instance Learning (BiAMIL) algorithm to detect BRCA status from hematoxylin and eosin (H&E) pathological images. A total of 319 histopathological slides from 254 breast cancer patients were included, comprising two dependent cohorts. Following image pre-processing, 633,484 tumor tiles from the training dataset were employed to train the self-developed deep-learning model. The performance of the network was evaluated in the internal and external test sets. BiAMIL achieved AUC values of 0.819 (95% CI [0.673-0.965]) in the internal test set, and 0.817 (95% CI [0.712-0.923]) in the external test set. To explore the relationship between BRCA status and interpretable morphological features in pathological images, we utilized Class Activation Mapping (CAM) technique and cluster analysis to investigate the connections between BRCA gene mutation status and tissue and cell features. Significantly, we observed that tumor-infiltrating lymphocytes and the morphological characteristics of tumor cells appeared to be potential features associated with BRCA status. An interpretable deep neural network model based on the attention mechanism was developed to predict the BRCA status in breast cancer. Keywords: Breast cancer, BRCA, deep learning, self-attention, interpretability.

Deep Learning Classification of Usual Interstitial Pneumonia Predicts Outcomes.

Rationale: Computed tomography (CT) enables noninvasive diagnosis of usual interstitial pneumonia (UIP), but enhanced image analyses are needed to overcome the limitations of visual assessment. Objectives: Apply multiple instance learning (MIL) to develop an explainable deep learning algorithm for prediction of UIP from CT and validate its performance in independent cohorts. Methods: We trained an MIL algorithm using a pooled dataset (n = 2,143) and tested it in three independent populations: data from a prior publication (n = 127), a single-institution clinical cohort (n = 239), and a national registry of patients with pulmonary fibrosis (n = 979). We tested UIP classification performance using receiver operating characteristic analysis, with histologic UIP as ground truth. Cox proportional hazards and linear mixed-effects models were used to examine associations between MIL predictions and survival or longitudinal FVC. Measurements and Main Results: In two cohorts with biopsy data, MIL improved accuracy for histologic UIP (area under the curve, 0.77 [n = 127] and 0.79 [n = 239]) compared with visual assessment (area under the curve, 0.65 and 0.71). In cohorts with survival data, MIL-UIP classifications were significant for mortality (n = 239, mortality to April 2021: unadjusted hazard ratio, 3.1; 95% confidence interval [CI], 1.96-4.91; P < 0.001; and n = 979, mortality to July 2022: unadjusted hazard ratio, 3.64; 95% CI, 2.66-4.97; P < 0.001). Individuals classified as UIP positive by the algorithm had a significantly greater annual decline in FVC than those classified as UIP negative (-88 ml/yr vs. -45 ml/yr; n = 979; P < 0.01), adjusting for extent of lung fibrosis. Conclusions: Computerized assessment using MIL identifies clinically significant features of UIP on CT. Such a method could improve confidence in radiologic assessment of patients with interstitial lung disease, potentially enabling earlier and more precise diagnosis.

An integrated model combined intra- and peritumoral regions for predicting chemoradiation response of non small cell lung cancers based on radiomics and deep learning

PurposeThe purpose of this study was to develop a model for predicting chemoradiation response in non-small cell lung cancer (NSCLC) patients by integrating radiomics and deep-learning features and combined intra- and peritumoral regions with pre-treated CT images. Materials and methodsThis study enrolled 462 patients with NSCLC who received chemoradiation. On the basis of pretreated CT images, we developed three models to compare the prediction of chemoradiation: intratumoral, peritumoral and combined regions. To further illustrate each model, we established different feature integration methods: a) radiomics model with 1500 features; b) deep learning model with a multiple instance learning algorithm; c) integrated model by integrating radiomic and deep learning features. For radiomics and integrated models, support vector machine and the least absolute shrinkage and selection operator were used to extract and select features. Transfer learning and max pooling algorithms were used to identify high informative features in deep learning models. We applied ten-fold cross validation in model training and testing. ResultsThe best area under the curve (AUC) of intratumoral, peritumoral and combined models were 0.89 (95% CI, 0.74–0.93), 0.86 (95% CI, 0.75–0.92) and 0.92 (95% CI, 0.81–0.95), respectively. It indicated the importance of the peritumoral region for treatment response prediction and should be used in combination with the intratumoral region. Integrated models gave better results than models with radiomics and deep learning features alone in all regions of interest and radiomics models outperformed deep learning models in any comparative models. ConclusionsThe model that integrate radiomic and deep learning features and combined intra- and peritumoral regions provide valuable information in predicting treatment response of chemoradiation. It can help oncologists customize personalized clinical treatment plans for NSCLC patients.

GNN-Based Embedded Framework for Consumer Affect Recognition Using Thermal Facial ROIs

Recently consumer electronics products for the human device or machine interaction in smart healthcare systems have been widespread due to progress in health monitoring hardware and remote diagnostic services. Affect recognition through thermal facial signatures is significant in real-time Human-machine Interaction (HMI) studies. The distribution of facial skin temperature displays explicit characteristics related to affect arousal. During human interacts with a machine or computer, it is challenging to detect face and track the facial regions of interest (ROI) in a thermal video because of head motion artifacts. Our proposed embedded portable HMI product integrated with the required hardware and deep learning architecture is used to recognize the human affect in real-time. The transfer learning approach (Faster R-CNN) and the Multiple Instance Learning (MIL) algorithm are applied to thermal video for thermal face and ROIs detection and tracking. The multivariate time series (MTS) features generated from specific ROIs' calculated mean intensity variations. This study proposes a k-Nearest Neighbor (k-NN) algorithm-based graph neural network (GNN) architecture that utilizes MTS features to recognize affects in the facial thermal video. The effectiveness of the proposed algorithm for the In-house and NVIE thermal data-set gives better accuracy of 76.05% and 71.91% compared with state-of-the-art approaches.

A robust and lightweight deep attention multiple instance learning algorithm for predicting genetic alterations.

Self-attention mechanism-based algorithms are attractive in digital pathology due to their interpretability, but suffer from computation complexity. This paper presents a novel, lightweight Attention-based Multiple Instance Mutation Learning (AMIML) model to allow small-scale attention operations for predicting gene mutations. Compared to the standard self-attention model, AMIML reduces the number of model parameters by approximately 70%. Using data for 24 clinically relevant genes from four cancer cohorts in TCGA studies (UCEC, BRCA, GBM, and KIRC), we compare AMIML with a standard self-attention model, five other deep learning models, and four traditional machine learning models. The results show that AMIML has excellent robustness and outperforms all the baseline algorithms in the vast majority of the tested genes. Conversely, the performance of the reference deep learning and machine learning models vary across different genes, and produce suboptimal prediction for certain genes. Furthermore, with the flexible and interpretable attention-based pooling mechanism, AMIML can further zero in and detect predictive image patches.

Explainable AI identifies diagnostic cells of genetic AML subtypes.

Explainable AI is deemed essential for clinical applications as it allows rationalizing model predictions, helping to build trust between clinicians and automated decision support tools. We developed an inherently explainable AI model for the classification of acute myeloid leukemia subtypes from blood smears and found that high-attention cells identified by the model coincide with those labeled as diagnostically relevant by human experts. Based on over 80,000 single white blood cell images from digitized blood smears of 129 patients diagnosed with one of four WHO-defined genetic AML subtypes and 60 healthy controls, we trained SCEMILA, a single-cell based explainable multiple instance learning algorithm. SCEMILA could perfectly discriminate between AML patients and healthy controls and detected the APL subtype with an F1 score of 0.86±0.05 (mean±s.d., 5-fold cross-validation). Analyzing a novel multi-attention module, we confirmed that our algorithm focused with high concordance on the same AML-specific cells as human experts do. Applied to classify single cells, it is able to highlight subtype specific cells and deconvolve the composition of a patient's blood smear without the need of single-cell annotation of the training data. Our large AML genetic subtype dataset is publicly available, and an interactive online tool facilitates the exploration of data and predictions. SCEMILA enables a comparison of algorithmic and expert decision criteria and can present a detailed analysis of individual patient data, paving the way to deploy AI in the routine diagnostics for identifying hematopoietic neoplasms.

An accurate prediction of the origin for bone metastatic cancer using deep learning on digital pathological images.

Determining the origin of bone metastatic cancer (OBMC) is of great significance to clinical therapeutics. It is challenging for pathologists to determine the OBMC with limited clinical information and bone biopsy. We designed a regional multiple-instance learning algorithm to predict the OBMC based on hematoxylin-eosin (H&E) staining slides alone. We collected 1041 cases from eight different hospitals and labeled 26,431 regions of interest to train the model. The performance of the model was assessed by ten-fold cross validation and external validation. Under the guidance of top3 predictions, we conducted an IHC test on 175 cases of unknown origins to compare the consistency of the results predicted by the model and indicated by the IHC markers. We also applied the model to identify whether there was tumor or not in a region, as well as distinguishing squamous cell carcinoma, adenocarcinoma, and neuroendocrine tumor. In the within-cohort, our model achieved a top1-accuracy of 91.35% and a top3-accuracy of 97.75%. In the external cohort, our model displayed a good generalizability with a top3-accuracy of 97.44%. The top1 consistency between the results of the model and the immunohistochemistry markers was 83.90% and the top3 consistency was 94.33%. The model obtained an accuracy of 98.98% to identify whether there was tumor or not and an accuracy of 93.85% to differentiate three types of cancers. Our model demonstrated good performance to predict the OBMC from routine histology and had great potential for assisting pathologists with determining the OBMC accurately. National Science Foundation of China (61875102 and 61975089), Natural Science Foundation of Guangdong province (2021A15-15012379 and 2022A1515 012550), Science and Technology Research Program of Shenzhen City (JCYJ20200109110606054 and WDZC20200821141349001), and Tsinghua University Spring Breeze Fund (2020Z99CFZ023).

Interpretable deep learning of myelin histopathology in age-related cognitive impairment

Age-related cognitive impairment is multifactorial, with numerous underlying and frequently co-morbid pathological correlates. Amyloid beta (Aβ) plays a major role in Alzheimer’s type age-related cognitive impairment, in addition to other etiopathologies such as Aβ-independent hyperphosphorylated tau, cerebrovascular disease, and myelin damage, which also warrant further investigation. Classical methods, even in the setting of the gold standard of postmortem brain assessment, involve semi-quantitative ordinal staging systems that often correlate poorly with clinical outcomes, due to imperfect cognitive measurements and preconceived notions regarding the neuropathologic features that should be chosen for study. Improved approaches are needed to identify histopathological changes correlated with cognition in an unbiased way. We used a weakly supervised multiple instance learning algorithm on whole slide images of human brain autopsy tissue sections from a group of elderly donors to predict the presence or absence of cognitive impairment (n = 367 with cognitive impairment, n = 349 without). Attention analysis allowed us to pinpoint the underlying subregional architecture and cellular features that the models used for the prediction in both brain regions studied, the medial temporal lobe and frontal cortex. Despite noisy labels of cognition, our trained models were able to predict the presence of cognitive impairment with a modest accuracy that was significantly greater than chance. Attention-based interpretation studies of the features most associated with cognitive impairment in the top performing models suggest that they identified myelin pallor in the white matter. Our results demonstrate a scalable platform with interpretable deep learning to identify unexpected aspects of pathology in cognitive impairment that can be translated to the study of other neurobiological disorders.

Interpretable prediction of necrotizing enterocolitis from machine learning analysis of premature infant stool microbiota

BackgroundNecrotizing enterocolitis (NEC) is a common, potentially catastrophic intestinal disease among very low birthweight premature infants. Affecting up to 15% of neonates born weighing less than 1500 g, NEC causes sudden-onset, progressive intestinal inflammation and necrosis, which can lead to significant bowel loss, multi-organ injury, or death. No unifying cause of NEC has been identified, nor is there any reliable biomarker that indicates an individual patient’s risk of the disease. Without a way to predict NEC in advance, the current medical strategy involves close clinical monitoring in an effort to treat babies with NEC as quickly as possible before irrecoverable intestinal damage occurs. In this report, we describe a novel machine learning application for generating dynamic, individualized NEC risk scores based on intestinal microbiota data, which can be determined from sequencing bacterial DNA from otherwise discarded infant stool. A central insight that differentiates our work from past efforts was the recognition that disease prediction from stool microbiota represents a specific subtype of machine learning problem known as multiple instance learning (MIL).ResultsWe used a neural network-based MIL architecture, which we tested on independent datasets from two cohorts encompassing 3595 stool samples from 261 at-risk infants. Our report also introduces a new concept called the “growing bag” analysis, which applies MIL over time, allowing incorporation of past data into each new risk calculation. This approach allowed early, accurate NEC prediction, with a mean sensitivity of 86% and specificity of 90%. True-positive NEC predictions occurred an average of 8 days before disease onset. We also demonstrate that an attention-gated mechanism incorporated into our MIL algorithm permits interpretation of NEC risk, identifying several bacterial taxa that past work has associated with NEC, and potentially pointing the way toward new hypotheses about NEC pathogenesis. Our system is flexible, accepting microbiota data generated from targeted 16S or “shotgun” whole-genome DNA sequencing. It performs well in the setting of common, potentially confounding preterm neonatal clinical events such as perinatal cardiopulmonary depression, antibiotic administration, feeding disruptions, or transitions between breast feeding and formula.ConclusionsWe have developed and validated a robust MIL-based system for NEC prediction from harmlessly collected premature infant stool. While this system was developed for NEC prediction, our MIL approach may also be applicable to other diseases characterized by changes in the human microbiota.

A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences

Infrared-Thermal Imaging is a non-contact mechanism for psychophysiological research and application in Human–Computer Interaction (HCI). Real-time detection of the face and tracking the Regions of Interest (ROI) in the thermal video during HCI is challenging due to head motion artifacts. This paper proposes a three-stage HCI framework for computing the multivariate time-series thermal video sequences to recognize human emotion and provides distraction suggestions. The first stage comprises of face, eye, and nose detection using a Faster R-CNN (region-based convolutional neural network) architecture and used Multiple Instance Learning (MIL) algorithm for tracking the face ROIs across the thermal video. The mean intensity of ROIs is calculated which forms a multivariate time series (MTS) data. In the second stage, the smoothed MTS data are passed to the Dynamic Time Warping (DTW) algorithm to classify emotional states elicited by video stimulus. During HCI, our proposed framework provides relevant suggestions from a psychological and physical distraction perspective in the third stage. Our proposed approach signifies better accuracy in comparison with other classification methods and thermal data-sets.

Random Graph-based Multiple Instance Learning for Structured IoT Smart City Applications

Because of the complex activities involved in IoT networks of a smart city, an important question arises: What are the core activities of the networks as a whole and its basic information flow structure? Identifying and discovering core activities and information flow is a crucial step that can facilitate the analysis. This is the question we are addressing—that is, to identify the core services as a common core substructure despite the probabilistic nature and the diversity of its activities. If this common substructure can be discovered, a systemic analysis and planning can then be performed and key policies related to the community can be developed. Here, a local IoT network can be represented as an attributed graph. From an ensemble of attributed graphs, identifying the common subgraph pattern is then critical in understanding the complexity. We introduce this as the common random subgraph (CRSG) modeling problem, aiming at identifying a subgraph pattern that is the structural “core” that conveys the probabilistically distributed graph characteristics. Given an ensemble of network samples represented as attributed graphs, the method generates a CRSG model that encompasses both structural and statistical characteristics from the related samples while excluding unrelated networks. In generating a CRSG model, our method using a multiple instance learning algorithm transforms an attributed graph (composed of structural elements as edges and their two endpoints) into a “bag” of instances in a vector space. Common structural components across positively labeled graphs are then identified as the common instance patterns among instances across different bags. The structure of the CRSG arises through the combining of common patterns. The probability distribution of the CRSG can then be estimated based on the connections and distributions from the common elements. Experimental results demonstrate that CRSG models are highly expressive in describing typical network characteristics.

Multiple-instance Learning based on Bernoulli Mixture Model

Multiple-instance learning (MIL) is a form of weakly supervised learning. Its instances are arranged in groups (called bags), and labels are provided for the entire bag after training. The classic MIL method represents examples with pre-calculated features, and the classification process is also cumbersome. In this paper, we use neural networks to extract features, parameterize all transformations, and use Bernoulli mixture model to construct MIL models for baggage tags, using simpler network structures and more accurately solving these problems. Experiments show that our results can be competitive with the classical MIL algorithm on the MINST dataset.

Multiple-Instance Learning Approach via Bayesian Extreme Learning Machine

Multiple-instance learning (MIL) can solve supervised learning tasks, where only a bag of multiple instances is labeled, instead of a single instance. It is considerably important to develop effective and efficient MIL algorithms, because real-world datasets usually contain large instances. Known for its good generalization performance, MIL based on extreme learning machines (ELM-MIL) has proven to be more efficient than several typical MIL classification methods. ELM-MIL selects the most qualified instances from each bag through a single hidden layer feedforward network (SLFN) and trains modified ELM models to update the output weights. This learning approach often performs susceptible to the number of hidden nodes and can easily suffer from over-fitting problem. Using Bayesian inferences, this study introduces a Bayesian ELM (BELM)-based MIL algorithm (BELM-MIL) to address MIL classification problems. First, weight self-learning method based on a Bayesian network is applied to determine the weights of instance features. The most qualified instances are then selected from each bag to represent the bag. Second, BELM can improve the classification model via regularization of automatic estimations to reduce possible over-fitting during the calibration process. Experiments and comparisons are conducted with several competing algorithms on Musk datasets, images datasets, and inductive logic programming datasets. Superior classification accuracy and performance are demonstrated by BELM-MIL.

Rotation-Invariant Feature Learning for Object Detection in VHR Optical Remote Sensing Images by Double-Net

Rotation-invariant feature extraction is crucial for object detection in Very High Resolution (VHR) optical remote sensing images. Although convolutional neural networks (CNNs) are good at extracting the translation-invariant features and have been widely applied in computer vision, it is still a challenging problem for CNNs to extract rotation-invariant features in VHR optical remote sensing images. In this paper we present a novel Double-Net with sample pairs from the same class as inputs to improve the performance of object detection and classification in VHR optical remote sensing images. Specifically, the proposed Double-Net contains multiple channels of CNNs in which each channel refers to a specific rotation direction and all CNNs share identical weights. Based on the output features of all channels, multiple instance learning algorithm is employed to extract the final rotation-invariant features. Experimental results on two publicly available benchmark datasets, namely Mnist-rot-12K and NWPU VHR-10, demonstrate that the presented Double-Net outperforms existing approaches on the performance of rotation-invariant feature extraction, which is especially effective under the situation of small training samples.

Adaptively Denoising Proposal Collection for Weakly Supervised Object Localization

In this paper, we address the problem of weakly supervised object localization (WSL), which trains a detection network on the dataset with only image-level annotations. The proposed approach is built on the observation that the proposal set from the training dataset is a collection of background, object parts, and objects. Several strategies are taken to adaptively eliminate the noisy proposals and generate pseudo object-level annotations for the weakly labeled dataset. A multiple instance learning (MIL) algorithm enhanced by mask-out strategy is adopted to collect the class-specific object proposals, which are then utilized to adapt a pre-trained classification network to a detection network. In addition, the detection results from the detection network are re-weighted by jointly considering the detection scores and the overlap ratio of proposals in a proposal subset optimization framework. The optimal proposals work as object-level labels that enable a pseudo-strongly supervised dataset for training the detection network. Consequently, we establish a fully adaptive detection network. Extensive evaluations on the PASCAL VOC 2007 and 2012 datasets demonstrate a significant improvement compared with the state-of-the-art methods.

Melanoma Detection by Means of Multiple Instance Learning.

We present an application to melanoma detection of a multiple instance learning (MIL) approach, whose objective, in the binary case, is to discriminate between positive and negative sets of items. In the MIL terminology these sets are called bags and the items inside the bags are called instances. Under the hypothesis that a bag is positive if at least one of its instances is positive and it is negative if all its instances are negative, the MIL paradigm fits very well with images classification, since an image (bag) is in general classified on the basis of some its subregions (instances). In this work we have applied a MIL algorithm on some clinical data constituted by color dermoscopic images, with the aim to discriminate between melanomas (positive images) and common nevi (negative images). In comparison with standard classification approaches, such as the well known support vector machine, our method performs very well in terms both of accuracy and sensitivity. In particular, using a leave-one-out validation on a data set constituted by 80 melanomas and 80 common nevi, we have obtained the following results: accuracy = 92.50%, sensitivity = 97.50% and specificity = 87.50%. Since the results appear promising, we conclude that a MIL technique could be at the basis of more sophisticated tools useful to physicians in melanoma detection.

Online Visual Tracking of Weighted Multiple Instance Learning via Neutrosophic Similarity-Based Objectness Estimation

An online neutrosophic similarity-based objectness tracking with a weighted multiple instance learning algorithm (NeutWMIL) is proposed. Each training sample is extracted surrounding the object location, and the distribution of these samples is symmetric. To provide a more robust weight for each sample in the positive bag, the asymmetry of the importance of the samples is considered. The neutrosophic similarity-based objectness estimation with object properties (super straddling) is applied. The neutrosophic theory is a new branch of philosophy for dealing with incomplete, indeterminate, and inconsistent information. By considering the surrounding information of the object, a single valued neutrosophic set (SVNS)-based segmentation parameter selection method is proposed, to produce a well-built set of superpixels which can better explain the object area at each frame. Then, the intersection and shape-distance criteria are proposed for weighting each superpixel in the SVNS domain, mainly via three membership functions, T (truth), I (indeterminacy), and F (falsity), for each criterion. After filtering out the superpixels with low response, the newly defined neutrosophic weights are utilized for weighting each sample. Furthermore, the objectness estimation information is also applied for estimating and alleviating the problem of tracking drift. Experimental results on challenging benchmark video sequences reveal the superior performance of our algorithm when confronting appearance changes and background clutters.

基于相关相似度的在线多示例学习目标跟踪算法

基于相关相似度的在线多示例学习目标跟踪算法

Automatic landmark discovery for learning agents under partial observability

AbstractIn the reinforcement learning context, a landmark is a compact information which uniquely couples a state, for problems with hidden states. Landmarks are shown to support finding good memoryless policies for Partially Observable Markov Decision Processes (POMDP) which contain at least one landmark. SarsaLandmark, as an adaptation of Sarsa(λ), is known to promise a better learning performance with the assumption that all landmarks of the problem are known in advance.In this paper, we propose a framework built upon SarsaLandmark, which is able to automatically identify landmarks within the problem during learning without sacrificing quality, and requiring no prior information about the problem structure. For this purpose, the framework fuses SarsaLandmark with a well-known multiple-instance learning algorithm, namely Diverse Density (DD). By further experimentation, we also provide a deeper insight into our concept filtering heuristic to accelerate DD, abbreviated as DDCF (Diverse Density with Concept Filtering), which proves itself to be suitable for POMDPs with landmarks. DDCF outperforms its antecedent in terms of computation speed and solution quality without loss of generality.The methods are empirically shown to be effective via extensive experimentation on a number of known and newly introduced problems with hidden state, and the results are discussed.

Leveraging Content Sensitiveness and User Trustworthiness to Recommend Fine-Grained Privacy Settings for Social Image Sharing

To configure successful privacy settings for social image sharing, two issues are inseparable: 1) content sensitiveness of the images being shared; and 2) trustworthiness of the users being granted to see the images. This paper aims to consider these two inseparable issues simultaneously to recommend fine-grained privacy settings for social image sharing. For achieving more compact representation of image content sensitiveness (privacy), two approaches are developed: 1) a deep network is adapted to extract 1024-D discriminative deep features; and 2) a deep multiple instance learning algorithm is adopted to identify 280 privacy-sensitive object classes and events. Second, users on the social network are clustered into a set of representative social groups to generate a discriminative dictionary for user trustworthiness characterization. Finally, both the image content sensitiveness and the user trustworthiness are integrated to train a tree classifier to recommend fine-grained privacy settings for social image sharing. Our experimental studies have demonstrated both the efficiency and the effectiveness of our proposed algorithms.