Multi-instance Learning Methods Research Articles

Hybrid multiple instance learning network for weakly supervised medical image classification and localization

Weakly supervised medical image analysis is of great significance for computer-aided diagnosis due to the difficulty in obtaining accurately labeled medical data. In this paper, we proposed a new Multi-instance Learning (MIL) framework called HybridMIL integrating CNN Convolutional Neural Networks (CNN) and Broad Learning Systems (BLS). Our HybridMIL can overcome several challenging issues over existing MIL methods based on either CNN or BLS alone: (i) Multiple levels (i.e., different resolutions) of feature information can be simultaneously extracted through a newly proposed instance-level feature enhancement (IFE) module; (ii) Global-level semantic information contained in the deep layers can be better represented under the global-level semantic enhancement (GSE) module; (iii) Hybrid feature fusion (HFF) module is newly designed to effectively fuse and align the multi-level outputs of IFE and global-level semantic information of GSE for subsequent classification and localization tasks. The proposed HybridMIL is evaluated on various public medical and MIL benchmark datasets. The results indicate that HybridMIL surpasses other recent MIL models in terms of classification and localization performance by up to 8.5% and 9.0%, respectively. Lastly, we demonstrate the highly competitive performance of HybridMIL in general MIL problems, going beyond weakly supervised medical image analysis.

Data-Driven Knowledge Fusion for Deep Multi-Instance Learning.

Multi-instance learning (MIL) is a widely applied technique in practical applications that involve complex data structures. MIL can be broadly categorized into two types: traditional methods and those based on deep learning. These approaches have yielded significant results, especially regarding their problem-solving strategies and experiment validation, providing valuable insights for researchers in the MIL field. However, considerable knowledge is often trapped within the algorithm, leading to subsequent MIL algorithms that rely solely on the model's data fitting to predict unlabeled samples. This results in a significant loss of knowledge and impedes the development of more powerful models. In this article, we propose a novel data-driven knowledge fusion for deep MIL (DKMIL) algorithm. DKMIL adopts a completely different idea from existing deep MIL methods by analyzing the decision-making of key samples in the dataset (referred to as the data-driven) and using the knowledge fusion module designed to extract valuable information from these samples to assist the model's learning. In other words, this module serves as a new interface between data and the model, providing strong scalability and enabling prior knowledge from existing algorithms to enhance the model's learning ability. Furthermore, to adapt the downstream modules of the model to more knowledge-enriched features extracted from the data-driven knowledge fusion (DDKF) module, we propose a two-level attention (TLA) module that gradually learns shallow-and deep-level features of the samples to achieve more effective classification. We will prove the scalability of the knowledge fusion module and verify the efficiency of the proposed architecture by conducting experiments on 62 datasets across five categories.

An active learning enhanced data programming (ActDP) framework for ECG time series

Supervised machine learning learns a mapping from input data to output labels, based on the patterns and relationships present in a huge labelled training data.Getting labelled data generally requires a substantial allocation of resources in terms of cost and time. In such scenarios, weak supervised learning techniques like data programming (DP) and active learning (AL) can be advantageous for time-series classification tasks. These paradigms can be used to assign data labels in an automated manner, and time-series classification can subsequently be carried out on the labeled data. This work proposes a novel framework titled AL enhanced data programming (ActDP). It uses a combination of DP and AL for electrocardiogram (ECG) beat classification using single-lead data. ECG beat classification is pivotal in cardiology and healthcare applications for diagnosing a broad spectrum of heart conditions and arrhythmias. To establish the usefulness of this proposed ActDP framework, the experiments have been conducted using the MIT-BIH dataset with 94,224 ECG beats. DP assigns a probabilistic label to each ECG beat using nine novel polar labelling functions and a generative model in this work. Further, AL improves the result of DP by replacing the labels for sampled ECG beats of a generative model with ground truth. Subsequently, a discriminative model is trained on these labels for each iteration. The experimental results show that by incorporating AL into DP in the ActDP framework, the accuracy of ECG classification strictly increases from 85.7% to 97.34% in 58 iterations. Comparatively, the proposed framework (ActDP) has demonstrated a higher classification accuracy of 97.34%. In contrast, DP with data augmentation (DA) achieves an accuracy of 92.2%, while DP without DA results in an accuracy of 85.7%, few-shot learning techniques yield 87.5%–89.2%, and multi-instance learning methods achieve accuracies in the range of 88.9%–94.1%

Superpixel-based multi-scale multi-instance learning for hyperspectral image classification

Superpixels can define meaningful local regions within a hyperspectral image (HSI) and have become the building blocks of various HSI classification methods. The superpixels in HSIs consist of multiple spectral pixels, sharing a similar structure with the data in multi-instance learning (MIL). However, the potential of MIL methods in the field of HSI classification has been rarely explored. In this paper, we propose the superpixel-based multi-scale multi-instance learning (MSMIL) framework, enhancing the superpixel representation with MIL for the first time. Segmenting the HSIs with superpixels, MSMIL converts the HSI classification into MIL problems and extracts superpixel representations via the MIL method, namely multi-instance factor analysis (MIFA). Compared with the existing methods focusing exclusively on the local information, MIFA utilizes the deviations from an overall generative model to describe the superpixels, retaining both the local and the global information. Moreover, MSMIL introduces multi-scale superpixels and a spectral–spatial decision fusion strategy for further refinement, where the results of multi-scale superpixel maps are weighted according to prediction certainty and spatial consistency. The proposed method is evaluated on four benchmark datasets and achieves competitive results. For example, MSMIL outperforms the comparison methods with a margin of 5% overall accuracy on the Indian Pines dataset, when 2% pixels are selected as the training set.

Double similarities weighted multi-instance learning kernel and its application

Multi-instance learning (MIL), as a special version of classification, focuses on labeled sets (bags) consisting of unlabeled instances and has drawn accumulative attention due to its significant importance in practical applications. However, most existing MIL methods just utilize partial information (bags or instances) of MIL data to construct the kernel function, resulting in deteriorated classification performance of MIL. In this paper, we propose a Double Similarities weighted Multi-Instance Learning (DSMIL) kernel framework, which utilizes the similarities of Bag-to-Bag (B2B) and Instance-to-Bag (I2B). In the proposed kernel framework, the similarities of B2B and I2B could be derived from the prototypes distance of inter-bag and similarity matrix of intra-bag, respectively, based on the affinity propagation (AP) clustering of the bag. Meanwhile, we give theoretical proof of the validity of the designed kernel function. Experimental results on benchmark and semi-synthetic datasets show that our proposed method obtains competitive classification performance and achieves robustness to parameters and noise.

Interpreting vulnerabilities of multi-instance learning to adversarial perturbations

Multi-instance learning (MIL) is a recent machine learning paradigm which is immensely useful in various real-life applications, like image analysis, video anomaly detection, text classification, etc. It is well known that most of the existing machine learning classifiers are highly vulnerable to adversarial perturbations. Since MIL is a weakly supervised learning, where information is available for a set of instances, called bag and not for every instance, adversarial perturbations can be fatal. In this paper, we have proposed two adversarial perturbation methods to analyze the effect of adversarial perturbations to interpret the vulnerabilities of MIL methods. Out of the two algorithms, one can be customized for every bag, and the other is a universal one, which can affect all bags in a given data set and thus has some generalizability. Furthermore, through simulations, we have demonstrated the efficacy of the proposed algorithms in fooling state-of-the-art MIL approaches, such that these models make incorrect predictions regarding the label assigned to the bag. Finally, we have discussed, through experiments, about taking care of these kind of adversarial perturbations through a simple strategy. Source codes are available athttps://github.com/InkiInki/MI-UAP.

A deep multi-instance neural network for dyeing-free inspection of yarn dyeing uniformity

Dyeing evenness is a decisive factor influencing the commercial values of polyester yarn. Thus, the inspection of yarn dyeing uniformity plays a vital role in the textile community. Due to the inefficiency of dyeing process, the traditional dyeing-based inspection methods are both laborious and time-consuming. To improve efficiency, this study attempts to develop a fast dyeing-free method for dyeing uniformity inspection based on imaging spectrometer and multi-instance learning (MIL). The relevant properties of yarn samples are first recorded in the hyperspectral images (HSIs). As the available labels are ambiguous, classifying the collected HSIs has become a MIL problem. Meanwhile, the correlation between the spectral pixels and the sample labels might be sophisticated. It might be difficult for the existing MIL methods to learn such data. In this paper, a deep Fisher score-based multi-instance neural network (DFSNet) is also proposed for classifying the captured HSIs. The DFSNet is able to learn a sophisticated correlation between deep instance features and bag representation. Specifically, a Fisher score-based MIL pooling layer is first developed to convert the instance-level features into bag-level features. The DFSNet is then developed with a ladder structure and the Fisher score-based MIL pooling layer. The proposed dyeing-free method and DFSNet are evaluated using the actual polyester samples. Moreover, the proposed DFSNet and the spectral range of HSI are further analyzed. The experiment results have indicated that the proposed method could achieve satisfactory performance, providing a potential solution to fast dyeing uniformity inspection.

Multi-instance embedding learning with deconfounded instance-level prediction

Confounded information is an objective fact when using multi-instance learning (MIL) to classify bags of instances, which may be inherited by MIL embedding methods and lead to questionable bag label prediction. To respond to this problem, we propose the multi-instance embedding learning with deconfounded instance-level prediction algorithm. Unlike traditional embedding-based strategies, we design a deconfounded optimization goal to maximize the distinction between instances in positive and negative bags. In addition, we present and use bag-level embedding with feature distillation to reduce the MIL classification task to a single-instance learning problem. Under the theoretical analysis, the embedding cohesiveness and feature magnitude metrics are developed to explain the benefits of the proposed deconfounded technique in MIL settings. Extensive experiments on thirty-four data sets demonstrate that our proposed method has the best overall performance over other state-of-the-art MIL methods. This strategy, in particular, has a substantial advantage on web data sets. Source codes are available at https://github.com/InkiInki/MEDI .

Prediction of Protein Subcellular Localization Based on Microscopic Images via Multi‐Task Multi‐Instance Learning

Protein localization information is essential for understanding protein functions and their roles in various biological processes. The image-based prediction methods of protein subcellular localization have emerged in recent years because of the advantages of microscopic images in revealing spatial expression and distribution of proteins in cells. However, the image-based prediction is a very challenging task, due to the multi-instance nature of the task and low quality of images. In this paper, we propose a multi-task learning strategy and mask generation to enhance the prediction performance. Furthermore, we also investigate effective multi-instance learning schemes. We collect a large-scale dataset from the Human Protein Atlas database, and the experimental results show that the proposed multi-task multi-instance learning model outperforms both single-instance learning and common multi-instance learning methods by large margins.

A fisher score-based multi-instance learning method assisted by mixture of factor analysis

Multi-instance learning (MIL) is a problem with significant importance due to the existence of ambiguous labels in many practical tasks. The training data in MIL problem is represented by labeled bags containing a various number of unlabeled instances. This inexact label information has brought complexities to the classification of instances and bags. Fisher scores have been utilized to extract fixed-length representation vectors from the MIL bags with the assistance of generative models. However, the generative model utilized by the existing methods might disregard the covariance information. In this paper, we propose a multi-instance learning method (miMFA) utilizing Fisher scores derived from the mixture of factor analysis (MFA) model. The MFA generates nonlinear data with a set of local factor analysis models, while each local model approximates the full covariance Gaussian using latent factors. Thus, the MFA could cover the data distribution and generate Fisher scores effectively. The MFA-based Fisher score is then utilized to form the bag representation. Moreover, a reinforcement vector and the complementary information from auxiliary MIL methods are introduced for performance refinement. The proposed miMFA is evaluated on 11 MIL datasets, and achieves satisfactory results compared with the existing methods.

Interventional Multi-Instance Learning with Deconfounded Instance-Level Prediction

When applying multi-instance learning (MIL) to make predictions for bags of instances, the prediction accuracy of an instance often depends on not only the instance itself but also its context in the corresponding bag. From the viewpoint of causal inference, such bag contextual prior works as a confounder and may result in model robustness and interpretability issues. Focusing on this problem, we propose a novel interventional multi-instance learning (IMIL) framework to achieve deconfounded instance-level prediction. Unlike traditional likelihood-based strategies, we design an Expectation-Maximization (EM) algorithm based on causal intervention, providing a robust instance selection in the training phase and suppressing the bias caused by the bag contextual prior. Experiments on pathological image analysis demonstrate that our IMIL method substantially reduces false positives and outperforms state-of-the-art MIL methods.

Bag dissimilarity regularized multi-instance learning

Multi-instance learning (MIL) is able to cope with the weakly supervised problems where the training data is represented by labeled bags consisting of multiple unlabeled instances. Due to its practical significance, MIL has recently drawn increasing attention. Introducing bag representations is an attractive way to learn MIL data. However, it is difficult for the existing MIL methods to utilize both implicit and explicit bag representations simultaneously. In this paper, we propose a bag dissimilarity regularized (BDR) framework that incorporates multiple bag representations regardless of explicitness or implicitness. Here, the implicit bag representations are incorporated into a regularization term that contains the intrinsic geometric information provided by the bag dissimilarities. The regularization term can be added to the objective function of supervised classifiers. An effective method for explicit bag embedding is also proposed, which exploits the Fisher score derived from factor analysis. Finally, we propose two specific BDR methods based on support vector machine and broad learning system. The proposed BDR methods are evaluated on 14 datasets, and have achieved competitive results with limited computation consumption. We also discuss the effectiveness and the characteristics of BDR framework.

Second-order multi-instance learning model for whole slide image classification

Whole slide histopathology images (WSIs) play a crucial role in diagnosing lymph node metastasis of breast cancer, which usually lack fine-grade annotations of tumor regions and have large resolutions (typically 105 × 105 pixels). Multi-instance learning has gradually become a dominant weakly supervised learning framework for WSI classification when only slide-level labels are available. In this paper, we develop a novel second-order multiple instances learning method (SoMIL) with an adaptive aggregator stacked by the attention mechanism and recurrent neural network (RNN) for histopathological image classification. To be specific, the proposed method applies a second-order pooling module (matrix power normalization covariance) for instance-level feature extraction of weakly supervised learning framework, attempting to explore second-order statistics of deep features for histopathological images. Additionally, we utilize an efficient channel attention mechanism to adaptively highlight the most discriminative instance features, followed by an RNN to update the final bag-level representation for the slide classification. Experimental results on the lymph node metastasis dataset of 2016 Camelyon grand challenge demonstrate the significant improvement of our proposed SoMIL framework compared with other state-of-the-art multi-instance learning methods. Moreover, in the external validation on 130 WSIs, SoMIL also achieves an impressive area under the curve performance that competitive to the fully-supervised framework.

Weakly supervised learning of RNA modifications from low-resolution epitranscriptome data.

MotivationIncreasing evidence suggests that post-transcriptional ribonucleic acid (RNA) modifications regulate essential biomolecular functions and are related to the pathogenesis of various diseases. Precise identification of RNA modification sites is essential for understanding the regulatory mechanisms of RNAs. To date, many computational approaches for predicting RNA modifications have been developed, most of which were based on strong supervision enabled by base-resolution epitranscriptome data. However, high-resolution data may not be available.ResultsWe propose WeakRM, the first weakly supervised learning framework for predicting RNA modifications from low-resolution epitranscriptome datasets, such as those generated from acRIP-seq and hMeRIP-seq. Evaluations on three independent datasets (corresponding to three different RNA modification types and their respective sequencing technologies) demonstrated the effectiveness of our approach in predicting RNA modifications from low-resolution data. WeakRM outperformed state-of-the-art multi-instance learning methods for genomic sequences, such as WSCNN, which was originally designed for transcription factor binding site prediction. Additionally, our approach captured motifs that are consistent with existing knowledge, and visualization of the predicted modification-containing regions unveiled the potentials of detecting RNA modifications with improved resolution.Availability implementationThe source code for the WeakRM algorithm, along with the datasets used, are freely accessible at: https://github.com/daiyun02211/WeakRMSupplementary information Supplementary data are available at Bioinformatics online.

Image Quality Assessment of Fetal Brain MRI Using Multi-Instance Deep Learning Methods.

Due to random motion of fetuses and maternal respirations, image quality of fetal brain MRIs varies considerably. To address this issue, visual inspection of the images is performed during acquisition phase and after 3D-reconstruction, and the images are re-acquired if they are deemed to be of insufficient quality. However, this process is time-consuming and subjective. Multi-instance (MI) deep learning methods (DLMs) may perform this task automatically. To propose an MI count-based DLM (MI-CB-DLM), an MI vote-based DLM (MI-VB-DLM), and an MI feature-embedding DLM (MI-FE-DLM) for automatic assessment of 3D fetal-brain MR image quality. To quantify influence of fetal gestational age (GA) on DLM performance. Retrospective. Two hundred and seventy-one MR exams from 211 fetuses (mean GA ± SD=30.9 ± 5.5 weeks). T2 -weighted single-shot fast spin-echo acquired at 1.5 T. The T2 -weighted images were reconstructed in 3D. Then, two fetal neuroradiologists, a clinical neuroscientist, and a fetal MRI technician independently labeled the reconstructed images as 1 or 0 based on image quality (1=high; 0=low). These labels were fused and served as ground truth. The proposed DLMs were trained and evaluated using three repeated 10-fold cross-validations (training and validation sets of 244 and 27 scans). To quantify GA influence, this variable was included as an input of the DLMs. DLM performance was evaluated using precision, recall, F-score, accuracy, and AUC values. Precision, recall, F-score, accuracy, and AUC averaged over the three cross validations were 0.85 ± 0.01, 0.85 ± 0.01, 0.85 ± 0.01, 0.85 ± 0.01, 0.93 ± 0.01, for MI-CB-DLM (without GA); 0.75 ± 0.03, 0.75 ± 0.03, 0.75 ± 0.03, 0.75 ± 0.03, 0.81 ± 0.03, for MI-VB-DLM (without GA); 0.81 ± 0.01, 0.81 ± 0.01, 0.81 ± 0.01, 0.81 ± 0.01, 0.89 ± 0.01, for MI-FE-DLM (without GA); and 0.86 ± 0.01, 0.86 ± 0.01, 0.86 ± 0.01, 0.86 ± 0.01, 0.93 ± 0.01, for MI-CB-DLM with GA. MI-CB-DLM performed better than other DLMs. Including GA as an input of MI-CB-DLM improved its performance. MI-CB-DLM may potentially be used to objectively and rapidly assess fetal MR image quality. 4 TECHNICAL EFFICACY: Stage 3.

Dynamic Pseudo-Label Generation for Weakly Supervised Object Detection in Remote Sensing Images

In recent years, fully supervised object detection methods in remote sensing images with good performance have been developed. However, this approach requires a large number of instance-level annotated samples that are relatively expensive to acquire. Therefore, weakly supervised learning using only image-level annotations has attracted much attention. Most of the weakly supervised object detection methods are based on multi-instance learning methods, and their performance depends on the process of scoring the candidate region proposals during training. In this process, the use of only image-level labels for supervision usually cannot obtain optimal results due to the lack of location information of the object. To address the above problem, a dynamic sample pseudo-label generation framework is proposed to generate pseudo-labels for each proposal without additional annotations. First, we propose the pseudo-label generation algorithm (PLG) to generate the category labels of the proposal by using the localization information of the object. Specifically, we propose to use the pixel average of the object’s localization map in the proposal as the proposal category confidence and calculate the pseudo-label by comparing the proposal category confidence with the preset threshold. In addition, an effective adaptive threshold selection strategy is designed to eliminate the effect of different category shape differences in computing sample pseudo-labels. Comparative experiments on the NWPU VHR-10 dataset demonstrate that our method can significantly improve the detection performance compared to existing methods.

A multi-instance networks with multiple views for classification of mammograms

Breast cancer is the most common malignant disease in women, and early screening of breast cancer is crucial for improving the survival rate. Mammography is one of the most popular imaging methods for breast cancer screening with the characteristics of practicality, effectiveness, and low cost. However, the classification of mammograms suffers from large image sizes, indistinct image characteristics of lesions, small proportion of abnormalities, and class imbalance. To address these difficulties, the multi-view input and weighted multi-instance learning (MIL) methods are proposed. A novel model called the weighted MIL DenseNet with multi-view input (WMDNet) is presented that integrates the two methods above. The multi-view inputs method is used to enhance the abnormalities of mammograms and obtain more potential features from mammograms with different views, simultaneously. The weighted MIL is designed to extract the most suspicious lesions from mammograms to resolve the problems of small proportion of abnormalities and class imbalance. To verify the effectiveness of the proposed methods, three binary classification models are evaluated on two public datasets, the INbreast and MIAS datasets. The experimental results demonstrate that the proposed methods can achieve preferably results compared with several other state-of-the-art approaches, especially the proposed WMDNet model gains the best classification results on both datasets.

Classification of Lentigo Maligna at Patient-Level by Means of Reflectance Confocal Microscopy Data

Reflectance confocal microscopy is an appropriate tool for the diagnosis of lentigo maligna. Compared with dermoscopy, this device can provide abundant information as a mosaic and/or a stack of images. In this particular context, the number of images per patient varied between 2 and 833 images and the objective, ultimately, is to be able to discern between benign and malignant classes. First, this paper evaluated classification at the image level, with the help of handcrafted methods derived from the literature and transfer learning methods. The transfer learning feature extraction methods outperformed the handcrafted feature extraction methods from literature, with a F 1 score value of 0.82. Secondly, this work proposed patient-level supervised methods based on image decisions and a comparison of these with multi-instance learning methods. This study achieved comparable results to those of the dermatologists, with an auc score of 0.87 for supervised patient diagnosis and an auc score of 0.88 for multi-instance learning patient diagnosis. According to these results, computer-aided diagnosis methods presented in this paper could be easily used in a clinical context to save time or confirm a diagnosis and can be oriented to detect images of interest. Also, this methodology can be used to serve future works based on multimodality.

A Transfer Learning-Based Multi-Instance Learning Method With Weak Labels.

In multi-instance learning (MIL), labels are associated with bags rather than the instances in the bag. Most of the previous MIL methods assume that each bag has the actual label in the training set. However, from the process of labeling work, the label of a bag is always evaluated by the calculation of the labels obtained from a number of labelers. In the calculation, the weight of each labeler is always unknown and people always assign the weight for each labeler by random or equally, and this may result in the ambiguous labels for the bags, which is called weak labels here. In addition, we always meet the problem of knowledge transfer from the source task to the target task, and this leads to the study of multiple instance transfer learning. In this article, we propose a new framework called transfer learning-based multiple instance learning (TMIL) framework to address the problem of multiple instance transfer learning in which both the source task and the target task contain the weak labels. We first construct a TMIL model with weak labels, which can transfer knowledge from the source task to the target task where both source and target tasks contain weak labels. We then put forward an iterative framework to solve the transfer learning model with weak labels so that we can update the label of the bag to improve the performance of multiple instance learning. We then present the convergence analysis of the proposed method. The experiments show that the proposed method outperforms the existing multiple instance learning methods and can correct the initial labels to obtain the actual labels for the bags.

Fragmentary Multi-Instance Classification.

Multi-instance learning (MIL) has been extensively applied to various real tasks involving objects with bags of instances, such as in drugs and images. Previous studies on MIL assume that data are entirely complete. However, in many real tasks, the instance is fragmentary. In this article, we present probably the first study on multi-instance classification with fragmentary data. In our proposed framework, called fragmentary multi-instance classification (FIC), the fragmentary data are completed and the multi-instance classifier is learned jointly. To facilitate the integration between the completion and classifier learning, FIC establishes the weighting mechanism to measure the importance levels of different instances. To validate the compatibility of our framework, four typical MIL methods, including multi-instance support vector machine (MI-SVM), expectation maximization diverse density (EM-DD), citation- K nearest neighbors (Citation-KNNs), and MIL with discriminative bag mapping (MILDM), are embedded into the framework to obtain the corresponding FIC versions. As an illustration, an efficient solving algorithm is developed to address the problem for MI-SVM, together with the proof of convergence behavior. The experimental results on various types of real-world datasets demonstrate the effectiveness.