Multiple Instance Learning Research Articles

Improved DeTraC Binary Coyote Net-Based Multiple Instance Learning for Predicting Lymph Node Metastasis of Breast Cancer From Whole-Slide Pathological Images.

Early detection of lymph node metastasis in breast cancer is vital for improving treatment outcomes and prognosis. This study introduces an Improved Decompose, Transfer, and Compose Binary Coyote Net-based Multiple Instance Learning (ImDeTraC-BCNet-MIL) method for predicting lymph node metastasis from Whole Slide Images (WSIs) using multiple instance learning. The method involves segmenting WSIs into patches using Otsu and double-dimensional clustering techniques. The developed multiple instance learning approach introduces a paradigm into computational pathology by shaping pathological data and constructing features. ImDeTraC-BCNet-MIL was utilised for feature generation during both training and testing to differentiate lymph node metastasis in WSIs. The proposed model achieves the highest accuracy of 95.3% and 99.8%, precision values of 98% and 99.8%, and recall rates of 92.9% and 99.8% on the Camelyon16 and Camelyon17 datasets. These findings underscore the effectiveness of ImDeTraC-BCNet-MIL in enhancing the early detection of lymph node metastasis in breast cancer.

Dual-Attention Multiple Instance Learning Framework for Pathology Whole-Slide Image Classification

Conventional methods for tumor diagnosis suffer from two inherent limitations: they are time-consuming and subjective. Computer-aided diagnosis (CAD) is an important approach for addressing these limitations. Pathology whole-slide images (WSIs) are high-resolution tissue images that have made significant contributions to cancer diagnosis and prognosis assessment. Due to the complexity of WSIs and the availability of only slide-level labels, multiple instance learning (MIL) has become the primary framework for WSI classification. However, most MIL methods fail to capture the interdependence among image patches within a WSI, which is crucial for accurate classification prediction. Moreover, due to the weak supervision of slide-level labels, overfitting may occur during the training process. To address these issues, this paper proposes a dual-attention-based multiple instance learning framework (DAMIL). DAMIL leverages the spatial relationships and channel information between WSI patches for classification prediction, without detailed pixel-level tumor annotations. The output of the model preserves the semantic variations in the latent space, enhances semantic disturbance invariance, and provides reliable class identification for the final slide-level representation. We validate the effectiveness of DAMIL on the most commonly used public dataset, Camelyon16. The results demonstrate that DAMIL outperforms the state-of-the-art methods in terms of classification accuracy (ACC), area under the curve (AUC), and F1-Score. Our model also allows for the examination of its interpretability by visualizing the dual-attention weights. To the best of our knowledge, this is the first attempt to use a dual-attention mechanism, considering both spatial and channel information, for whole-slide image classification.

Capturing cell heterogeneity in representations of cell populations for image-based profiling using contrastive learning.

Image-based cell profiling is a powerful tool that compares perturbed cell populations by measuring thousands of single-cell features and summarizing them into profiles. Typically a sample is represented by averaging across cells, but this fails to capture the heterogeneity within cell populations. We introduce CytoSummaryNet: a Deep Sets-based approach that improves mechanism of action prediction by 30-68% in mean average precision compared to average profiling on a public dataset. CytoSummaryNet uses self-supervised contrastive learning in a multiple-instance learning framework, providing an easier-to-apply method for aggregating single-cell feature data than previously published strategies. Interpretability analysis suggests that the model achieves this improvement by downweighting small mitotic cells or those with debris and prioritizing large uncrowded cells. The approach requires only perturbation labels for training, which are readily available in all cell profiling datasets. CytoSummaryNet offers a straightforward post-processing step for single-cell profiles that can significantly boost retrieval performance on image-based profiling datasets.

Artificial intelligence-based morphologic classification and molecular characterization of neuroblastic tumors from digital histopathology

A deep learning model using attention-based multiple instance learning (aMIL) and self-supervised learning (SSL) was developed to perform pathologic classification of neuroblastic tumors and assess MYCN-amplification status using H&E-stained whole slide images from the largest reported cohort to date. The model showed promising performance in identifying diagnostic category, grade, mitosis-karyorrhexis index (MKI), and MYCN-amplification with validation on an external test dataset, suggesting potential for AI-assisted neuroblastoma classification.

Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Bag-Level Classifier is a Good Instance-Level Teacher.

Multiple Instance Learning (MIL) has demonstrated promise in Whole Slide Image (WSI) classification. However, a major challenge persists due to the high computational cost associated with processing these gigapixel images. Existing methods generally adopt a two-stage approach, comprising a non-learnable feature embedding stage and a classifier training stage. Though it can greatly reduce memory consumption by using a fixed feature embedder pre-trained on other domains, such a scheme also results in a disparity between the two stages, leading to suboptimal classification accuracy. To address this issue, we propose that a bag-level classifier can be a good instance-level teacher. Based on this idea, we design Iteratively Coupled Multiple Instance Learning (ICMIL) to couple the embedder and the bag classifier at a low cost. ICMIL initially fixes the patch embedder to train the bag classifier, followed by fixing the bag classifier to fine-tune the patch embedder. The refined embedder can then generate better representations in return, leading to a more accurate classifier for the next iteration. To realize more flexible and more effective embedder fine-tuning, we also introduce a teacher-student framework to efficiently distill the category knowledge in the bag classifier to help the instance-level embedder fine-tuning. Intensive experiments were conducted on four distinct datasets to validate the effectiveness of ICMIL. The experimental results consistently demonstrated that our method significantly improves the performance of existing MIL backbones, achieving state-of-the-art results. The code and the organized datasets can be accessed by: https://github.com/Dootmaan/ICMIL/tree/confidence-based.

Automated Detection of Filamentous Fungal Keratitis on Whole Slide Images of Potassium Hydroxide Smears with Multiple Instance Learning

Automated Detection of Filamentous Fungal Keratitis on Whole Slide Images of Potassium Hydroxide Smears with Multiple Instance Learning

Weakly supervised multiple instance learning model with generalization ability for clinical adenocarcinoma screening on serous cavity effusion pathology

Accurate and rapid screening of adenocarcinoma cells in serous cavity effusion is vital in diagnosing the stage of metastatic tumors and providing prompt medical treatment. However, it is often difficult for pathologists to screen serous cavity effusion. Fixed agglutination cell block can help to improve diagnostic sensitivity in malignant tumor cells through analyzing larger volumes of serous cavity effusion, though it could accordingly lead to screening on more cells for pathologists. With the advent of whole slide imaging and development of artificial intelligence, advanced deep learning models are expected to assist pathologists in improving the diagnostic efficiency and accuracy. In this study, so far as we known, it is the first time to use cell block technology combined with a proposed weakly supervised deep learning model with multiple instance learning method to screen serous adenocarcinoma. The comparative experiments were implemented through five-fold cross-validation, and the results demonstrated that our proposed model not only achieves state-of-the-art performance under weak supervision while balancing the number of learnable parameters and computational costs and reduces the workload of pathologists, but also presents a quantitative and interpretable cellular pathological scene of serous adenocarcinoma with superior interpretability and strong generalization capability. The performances and features of the model indicate its effectiveness in the rapid screening and diagnosis of serous cavity effusion and its potential in broad clinical application prospects, e.g., in precision medical applications. Moreover, the constructed two real-world pathological datasets would be the first public WSI datasets of serous cavity effusion with adenocarcinoma based on cell block sections, which can help to assist colleagues.

Deep Learning Predicts Lymphovascular Invasion Status in Muscle Invasive Bladder Cancer Histopathology.

Lymphovascular invasion (LVI) is linked to poor prognosis in patients with muscle-invasive bladder cancer (MIBC). Accurately identifying the LVI status in MIBC patients is crucial for effective risk stratification and precision treatment. We aim to develop a deep learning model to identify the LVI status in whole-slide images (WSIs) of MIBC patients. A cohort from The Cancer Genome Atlas (TCGA) database was used to train a deep learning model, slide-based lymphovascular invasion predictor (SBLVIP), based on multiple-instance learning. This model was externally validated using the Renmin Hospital of Wuhan University (RHWU) and People's Hospital of Hanchuan City (PHHC) cohorts. Kaplan-Meier curves, along with univariate and multivariate Cox models, were employed to evaluate the association between the LVI status predicted by SBLVIP and the survival outcomes of MIBC patients. In the TCGA cohort, the SBLVIP model achieved an average accuracy of 0.804 [95% confidence interval (CI) 0.712-0.895] and an area under the receiver operating characteristic curve (AUC) of 0.77 (95% CI 0.63-0.84) in the training set. In the internal validation set, the model's average accuracy and AUC were 0.774 (95% CI, 0.701-0.846) and 0.76 (95% CI, 0.60-0.83), respectively. In the RHWU cohort, the SBLVIP model achieved an average accuracy of 0.807 (95% CI 0.734-0.880) and an AUC of 0.74 (95% CI 0.55-0.83). In the PHHC cohort, SBLVIP demonstrated an average accuracy of 0.821 (95% CI 0.737-0.909) and an AUC of 0.74 (95% CI 0.58-0.89). Moreover, the LVI status predicted by SBLVIP showed significant independent prognostic value (P = 1 × 10-6). We developed a deep learning model named SBLVIP to predict the LVI status in routine WSIs of MIBC patients.

An interpretable deep learning model for detecting BRCA pathogenic variants of breast cancer from hematoxylin and eosin-stained pathological images.

Determining the status of breast cancer susceptibility genes (BRCA) is crucial for guiding breast cancer treatment. Nevertheless, the need for BRCA genetic testing among breast cancer patients remains unmet due to high costs and limited resources. This study aimed to develop a Bi-directional Self-Attention Multiple Instance Learning (BiAMIL) algorithm to detect BRCA status from hematoxylin and eosin (H&E) pathological images. A total of 319 histopathological slides from 254 breast cancer patients were included, comprising two dependent cohorts. Following image pre-processing, 633,484 tumor tiles from the training dataset were employed to train the self-developed deep-learning model. The performance of the network was evaluated in the internal and external test sets. BiAMIL achieved AUC values of 0.819 (95% CI [0.673-0.965]) in the internal test set, and 0.817 (95% CI [0.712-0.923]) in the external test set. To explore the relationship between BRCA status and interpretable morphological features in pathological images, we utilized Class Activation Mapping (CAM) technique and cluster analysis to investigate the connections between BRCA gene mutation status and tissue and cell features. Significantly, we observed that tumor-infiltrating lymphocytes and the morphological characteristics of tumor cells appeared to be potential features associated with BRCA status. An interpretable deep neural network model based on the attention mechanism was developed to predict the BRCA status in breast cancer. Keywords: Breast cancer, BRCA, deep learning, self-attention, interpretability.

Deep learning-based image quality assessment for optical coherence tomography macular scans: a multicentre study

AimsTo develop and externally test deep learning (DL) models for assessing the image quality of three-dimensional (3D) macular scans from Cirrus and Spectralis optical coherence tomography devices.MethodsWe retrospectively collected two...

Ensemble transformer-based multiple instance learning to predict pathological subtypes and tumor mutational burden from histopathological whole slide images of endometrial and colorectal cancer

In endometrial cancer (EC) and colorectal cancer (CRC), in addition to microsatellite instability, tumor mutational burden (TMB) has gradually gained attention as a genomic biomarker that can be used clinically to determine which patients may benefit from immune checkpoint inhibitors. High TMB is characterized by a large number of mutated genes, which encode aberrant tumor neoantigens, and implies a better response to immunotherapy. Hence, a part of EC and CRC patients associated with high TMB may have higher chances to receive immunotherapy. TMB measurement was mainly evaluated by whole-exome sequencing or next-generation sequencing, which was costly and difficult to be widely applied in all clinical cases. Therefore, an effective, efficient, low-cost and easily accessible tool is urgently needed to distinguish the TMB status of EC and CRC patients. In this study, we present a deep learning framework, namely Ensemble Transformer-based Multiple Instance Learning with Self-Supervised Learning Vision Transformer feature encoder (ETMIL-SSLViT), to predict pathological subtype and TMB status directly from the H&E stained whole slide images (WSIs) in EC and CRC patients, which is helpful for both pathological classification and cancer treatment planning. Our framework was evaluated on two different cancer cohorts, including an EC cohort with 918 histopathology WSIs from 529 patients and a CRC cohort with 1495 WSIs from 594 patients from The Cancer Genome Atlas. The experimental results show that the proposed methods achieved excellent performance and outperforming seven state-of-the-art (SOTA) methods in cancer subtype classification and TMB prediction on both cancer datasets. Fisher’s exact test further validated that the associations between the predictions of the proposed models and the actual cancer subtype or TMB status are both extremely strong (p<0.001). These promising findings show the potential of our proposed methods to guide personalized treatment decisions by accurately predicting the EC and CRC subtype and the TMB status for effective immunotherapy planning for EC and CRC patients.

Multiple instance learning for WSI: A comparative analysis of attention-based approaches

Whole slide images (WSI), obtained by high-resolution digital scanning of microscope slides at multiple scales, are the cornerstone of modern Digital Pathology. However, they represent a particular challenge to artificial intelligence (AI)-based/AI-mediated analysis because pathology labeling is typically done at slide-level, instead of tile-level. It is not just that medical diagnostics is recorded at the specimen level, the detection of oncogene mutation is also experimentally obtained, and recorded by initiatives like The Cancer Genome Atlas (TCGA), at the slide level. This configures a dual challenge: (a) accurately predicting the overall cancer phenotype and (b) finding out what cellular morphologies are associated with it at the tile level. To better understand and address these challenges, two existing weakly supervised Multiple Instance Learning (MIL) approaches were explored and compared: Attention MIL (AMIL) and Additive MIL (AdMIL). These architectures were analyzed on tumor detection (a task where these models obtained good results previously) and TP53 mutation detection (a much less explored task). For tumor detection, we built a dataset from Lung Squamous Cell Carcinoma (TCGA-LUSC) slides, with 349 positive and 349 negative slides. The patches were extracted from 5× magnification. For TP53 mutation detection, we explored a dataset built from Invasive Breast Carcinoma (TCGA-BRCA) slides, with 347 positive and 347 negative slides. In this case, we explored three different magnification levels: 5×, 10×, and 20×. Our results show that a modified additive implementation of MIL matched the performance of reference implementation (AUC 0.96), and was only slightly outperformed by AMIL (AUC 0.97) on the tumor detection task. TP53 mutation was most sensitive to features at the higher applications where cellular morphology is resolved. More interestingly from the perspective of the molecular pathologist, we highlight the possible ability of these MIL architectures to identify distinct sensitivities to morphological features (through the detection of regions of interest, ROIs) at different amplification levels. This ability for models to obtain tile-level ROIs is very appealing to pathologists as it provides the possibility for these algorithms to be integrated in a digital staining application for analysis, facilitating the navigation through these high-dimensional images and the diagnostic process.

Weakly supervised deep learning for diagnosis of prostate cancer after radical prostatectomy: a population- based, multicenter study

Abstract Introduction/Objective The pathological diagnosis of prostate cancer after radical prostatectomy (RP) is a challenging and time-consuming process. Artificial intelligence has the potential to enhance this situation significantly. Our study aimed to assess the efficacy of weakly supervised deep learning, specifically focusing on practical diagnostic tasks for RP specimens. Methods/Case Report We obtained slices from clinical centers across four regions in China. The model was developed based on a multiple instance learning framework. We evaluated the model’s predictive accuracy, covering both slice-level and lesion-level diagnoses. Additionally, we investigated lesion localization using attention visualization. Slice-level diagnosis included prostate cancer detection and Gleason grading, while lesion-level diagnosis involved recognizing index lesions and non-index lesions. Results (if a Case Study enter NA) The study involved 13,245 slides from 304 patients. The area under the curve values in four validation sets ranged from 0.921 (95% CI 0.913–0.930) to 0.964 (95% CI 0.959–0.968). Regarding Gleason grading, the model demonstrated quadratic Weighted Kappa values of 0.748 and 0.720 for internal and external validation, respectively. The model predicted 1,683 (35.3%) positive and 3,081 (64.7%) negative slices in all validation sets. The median area of index lesions and non-index lesions was 92.6mm2 (IQR: 54.2–130.0mm2) and 3.5mm2 (IQR: 1.4–10.9mm2) (p&amp;lt;0.001), respectively. The model accurately predicted index lesions in all cases for three validation sets. Furthermore, the model achieved 100% accuracy in predicting slides with the maximum tumor area for index lesions in two validation sets. In visual assessment, 200 randomly selected slices predicted as true positive demonstrated that the model could identify 97.5% (195/200) of the index lesions. Conclusion The weakly supervised deep learning model demonstrated remarkably high diagnostic and localization capabilities in identifying index lesions within RP specimens. This targeted approach would effectively alleviate the diagnostic workload associated with pathology assessments for prostate cancer.

Prediction of homologous recombination deficiency from routine histology with attention-based multiple instance learning in nine different tumor types.

Homologous recombination deficiency (HRD) is recognized as a pan-cancer predictive biomarker that potentially indicates who could benefit from treatment with PARP inhibitors (PARPi). Despite its clinical significance, HRD testing is highly complex. Here, we investigated in a proof-of-concept study whether Deep Learning (DL) can predict HRD status solely based on routine hematoxylin & eosin (H&E) histology images across nine different cancer types. We developed a deep learning pipeline with attention-weighted multiple instance learning (attMIL) to predict HRD status from histology images. As part of our approach, we calculated a genomic scar HRD score by combining loss of heterozygosity (LOH), telomeric allelic imbalance (TAI), and large-scale state transitions (LST) from whole genome sequencing (WGS) data of n = 5209 patients across two independent cohorts. The model's effectiveness was evaluated using the area under the receiver operating characteristic curve (AUROC), focusing on its accuracy in predicting genomic HRD against a clinically recognized cutoff value. Our study demonstrated the predictability of genomic HRD status in endometrial, pancreatic, and lung cancers reaching cross-validated AUROCs of 0.79, 0.58, and 0.66, respectively. These predictions generalized well to an external cohort, with AUROCs of 0.93, 0.81, and 0.73. Moreover, a breast cancer-trained image-based HRD classifier yielded an AUROC of 0.78 in the internal validation cohort and was able to predict HRD in endometrial, prostate, and pancreatic cancer with AUROCs of 0.87, 0.84, and 0.67, indicating that a shared HRD-like phenotype occurs across these tumor entities. This study establishes that HRD can be directly predicted from H&E slides using attMIL, demonstrating its applicability across nine different tumor types.

Transformer-based representation learning and multiple-instance learning for cancer diagnosis exclusively from raw sequencing fragments of bisulfite-treated plasmacell-free DNA.

Early cancer diagnosis from bisulfite-treated cell-free DNA (cfDNA) fragments requires tedious data analytical procedures. Here, we present a deep-learning-based approach for early cancer interception and diagnosis (DECIDIA) that can achieve accurate cancer diagnosis exclusively from bisulfite-treated cfDNA sequencing fragments. DECIDIA relies on transformer-based representation learning of DNA fragments and weakly supervised multiple-instance learning for classification. We systematically evaluate the performance of DECIDIA for cancer diagnosis and cancer type prediction on a curated dataset of 5389 samples that consist of colorectal cancer (CRC; n = 1574), hepatocellular cell carcinoma (HCC; n = 1181), lung cancer (n = 654), and non-cancer control (n = 1980). DECIDIA achieved an area under the receiver operating curve (AUROC) of 0.980 (95% CI, 0.976-0.984) in 10-fold cross-validation settings on the CRC dataset by differentiating cancer patients from cancer-free controls, outperforming benchmarked methods that are based on methylation intensities. Noticeably, DECIDIA achieved an AUROC of 0.910 (95% CI, 0.896-0.924) on the externally independent HCC testing set in distinguishing HCC patients from cancer-free controls, although there was no HCC data used in model development. In the settings of cancer-type classification, we observed that DECIDIA achieved a micro-average AUROC of 0.963 (95% CI, 0.960-0.966) and an overall accuracy of 82.8% (95% CI, 81.8-83.9). In addition, we distilled four sequence signatures from the raw sequencing reads that exhibited differential patterns in cancer versus control and among different cancer types. Our approach represents a new paradigm towards eliminating the tedious data analytical procedures for liquid biopsy that uses bisulfite-treated cfDNA methylome.

Ranking attention multiple instance learning for lymph node metastasis prediction on multicenter cervical cancer MRI.

In the current clinical diagnostic process, the gold standard for lymph node metastasis (LNM) diagnosis is histopathological examination following surgical lymphadenectomy. Developing a non-invasive and preoperative method for predicting LNM is necessary and holds significant clinical importance. We develop a ranking attention multiple instance learning (RA-MIL) model that integrates convolutional neural networks (CNNs) and ranking attention pooling to diagnose LNM from T2 MRI. Our RA-MIL model applies the CNNs to derive imaging features from 2D MRI slices and employs ranking attention pooling to create patient-level feature representation for diagnostic classification. Based on the MIL and attention theory, informative regions of top-ranking MRI slices from LNM-positive patients are visualized to enhance the interpretability of automatic LNM prediction. This retrospective study collected 300 female patients with cervical cancer who underwent T2-weighted magnetic resonance imaging (MRI) scanning and histopathological diagnosis from one hospital (289 patients) and one open-source dataset (11 patients). Our RA-MIL model delivers promising LNM prediction performance, achieving the area under the receiver operating characteristic curve (AUC) of 0.809 on the internal test set and 0.833 on the public dataset. Experiments show significant improvements in LNM status prediction using the proposed RA-MIL model compared with other state-of-the-art (SOTA) comparative deep learning models. The developed RA-MIL model has the potential to serve as a non-invasive auxiliary tool for preoperative LNM prediction, offering visual interpretability regarding informative MRI slices and regions in LNM-positive patients.

LungHist700: A dataset of histological images for deep learning in pulmonary pathology

Accurate detection and classification of lung malignancies are crucial for early diagnosis, treatment planning, and patient prognosis. Conventional histopathological analysis is time-consuming, limiting its clinical applicability. To address this, we present a dataset of 691 high-resolution (1200 × 1600 pixels) histopathological lung images, covering adenocarcinomas, squamous cell carcinomas, and normal tissues from 45 patients. These images are subdivided into three differentiation levels for both pathological types: well, moderately, and poorly differentiated, resulting in seven classes for classification. The dataset includes images at 20x and 40x magnification, reflecting real clinical diversity. We evaluated image classification using deep neural network and multiple instance learning approaches. Each method was used to classify images at 20x and 40x magnification into three superclasses. We achieved accuracies between 81% and 92%, depending on the method and resolution, demonstrating the dataset’s utility.

Deep learning-based analysis of EGFR mutation prevalence in lung adenocarcinoma H&E whole slide images.

EGFR mutations are a major prognostic factor in lung adenocarcinoma. However, current detection methods require sufficient samples and are costly. Deep learning is promising for mutation prediction in histopathological image analysis but has limitations in that it does not sufficiently reflect tumor heterogeneity and lacks interpretability. In this study, we developed a deep learning model to predict the presence of EGFR mutations by analyzing histopathological patterns in whole slide images (WSIs). We also introduced the EGFR mutation prevalence (EMP) score, which quantifies EGFR prevalence in WSIs based on patch-level predictions, and evaluated its interpretability and utility. Our model estimates the probability of EGFR prevalence in each patch by partitioning the WSI based on multiple-instance learning and predicts the presence of EGFR mutations at the slide level. We utilized a patch-masking scheduler training strategy to enable the model to learn various histopathological patterns of EGFR. This study included 868 WSI samples from lung adenocarcinoma patients collected from three medical institutions: Hallym University Medical Center, Inha University Hospital, and Chungnam National University Hospital. For the test dataset, 197 WSIs were collected from Ajou University Medical Center to evaluate the presence of EGFR mutations. Our model demonstrated prediction performance with an area under the receiver operating characteristic curve of 0.7680 (0.7607-0.7720) and an area under the precision-recall curve of 0.8391 (0.8326-0.8430). The EMP score showed Spearman correlation coefficients of 0.4705 (p = 0.0087) for p.L858R and 0.5918 (p = 0.0037) for exon 19 deletions in 64 samples subjected to next-generation sequencing analysis. Additionally, high EMP scores were associated with papillary and acinar patterns (p = 0.0038 and p = 0.0255, respectively), whereas low EMP scores were associated with solid patterns (p = 0.0001). These results validate the reliability of our model and suggest that it can provide crucial information for rapid screening and treatment plans.

A MIL-based approach for welding defect classification

Welding quality plays a prominent role in many fabrication processes, such as those in the automotive, aerospace, shipbuilding, and electronics industries. Deep learning facilitates the automatic classification of welding quality using non-invasive techniques such as X-rays, thermal imaging, and ultrasonic techniques. In this study, we investigate the ability of deep learning to detect welding quality directly from the welded workpieces. Typically, good and bad weld samples can be seen together along with surrounding material as a background, which makes traditional classification based using computer vision difficult. Therefore, we propose a new framework based on Multiple Instance Learning (MIL) that reveals the distribution of each small piece of the image and serves as an effective guide for welding quality classification. Furthermore, a new collection of welding image datasets is developed to verify the baseline model and our proposed method. The results show that our proposed method is effective in classifying welding quality in welding workpiece images and identifying welding features in each small section of the welding area in workpiece images. This study demonstrates the effectiveness of incorporating MIL in weakly supervised welding quality classification and reveals the potential for performing quality control on welding in industrial pipelines automatically.

Multimodal and multiscale feature fusion for weakly supervised video anomaly detection

Weakly supervised video anomaly detection aims to detect anomalous events with only video-level labels. In the absence of boundary information for anomaly segments, most existing methods rely on multiple instance learning. In these approaches, the predictions for unlabeled video snippets are guided by the classification of labeled untrimmed videos. However, these methods do not account for issues such as video blur and visual occlusion, which can hinder accurate anomaly detection. To address these issues, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, RGB and optical flow snippets are input into pre-trained I3D to extract appearance and motion features. Then, we introduce an Attention De-redundancy (AD) module, which employs an attention mechanism to filter out task-irrelevant redundancy in these appearance and motion features. Next, to mitigate the effects of video blurring and visual occlusion, we propose a Multi-scale Feature Learning module. This module captures long-term and short-term temporal dependencies among video snippets to provide global and local guidance for blurred or occluded video snippets. Finally, to effectively utilize the discriminative features of different modalities, we propose an Adaptive Feature Fusion module. This module adaptively fuses appearance and motion features based on their respective feature weights. Extensive experimental results demonstrate that our proposed method outperforms mainstream unsupervised and weakly supervised methods in terms of AUC. Specifically, our proposed method achieves 97.00% AUC and 85.31% AUC on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime, respectively.