Multiple Instance Learning Framework Research Articles

RLogist: Fast Observation Strategy on Whole-Slide Images with Deep Reinforcement Learning

Whole-slide images (WSI) in computational pathology have high resolution with gigapixel size, but are generally with sparse regions of interest, which leads to weak diagnostic relevance and data inefficiency for each area in the slide. Most of the existing methods rely on a multiple instance learning framework that requires densely sampling local patches at high magnification. The limitation is evident in the application stage as the heavy computation for extracting patch-level features is inevitable. In this paper, we develop RLogist, a benchmarking deep reinforcement learning (DRL) method for fast observation strategy on WSIs. Imitating the diagnostic logic of human pathologists, our RL agent learns how to find regions of observation value and obtain representative features across multiple resolution levels, without having to analyze each part of the WSI at the high magnification. We benchmark our method on two whole-slide level classification tasks, including detection of metastases in WSIs of lymph node sections, and subtyping of lung cancer. Experimental results demonstrate that RLogist achieves competitive classification performance compared to typical multiple instance learning algorithms, while having a significantly short observation path. In addition, the observation path given by RLogist provides good decision-making interpretability, and its ability of reading path navigation can potentially be used by pathologists for educational/assistive purposes. Our code is available at: https://github.com/tencent-ailab/RLogist.

Multi-modal deep learning improves grain yield prediction in wheat breeding by fusing genomics and phenomics.

Developing new crop varieties with superior performance is highly important to ensure robust and sustainable global food security. The speed of variety development is limited by long field cycles and advanced generation selections in plant breeding programs. While methods to predict yield from genotype or phenotype data have been proposed, improved performance and integrated models are needed. We propose a machine learning model that leverages both genotype and phenotype measurements by fusing genetic variants with multiple data sources collected by unmanned aerial systems. We use a deep multiple instance learning framework with an attention mechanism that sheds light on the importance given to each input during prediction, enhancing interpretability. Our model reaches 0.754 ± 0.024 Pearson correlation coefficient when predicting yield in similar environmental conditions; a 34.8% improvement over the genotype-only linear baseline (0.559 ± 0.050). We further predict yield on new lines in an unseen environment using only genotypes, obtaining a prediction accuracy of 0.386 ± 0.010, a 13.5% improvement over the linear baseline. Our multi-modal deep learning architecture efficiently accounts for plant health and environment, distilling the genetic contribution and providing excellent predictions. Yield prediction algorithms leveraging phenotypic observations during training therefore promise to improve breeding programs, ultimately speeding up delivery of improved varieties. Available at https://github.com/BorgwardtLab/PheGeMIL (code) and https://doi.org/doi:10.5061/dryad.kprr4xh5p (data).

Generalized pancreatic cancer diagnosis via multiple instance learning and anatomically-guided shape normalization.

Pancreatic cancer is a highly malignant cancer type with a high mortality rate. As no obvious symptoms are associated with this cancer type, most of the diagnoses are made when the patients are already in a late stage. In this work, we propose an automated method for effective early diagnosis of pancreatic cancer based on multiple instance learning with contrast-enhanced CT images. In this method, diagnosis stability and generalizability were improved through shape normalization based on anatomical structures as well as instance-level contrastive learning. Specifically, anatomically-guided shape normalization were developed to reconstruct the pancreatic regions of interest by spatial transformations, account for larger tumor parts in these regions, and hence enhance the extraction of pancreatic features. Moreover, instance-level contrastive learning was employed to aggregate different types of tumor features within the multiple instance learning framework. This learning approach can maintain the tumor feature integrity and enhance the diagnosis stability. Finally, a balance-adjustment strategy was designed to alleviate the class imbalance problem caused by the scarcity of tumor samples. Extensive experimental results demonstrated remarkable performance of our method when conducted cross-validation on an in-house dataset with 310 patients and independent test on two unseen datasets (a private test set with 316 and a publicly-available test set with 281). The proposed strategies also led to significant improvements in generalizability. Besides, the clinical significance of the proposed method was further verified through two independent test results in which tumors smaller than 2cm in diameter were identified at accuracies of 80.9% and 90.1%, respectively. Overall, our method provides a potentially successful tool for early diagnosis of pancreatic cancer. Our source codes will be released at https://github.com/SJTUBME-QianLab/MIL_PAdiagnosis.

Weakly supervised histopathology image segmentation with self-attention.

Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.

Histopathological auxiliary system for brain tumour (HAS-Bt) based on weakly supervised learning using a WHO CNS5-style pipeline.

Classification and grading of central nervous system (CNS) tumours play a critical role in the clinic. When WHO CNS5 simplifies the histopathology diagnosis and places greater emphasis on molecular pathology, artificial intelligence (AI) has been widely used to meet the increased need for an automatic histopathology scheme that could liberate pathologists from laborious work. This study was to explore the diagnosis scope and practicality of AI. A one-stop Histopathology Auxiliary System for Brain tumours (HAS-Bt) is introduced based on a pipeline-structured multiple instance learning (pMIL) framework developed with 1,385,163 patches from 1038 hematoxylin and eosin (H&E) slides. The system provides a streamlined service including slide scanning, whole-slide image (WSI) analysis and information management. A logical algorithm is used when molecular profiles are available. The pMIL achieved an accuracy of 0.94 in a 9-type classification task on an independent dataset composed of 268 H&E slides. Three auxiliary functions are developed and a built-in decision tree with multiple molecular markers is used to automatically formed integrated diagnosis. The processing efficiency was 443.0s per slide. HAS-Bt shows outstanding performance and provides a novel aid for the integrated neuropathological diagnostic workflow of brain tumours using CNS 5 pipeline.

Robust fall detection in video surveillance based on weakly supervised learning

Fall event detection has been a research hotspot in recent years in the fields of medicine and health. Currently, vision-based fall detection methods have been considered the most promising methods due to their advantages of a non-contact characteristic and easy deployment. However, the existing vision-based fall detection methods mainly use supervised learning in model training and require much time and energy for data annotations. To address these limitations, this work proposes a detection method that uses a weakly supervised learning-based dual-modal network. The proposed method adopts a deep multiple instance learning framework to learn the fall events using weak labels. As a result, the proposed method does not require time-consuming fine-grained annotations. The final detection result of each video is obtained by integrating the information obtained from two streams of the dual-modal network using the proposed dual-modal fusion strategy. Experimental results on two public benchmark datasets and a proposed dataset demonstrate the superiority of the proposed method over the current state-of-the-art methods.

Abstract 5414: Automated detection of high-grade urothelial carcinoma from urine cytology slides using attention-based deep learning

Abstract Urine cytology has long been an effective and non-invasive test for the detection of bladder urothelial carcinomas (UC) routinely performed in cases of unexplained hematuria or for monitoring patients with UC. In cytopathology practice, urine cytology specimens are examined manually with a light microscope to identify morphologic features associated with different diagnostic categories based on the Paris System (TPS) for Reporting Urinary Cytology. Specifically, the diagnosis of high-grade urothelial carcinoma (HGUC) requires the identification of > 5-10 cells with a nuclear/cytoplasm ratio of 0.7 or greater and hyperchromasia together with coarse chromatin or irregular nuclear membranes. However, the task of identifying HGUC involves a substantial degree of manual review and is often associated with intra-and inter-observer variability. To address this, we have designed an accurate and efficient deep learning system capable of automatically distinguishing between HGUC and non-HGUC using digitized cytology slides. Our model has been developed using a retrospective cohort of 158 digitized urine ThinPrep cytology slides consisting of HGUC (n=98) and negative for HGUC (n=60). The model was then prospectively validated on a cohort of 105 urine cytology slides that were also independently reviewed prospectively in a blinded manner by a cytopathologist and cytotechnologist. Our system uses Otsu’s method for automatic image thresholding followed by dividing images into non-overlapping tiles of 500 × 500 pixels at the highest magnification. Subsequently, we use a pre-trained ResNet50 model to extract features which are used for training our attention-based multiple instance learning framework. For the training task, our retrospective cohort (158 slides) has been divided into 10 different splits each consisting of training (70%), validation (15%), and testing (15%) sets. The training and validation sets were used for the model training and optimalization, respectively, while the testing set was used for assessing the performance. This process yielded 10 different models with an average Area Under the ROC Curve (AUC) of 0.80 in the testing set. The best performing model had an AUC of 0.90 and an accuracy of 0.88. This model was subsequently validated prospectively in an independent testing cohort with 105 slides. In the prospective testing cohort, the model was able to accurately distinguish between HGUC and non-HGUC with an AUC of 0.83, accuracy of 0.76, sensitivity of 0.89, and specificity of 0.62. Additionally, our system can detect slide regions with high attention score for HGUC which are enriched in atypical urothelial cells. These findings show that our system can be utilized to assist cytopathologists in assessing urine cytology slides and to detect regions with high-diagnostic relevance for further assessment which is expected to reduce the time needed for manual review. Citation Format: Mohamed Omar, David Kim, Luigi Marchionni, Momin T. Siddiqui. Automated detection of high-grade urothelial carcinoma from urine cytology slides using attention-based deep learning. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5414.

Abstract 5369: Detection of ERG:TMPRSS2 gene fusion in prostate cancer from histopathology slides using attention-based deep learning

Abstract ERG:TMPRSS2 fusion is present in almost 50% of prostate cancer (PCa) cases of European ascent and plays an important role in carcinogenesis and disease progression. ERG status is detected using fluorescence in situ hybridization (FISH) or reverse transcription-polymerase chain reaction (RT-PCR), and since these tests are costly and require special training, there is need for innovative tools to decrease the cost and streamline the diagnostic process. For this reason, we have developed a deep learning (DL) system capable of inferring ERG fusion status using only digitized hematoxylin and eosin (H&E)-stained slides from PCa patients, and detecting tissue regions of high diagnostic relevance. To develop the model, we used the PCa TCGA dataset which includes 436 formalin-fixed paraffin-embedded (FFPE) H&E-stained whole slide images (WSIs) from 393 PCa patients who underwent radical prostatectomy. Subsequently, we evaluated the model’s performance on an independent cohort of 314 WSIs provided by the Johns Hopkins University (natural history cohort). Slides were divided into tiles of 500 × 500 px from which feature extraction was performed using a pre-trained ResNet50 model. Feature vectors were then used to train attention-based multiple instance learning framework to predict the slide-level label as either ERG-positive or negative, and to score slide regions based on their contribution to the slide-level representation. Our model can predict the ERG fusion status with an Area Under the ROC Curve (AUC) of 0.84 and 0.73 in the training data and the independent testing cohort, respectively. Also, the model detects tissue regions with high attention score for each class. To decipher the cellular composition in these highly relevant regions for the cases predicted as ERG-positive or negative, we used HoVer-Net model to perform nuclear segmentation and classification into five categories: benign epithelial, tumor, stroma, immune, and necrotic. Notably, we found that the cellular composition of the highly relevant patches can capture prognostic information. Specifically, In the TCGA dataset, a high ratio of neoplastic cells in the relevant patches was significantly associated with worse progression free survival (PFS), while high ratios of necrotic, stromal, and stromal to neoplastic cells were significantly associated with better PFS. Similar findings were also obtained in the natural history cohort in which a high ratio of neoplastic cells was significantly associated with worse overall survival (OS) and metastasis free survival (MFS), while high ratios of immune, stromal, and stromal to neoplastic cells were significantly associated with longer OS and MFS. These results show that ERG fusion status can be inferred from H&E-stained WSIs, ultimately demonstrating the benefit of DL systems in extracting tissue morphological features of high diagnostic and prognostic relevance. Citation Format: Mohamed Omar, Zhuoran Xu, Sophie B. Rand, Daniela C. Salles, Edward M. Schaeffer, Tamara L. Lotan, Massimo Loda, Luigi Marchionni. Detection of ERG:TMPRSS2 gene fusion in prostate cancer from histopathology slides using attention-based deep learning. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5369.

Convolutional Multiple Instance Learning for Sleep Spindle Detection With Label Refinement

Sleep spindles are closely associated with cognitive functions and neurological disorders; thus, spindle detection has been an important topic in sleep medicine. Recently, machine learning approaches have shown the potential in automatic sleep spindle detection by learning optimized features in a data-driven way, while they highly rely on labeled data, and the performance can be degraded when labels are inaccurate. However, accurate annotation of the spindle is usually difficult to obtain and high intraexpert and interexpert variance exist. In this work, we propose a convolutional neural network (CNN) with a label refinement component to learn an effective spindle detector with imperfect labels. Our approach consists of two stages: 1) a feature learning stage and 2) a label refinement stage. In the feature learning stage, a CNN-based multiple instance learning framework (CNN-MIL) is built for spindle feature learning. By assuming only parts of each labeled spindle segment contain true spindle patterns, the CNN-MIL model can learn most-likely spindle-related features from ambiguous labels. In the label refinement stage, we adjust the spindle labels by merging the original labels and labels predicted by CNN-MIL, and the modified labels are then used in the next round CNN-MIL feature learning. The two stages perform alternately for detector optimization. Extensive experiments demonstrated that our approach achieved the state-of-the-art performance.

Multiple Temporal Pooling Mechanisms for Weakly Supervised Temporal Action Localization

Recent action localization works learn in a weakly supervised manner to avoid the expensive cost of human labeling. Those works are mostly based on the Multiple Instance Learning framework, where temporal pooling is an indispensable part that usually relies on the guidance of snippet-level Class Activation Sequences (CAS) . However, we observe that previous works only leverage a simple convolutional neural network for the generation of CAS, which ignores the weak discriminative foreground action segments and the background ones, and meanwhile, the relationship between different actions has not been considered. To solve this problem, we propose multiple temporal pooling mechanisms (MTP) for a more sufficient information utilization. Specifically, with the design of the Foreground Variance Branch, Dual Foreground Attention Branch and Hybrid Attention Fine-tuning Branch, MTP can leverage more effective information from different aspects and generate different CASs to guide the learning of temporal pooling. Moreover, different loss functions are designed for a better optimization of individual branches, aiming to effectively distinguish the action from the background. Our method shows excellent results on the THUMOS14 and ActivityNet1.2 datasets.

The devil is in the details: a small-lesion sensitive weakly supervised learning framework for prostate cancer detection and grading.

Prostate cancer (PCa) is a significant health concern in aging males, and the diagnosis depends primarily on histopathological assessments to determine tumor size and Gleason score. This process is highly time-consuming, subjective, and relies on the extensive experience of the pathologists. Deep learning based artificial intelligence shows an ability to match pathologists on many prostate cancer diagnostic scenarios. However, it is easy to make mistakes on some hard cases with small tumor areas considering the extensively high-resolution of whole slide images (WSIs). The absence of fine-grained and large-scale annotations of such small tumor lesions makes this problem more challenging. Existing methods usually perform uniform cropping of the foreground of WSI and then use convolutional neural networks as the backbone network to predict the classification results. However, cropping can damage the structure of tiny tumors, which affects classification accuracy. To solve this problem, we propose an Intensive-Sampling Multiple Instance Learning Framework (ISMIL), which focuses on tumor regions and improves the recognition of small tumor regions by intensively sampling the crucial regions. Experiments of prostate cancer detection show that our method achieves an area under the receiver operating characteristic curve (AUC) of 0.987 on the PANDA sets, which improves recall by at least 33% with higher specificity over the current primary methods for hard cases. The ISMIL also demonstrates comparable abilities to human experts on the prostate cancer grading task. Moreover, ISMIL has shown good robustness in independent cohorts, which makes it a potential tool to improve the diagnostic efficiency of pathologists.

Deep semi-supervised multiple instance learning with self-correction for DME classification from OCT images.

Supervised deep learning has achieved prominent success in various diabetic macular edema (DME) recognition tasks from optical coherence tomography (OCT) volumetric images. A common problematic issue that frequently occurs in this field is the shortage of labeled data due to the expensive fine-grained annotations, which increases substantial difficulty in accurate analysis by supervised learning. The morphological changes in the retina caused by DME might be distributed sparsely in B-scan images of the OCT volume, and OCT data is often coarsely labeled at the volume level. Hence, the DME identification task can be formulated as a multiple instance classification problem that could be addressed by multiple instance learning (MIL) techniques. Nevertheless, none of previous studies utilize unlabeled data simultaneously to promote the classification accuracy, which is particularly significant for a high quality of analysis at the minimum annotation cost. To this end, we present a novel deep semi-supervised multiple instance learning framework to explore the feasibility of leveraging a small amount of coarsely labeled data and a large amount of unlabeled data to tackle this problem. Specifically, we come up with several modules to further improve the performance according to the availability and granularity of their labels. To warm up the training, we propagate the bag labels to the corresponding instances as the supervision of training, and propose a self-correction strategy to handle the label noise in the positive bags. This strategy is based on confidence-based pseudo-labeling with consistency regularization. The model uses its prediction to generate the pseudo-label for each weakly augmented input only if it is highly confident about the prediction, which is subsequently used to supervise the same input in a strongly augmented version. This learning scheme is also applicable to unlabeled data. To enhance the discrimination capability of the model, we introduce the Student-Teacher architecture and impose consistency constraints between two models. For demonstration, the proposed approach was evaluated on two large-scale DME OCT image datasets. Extensive results indicate that the proposed method improves DME classification with the incorporation of unlabeled data and outperforms competing MIL methods significantly, which confirm the feasibility of deep semi-supervised multiple instance learning at a low annotation cost.

Label Cleaning Multiple Instance Learning: Refining Coarse Annotations on Single Whole-Slide Images.

Annotating cancerous regions in whole-slide images (WSIs) of pathology samples plays a critical role in clinical diagnosis, biomedical research, and machine learning algorithms development. However, generating exhaustive and accurate annotations is labor-intensive, challenging, and costly. Drawing only coarse and approximate annotations is a much easier task, less costly, and it alleviates pathologists' workload. In this paper, we study the problem of refining these approximate annotations in digital pathology to obtain more accurate ones. Some previous works have explored obtaining machine learning models from these inaccurate annotations, but few of them tackle the refinement problem where the mislabeled regions should be explicitly identified and corrected, and all of them require a - often very large - number of training samples. We present a method, named Label Cleaning Multiple Instance Learning (LC-MIL), to refine coarse annotations on a single WSI without the need for external training data. Patches cropped from a WSI with inaccurate labels are processed jointly within a multiple instance learning framework, mitigating their impact on the predictive model and refining the segmentation. Our experiments on a heterogeneous WSI set with breast cancer lymph node metastasis, liver cancer, and colorectal cancer samples show that LC-MIL significantly refines the coarse annotations, outperforming state-of-the-art alternatives, even while learning from a single slide. Moreover, we demonstrate how real annotations drawn by pathologists can be efficiently refined and improved by the proposed approach. All these results demonstrate that LC-MIL is a promising, lightweight tool to provide fine-grained annotations from coarsely annotated pathology sets.

Su-MICL: Severity-Guided Multiple Instance Curriculum Learning for Histopathology Image Interpretable Classification.

Histopathology image classification plays a critical role in clinical diagnosis. However, due to the absence of clinical interpretability, most existing image-level classifiers remain impractical. To acquire the essential interpretability, lesion-level diagnosis is also provided, relying on detailed lesion-level annotations. Although the multiple-instance learning (MIL)-based approach can identify lesions by only utilizing image-level annotations, it requires overly strict prior information and has limited accuracy in lesion-level tasks. Here, we present a novel severity-guided multiple instance curriculum learning (Su-MICL) strategy to avoid tedious labeling. The proposed Su-MICL is under a MIL framework with a neglected prior: disease severity to define the learning difficulty of training images. Based on the difficulty degree, a curriculum is developed to train a model utilizing images from easy to hard. The experimental results for two histopathology image datasets demonstrate that Su-MICL achieves comparable performance to the state-of-the-art weakly supervised methods for image-level classification, and its performance for identifying lesions is closest to the supervised learning method. Without tedious lesion labeling, the Su-MICL approach can provide an interpretable diagnosis, as well as an effective insight to aid histopathology image diagnosis.

Contrastive Multiple Instance Learning: An Unsupervised Framework for Learning Slide-Level Representations of Whole Slide Histopathology Images without Labels.

Recent methods in computational pathology have trended towards semi- and weakly-supervised methods requiring only slide-level labels. Yet, even slide-level labels may be absent or irrelevant to the application of interest, such as in clinical trials. Hence, we present a fully unsupervised method to learn meaningful, compact representations of WSIs. Our method initially trains a tile-wise encoder using SimCLR, from which subsets of tile-wise embeddings are extracted and fused via an attention-based multiple-instance learning framework to yield slide-level representations. The resulting set of intra-slide-level and inter-slide-level embeddings are attracted and repelled via contrastive loss, respectively. This resulted in slide-level representations with self-supervision. We applied our method to two tasks- (1) non-small cell lung cancer subtyping (NSCLC) as a classification prototype and (2) breast cancer proliferation scoring (TUPAC16) as a regression prototype-and achieved an AUC of 0.8641 ± 0.0115 and correlation (R2) of 0.5740 ± 0.0970, respectively. Ablation experiments demonstrate that the resulting unsupervised slide-level feature space can be fine-tuned with small datasets for both tasks. Overall, our method approaches computational pathology in a novel manner, where meaningful features can be learned from whole-slide images without the need for annotations of slide-level labels. The proposed method stands to benefit computational pathology, as it theoretically enables researchers to benefit from completely unlabeled whole-slide images.

Detection of m6A from direct RNA sequencing using a multiple instance learning framework

RNA modifications such as m6A methylation form an additional layer of complexity in the transcriptome. Nanopore direct RNA sequencing can capture this information in the raw current signal for each RNA molecule, enabling the detection of RNA modifications using supervised machine learning. However, experimental approaches provide only site-level training data, whereas the modification status for each single RNA molecule is missing. Here we present m6Anet, a neural-network-based method that leverages the multiple instance learning framework to specifically handle missing read-level modification labels in site-level training data. m6Anet outperforms existing computational methods, shows similar accuracy as experimental approaches, and generalizes with high accuracy to different cell lines and species without retraining model parameters. In addition, we demonstrate that m6Anet captures the underlying read-level stoichiometry, which can be used to approximate differences in modification rates. Overall, m6Anet offers a tool to capture the transcriptome-wide identification and quantification of m6A from a single run of direct RNA sequencing.

DEEMD: Drug Efficacy Estimation Against SARS-CoV-2 Based on Cell Morphology With Deep Multiple Instance Learning

Drug repurposing can accelerate the identification of effective compounds for clinical use against SARS-CoV-2, with the advantage of pre-existing clinical safety data and an established supply chain. RNA viruses such as SARS-CoV-2 manipulate cellular pathways and induce reorganization of subcellular structures to support their life cycle. These morphological changes can be quantified using bioimaging techniques. In this work, we developed DEEMD: a computational pipeline using deep neural network models within a multiple instance learning framework, to identify putative treatments effective against SARS-CoV-2 based on morphological analysis of the publicly available RxRx19a dataset. This dataset consists of fluorescence microscopy images of SARS-CoV-2 non-infected cells and infected cells, with and without drug treatment. DEEMD first extracts discriminative morphological features to generate cell morphological profiles from the non-infected and infected cells. These morphological profiles are then used in a statistical model to estimate the applied treatment efficacy on infected cells based on similarities to non-infected cells. DEEMD is capable of localizing infected cells via weak supervision without any expensive pixel-level annotations. DEEMD identifies known SARS-CoV-2 inhibitors, such as Remdesivir and Aloxistatin, supporting the validity of our approach. DEEMD can be explored for use on other emerging viruses and datasets to rapidly identify candidate antiviral treatments in the future. Our implementation is available online at https://www.github.com/Sadegh-Saberian/DEEMD.

Multiple instance-based video anomaly detection using deep temporal encoding–decoding

In this paper, we propose a weakly supervised deep temporal encoding–decoding solution for anomaly detection in surveillance videos using multiple instance learning. The proposed approach uses both abnormal and normal video clips during the training phase which is developed in the multiple instance learning framework where we treat the video as a bag and video clips as instances in the bag. Our main contribution lies in the proposed novel approach to consider temporal relations between video instances in a weakly supervised setting. We deal with video instances (clips) as sequential visual data rather than a set of independent instances. We employ a deep temporal encoding–decoding network that is designed to capture spatio-temporal evolution of video instances over time. We also propose a new loss function that maximizes the mean distance between normal and abnormal instance predictions. The new loss function ensures a low false alarm rate which is crucial in practical surveillance applications. The proposed temporal encoding–decoding approach with the modified loss is benchmarked against the state of the art. The results show that the proposed method performs similar to, or better than the state-of-the-art solutions for anomaly detection in video surveillance applications and achieves the lowest false alarm rate on UCF-Crime dataset.

Constrained multiple instance learning for ulcerative colitis prediction using histological images

Background and Objective: Ulcerative colitis (UC) is an inflammatory bowel disease (IBD) affecting the colon and the rectum characterized by a remitting-relapsing course. To detect mucosal inflammation associated with UC, histology is considered the most stringent criteria. In turn, histologic remission (HR) correlates with improved clinical outcomes and has been recently recognized as a desirable treatment target. The leading biomarker for assessing histologic remission is the presence or absence of neutrophils. Therefore, the finding of this cell in specific colon structures indicates that the patient has UC activity. However, no previous studies based on deep learning have been developed to identify UC based on neutrophils detection using whole-slide images (WSI).Methods: The methodological core of this work is a novel multiple instance learning (MIL) framework with location constraints able to determine the presence of UC activity using WSI. In particular, we put forward an effective way to introduce constraints about positive instances to effectively explore additional weakly supervised information that is easy to obtain and enjoy a significant boost to the learning process. In addition, we propose a new weighted embedding to enlarge the relevance of the positive instances.Results: Extensive experiments on a multi-center dataset of colon and rectum WSIs, PICASSO-MIL, demonstrate that using the location information we can improve considerably the results at WSI-level. In comparison with prior MIL settings, our method allows for 10% improvements in bag-level accuracy.Conclusion: Our model, which introduces a new form of constraints, surpass the results achieved from current state-of-the-art methods that focus on the MIL paradigm. Our method can be applied to other histological concerns where the morphological features determining a positive WSI are tiny and similar to others in the image.

Attention-based anomaly detection in multi-view surveillance videos

Anomaly detection is one of the most challenging tasks in visual understanding because anomalous events are diverse and complicated. In this paper, we propose a future frame prediction framework and a Multiple Instance Learning (MIL) framework by leveraging attention schemes to learn anomalies. In both frameworks, we utilize the attention-based module to better localize anomalies. Further, we introduce a memory addressing module for the future frame prediction framework, and a novel loss function for the MIL framework, respectively. We also introduce a new multi-view dataset of 170 videos with 10 realistic anomalies that pose a serious threat to security such as intrusion, crowd, accident, weapon, arson, etc., as well as normal activities. The experimental results demonstrate the effectiveness of our methods on the proposed dataset and multiple benchmark datasets. The proposed dataset brings more opportunities and challenges to future work on anomaly detection.