Multi-instance Learning Framework Research Articles

Multi-Instance Multi-Task Learning for Joint Clinical Outcome and Genomic Profile Predictions From the Histopathological Images.

With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range from cancer diagnosis, survival prediction and estimating the transcriptional profile of gene TP53. The experimental results demonstrated that HistMIMT can yield better outcome on all clinical prediction tasks than its competitors.

Use of Artificial Intelligence in the Prediction of Chiari Malformation Type 1 Recurrence After Posterior Fossa Decompressive Surgery.

Purpose The purpose of this study was to train a deep learning-based method for the prediction of postoperative recurrence of symptoms in Chiari malformation type 1 (CM1) patients undergoing surgery. Studies suggest that certain radiological and clinical features do exist in patients with treatment failure, though these are inconsistent and poorly defined. Methodology This study was a retrospective cohort study of patients who underwent primary surgical intervention for CM1 from January 2010 to May 2020. Only patients who completed pre- and postoperative 12-item short form(SF-12) surveys were included and these were used to classify the recurrence or persistence of symptoms. Forty patients had an improvement in overall symptoms while 17 had recurrence or persistence. After magnetic resonance imaging (MRI) data augmentation, a ResNet50, pre-trained on the ImageNet dataset, was used for feature extraction, and then clustering-constrained attention multiple instance learning (CLAM), a weakly supervised multi-instance learning framework, was trained for prediction of recurrence. Five-fold cross-validation was used for the development of MRI only, clinical features only, and a combined machine learning model. Results This study included 57 patients who underwent CM1 decompression. The recurrence rate was 30%. The combined model incorporating MRI, pre-operative SF-12 physical component scale (PCS), and extent of cerebellar ectopia performed best with an area under the curve (AUC) of 0.71 and an F1 score of 0.74. Conclusion This is the first study to our knowledge to explore the prediction of postoperative recurrence of symptoms in CM1 patients using machine learning methods and represents the first steptowarddeveloping a clinically useful prognostication machine learning model. Further studies utilizing a similar deep learning approach with a larger sample size are needed to improve the performance.

MAMILNet: advancing precision oncology with multi-scale attentional multi-instance learning for whole slide image analysis.

Whole Slide Image (WSI) analysis, driven by deep learning algorithms, has the potential to revolutionize tumor detection, classification, and treatment response prediction. However, challenges persist, such as limited model generalizability across various cancer types, the labor-intensive nature of patch-level annotation, and the necessity of integrating multi-magnification information to attain a comprehensive understanding of pathological patterns. In response to these challenges, we introduce MAMILNet, an innovative multi-scale attentional multi-instance learning framework for WSI analysis. The incorporation of attention mechanisms into MAMILNet contributes to its exceptional generalizability across diverse cancer types and prediction tasks. This model considers whole slides as "bags" and individual patches as "instances." By adopting this approach, MAMILNet effectively eliminates the requirement for intricate patch-level labeling, significantly reducing the manual workload for pathologists. To enhance prediction accuracy, the model employs a multi-scale "consultation" strategy, facilitating the aggregation of test outcomes from various magnifications. Our assessment of MAMILNet encompasses 1171 cases encompassing a wide range of cancer types, showcasing its effectiveness in predicting complex tasks. Remarkably, MAMILNet achieved impressive results in distinct domains: for breast cancer tumor detection, the Area Under the Curve (AUC) was 0.8872, with an Accuracy of 0.8760. In the realm of lung cancer typing diagnosis, it achieved an AUC of 0.9551 and an Accuracy of 0.9095. Furthermore, in predicting drug therapy responses for ovarian cancer, MAMILNet achieved an AUC of 0.7358 and an Accuracy of 0.7341. The outcomes of this study underscore the potential of MAMILNet in driving the advancement of precision medicine and individualized treatment planning within the field of oncology. By effectively addressing challenges related to model generalization, annotation workload, and multi-magnification integration, MAMILNet shows promise in enhancing healthcare outcomes for cancer patients. The framework's success in accurately detecting breast tumors, diagnosing lung cancer types, and predicting ovarian cancer therapy responses highlights its significant contribution to the field and paves the way for improved patient care.

MA-MIL: Sampling point-level abnormal ECG location method via weakly supervised learning

Background and ObjectiveCurrent automatic electrocardiogram (ECG) diagnostic systems could provide classification outcomes but often lack explanations for these results. This limitation hampers their application in clinical diagnoses. Previous supervised learning could not highlight abnormal segmentation output accurately enough for clinical application without manual labeling of large ECG datasets. MethodIn this study, we present a multi-instance learning framework called MA-MIL, which has designed a multi-layer and multi-instance structure that is aggregated step by step at different scales. We evaluated our method using the public MIT-BIH dataset and our private dataset. ResultsThe results show that our model performed well in both ECG classification output and heartbeat level, sub-heartbeat level abnormal segment detection, with accuracy and F1 scores of 0.987 and 0.986 for ECG classification and 0.968 and 0.949 for heartbeat level abnormal detection, respectively. Compared to visualization methods, the IoU values of MA-MIL improved by at least 17 % and at most 31 % across all categories. ConclusionsMA-MIL could accurately locate the abnormal ECG segment, offering more trustworthy results for clinical application.

DeepLION2: deep multi-instance contrastive learning framework enhancing the prediction of cancer-associated T cell receptors by attention strategy on motifs.

T cell receptor (TCR) repertoires provide valuable insights into complex human diseases, including cancers. Recent advancements in immune sequencing technology have significantly improved our understanding of TCR repertoire. Some computational methods have been devised to identify cancer-associated TCRs and enable cancer detection using TCR sequencing data. However, the existing methods are often limited by their inadequate consideration of the correlations among TCRs within a repertoire, hindering the identification of crucial TCRs. Additionally, the sparsity of cancer-associated TCR distribution presents a challenge in accurate prediction. To address these issues, we presented DeepLION2, an innovative deep multi-instance contrastive learning framework specifically designed to enhance cancer-associated TCR prediction. DeepLION2 leveraged content-based sparse self-attention, focusing on the top k related TCRs for each TCR, to effectively model inter-TCR correlations. Furthermore, it adopted a contrastive learning strategy for bootstrapping parameter updates of the attention matrix, preventing the model from fixating on non-cancer-associated TCRs. Extensive experimentation on diverse patient cohorts, encompassing over ten cancer types, demonstrated that DeepLION2 significantly outperformed current state-of-the-art methods in terms of accuracy, sensitivity, specificity, Matthews correlation coefficient, and area under the curve (AUC). Notably, DeepLION2 achieved impressive AUC values of 0.933, 0.880, and 0.763 on thyroid, lung, and gastrointestinal cancer cohorts, respectively. Furthermore, it effectively identified cancer-associated TCRs along with their key motifs, highlighting the amino acids that play a crucial role in TCR-peptide binding. These compelling results underscore DeepLION2's potential for enhancing cancer detection and facilitating personalized cancer immunotherapy. DeepLION2 is publicly available on GitHub, at https://github.com/Bioinformatics7181/DeepLION2, for academic use only.

A patch distribution-based active learning method for multiple instance Alzheimer's disease diagnosis

Medical data, particularly the complex brain imaging structures, acquisition presents significant difficulties and high diagnostic expenses, resulting in a scarcity of the trainable samples in the real-world scenarios. To overcome this limitation, we present an active learning-based sampling strategy that selects the most informative samples from the unlabeled candidate sample pool for expert annotation, leading to high classification performance with a reduced number of training samples. This study adopts a patch-level perspective and introduces a multi-instance learning framework for Alzheimer's Disease diagnosis. Initially, a patch pre-selection module is designed to identify pathology-prone regions while excluding background areas and irrelevant information. Subsequently, an inner-patch local attention mechanism block and an outer-patch global attention mechanism block are developed to enhance the extraction of discriminative local and global information by the network model. Finally, an active learning sampling strategy is devised to minimize the costs associated with data acquisition and expert annotation in medical domain. The effectiveness of the proposed network framework and active learning strategy was validated through four sets of control experiments on the ADNI dataset.

Double branch synergies with modal reinforcement for weakly supervised temporal action detection

Weakly supervised Temporal Action localization (WTAL) aims to locate the action instances and identify their corresponding labels. Most current methods rely on a Multi-Instance Learning (MIL) framework to predict start and end boundaries of each action in a video. However, they have shortcomings of incomplete positioning and context confusion. Therefore, we propose an algorithm of Double Branch Synergies with Modal Reinforcement (DBSMR), which utilizes long-short temporal attention to model contextual relationships and refines segmental features to encourage more distinguishable segment classification. In terms of blur boundaries between actions and camouflage background in complex scenes and easily resulting in wrong positioning, we construct a sparse graph focusing on the effective representation of context motion by optical flow modal learning, further enhancing the representation of the active region to be examined, and suppressing the interference from the background. Finally, with the idea of ”All roads lead to Rome”, we design motion-guided loss constraints to balance the long-short temporal module and graph reinforcement module, by which the two branches can converge to almost the same detection goal, thus to achieve an agreement of near ground truth localization result. The algorithm achieves mAP of 69.1% and 42.0% detection performances on the THUMOS14 and ActivityNet1.2 datasets respectively. We also verify its effectiveness by comparing it with the state-of-the-art methods.

P995 Development and validation of a deep learning model based on histological characteristics to predict primary non-response of infliximab in Crohn's disease

Abstract Background Predicting primary non-response to infliximab (IFX) is important to select the optimal therapy and utilize personalized management in patients of Crohn’s disease (CD). This study aimed to develop and validate a deep learning (DL) model based on histological characteristics for predicting primary non-response to IFX in CD. Methods A training cohort and a validation cohort were recruited from the First Affiliated Hospital of Sun Yat-sen University. Clinical features, serological biomarkers, imaging features and histological characteristics of colonoscopy biopsy at baseline of IFX therapy were obtained as candidate predictors. Primary non-response was assessed at week 14. We developed a model using a multi-instance learning framework. This model combines patch-level features from whole slide images (WSI) with multi-modal information, together with pre-treatment image features and clinical features, to predict the response to infliximab (IFX) therapy. Finally, the model was validated for area under the curve (AUC) in a reference validation cohort. Results A total of 131 patients were included in the derivation cohort. The DL model based on pathologic features only had an under-the-curve receiver operating characteristic (AUROC) value of 0.87 in the 10-fold cross-validation cohort. The mean AUROC for the model combining pathologic and radiographic features was 0.88. The DL model achieved the highest AUROC value of 0.91 when combining pathological, radiomics and clinical features (Figure 1). When the pathological film was visualized by attention thermography, it was found that the ulcerated tissue was focused on, especially in areas dominated by neutrophils, lymphocyte aggregation and abnormal crypt structure. Conclusion The study developed several accurate DL models for predicting primary non-response to IFX by WSI. The DL model combining pathological, radiomics and clinical features achieving best performance. This model may help clinicians select appropriate therapeutic strategy and avoid unnecessary exposure to IFX in CD patients.

Superpixel-based multi-scale multi-instance learning for hyperspectral image classification

Superpixels can define meaningful local regions within a hyperspectral image (HSI) and have become the building blocks of various HSI classification methods. The superpixels in HSIs consist of multiple spectral pixels, sharing a similar structure with the data in multi-instance learning (MIL). However, the potential of MIL methods in the field of HSI classification has been rarely explored. In this paper, we propose the superpixel-based multi-scale multi-instance learning (MSMIL) framework, enhancing the superpixel representation with MIL for the first time. Segmenting the HSIs with superpixels, MSMIL converts the HSI classification into MIL problems and extracts superpixel representations via the MIL method, namely multi-instance factor analysis (MIFA). Compared with the existing methods focusing exclusively on the local information, MIFA utilizes the deviations from an overall generative model to describe the superpixels, retaining both the local and the global information. Moreover, MSMIL introduces multi-scale superpixels and a spectral–spatial decision fusion strategy for further refinement, where the results of multi-scale superpixel maps are weighted according to prediction certainty and spatial consistency. The proposed method is evaluated on four benchmark datasets and achieves competitive results. For example, MSMIL outperforms the comparison methods with a margin of 5% overall accuracy on the Indian Pines dataset, when 2% pixels are selected as the training set.

Masked autoencoders with handcrafted feature predictions: Transformer for weakly supervised esophageal cancer classification

Background and ObjectiveEsophageal cancer is a serious disease with a high prevalence in Eastern Asia. Histopathology tissue analysis stands as the gold standard in diagnosing esophageal cancer. In recent years, there has been a shift towards digitizing histopathological images into whole slide images (WSIs), progressively integrating them into cancer diagnostics. However, the gigapixel sizes of WSIs present significant storage and processing challenges, and they often lack localized annotations. To address this issue, multi-instance learning (MIL) has been introduced for WSI classification, utilizing weakly supervised learning for diagnosis analysis. By applying the principles of MIL to WSI analysis, it is possible to reduce the workload of pathologists by facilitating the generation of localized annotations. Nevertheless, the approach's effectiveness is hindered by the traditional simple aggregation operation and the domain shift resulting from the prevalent use of convolutional feature extractors pretrained on ImageNet. MethodsWe propose a MIL-based framework for WSI analysis and cancer classification. Concurrently, we introduce employing self-supervised learning, which obviates the need for manual annotation and demonstrates versatility in various tasks, to pretrain feature extractors. This method enhances the extraction of representative features from esophageal WSI for MIL, ensuring more robust and accurate performance. ResultsWe build a comprehensive dataset of whole esophageal slide images and conduct extensive experiments utilizing this dataset. The performance on our dataset demonstrates the efficiency of our proposed MIL framework and the pretraining process, with our framework outperforming existing methods, achieving an accuracy of 93.07% and AUC (area under the curve) of 95.31%. ConclusionThis work proposes an effective MIL method to classify WSI of esophageal cancer. The promising results indicate that our cancer classification framework holds great potential in promoting the automatic whole esophageal slide image analysis.

Shared-Specific Feature Learning With Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.

The fusion of multi-modal medical data is essential to assist medical experts to make treatment decisions for precision medicine. For example, combining the whole slide histopathological images (WSIs) and tabular clinical data can more accurately predict the lymph node metastasis (LNM) of papillary thyroid carcinoma before surgery to avoid unnecessary lymph node resection. However, the huge-sized WSI provides much more high-dimensional information than low-dimensional tabular clinical data, making the information alignment challenging in the multi-modal WSI analysis tasks. This paper presents a novel transformer-guided multi-modal multi-instance learning framework to predict lymph node metastasis from both WSIs and tabular clinical data. We first propose an effective multi-instance grouping scheme, named siamese attention-based feature grouping (SAG), to group high-dimensional WSIs into representative low-dimensional feature embeddings for fusion. We then design a novel bottleneck shared-specific feature transfer module (BSFT) to explore the shared and specific features between different modalities, where a few learnable bottleneck tokens are utilized for knowledge transfer between modalities. Moreover, a modal adaptation and orthogonal projection scheme were incorporated to further encourage BSFT to learn shared and specific features from multi-modal data. Finally, the shared and specific features are dynamically aggregated via an attention mechanism for slide-level prediction. Experimental results on our collected lymph node metastasis dataset demonstrate the efficiency of our proposed components and our framework achieves the best performance with AUC (area under the curve) of 97.34%, outperforming the state-of-the-art methods by over 1.27%.

A Structure-aware Hierarchical Graph-based Multiple Instance Learning Framework for pT Staging in Histopathological Image.

Pathological primary tumor (pT) stage focuses on the infiltration degree of the primary tumor to surrounding tissues, which relates to the prognosis and treatment choices. The pT staging relies on the field-of-views from multiple magnifications in the gigapixel images, which makes pixel-level annotation difficult. Therefore, this task is usually formulated as a weakly supervised whole slide image (WSI) classification task with the slide-level label. Existing weakly-supervised classification methods mainly follow the multiple instance learning paradigm, which takes the patches from single magnification as the instances and extracts their morphological features independently. However, they cannot progressively represent the contextual information from multiple magnifications, which is critical for pT staging. Therefore, we propose a structure-aware hierarchical graph-based multi-instance learning framework (SGMF) inspired by the diagnostic process of pathologists. Specifically, a novel graph-based instance organization method is proposed, namely structure-aware hierarchical graph (SAHG), to represent the WSI. Based on that, we design a novel hierarchical attention-based graph representation (HAGR) network to capture the critical patterns for pT staging by learning cross-scale spatial features. Finally, the top nodes of SAHG are aggregated by a global attention layer for bag-level representation. Extensive studies on three large-scale multi-center pT staging datasets with two different cancer types demonstrate the effectiveness of SGMF, which outperforms state-of-the-art up to 5.6% in the F1 score.

LTACL: long-tail awareness contrastive learning for distantly supervised relation extraction

Distantly supervised relation extraction is an automatically annotating method for large corpora by classifying a bound of sentences with two same entities and the relation. Recent works exploit sound performance by adopting contrastive learning to efficiently obtain instance representations under the multi-instance learning framework. Though these methods weaken the impact of noisy labels, it ignores the long-tail distribution problem in distantly supervised sets and fails to capture the mutual information of different parts. We are thus motivated to tackle these issues and establishing a long-tail awareness contrastive learning method for efficiently utilizing the long-tail data. Our model treats major and tail parts differently by adopting hyper-augmentation strategies. Moreover, the model provides various views by constructing novel positive and negative pairs in contrastive learning for gaining a better representation between different parts. The experimental results on the NYT10 dataset demonstrate our model surpasses the existing SOTA by more than 2.61% AUC score on relation extraction. In manual evaluation datasets including NYT10m and Wiki20m, our method obtains competitive results by achieving 59.42% and 79.19% AUC scores on relation extraction, respectively. Extensive discussions further confirm the effectiveness of our approach.

Multi-instance learning with application to the profiling of multi-victim homicides

Homicide involving multiple victims has a significant negative effect on society. Criminal profiling consists of determining the traits of an unknown offender based on those of the crime and the victims, with a view to their identification. To provide the most likely profile of the perpetrator of a multi-victim homicide, we propose a predictive model of supervised machine learning based on a Bayesian Network. Conventional classifiers can generate the perpetrator’s profile according to the traits of each of the victims of the same homicide, but the profiles may differ from one another. To address this issue, we consider the Multi-Instance (MI) learning framework, in which the victims of the same incident form a bag, and each bag is associated with a unique label for each of the perpetrator’s features. We introduce the unanimity MI assumption in this domain, and accordingly allocate a label to the bag based on the labels and probabilities the Bayesian Network has assigned its instances, using a combination rule from those of the ensemble of classifiers. We apply this methodology to the Federal Bureau of Investigation (FBI) homicide database to compare three combination rules empirically in the validation process, as well as theoretically, using the one that ultimately proves to be the best to build the final model, which is then applied in some illustrative examples to achieve the criminal profile.

Childhood Leukemia Classification via Information Bottleneck Enhanced Hierarchical Multi-Instance Learning.

Leukemia classification relies on a detailed cytomorphological examination of Bone Marrow (BM) smear. However, applying existing deep-learning methods to it is facing two significant limitations. Firstly, these methods require large-scale datasets with expert annotations at the cell level for good results and typically suffer from poor generalization. Secondly, they simply treat the BM cytomorphological examination as a multi-class cell classification task, thus failing to exploit the correlation among leukemia subtypes over different hierarchies. Therefore, BM cytomorphological estimation as a time-consuming and repetitive process still needs to be done manually by experienced cytologists. Recently, Multi-Instance Learning (MIL) has achievedmuch progress in data-efficient medical image processing, which only requires patient-level labels (which can be extracted from the clinical reports). In this paper, we propose a hierarchical MIL framework and equip it with Information Bottleneck (IB) to tackle the above limitations. First, to handle the patient-level label, our hierarchical MIL framework uses attention-based learning to identify cells with high diagnostic values for leukemia classification in different hierarchies. Then, following the information bottleneck principle, we propose a hierarchical IB to constrain and refine the representations of different hierarchies for better accuracy and generalization. By applying our framework to a large-scale childhood acute leukemia dataset with corresponding BM smear images and clinical reports, we show that it can identify diagnostic-related cells without the need for cell-level annotations and outperforms other comparison methods. Furthermore, the evaluation conducted on an independent test cohort demonstrates the high generalizability of our framework.

Pyramid multi-loss vision transformer for thyroid cancer classification using cytological smear

Multi-instance learning, a commonly used technique in artificial intelligence for analyzing slides, can be applied to diagnose thyroid cancer based on cytological smears. Since smears do not have multidimensional histological features similar to histopathology, mining potential contextual information and diversity of features is crucial for better classification performance. In this paper, we propose a pyramid multi-loss vision transformer model called PyMLViT, a novel algorithm with two core modules to address these issues. Specifically, we design a pyramid token extraction module to acquire potential contextual information on smears. The pyramid token structure extracts multi-scale local features, and the vision transformer structure further obtains global information through the self-attention mechanism. Furthermore, we construct multi-loss fusion module based on the conventional multi-instance learning framework. With carefully designed bag and patch weight allocation strategies, we incorporate slide-level annotations as pseudo-labels for patches to participate in training, thus enhancing the diversity of supervised information. Extensive experimental results on the real-world dataset show that PyMLViT has a high performance and a competitive number of parameters compared to popular methods for diagnosing thyroid cancer in cytological smears.

Colorectal cancer lymph node metastasis prediction with weakly supervised transformer-based multi-instance learning

Lymph node metastasis examined by the resected lymph nodes is considered one of the most important prognostic factors for colorectal cancer (CRC). However, it requires careful and comprehensive inspection by expert pathologists. To relieve the pathologists’ burden and speed up the diagnostic process, in this paper, we develop a deep learning system with the binary positive/negative labels of the lymph nodes to solve the CRC lymph node classification task. The multi-instance learning (MIL) framework is adopted in our method to handle the whole slide images (WSIs) of gigapixels in size at once and get rid of the labor-intensive and time-consuming detailed annotations. First, a transformer-based MIL model, DT-DSMIL, is proposed in this paper based on the deformable transformer backbone and the dual-stream MIL (DSMIL) framework. The local-level image features are extracted and aggregated with the deformable transformer, and the global-level image features are obtained with the DSMIL aggregator. The final classification decision is made based on both the local and the global-level features. After the effectiveness of our proposed DT-DSMIL model is demonstrated by comparing its performance with its predecessors, a diagnostic system is developed to detect, crop, and finally identify the single lymph nodes within the slides based on the DT-DSMIL and the Faster R-CNN model. The developed diagnostic model is trained and tested on a clinically collected CRC lymph node metastasis dataset composed of 843 slides (864 metastasis lymph nodes and 1415 non-metastatic lymph nodes), achieving the accuracy of 95.3% and the area under the receiver operating characteristic curve (AUC) of 0.9762 (95% confidence interval [CI]: 0.9607–0.9891) for the single lymph node classification. As for the lymph nodes with micro-metastasis and macro-metastasis, our diagnostic system achieves the AUC of 0.9816 (95% CI: 0.9659–0.9935) and 0.9902 (95% CI: 0.9787–0.9983), respectively. Moreover, the system shows reliable diagnostic region localizing performance: the model can always identify the most likely metastases, no matter the model’s predictions or manual labels, showing great potential in avoiding false negatives and discovering incorrectly labeled slides in actual clinical use.Graphical

Automatic recognition of protein subcellular location patterns in single cells from immunofluorescence images based on deep learning.

With the improvement of single-cell measurement techniques, there is a growing awareness that individual differences exist among cells, and protein expression distribution can vary across cells in the same tissue or cell line. Pinpointing the protein subcellular locations in single cells is crucial for mapping functional specificity of proteins and studying related diseases. Currently, research about single-cell protein location is still in its infancy, and most studies and databases do not annotate proteins at the cell level. For example, in the human protein atlas database, an immunofluorescence image stained for a particular protein shows multiple cells, but the subcellular location annotation is for the whole image, ignoring intercellular difference. In this study, we used large-scale immunofluorescence images and image-level subcellular locations to develop a deep-learning-based pipeline that could accurately recognize protein localizations in single cells. The pipeline consisted of two deep learning models, i.e. an image-based model and a cell-based model. The former used a multi-instance learning framework to comprehensively model protein distribution in multiple cells in each image, and could give both image-level and cell-level predictions. The latter firstly used clustering and heuristics algorithms to assign pseudo-labels of subcellular locations to the segmented cell images, and then used the pseudo-labels to train a classification model. Finally, the image-based model was fused with the cell-based model at the decision level to obtain the final ensemble model for single-cell prediction. Our experimental results showed that the ensemble model could achieve higher accuracy and robustness on independent test sets than state-of-the-art methods.

HRD-MILN: Accurately estimate tumor homologous recombination deficiency status from targeted panel sequencing data.

Homologous recombination deficiency (HRD) is a critical feature guiding drug and treatment selection, mainly for ovarian and breast cancers. As it cannot be directly observed, HRD status is estimated on a small set of genomic instability features from sequencing data. The existing methods often perform poorly when handling targeted panel sequencing data; however, the targeted panel is the most popular sequencing strategy in clinical practices. Thus, we proposed HRD-MILN to overcome the computational challenges from targeted panel sequencing. HRD-MILN incorporated a multi-instance learning framework to discover as many loss of heterozygosity (LOH) associated with HRD status to cluster as possible. Then the HRD score is obtained based on the association between the LOHs and the cluster in the sample to be estimated, and finally, the HRD status is estimated based on the score.In comparison experiments on targeted panel sequencing data, the Precision of HRD-MILN could achieve 87%, significantly improved from 63% reported by the existing methods, where the highest margin of improvement reached 14%. It also presented advantages on whole exome sequencing data. Based on our best knowledge, HRD-MILN is the first practical tool for estimating HRD status from targeted panel sequencing data and could benefit clinical applications.

End-to-End diagnosis of breast biopsy images with transformers.

Diagnostic disagreements among pathologists occur throughout the spectrum of benign to malignant lesions. A computer-aided diagnostic system capable of reducing uncertainties would have important clinical impact. To develop a computer-aided diagnosis method for classifying breast biopsy images into a range of diagnostic categories (benign, atypia, ductal carcinoma in situ, and invasive breast cancer), we introduce a transformer-based hollistic attention network called HATNet. Unlike state-of-the-art histopathological image classification systems that use a two pronged approach, i.e., they first learn local representations using a multi-instance learning framework and then combine these local representations to produce image-level decisions, HATNetstreamlines the histopathological image classification pipeline and shows how to learn representations from gigapixel size images end-to-end. HATNetextends the bag-of-words approach and uses self-attention to encode global information, allowing it to learn representations from clinically relevant tissue structures without any explicit supervision. It outperforms the previous best network Y-Net, which uses supervision in the form of tissue-level segmentation masks, by 8%. Importantly, our analysis reveals that HATNetlearns representations from clinically relevant structures, and it matches the classification accuracy of 87U.S. pathologists for this challenging test set.