Multi-instance Learning Research Articles

Deep learning-enabled classification of kidney allograft rejection on whole slide histopathologic images.

Diagnosis of kidney transplant rejection currently relies on manual histopathological assessment, which is subjective and susceptible to inter-observer variability, leading to limited reproducibility. We aim to develop a deep learning system for automated assessment of whole-slide images (WSIs) from kidney allograft biopsies to enable detection and subtyping of rejection and to predict the prognosis of rejection. We collected H&E-stained WSIs of kidney allograft biopsies at 400x magnification from January 2015 to September 2023 at two hospitals. These biopsy specimens were classified as T cell-mediated rejection, antibody-mediated rejection, and other lesions based on the consensus reached by two experienced transplant pathologists. To achieve feature extraction, feature aggregation, and global classification, we employed multi-instance learning and common convolution neural networks (CNNs). The performance of the developed models was evaluated using various metrics, including confusion matrix, receiver operating characteristic curves, the area under the curve (AUC), classification map, heat map, and pathologist-machine confrontations. In total, 906 WSIs from 302 kidney allograft biopsies were included for analysis. The model based on multi-instance learning enables detection and subtyping of rejection, named renal rejection artificial intelligence model (RRAIM), with the overall 3-category AUC of 0.798 in the independent test set, which is superior to that of three transplant pathologists under nearly routine assessment conditions. Moreover, the prognosis models accurately predicted graft loss within 1 year following rejection and treatment response for rejection, achieving AUC of 0.936 and 0.756, respectively. We first developed deep-learning models utilizing multi-instance learning for the detection and subtyping of rejection and prediction of rejection prognosis in kidney allograft biopsies. These models performed well and may be useful in assisting the pathological diagnosis.

Boosting Weakly Supervised Object Localization and Segmentation With Domain Adaption.

Weakly supervised object localization (WSOL), adopting only image-level annotations to learn the pixel-level localization model, can release human resources in the annotation process. Most one-stage WSOL methods learn the localization model with multi-instance learning, making them only activate discriminative object parts rather than the whole object. In our work, we attribute this problem to the domain shift between the training and test process of WSOL and provide a novel perspective that views WSOL as a domain adaption (DA) task. Under this perspective, a DA-WSOL pipeline is elaborated to better assist WSOL with DA approaches by considering the specificities for the adaption of WSOL. Our DA-WSOL pipeline can discern the source-related and the Universum samples from other target samples based on a proposed target sampling strategy and then utilize them to solve the sample unbalancing and label unmatching between the source and target domain of WSOL. Experiments show that our pipeline outperforms SOTA methods on three WSOL benchmarks and can improve the performance of downstream weakly supervised semantic segmentation tasks. Codes are available at https://github.com/zh460045050/dawsol.

Weakly supervised large-scale pancreatic cancer detection using multi-instance learning.

Early detection of pancreatic cancer continues to be a challenge due to the difficulty in accurately identifying specific signs or symptoms that might correlate with the onset of pancreatic cancer. Unlike breast or colon or prostate cancer where screening tests are often useful in identifying cancerous development, there are no tests to diagnose pancreatic cancers. As a result, most pancreatic cancers are diagnosed at an advanced stage, where treatment options, whether systemic therapy, radiation, or surgical interventions, offer limited efficacy. A two-stage weakly supervised deep learning-based model has been proposed to identify pancreatic tumors using computed tomography (CT) images from Henry Ford Health (HFH) and publicly available Memorial Sloan Kettering Cancer Center (MSKCC) data sets. In the first stage, the nnU-Net supervised segmentation model was used to crop an area in the location of the pancreas, which was trained on the MSKCC repository of 281 patient image sets with established pancreatic tumors. In the second stage, a multi-instance learning-based weakly supervised classification model was applied on the cropped pancreas region to segregate pancreatic tumors from normal appearing pancreas. The model was trained, tested, and validated on images obtained from an HFH repository with 463 cases and 2,882 controls. The proposed deep learning model, the two-stage architecture, offers an accuracy of 0.907 0.01, sensitivity of 0.905 0.01, specificity of 0.908 0.02, and AUC (ROC) 0.903 0.01. The two-stage framework can automatically differentiate pancreatic tumor from non-tumor pancreas with improved accuracy on the HFH dataset. The proposed two-stage deep learning architecture shows significantly enhanced performance for predicting the presence of a tumor in the pancreas using CT images compared with other reported studies in the literature.

SAFE-MIL: a statistically interpretable framework for screening potential targeted therapy patients based on risk estimation.

Patients with the target gene mutation frequently derive significant clinical benefits from target therapy. However, differences in the abundance level of mutations among patients resulted in varying survival benefits, even among patients with the same target gene mutations. Currently, there is a lack of rational and interpretable models to assess the risk of treatment failure. In this study, we investigated the underlying coupled factors contributing to variations in medication sensitivity and established a statistically interpretable framework, named SAFE-MIL, for risk estimation. We first constructed an effectiveness label for each patient from the perspective of exploring the optimal grouping of patients' positive judgment values and sampled patients into 600 and 1,000 groups, respectively, based on multi-instance learning (MIL). A novel and interpretable loss function was further designed based on the Hosmer-Lemeshow test for this framework. By integrating multi-instance learning with the Hosmer-Lemeshow test, SAFE-MIL is capable of accurately estimating the risk of drug treatment failure across diverse patient cohorts and providing the optimal threshold for assessing the risk stratification simultaneously. We conducted a comprehensive case study involving 457 non-small cell lung cancer patients with EGFR mutations treated with EGFR tyrosine kinase inhibitors. Results demonstrate that SAFE-MIL outperforms traditional regression methods with higher accuracy and can accurately assess patients' risk stratification. This underscores its ability to accurately capture inter-patient variability in risk while providing statistical interpretability. SAFE-MIL is able to effectively guide clinical decision-making regarding the use of drugs in targeted therapy and provides an interpretable computational framework for other patient stratification problems. The SAFE-MIL framework has proven its effectiveness in capturing inter-patient variability in risk and providing statistical interpretability. It outperforms traditional regression methods and can effectively guide clinical decision-making in the use of drugs for targeted therapy. SAFE-MIL offers a valuable interpretable computational framework that can be applied to other patient stratification problems, enhancing the precision of risk assessment in personalized medicine. The source code for SAFE-MIL is available for further exploration and application at https://github.com/Nevermore233/SAFE-MIL.

Fast Broad Multiview Multi-Instance Multilabel Learning (FBM3L) With Viewwise Intercorrelation.

Multiview multi-instance multilabel learning (M3L) is a popular research topic during the past few years in modeling complex real-world objects such as medical images and subtitled video. However, existing M3L methods suffer from relatively low accuracy and training efficiency for large datasets due to several issues: 1) the viewwise intercorrelation (i.e., the correlations of instances and/or bags between different views) are neglected; 2) the diverse correlations (e.g., viewwise intercorrelation, interinstance correlation, and interlabel correlation) are not jointly considered; and 3) high computation burden for training process over bags, instances, and labels across different views. To resolve these issues, a novel framework called fast broad M3L (FBM3L) is proposed with three innovations: 1) utilization of viewwise intercorrelation for better modeling of M3L tasks while existing M3L methods have not considered; 2) based on graph convolutional network (GCN) and broad learning system (BLS), a viewwise subnetwork is newly designed to achieve joint learning among the diverse correlations; and 3) under BLS platform, FBM3L can learn multiple subnetworks jointly across all views with significantly less training time. Experiments show that FBM3L is highly competitive (or even better than) in all evaluation metrics up to 64% in average precision (AP) and much faster than most M3L (or MIML) methods (up to 1030 times), especially on large multiview datasets ( ≥ 260 K objects).

A Robust Open-Set Multi-Instance Learning for Defending Adversarial Attacks in Digital Image

A Robust Open-Set Multi-Instance Learning for Defending Adversarial Attacks in Digital Image

Explaining Anomalous Events in Flight Data of UAV With Deep Attention-Based Multi-Instance Learning

Explaining Anomalous Events in Flight Data of UAV With Deep Attention-Based Multi-Instance Learning

Automated identification of protein expression intensity and classification of protein cellular locations in mouse brain regions from immunofluorescence images.

Knowledge of protein expression in mammalian brains at regional and cellular levels can facilitate understanding of protein functions and associated diseases. As the mouse brain is a typical mammalian brain considering cell type and structure, several studies have been conducted to analyze protein expression in mouse brains. However, labeling protein expression using biotechnology is costly and time-consuming. Therefore, automated models that can accurately recognize protein expression are needed. Here, we constructed machine learning models to automatically annotate the protein expression intensity and cellular location in different mouse brain regions from immunofluorescence images. The brain regions and sub-regions were segmented through learning image features using an autoencoder and then performing K-means clustering and registration to align with the anatomical references. The protein expression intensities for those segmented structures were computed on the basis of the statistics of the image pixels, and patch-based weakly supervised methods and multi-instance learning were used to classify the cellular locations. Results demonstrated that the models achieved high accuracy in the expression intensity estimation, and the F1 score of the cellular location prediction was 74.5%. This work established an automated pipeline for analyzing mouse brain images and provided a foundation for further study of protein expression and functions.

DCAMIL: Eye-tracking guided dual-cross-attention multi-instance learning for refining fundus disease detection

Deep neural networks (DNNs) have facilitated the development of computer-aided diagnosis (CAD) systems for fundus diseases, helping ophthalmologists to reduce missed diagnoses and misdiagnosis rates. However, the majority of CAD systems are data-driven, but lack the prior medical knowledge that can be performance-friendly. In this regard, we innovatively proposed a human-in-the-loop (HITL) CAD system by leveraging ophthalmologists’ eye-tracking information. Concretely, the HITL CAD system was implemented on the multi-instance learning (MIL), where clinicians’ gaze maps were beneficial to cherry-pick diagnosis-related instances. Furthermore, the dual-cross-attention MIL (DCAMIL) network was utilized to curb the adverse effects of noisy instances. Meanwhile, both the sequence augmentation (SA) module and the domain adversarial network (DAN) were introduced to enrich and standardize the instances in the training bag, respectively, thereby enhancing the robustness of our method. We conduct comparative experiments on our newly-constructed datasets (namely, AMD-Gaze and DR-Gaze) for the AMD and early DR detection, respectively. Rigorous experiments demonstrate the feasibility of our HITL CAD system and the superiority of the proposed DCAMIL, which fully exploits ophthalmologists’ eye-tracking information. These investigations indicate that clinicians’ gaze maps, as prior medical knowledge, is potential to contribute to the CAD systems of clinical diseases.

Identifying Student Profiles Within Online Judge Systems Using Explainable Artificial Intelligence

Online Judge (OJ) systems are typically considered within programming-related courses as they yield fast and objective assessments of the code developed by the students. Such an evaluation generally provides a single decision based on a rubric, most commonly whether the submission successfully accomplished the assignment. Nevertheless, since in an educational context such information may be deemed insufficient, it would be beneficial for both the student and the instructor to receive additional feedback about the overall development of the task. This work aims to tackle this limitation by considering the further exploitation of the information gathered by the OJ and automatically inferring feedback for both the student and the instructor. More precisely, we consider the use of learning-based schemes—particularly, Multi-Instance Learning and classical Machine Learning formulations—to model student behaviour. Besides, Explainable Artificial Intelligence is contemplated to provide human-understandable feedback. The proposal has been evaluated considering a case of study comprising 2,500 submissions from roughly 90 different students from a programming-related course in a Computer Science degree. The results obtained validate the proposal: the model is capable of significantly predicting the user outcome (either passing or failing the assignment) solely based on the behavioural pattern inferred by the submissions provided to the OJ. Moreover, the proposal is able to identify prone-to-fail student groups and profiles as well as other relevant information, which eventually serves as feedback to both the student and the instructor.

A Causality-Driven Graph Convolutional Network for Postural Abnormality Diagnosis in Parkinsonians.

Abnormal posture is a common movement disorder in the progress of Parkinson's disease (PD), and this abnormality can increase the risk of falls or even disabilities. The conventional assessment approach depends on the judgment of well-trained experts via canonical scales. However, this approach requires extensive clinical expertise and is highly subjective. Considering the potential of quantitative susceptibility mapping (QSM) in PD diagnosis, this study explored the QSM-based method for the automated classification between PD patients with and without postural abnormalities. Nevertheless, a major challenge is that unstable non-causal features typically lead to less reliable performance. Therefore, we propose a causality-driven graph-convolutional-network framework based on multi-instance learning, where performance stability is enhanced through the invariant prediction principle and causal interventions. Specifically, we adopt an intervention strategy that combines a non-causal intervenor with causal prediction. A stability constraint is proposed to ensure robust integrated prediction under different interventions. Moreover, an intra-class homogeneity constraint is enforced for each individually-learned causality scoring module to promote the extraction of group-level general features, and hence achieve a balance between subject-specific and group-level features. The proposed method demonstrated promising performance through extensive experiments on a real clinical dataset. Also, the features extracted by our method coincide with those reported in previous medical studies on PD posture abnormalities. In general, our work provides a clinically-valuable approach for automated, objective, and reliable diagnosis of postural abnormalities in Parkinsonians. Our source code is publicly available at https://github.com/SJTUBME-QianLab/CausalGCN-PDPA.

Regularized Optimal Transport Layers for Generalized Global Pooling Operations.

Global pooling is one of the most significant operations in many machine learning models and tasks, which works for information fusion and structured data (like sets and graphs) representation. However, without solid mathematical fundamentals, its practical implementations often depend on empirical mechanisms and thus lead to sub-optimal, even unsatisfactory performance. In this work, we develop a novel and generalized global pooling framework through the lens of optimal transport. The proposed framework is interpretable from the perspective of expectation-maximization. Essentially, it aims at learning an optimal transport across sample indices and feature dimensions, making the corresponding pooling operation maximize the conditional expectation of input data. We demonstrate that most existing pooling methods are equivalent to solving a regularized optimal transport (ROT) problem with different specializations, and more sophisticated pooling operations can be implemented by hierarchically solving multiple ROT problems. Making the parameters of the ROT problem learnable, we develop a family of regularized optimal transport pooling (ROTP) layers. We implement the ROTP layers as a new kind of deep implicit layer. Their model architectures correspond to different optimization algorithms. We test our ROTP layers in several representative set-level machine learning scenarios, including multi-instance learning (MIL), graph classification, graph set representation, and image classification. Experimental results show that applying our ROTP layers can reduce the difficulty of the design and selection of global pooling - our ROTP layers may either imitate some existing global pooling methods or lead to some new pooling layers fitting data better. The code is available at https://github.com/SDS-Lab/ROT-Pooling.

Automated Machine Learning-Based Diagnosis and Molecular Characterization of Acute Leukemias using Flow Cytometry Data

Abstract Current flow cytometric analysis of blood and bone marrow samples for the diagnosis of acute leukemias relies heavily on manual intervention in both the processing and analysis steps, introducing significant subjectivity into the resulting diagnosis and increasing diagnostic turn-around time. Additionally, concurrent molecular characterization of these samples via cytogenetics and targeted sequencing panels can take multiple days, thereby delaying patient diagnosis and treatment. Attention-based multi-instance learning models are machine learning models that can make accurate predictions and generate interpretable insights regarding the classification of a sample from multiple events/cells; while these models have been developed for anatomic pathology applications, they have yet to be applied to flow cytometry data. By utilizing 1,820 flow cytometry samples from 2019-2022 at Brigham and Women’s Hospital, we developed attention-based multi-instance machine learning models for automated diagnosis of acute leukemia, including differentiation of acute myeloid leukemia (AML) from B-lymphoblastic leukemia/lymphoma (B-ALL). Additionally, using concurrent cytogenetic and targeted sequencing data from 674 acute leukemia samples, machine learning models for prediction of molecular aberrancies from flow cytometry data were developed. Machine learning models were created using the TabNet deep learning architecture and VIME self-supervised training algorithm, which are state-of-the-art approaches towards machine learning from tabular data including flow cytometry data. Attention-based multi-instance models demonstrated strong performance for the automated diagnosis of acute leukemia versus non-leukemia samples (AUROC 0.869), as well as the separation of AML from B-ALL samples (AUROC 0.971). These models also accurately predicted cytogenetic aberrancies among AML samples, including t(15;17);PML::RARA (AUROC 1.00), as well as mutations including NPM1 (AUROC 0.725). These models do not require any manual intervention including compensation or gating, and additionally provide quantitative scores for the relative importance of different flow cytometry events and markers for the diagnosis of a particular sample. These importance scores can be integrated into flow cytometry analysis software for visualization and interpretation by hematopathologists. In this study, we have demonstrated the capability of machine learning models to provide automated diagnoses of acute leukemia, as well as accurately predict cytogenetic and molecular aberrancies in blood and bone marrow samples using flow cytometry data. This automated workflow can significantly decrease diagnostic turn-around time and ultimately improve patient outcomes.

Masked autoencoders with handcrafted feature predictions: Transformer for weakly supervised esophageal cancer classification

Background and ObjectiveEsophageal cancer is a serious disease with a high prevalence in Eastern Asia. Histopathology tissue analysis stands as the gold standard in diagnosing esophageal cancer. In recent years, there has been a shift towards digitizing histopathological images into whole slide images (WSIs), progressively integrating them into cancer diagnostics. However, the gigapixel sizes of WSIs present significant storage and processing challenges, and they often lack localized annotations. To address this issue, multi-instance learning (MIL) has been introduced for WSI classification, utilizing weakly supervised learning for diagnosis analysis. By applying the principles of MIL to WSI analysis, it is possible to reduce the workload of pathologists by facilitating the generation of localized annotations. Nevertheless, the approach's effectiveness is hindered by the traditional simple aggregation operation and the domain shift resulting from the prevalent use of convolutional feature extractors pretrained on ImageNet. MethodsWe propose a MIL-based framework for WSI analysis and cancer classification. Concurrently, we introduce employing self-supervised learning, which obviates the need for manual annotation and demonstrates versatility in various tasks, to pretrain feature extractors. This method enhances the extraction of representative features from esophageal WSI for MIL, ensuring more robust and accurate performance. ResultsWe build a comprehensive dataset of whole esophageal slide images and conduct extensive experiments utilizing this dataset. The performance on our dataset demonstrates the efficiency of our proposed MIL framework and the pretraining process, with our framework outperforming existing methods, achieving an accuracy of 93.07% and AUC (area under the curve) of 95.31%. ConclusionThis work proposes an effective MIL method to classify WSI of esophageal cancer. The promising results indicate that our cancer classification framework holds great potential in promoting the automatic whole esophageal slide image analysis.

A multi-instance multi-label learning algorithm based on radial basis functions and multi-objective particle swarm optimization

Radial basis function (RBF) neural networks for Multi-Instance Multi-Label (MIML) directly can exploit the connections between instances and labels so that they can preserve useful prior information, but they only adopt Gaussian radial basis function as their RBF whose parameters are difficult to determine. In this paper, parameters can be obtained by multi-objective optimization methods with multi performance measures treated as objectives, specifically, parameter estimation of different RBFs by an improved multi-objective particle swarm optimization (MOPSO) is proposed where Recall rate and Precision rate are chosen to obtain the most desirable Pareto optimal solution set. Furthermore, share-learning factor is proposed to modify the particle velocity in standard MOPSO to improve the global search ability and group cooperative ability. It is experimentally demonstrated that the proposed method can estimate the reliable parameters of different RBFs, and it is also very competitive with the state of art MIML methods.

Classification of Emotional and Immersive Outcomes in the Context of Virtual Reality Scene Interactions.

The constantly evolving technological landscape of the Metaverse has introduced a significant concern: cybersickness (CS). There is growing academic interest in detecting and mitigating these adverse effects within virtual environments (VEs). However, the development of effective methodologies in this field has been hindered by the lack of sufficient benchmark datasets. In pursuit of this objective, we meticulously compiled a comprehensive dataset by analyzing the impact of virtual reality (VR) environments on CS, immersion levels, and EEG-based emotion estimation. Our dataset encompasses both implicit and explicit measurements. Implicit measurements focus on brain signals, while explicit measurements are based on participant questionnaires. These measurements were used to collect data on the extent of cybersickness experienced by participants in VEs. Using statistical methods, we conducted a comparative analysis of CS levels in VEs tailored for specific tasks and their immersion factors. Our findings revealed statistically significant differences between VEs, highlighting crucial factors influencing participant engagement, engrossment, and immersion. Additionally, our study achieved a remarkable classification performance of 96.25% in distinguishing brain oscillations associated with VR scenes using the multi-instance learning method and 95.63% in predicting emotions within the valence-arousal space with four labels. The dataset presented in this study holds great promise for objectively evaluating CS in VR contexts, differentiating between VEs, and providing valuable insights for future research endeavors.

Automated Deep Learning-Based Diagnosis and Molecular Characterization of Acute Myeloid Leukemia Using Flow Cytometry

The current flow cytometric analysis of blood and bone marrow samples for diagnosis of acute myeloid leukemia (AML) relies heavily on manual intervention in the processing and analysis steps, introducing significant subjectivity into resulting diagnoses and necessitating highly trained personnel. Furthermore, concurrent molecular characterization via cytogenetics and targeted sequencing can take multiple days, delaying patient diagnosis and treatment. Attention-based multi-instance learning models (ABMILMs) are deep learning models that make accurate predictions and generate interpretable insights regarding the classification of a sample from individual events/cells; nonetheless, these models have yet to be applied to flow cytometry data. In this study, we developed a computational pipeline using ABMILMs for the automated diagnosis of AML cases based exclusively on flow cytometric data. Analysis of 1820 flow cytometry samples shows that this pipeline provides accurate diagnoses of acute leukemia (area under the receiver operating characteristic curve [AUROC] 0.961) and accurately differentiates AML vs B- and T-lymphoblastic leukemia (AUROC 0.965). Models for prediction of 9 cytogenetic aberrancies and 32 pathogenic variants in AML provide accurate predictions, particularly for t(15;17)(PML::RARA) [AUROC 0.929], t(8;21)(RUNX1::RUNX1T1) (AUROC 0.814), and NPM1 variants (AUROC 0.807). Finally, we demonstrate how these models generate interpretable insights into which individual flow cytometric events and markers deliver optimal diagnostic utility, providing hematopathologists with a data visualization tool for improved data interpretation, as well as novel biological associations between flow cytometric marker expression and cytogenetic/molecular variants in AML. Our study is the first to illustrate the feasibility of using deep learning-based analysis of flow cytometric data for automated AML diagnosis and molecular characterization.

Machine Learning Models Predict Molecular Genetic Subtypes of Multiple Myeloma from Whole-Slide Bone Marrow Aspirate Smears

Multiple myeloma (MM) is a plasma cell neoplasm and the second most common hematologic malignancy. Current guidelines recommend multiple myeloma cases undergo karyotyping and fluorescence in situ hybridization (FISH) analysis on bone marrow biopsy samples to assess for specific recurrent genetic abnormalities. Results of this testing help to predict differences in patient prognosis over time and to guide appropriate selection of therapy. Refinements in risk stratification incorporated additional complex molecular testing which, along with karyotyping and FISH, are costly and not available to all patients. Myeloma cell cytomorphologies have been associated in a subset of cases with more aggressive disease, when plasmablastic, and with a subset of t(11;14) cases, when lymphoplasmacytoid. Nonetheless, cellular features as assessed by manual microscopy have not been shown to identify biologic subtypes of disease reproducibly and broadly, such that these features are not a component of current prognostic systems. Whether accurate risk-stratification information can be extracted from microscopic images of myeloma cells through machine-based approaches is uncertain. Thus, in this study, we tested the feasibility of using machine learning models to predict multiple myeloma molecular genetic subtypes from analysis of neoplastic plasma cell morphology on Wright-stained aspirate smears. We first improved upon a previously developed computational pipeline that identifies and classifies hematopoietic cells from scanned whole-slide images (400x) of bone marrow aspirate smears, including classification of plasma cells with 94% accuracy. Using this pipeline, we obtained images from all plasma cells (up to 68,569 plasma cells per slide) from aspirate smears of 18 patients without plasma cell neoplasms and 96 patients with plasma cell neoplasms (including 20 with t(11;14) and 23 with gain(1q); 44 with standard risk FISH and 22 with high risk FISH (t(4;14), t(14;16), del(17p), or >4 copies 1q+)). We then developed convolutional neural network (CNN)-based multi-instance machine learning models to perform patient-level classifications of disease status and molecular genetic classification from image-level analysis of plasma cell cytomorphology. These models demonstrated strong performance for classifying patients without versus with plasma cell neoplasms (AUROC 0.80), and are capable of making patient-level predictions of multiple myeloma molecular genetic subtypes (t(11;14), AUROC 0.73; gain(1q), AUROC 0.71) as well as prediction of FISH risk level (AUROC 0.68). Finally, by assessing cell attention scores and CNN model weights, this machine learning pipeline can assess which individual plasma cells and which specific morphologic features provided the most utility in the prediction of molecular genetic subtypes for individual patients, hence providing explainability for model output. Importantly, our findings suggest that cytomorphologic features of plasma cells on routine aspirate smear slides contain information, extractable through machine-based approaches, that correlates with molecular genetic subtypes employed for risk stratification. With further refinement these promising computational digital pathology models can potentially yield tools not only for use in low resource settings, but also provide a potential basis for development of multimodal models that incorporate and improve upon results of currently used risk stratification tools.

Shared-Specific Feature Learning With Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis.

The fusion of multi-modal medical data is essential to assist medical experts to make treatment decisions for precision medicine. For example, combining the whole slide histopathological images (WSIs) and tabular clinical data can more accurately predict the lymph node metastasis (LNM) of papillary thyroid carcinoma before surgery to avoid unnecessary lymph node resection. However, the huge-sized WSI provides much more high-dimensional information than low-dimensional tabular clinical data, making the information alignment challenging in the multi-modal WSI analysis tasks. This paper presents a novel transformer-guided multi-modal multi-instance learning framework to predict lymph node metastasis from both WSIs and tabular clinical data. We first propose an effective multi-instance grouping scheme, named siamese attention-based feature grouping (SAG), to group high-dimensional WSIs into representative low-dimensional feature embeddings for fusion. We then design a novel bottleneck shared-specific feature transfer module (BSFT) to explore the shared and specific features between different modalities, where a few learnable bottleneck tokens are utilized for knowledge transfer between modalities. Moreover, a modal adaptation and orthogonal projection scheme were incorporated to further encourage BSFT to learn shared and specific features from multi-modal data. Finally, the shared and specific features are dynamically aggregated via an attention mechanism for slide-level prediction. Experimental results on our collected lymph node metastasis dataset demonstrate the efficiency of our proposed components and our framework achieves the best performance with AUC (area under the curve) of 97.34%, outperforming the state-of-the-art methods by over 1.27%.

Multi-Instance Learning Approach to the Modeling of Enantioselectivity of Conformationally Flexible Organic Catalysts.

Computational design of chiral organic catalysts for asymmetric synthesis is a promising technology that can significantly reduce the material and human resources required for the preparation of enantiopure compounds. Herein, for the modeling of catalysts' enantioselectivity, we propose to use the multi-instance learning approach accounting for multiple catalyst conformers and requiring neither conformer selection nor their spatial alignment. A catalyst was represented by an ensemble of conformers, each encoded by three-dimesinonal (3D) pmapper descriptors. A catalyzed reactant transformation was converted into a single molecular graph, a condensed graph of reaction, encoded by 2D fragment descriptors. A whole chemical reaction was finally encoded by concatenated 3D catalyst and 2D transformation descriptors. The performance of the proposed method was demonstrated in the modeling of the enantioselectivity of homogeneous and phase-transfer reactions and compared with the state-of-the-art approaches.