Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Multi-modal deep learning for automated assembly of periapical radiographs

This study aimed to develop and validate a multimodal deep learning model that leverages 2D grayscale ultrasound (US) images alongside readily available clinical data to improve diagnostic performance for ovarian cancer (OC). A retrospective analysis was conducted involving 1899 patients who underwent preoperative US examinations and subsequent surgeries for adnexal masses between 2019 and 2024. A multimodal deep learning model was constructed for OC diagnosis and extracting US morphological features from the images. The model's performance was evaluated using metrics such as receiver operating characteristic (ROC) curves, accuracy, and F1 score. The multimodal deep learning model exhibited superior performance compared to the image-only model, achieving areas under the curves (AUCs) of 0.9393 (95% CI 0.9139-0.9648) and 0.9317 (95% CI 0.9062-0.9573) in the internal and external test sets, respectively. The model significantly improved the AUCs for OC diagnosis by radiologists and enhanced inter-reader agreement. Regarding US morphological feature extraction, the model demonstrated robust performance, attaining accuracies of 86.34% and 85.62% in the internal and external test sets, respectively. Multimodal deep learning has the potential to enhance the diagnostic accuracy and consistency of radiologists in identifying OC. The model's effective feature extraction from ultrasound images underscores the capability of multimodal deep learning to automate the generation of structured ultrasound reports.

To develop and validate a multimodal deep learning model integrating clinical data, contrast-enhanced CT, and laryngoscopic images for differentiating early-stage (I-II) from advanced-stage (III-IV) laryngeal squamous cell carcinoma (LSCC). This retrospective multicenter study included 450 patients with pathologically confirmed LSCC from two Chinese medical centers. All patients had contrast-enhanced CT, white-light laryngoscopy, and clinical records. They were divided into training (n = 235), internal validation (n = 101), and external validation (n = 114) cohorts. Three single-modality models (CT-based deep learning [CT-DL], laryngoscopy-based multiple instance learning [L-MIL], and a clinical logistic regression model [CL]) and their combinations were compared. A feature-level fusion strategy was applied, and the final integrated multimodal model (CL + CT + L) was built using a stochastic gradient descent (SGD) classifier. Performance was evaluated by AUC, accuracy, sensitivity, specificity, calibration, and decision curve analysis (DCA), with prognostic value assessed by Kaplan-Meier and concordance index (C-index). A total of 450 patients were included (median age, 62 years [range, 31-88]; 365 men). The integrated multimodal model achieved AUCs of 0.902 (0.833-0.954) in the internal cohort and 0.888 (0.826-0.944) in the external cohort, outperforming all single- and dual-modality models (p < 0.05). Calibration and DCA confirmed strong consistency and clinical utility. The model categorized patients into distinct risk groups, which exhibited notable differences in progression-free survival (C-index = 0.584, p = 0.036). The integrated multimodal model showed high accuracy and generalizability for preoperative LSCC staging and may aid individualized treatment planning. Question Can a multimodal deep learning model combining clinical, CT, and laryngoscopic data improve preoperative staging accuracy of LSCC? Findings The integrated multimodal model achieved higher diagnostic accuracy and provided reliable prognostic stratification compared with conventional approaches. Clinical relevance This multimodal model offers a non-invasive, accurate, and generalizable tool for LSCC staging, supporting individualized treatment planning and enhancing patient management.

In general, water level prediction models using deep learning techniques have been developed using time-series water level observation data from upstream water level stations and target water level stations even though many of physical data are necessary to predict water level. The changes of the water level are greatly affected by rainfall in the basin, therefore rainfall information is needed to more accurately predict the water level. In particular, radar data has the advantage of being able to directly acquire the amount of rainfall occurring within a watershed. This study aims to develop the multimodal deep learning model to predict the water level using 2D grid radar rainfall data and 1D time-series water level observation data. This study proposed two multimodal deep learning models which have different structures. Both multimodal deep learning models predict the water level by simultaneously using the observed water level data up to the present time and the radar rainfall data that affects the water level in the future. The first proposed model consists of a deep learning network that links 2D Average Pooling (AvgPool2D), which compresses 2D radar data to 1D data, and Long Short-Term Memory (LSTM), which predicts 1D time series water level data. The second proposed model consists of a deep learning network that predicts water levels by linking Conv2DLSTM and LSTM, which can reflect the characteristics of 2D radar data without deformation.&amp;#160; The two proposed multimodal deep learning models were learned and evaluated in the upper basin of Hantan River. In addition, it was compared with the results of single-modal LSTM using only water level data. There are three water level stations in the study area, and the objective was to predict the water level of the downstream station up to 180 minutes in advance. For learning and verification of the deep learning model, 10-minute water level and radar rainfall data were collected from May 2019 to October 2021. For the radar data used as input, the grid data included in the target watershed were extracted and used among composite radar data with a resolution of 1 km operating by Ministry of Environment. As a result of evaluating each learned deep learning model, two multimodal models had higher prediction accuracy than the single-modal using only water level data. In particular, second proposed model (Conv2dLSTM+LSTM) had better predictive performance than first proposed model (AvgPool2D+LSTM) at the time of the sudden rise in water level due to rainfall.AcknowledgmentsResearch for this paper was carried out under the KICT Research Program (project no. 202200175-001, Development of future-leading technologies solving water crisis against to water disasters affected by climate change) funded by the Ministry of Science and ICT.

To investigate the value of a multimodal deep learning (MDL) model based on computed tomography (CT) and magnetic resonance imaging (MRI) for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC). A total of 287 patients with HCC from our institution and 58 patients from another individual institution were included. Among these, 119 patients with only CT data and 116 patients with only MRI data were selected for single-modality deep learning model development, after which select parameters were migrated for MDL model development with transfer learning (TL). In addition, 110 patients with simultaneous CT and MRI data were divided into a training cohort (n = 66) and a validation cohort (n = 44). We input the features extracted from DenseNet121 into an extreme learning machine (ELM) classifier to construct a classification model. The area under the curve (AUC) of the MDL model was 0.844, which was superior to that of the single-phase CT (AUC = 0.706-0.776, P < 0.05), single-sequence MRI (AUC = 0.706-0.717, P < 0.05), single-modality DL model (AUCall-phase CT = 0.722, AUCall-sequence MRI = 0.731; P < 0.05), clinical (AUC = 0.648, P < 0.05), but not to that of the delay phase (DP) and in-phase (IP) MRI and portal venous phase (PVP) CT models. The MDL model achieved better performance than models described above (P < 0.05). When combined with clinical features, the AUC of the MDL model increased from 0.844 to 0.871. A nomogram, combining deep learning signatures (DLS) and clinical indicators for MDL models, demonstrated a greater overall net gain than the MDL models (P < 0.05). The MDL model is a valuable noninvasive technique for preoperatively predicting MVI in HCC.

BackgroundThere is a critical unmet need for predictive biomarkers of cancer immunotherapy. The tumor microenvironment (TME) plays an important role in determining immunotherapy response and outcomes. Here, we aimed to...

Preoperative Prediction of Axillary Lymph Node Metastasis in Patients With Breast Cancer Through Multimodal Deep Learning Based on Ultrasound and Magnetic Resonance Imaging Images

ObjectiveThe objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data.Materials and methodsThe research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients’ pathological images and clinical data, respectively.ResultsA total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS).ConclusionsOur study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.

BackgroundAn increase in the incidence of central venous catheter (CVC)-associated deep venous thrombosis (CADVT) has been reported in pediatric patients over the past decade. At the same time, current screening guidelines for venous thromboembolism risk have low sensitivity for CADVT in hospitalized children. This study utilized a multimodal deep learning model to predict CADVT before it occurs.MethodsChildren who were admitted to intensive care units (ICUs) between December 2015 and December 2018 and with CVC placement at least 3 days were included. The variables analyzed included demographic characteristics, clinical conditions, laboratory test results, vital signs and medications. A multimodal deep learning (MMDL) model that can handle temporal data using long short-term memory (LSTM) and gated recurrent units (GRUs) was proposed for this prediction task. Four benchmark machine learning models, logistic regression (LR), random forest (RF), gradient boosting decision tree (GBDT) and a published cutting edge MMDL, were used to compare and evaluate the models with a fivefold cross-validation approach. Accuracy, recall, area under the ROC curve (AUC), and average precision (AP) were used to evaluate the discrimination of each model at three time points (24 h, 48 h and 72 h) before CADVT occurred. Brier score and Spiegelhalter’s z test were used measure the calibration of these prediction models.ResultsA total of 1830 patients were included in this study, and approximately 15% developed CADVT. In the CADVT prediction task, the model proposed in this paper significantly outperforms both traditional machine learning models and existing multimodal deep learning models at all 3 time points. It achieved 77% accuracy and 90% recall at 24 h before CADVT was discovered. It can be used to accurately predict the occurrence of CADVT 72 h in advance with an accuracy of greater than 75%, a recall of more than 87%, and an AUC value of 0.82.ConclusionIn this study, a machine learning method was successfully established to predict CADVT in advance. These findings demonstrate that artificial intelligence (AI) could provide measures for thromboprophylaxis in a pediatric intensive care setting.

BackgroundProper stratification of recurrence risk in breast cancer is crucial for guiding treatment decisions. This study aims to predict the recurrence risk of breast cancer patients using a multimodal deep learning model that integrates multiple sequence MRI imaging features with clinicopathologic characteristics.MethodsIn this retrospective study, we enrolled 574 patients with non-metastatic invasive breast cancer from two Chinese institutions between September 2012 and July 2019. We developed a multimodal deep learning (MDL) model by constructing a multi-instance learning framework based on convolutional neural networks. We integrated imaging features from T2WI, DWI, and DCE-MRI sequences with clinicopathologic features for breast cancer recurrence risk stratification. Subsequently, the performance of the MDL model was evaluated using receiver operating characteristic (ROC) curves, the Hosmer–Lemeshow test, calibration curves, and decision curve analysis (DCA). Survival analysis was conducted with Kaplan–Meier survival curves to stratify breast cancer patients into high and low-recurrence risk groups. Time-dependent ROC curves were used to assess 3-year, 5-year, and 7-year recurrence-free survival (RFS) for breast cancer patients. Additionally, we performed differential and enrichment analyses on Oncotype DX genes. We correlated these genes with clinicopathologic features and deep-learning radiographic features using univariate Cox regression and Pearson correlation analysis.ResultsThe MDL model demonstrated good performance in predicting breast cancer recurrence risk and accurately differentiated between high- and low-recurrence risk groups, with an AUC as high as 0.915 (95% CI 0.8448–0.9856). The C-index of prediction models was 0.803 in the testing cohort. The AUCs for 5-year and 7-year RFS were 0.936 (95% CI 0.876–0.997) and 0.956 (95% CI 0.902–1.000) in the validation cohort. In the testing cohort, these AUCs were 0.836 (95% CI 0.763–0.909) and 0.783 (95% CI 0.676–0.891). This study found a significant correlation between Oncotype DX gene expression, clinicopathologic features, and deep-learning radiographic features (p < 0.05).ConclusionsThis study validated the robust predictive accuracy of the MDL model in identifying high- and low-risk groups for recurrence. The correlations identified between Oncotype DX genes, clinicopathologic features, and deep-learning radiographic features offer novel insights for future biomarker research in breast cancer.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13058-025-02129-z.

Tuberculosis (TB) continues to be a major global health issue, especially in areas with limited resources where diagnostic tools are often insufficient. Traditional TB detection methods are slow and lack sensitivity, particularly for early-stage or low bacterial load cases. This study introduces a new multimodal deep learning model that integrates sputum image segmentation across RGB, hue, saturation, and value (HSV), and CIELAB color channels, using the YOLOv8 model for real-time detection and segmentation. The model uses the International Union Against Tuberculosis and Lung Disease (IUATLD) grading scale for accurate Mycobacterium tuberculosis (MTB) classification. Our approach shows high accuracy (92.24%) and precise forecasting (mean absolute percent error (MAPE) of 0.23%), greatly enhancing diagnostic speed and reliability. This research offers a novel method for classifying MTB using a multimodal deep learning model that integrates sputum image segmentation across RGB, HSV, and CIELAB color channels. By using the YOLOv8 model for real-time bounding box detection and segmentation, and the IUATLD grading scale for classification, our method achieves high accuracy and precision in identifying TB bacteria. Our findings indicate that this multimodal deep learning approach significantly improves diagnostic accuracy and speed, providing a reliable tool for early TB detection.

The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim 18F-fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL. Initially, 205 DLBCL patients undergoing interim [18F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance. The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability. The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment. • The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients. • Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance. • Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.

Performance of deep learning in mapping water quality of Lake Simcoe with long-term Landsat archive

Abdominal aortic aneurysm (AAA) progression carries a significant rupture risk, demanding accurate prediction models beyond traditional methods that rely on limited clinical parameters and often overlook complex factor interplay. We aimed to enhance prediction by developing and validating an end-to-end multi-modal deep learning (DL) model that integrates features extracted using ResNet from computed tomography (CT) images, geometric features derived from radiomics based on CT annotations, and clinical features obtained from clinical records. This retrospective study utilized data from 561 AAA patients sourced from Boramae Medical Center and Seoul National University Hospital, including 14,252 annotated CT axial images alongside detailed clinical information. Patients were categorized into rapid or slow progression groups based on an annual growth rate threshold of 2.5 mm/year. The multi-modal DL model that incorporated CT images, clinical features, and geometric features demonstrated superior predictive performance for rapid progression, achieving an area under the receiver operating characteristic curve (AUC) of 0.807 and an accuracy of 0.758. This significantly outperformed traditional machine learning models utilizing only clinical data (AUC: 0.716) or only geometric features (AUC: 0.715). The improvement in AUC was statistically significant according to DeLong’s test. This study underscores the value of AI-driven, multi-modal approaches for enhancing patient-specific AAA risk stratification, potentially enabling more precise monitoring and optimized timing for clinical interventions.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-22167-z.

The aging process of microplastics (MPs) affects their surface physicochemical properties, thereby influencing their behaviors in releasing harmful chemicals, adsorption of organic contaminants, sinking, and more. Understanding the aging process is crucial for evaluating MPs' environmental behaviors and risks, but tracing the aging process remains challenging. Here, we propose a multimodal deep learning model to trace typical aging factors of aged MPs based on MPs' physicochemical characteristics. A total of 1353 surface morphology images and 1353 Fourier transform infrared spectroscopy spectra were achieved from 130 aged MPs undergoing different aging processes, demonstrating that physicochemical properties of aged MPs vary from aging processes. The multimodal deep learning model achieved an accuracy of 93% in predicting the major aging factors of aged MPs. The multimodal deep learning model improves the model's accuracy by approximately 5-20% and reduces prediction bias compared to the single-modal model. In practice, the established model was performed to predict the major aging factors of naturally aged MPs collected from typical environment matrices. The prediction results aligned with the aging conditions of specific environments, as reported in previous studies. Our findings provide new insights into tracing and understanding the plastic aging process, contributing more accurately to the environmental risk assessment of aged MPs.