Multimodal Deep Learning Research Articles

Open-world disaster information identification from multimodal social media

The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification.

Damage‐level classification considering both correlation between image and text data and confidence of attention map

AbstractIn damage‐level classification, deep learning. models are more likely to focus on regions unrelated to classification targets because of the complexities inherent in real data, such as the diversity of damages (e.g., crack, efflorescence, and corrosion). This causes performance degradation. To solve this problem, it is necessary to handle data complexity and uncertainty. This study proposes a multimodal deep learning model that can focus on damaged regions using text data related to damage in images, such as materials and components. Furthermore, by adjusting the effect of attention maps on damage‐level classification performance based on the confidence calculated when estimating these maps, the proposed method realizes an accurate damage‐level classification. Our contribution is the development of a model with an end‐to‐end multimodal attention mechanism that can simultaneously consider both text and image data and the confidence of the attention map. Finally, experiments using real images validate the effectiveness of the proposed method.

Unveiling the distinctive variations in multi-omics triggered by TP53 mutation in lung cancer subtypes: An insight from interaction among intratumoral microbiota, tumor microenvironment, and pathology

The TP53 mutation is one of the most common gene mutations in non-small cell lung cancer (NSCLC) and plays a significant role in the occurrence, development, and prognosis of both lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). Recent studies have also suggested the predictive value of TP53 mutations in the response to immunotherapy for NSCLC. It is known that intratumoral microbiota, tumor immune microenvironment (TIME) and histology are associated with the roles of TP53 mutation in NSCLC. However, the intrinsic associations among these three factors and their underlying interaction with TP53 mutation are not well understood. Additionally, the potential of predicting TP53 mutations using deep learning methods has not yet been fully evaluated. In this paper, we comprehensively evaluated the tissue microbiome, host gene expression characteristics, and histopathological slides of 992 NSCLC patients obtained from the cancer genome atlas (TCGA) and validated the findings using multi-omics data of 332 NSCLC patients from the Clinical Proteomic Tumor Analysis Consortium (CPTAC). Compared to LUSC, LUAD exhibited more substantial differences between patients with and without TP53 mutation in all three aspects. In LUAD, our results imply underlying links between the tissue microbiome and immune cell components in the TIME, and show that the abundance of immune cells is reflected in histology slides. Furthermore, we propose a novel multimodal deep learning model that focuses on histopathology images, which achieves an area under the curve (AUC) of 0.84 in LUAD. In summary, TP53 mutation of LUAD resulted more significant changes in intratumoral microbiota, TIME and histology than that of LUSC. And histopathology images can be used to predict TP53 mutation in LUAD with reasonable accuracy.

A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding.

Deep learning methods have been applied when working to enhance the prediction accuracy of traditional statistical methods in the field of plant breeding. Although deep learning seems to be a promising approach for genomic prediction, it has proven to have some limitations, since its conventional methods fail to leverage all available information. Multimodal deep learning methods aim to improve the predictive power of their unimodal counterparts by introducing several modalities (sources) of input information. In this review, we introduce some theoretical basic concepts of multimodal deep learning and provide a list of the most widely used neural network architectures in deep learning, as well as the available strategies to fuse data from different modalities. We mention some of the available computational resources for the practical implementation of multimodal deep learning problems. We finally performed a review of applications of multimodal deep learning to genomic selection in plant breeding and other related fields. We present a meta-picture of the practical performance of multimodal deep learning methods to highlight how these tools can help address complex problems in the field of plant breeding. We discussed some relevant considerations that researchers should keep in mind when applying multimodal deep learning methods. Multimodal deep learning holds significant potential for various fields, including genomic selection. While multimodal deep learning displays enhanced prediction capabilities over unimodal deep learning and other machine learning methods, it demands more computational resources. Multimodal deep learning effectively captures intermodal interactions, especially when integrating data from different sources. To apply multimodal deep learning in genomic selection, suitable architectures and fusion strategies must be chosen. It is relevant to keep in mind that multimodal deep learning, like unimodal deep learning, is a powerful tool but should be carefully applied. Given its predictive edge over traditional methods, multimodal deep learning is valuable in addressing challenges in plant breeding and food security amid a growing global population.

A multimodal deep learning-based algorithm for specific fetal heart rate events detection.

This study aims to develop a multimodal deep learning-based algorithm for detecting specific fetal heart rate (FHR) events, to enhance automatic monitoring and intelligent assessment of fetal well-being. We analyzed FHR and uterine contraction signals by combining various feature extraction techniques, including morphological features, heart rate variability features, and nonlinear domain features, with deep learning algorithms. This approach enabled us to classify four specific FHR events (bradycardia, tachycardia, acceleration, and deceleration) as well as four distinct deceleration patterns (early, late, variable, and prolonged deceleration). We proposed a multi-model deep neural network and a pre-fusion deep learning model to accurately classify the multimodal parameters derived from Cardiotocography signals. These accuracy metrics were calculated based on expert-labeled data. The algorithm achieved a classification accuracy of 96.2 % for acceleration, 94.4 % for deceleration, 90.9 % for tachycardia, and 85.8 % for bradycardia. Additionally, it achieved 67.0 % accuracy in classifying the four distinct deceleration patterns, with 80.9 % accuracy for late deceleration and 98.9 % for prolonged deceleration. The proposed multimodal deep learning algorithm serves as a reliable decision support tool for clinicians, significantly improving the detection and assessment of specific FHR events, which are crucial for fetal health monitoring.

Reconstructing urban vegetation evolution in China using multimodal deep learning and 30-years Landsat archive

To retrospect the impact of urbanization on vegetation, consistent and reliable maps that provide urban vegetation estimates are required. In this study, we make the first attempt to employ deep learning for urban vegetation mapping of 2120 cities in China using a 30-years Landsat archive (UV-30). We present a multimodal deep learning (MDL) model to combine features from 5 categories, namely reflective bands, remote sensing indices, texture variables, topographical variables, and time labels. We normalized elevation data locally to remove the inter-city differences, while retaining topographic details. We used time labels to account for spectral variability over time and Landsat sensors. The performance of the MDL models was compared with a vanilla deep neural network, random forest, and support vector regression. Results indicated that the fusion of different categories of features improved the prediction accuracy. The MDL performed best in most validation cities, and our results were superior to the fractional maps aggregated from high-resolution datasets in homogeneous urban landscapes. We further analyzed urban vegetation dynamics across the urban core-new town-outskirt gradients of different geographical regions and city sizes. We found a gradually diminishing de-greening trend in urban China, especially in urban cores and new towns. Nevertheless, we observed a more pronounced degradation of urban vegetation in the south regions of China compared to the north, and smaller cities show no evident urban vegetation increase. The suggested framework is valuable for urban vegetation assessments and facilitates a thorough understanding and management of urban ecosystems.

A Multimodal Deep Learning Nomogram for the Identification of Clinically Significant Prostate Cancer in Patients with Gray-Zone PSA Levels: Comparison with Clinical and Radiomics Models

ObjectiveTo establish a multimodal deep learning nomogram for predicting clinically significant prostate cancer in patients with grey-zone PSA levels. MethodsThis retrospective study enrolled 303 patients with pathological results between January 2018 and December 2022. Clinical variables and the PI-RADS v2.1 score were used to construct a clinical model. Radiomics and deep learning features from bp-MRI were used to develop a radiomics model with SVM and a deep learning model, respectively. A hybrid fusion approach was used to integrate the multimodal data and construct combined models (Comb.Rad.model and Comb.DL.model). The robustness of the radiomics model with XGBoost was validated and compared. Model efficacy was assessed through ROC curve and decision curve analysis. A nomogram was developed based on the best-performing model. ResultsThe clinical model had AUCs of 0.845 and 0.779 in the training and testing set. The radiomics model with SVM and the deep learning model achieved AUCs of 0.825 and 0.933 in the training set and 0.811 and 0.907 in the testing set, respectively. The diagnostic performance of the combined models was significantly improved, with Comb.DL.model having a higher AUC than Comb.Rad.model in both the training (0.986 vs. 0.924, P = 0.008) and testing (0.965 vs. 0.859, P = 0.005) set. The diagnostic efficiency of both the radiomics model and Comb.Rad.model with XGBoost were comparable to that of SVM, confirming the robustness of the established model. ConclusionThe integrated nomogram combining deep learning features, PI-RADS score, and clinical variables significantly outperformed the traditional radiomics and clinical models.

Multimodal Deep Learning Integration of Image, Weather, and Phenotypic Data Under Temporal Effects for Early Prediction of Maize Yield

Maize (Zea mays L.) has been shown to be sensitive to temperature deviations, influencing its yield potential. The development of new maize hybrids resilient to unfavourable weather is a desirable aim for crop breeders. In this paper, we showcase the development of a multimodal deep learning model using RGB images, phenotypic, and weather data under temporal effects to predict the yield potential of maize before or during anthesis and silking stages. The main objective of this study was to assess if the inclusion of historical weather data, maize growth captured through imagery, and important phenotypic traits would improve the predictive power of an established multimodal deep learning model. Evaluation of the model performance when training from scratch showed its ability to accurately predict ~89% of hybrids with high-yield potential and demonstrated enhanced explanatory power compared with previously published models. Shapley Additive explanations (SHAP) analysis indicated the top influential features include plant density, hybrid placement in the field, date to anthesis, parental line, temperature, humidity, and solar radiation. Including weather historical data was important for model performance, significantly enhancing the predictive and explanatory power of the model. For future research, the use of the model can move beyond maize yield prediction by fine-tuning the model on other crop data, serving as a potential decision-making tool for crop breeders to determine high-performing individuals from diverse crop types.

A cross-attention-based deep learning approach for predicting functional stroke outcomes using 4D CTP imaging and clinical metadata

Acute ischemic stroke (AIS) remains a global health challenge, leading to long-term functional disabilities without timely intervention. Spatio-temporal (4D) Computed Tomography Perfusion (CTP) imaging is crucial for diagnosing and treating AIS due to its ability to rapidly assess the ischemic core and penumbra. Although traditionally used to assess acute tissue status in clinical settings, 4D CTP has also been explored in research for predicting stroke tissue outcomes. However, its potential for predicting functional outcomes, especially in combination with clinical metadata, remains unexplored. Thus, this work aims to develop and evaluate a novel multimodal deep learning model for predicting functional outcomes (specifically, 90-day modified Rankin Scale) in AIS patients by combining 4D CTP and clinical metadata. To achieve this, an intermediate fusion strategy with a cross-attention mechanism is introduced to enable a selective focus on the most relevant features and patterns from both modalities. Evaluated on a dataset comprising 70 AIS patients who underwent endovascular mechanical thrombectomy, the proposed model achieves an accuracy (ACC) of 0.77, outperforming conventional late fusion strategies (ACC = 0.73) and unimodal models based on either 4D CTP (ACC = 0.61) or clinical metadata (ACC = 0.71). The results demonstrate the superior capability of the proposed model to leverage complex inter-modal relationships, emphasizing the value of advanced multimodal fusion techniques for predicting functional stroke outcomes.

DAB-LSTMNN: A novel respiratory disease diagnosis deep CNN framework based on attention scheme and BLSTM neural networks

Respiratory diseases have a significant impact on modern society, posing considerable public health risks. With the emergence of the COVID-19 pandemic, addressing respiratory issues has become increasingly urgent and important. Recently, artificial intelligence-based methods utilizing the acoustic recordings of suspected patients have shown promise in the diagnosis of various respiratory diseases, enabling localized treatment and containment of their spread. As the existing methods cannot efficiently extract subtle differences between the sound samples, thereby limiting their generalization ability. To improve the diagnosis accuracy, this paper proposes a novel multi-channel, multi-modal deep learning architecture based on the attention mechanism. The proposed framework combines a deep convolutional neural network (DCNN) with a bidirectional long short-term memory (BLSTM) network, and also utilizes the attention scheme to extract temporal and spectral features from different modalities of speech data (e.g. coughs, counting sounds and sustained vowel articulation). The proposed method effectively classifies COVID-19 patients, asthma patients and healthy individuals, with a test accuracy of 89.27 ± 0.1%, and an F1 score of 85.42 ± 0.2%. The experimental results validate the feasibility of our method, and also indicate that it is competitive with the existing deep networks.

HyMNet: A Multimodal Deep Learning System for Hypertension Prediction Using Fundus Images and Cardiometabolic Risk Factors

Study Objectives: This study aimed to develop a multimodal deep learning (MMDL) system called HyMNet, integrating fundus images and cardiometabolic factors (age and sex) to enhance hypertension (HTN) detection. Methods: HyMNet employed RETFound, a model pretrained on 1.6 million retinal images, for the fundus data, in conjunction with a fully connected neural network for age and sex. The two pathways were jointly trained by joining their feature vectors into a fusion network. The system was trained on 5016 retinal images from 1243 individuals provided by the Saudi Ministry of National Guard Health Affairs. The influence of diabetes on HTN detection was also assessed. Results: HyMNet surpassed the unimodal system, achieving an F1 score of 0.771 compared to 0.745 for the unimodal model. For diabetic patients, the F1 score was 0.796, while it was 0.466 for non-diabetic patients. Conclusions: HyMNet exhibited superior performance relative to unimodal approaches, with an F1 score of 0.771 for HyMNet compared to 0.752 for models trained on demographic data alone, underscoring the advantages of MMDL systems in HTN detection. The findings indicate that diabetes significantly impacts HTN prediction, enhancing detection accuracy among diabetic patients. Utilizing MMDL with diverse data sources could improve clinical applicability and generalization.

Drug–drug interaction extraction based on multimodal feature fusion by Transformer and BiGRU

Understanding drug–drug interactions (DDIs) plays a vital role in the fields of drug disease treatment, drug development, preventing medical error, and controlling health care-costs. Extracting potential from biomedical corpora is a major complement of existing DDIs. Most existing DDI extraction (DDIE) methods do not consider the graph and structure of drug molecules, which can improve the performance of DDIE. Considering the different advantages of bi-directional gated recurrent units (BiGRU), Transformer, and attention mechanisms in DDIE tasks, a multimodal feature fusion model combining BiGRU and Transformer (BiGGT) is here constructed for DDIE. In BiGGT, the vector embeddings of medical corpora, drug molecule topology graphs, and structure are conducted by Word2vec, Mol2vec, and GCN, respectively. BiGRU and multi-head self-attention (MHSA) are integrated into Transformer to extract the local–global contextual DDIE features, which is important for DDIE. The extensive experiment results on the DDIExtraction 2013 shared task dataset show that the BiGGT-based DDIE method outperforms state-of-the-art DDIE approaches with a precision of 78.22%. BiGGT expands the application of multimodal deep learning in the field of multimodal DDIE.

Integrating Multimodal Deep Learning with Multipoint Statistics for 3D Crustal Modeling: A Case Study of the South China Sea

Constructing an accurate three-dimensional (3D) geological model is crucial for advancing our understanding of subsurface structures and their evolution, particularly in complex regions such as the South China Sea (SCS). This study introduces a novel approach that integrates multimodal deep learning with multipoint statistics (MPS) to develop a high-resolution 3D crustal P-wave velocity structure model of the SCS. Our method addresses the limitations of traditional algorithms in capturing non-stationary geological features and effectively incorporates heterogeneous data from multiple geophysical sources, including 44 wide-angle seismic crustal structure profiles obtained by ocean bottom seismometers (OBSs), gravity anomalies, magnetic anomalies, and topographic data. The proposed model is rigorously validated against existing methods such as Kriging interpolation and MPS alone, demonstrating superior performance in reconstructing both global and local spatial features of the crustal structure. The integration of diverse datasets significantly enhances the model’s accuracy, reducing errors and improving the alignment with known geological information. The resulting 3D model provides a detailed and reliable representation of the SCS crust, offering critical insights for studies on tectonic evolution, resource exploration, and geodynamic processes. This work highlights the potential of combining deep learning with geostatistical methods for geological modeling, providing a robust framework for future applications in geosciences. The flexibility of our approach also suggests its applicability to other regions and geological attributes, paving the way for more comprehensive and data-driven investigations of Earth’s subsurface.

Real-time deep learning-assisted mechano-acoustic system for respiratory diagnosis and multifunctional classification

Epidermally mounted sensors using triaxial accelerometers have been previously used to monitor physiological processes with the implementation of machine learning (ML) algorithm interfaces. The findings from these previous studies have established a strong foundation for the analysis of high-resolution, intricate signals, typically through frequency domain conversion. In this study we integrate a wireless mechano-acoustic sensor with a multi-modal deep learning system for the real-time analysis of signals emitted by the laryngeal prominence area of the thyroid cartilage at frequency ranges up to 1 kHz. This interface provides real-time data visualization and communication with the ML server, creating a system that assesses severity of chronic obstructive pulmonary disease and analyzes the user’s speech patterns.

Impact of metadata in multimodal classification of bone tumours

The accurate classification of bone tumours is crucial for guiding clinical decisions regarding treatment and follow-up. However, differentiating between various tumour types is challenging due to the rarity of certain entities, high intra-class variability, and limited training data in clinical practice. This study proposes a multimodal deep learning model that integrates clinical metadata and X-ray imaging to improve the classification of primary bone tumours. The dataset comprises 1,785 radiographs from 804 patients collected between 2000 and 2020, including metadata such as age, affected bone site, tumour position, and gender. Ten tumour types were selected, with histopathology or tumour board decisions serving as the reference standard.MethodsOur model is based on the NesT image classification model and a multilayer perceptron with a joint fusion architecture. Descriptive statistics included incidence and percentage ratios for discrete parameters, and mean, standard deviation, median, and interquartile range for continuous parameters.ResultsThe mean age of the patients was 33.62 ± 18.60 years, with 54.73% being male. Our multimodal deep learning model achieved 69.7% accuracy in classifying primary bone tumours, outperforming the Vision Transformer model by five percentage points. SHAP values indicated that age had the most substantial influence among the considered metadata.ConclusionThe joint fusion approach developed in this study, integrating clinical metadata and imaging data, outperformed state-of-the-art models in classifying primary bone tumours. The use of SHAP values provided insights into the impact of different metadata on the model’s performance, highlighting the significant role of age. This approach has potential implications for improving diagnostic accuracy and understanding the influence of clinical factors in tumour classification.

Multimodal deep learning radiomics model for predicting postoperative progression in solid stage I non-small cell lung cancer

PurposeTo explore the application value of a multimodal deep learning radiomics (MDLR) model in predicting the risk status of postoperative progression in solid stage I non-small cell lung cancer (NSCLC).Materials and MethodsA total of 459 patients with histologically confirmed solid stage I NSCLC who underwent surgical resection in our institution from January 2014 to September 2019 were reviewed retrospectively. At another medical center, 104 patients were reviewed as an external validation cohort according to the same criteria. A univariate analysis was conducted on the clinicopathological characteristics and subjective CT findings of the progression and non-progression groups. The clinicopathological characteristics and subjective CT findings that exhibited significant differences were used as input variables for the extreme learning machine (ELM) classifier to construct the clinical model. We used the transfer learning strategy to train the ResNet18 model, used the model to extract deep learning features from all CT images, and then used the ELM classifier to classify the deep learning features to obtain the deep learning signature (DLS). A MDLR model incorporating clinicopathological characteristics, subjective CT findings and DLS was constructed. The diagnostic efficiencies of the clinical model, DLS model and MDLR model were evaluated by the area under the curve (AUC).ResultsUnivariate analysis indicated that size (p = 0.004), neuron-specific enolase (NSE) (p = 0.03), carbohydrate antigen 19 − 9 (CA199) (p = 0.003), and pathological stage (p = 0.027) were significantly associated with the progression of solid stage I NSCLC after surgery. Therefore, these clinical characteristics were incorporated into the clinical model to predict the risk of progression in postoperative solid-stage NSCLC patients. A total of 294 deep learning features with nonzero coefficients were selected. The DLS in the progressive group was (0.721 ± 0.371), which was higher than that in the nonprogressive group (0.113 ± 0.350) (p < 0.001). The combination of size、NSE、CA199、pathological stage and DLS demonstrated the superior performance in differentiating postoperative progression status. The AUC of the MDLR model was 0.885 (95% confidence interval [CI]: 0.842–0.927), higher than that of the clinical model (0.675 (95% CI: 0.599–0.752)) and DLS model (0.882 (95% CI: 0.835–0.929)). The DeLong test and decision in curve analysis revealed that the MDLR model was the most predictive and clinically useful model.ConclusionMDLR model is effective in predicting the risk of postoperative progression of solid stage I NSCLC, and it is helpful for the treatment and follow-up of solid stage I NSCLC patients.

Enhancing societal security: a multimodal deep learning approach for a public person identification and tracking system

In public spaces, threats to societal security are a major concern, and emerging technologies offer potential countermeasures. The proposed intelligent person identification system monitors and identifies individuals in public spaces using gait, face, and iris recognition. The system employs a multimodal approach for secure identification and utilises deep convolutional neural networks (DCNNs) that have been pretrained to predict individuals. For increased accuracy, the proposed system is implemented on a cloud server and integrated with citizen identification systems such as Aadhar/SSN. The performance of the system is determined by the rate of accuracy achieved when identifying individuals in a public space. The proposed multimodal secure identification system achieves a 94% accuracy rate, which is higher than that of existing public space person identification systems. Integration with citizen identification systems improves precision and provides immediate life-saving assistance to those in need. Utilising secure deep learning techniques for precise person identification, the proposed system offers a promising solution to security threats in public spaces. This research is necessary to investigate the efficacy and potential applications of the proposed system, including accident identification, theft identification, and intruder identification in public spaces.

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

In breast cancer research, diverse data types and formats, such as radiological images, clinical records, histological data, and expression analysis, are employed. Given the intricate nature of natural phenomena, relying on the features of a single modality is seldom sufficient for comprehensive analysis. Therefore, it is possible to guarantee medical relevance and achieve improved clinical outcomes by combining several modalities. The presen study carefully maps and reviews 47 primary articles from six well-known digital libraries that were published between 2018 and 2023 for breast cancer classification based on multimodal deep learning fusion (MDLF) techniques. This systematic literature review encompasses various aspects, including the medical modalities combined, the datasets utilized in these studies, the techniques, models, and architectures used in MDLF and it also discusses the advantages and limitations of each approach. The analysis of selected papers has revealed a compelling trend: the emergence of new modalities and combinations that were previously unexplored in the context of breast cancer classification. This exploration has not only expanded the scope of predictive models but also introduced fresh perspectives for addressing diverse targets, ranging from screening to diagnosis and prognosis. The practical advantages of MDLF are evident in its ability to enhance the predictive capabilities of machine learning models, resulting in improved accuracy across diverse applications. The prevalence of deep learning models underscores their success in autonomously discerning complex patterns, offering a substantial departure from traditional machine learning approaches. Furthermore, the paper explores the challenges and future directions in this field, including the need for larger datasets, the use of ensemble learning methods, and the interpretation of multimodal models.

Deep learning-based CT image for pulmonary nodule classification with intrathoracic fat: A multicenter study

Correctly distinguishing between benign and malignant pulmonary nodules can avoid unnecessary invasive procedures. However, existing models exhibit a deficiency in the study of fat regions. We aim to create a multimodal deep learning nomogram (DLN) using pulmonary nodule and intrathoracic fat (ITF) features to distinguish between benign and malignant nodules and further demonstrate that intrathoracic fat can enhance the performance of lung cancer prediction models. A segmentation methodology was devised for precisely delineating intranodular and perinodular region (IPN). The Swin Transformer was used for feature extraction. The least absolute shrinkage and selection operator (LASSO) regression was employed for feature selection. The nomogram integrating IPN and ITF signatures revealed better diagnostic accuracy than IPN signature alone (net reclassification improvement = 0.134, P<0.05, and integrated discrimination improvement = 0.078, P<0.05). In the internal test cohort, DLN obtained an area under the curve (AUC) of 0.913 (95 % CI: 0.870, 0.952). In the external test cohort, the AUC was 0.906 (95 % CI: 0.867, 0.938). There was no significant difference between the calibrated curve and perfectly calibrated curve in all cohorts (P>0.05). Decision curve analysis showed that the DLN was clinically useful. The DLN demonstrated substantial diagnostic value for distinguishing between benign and malignant pulmonary nodules. The incorporation of the ITF signature has enhanced the performance of DLN.

Prostate Cancer Risk Stratification in NRG Oncology Phase III Randomized Trials Using Multimodal Deep Learning With Digital Histopathology.

Current clinical risk stratification methods for localized prostate cancer are suboptimal, leading to over- and undertreatment. Recently, machine learning approaches using digital histopathology have shown superior prognostic ability in phase III trials. This study aims to develop a clinically usable risk grouping system using multimodal artificial intelligence (MMAI) models that outperform current National Comprehensive Cancer Network (NCCN) risk groups. The cohort comprised 9,787 patients with localized prostate cancer from eight NRG Oncology randomized phase III trials, treated with radiation therapy, androgen deprivation therapy, and/or chemotherapy. Locked MMAI models, which used digital histopathology images and clinical data, were applied to each patient. Expert consensus on cut points defined low-, intermediate-, and high-risk groups on the basis of 10-year distant metastasis rates of 3% and 10%, respectively. The MMAI's reclassification and prognostic performance were compared with the three-tier NCCN risk groups. The median follow-up for censored patients was 7.9 years. According to NCCN risk categories, 30.4% of patients were low-risk, 25.5% intermediate-risk, and 44.1% high-risk. The MMAI risk classification identified 43.5% of patients as low-risk, 34.6% as intermediate-risk, and 21.8% as high-risk. MMAI reclassified 1,039 (42.0%) patients initially categorized by NCCN. Despite the MMAI low-risk group being larger than the NCCN low-risk group, the 10-year metastasis risks were comparable: 1.7% (95% CI, 0.2 to 3.2) for NCCN and 3.2% (95% CI, 1.7 to 4.7) for MMAI. The overall 10-year metastasis risk for NCCN high-risk patients was 16.6%, with MMAI further stratifying this group into low-, intermediate-, and high-risk, showing metastasis rates of 3.4%, 8.2%, and 26.3%, respectively. The MMAI risk grouping system expands the population of men identified as having low metastatic risk and accurately pinpoints a high-risk subset with elevated metastasis rates. This approach aims to prevent both overtreatment and undertreatment in localized prostate cancer, facilitating shared decision making.