Multimodal Deep Learning Model Research Articles

Zero-shot prediction of mutation effects with multimodal deep representation learning guides protein engineering.

Mutations in amino acid sequences can provoke changes in protein function. Accurate and unsupervised prediction of mutation effects is critical in biotechnology and biomedicine, but remains a fundamental challenge. To resolve this challenge, here we present Protein Mutational Effect Predictor (ProMEP), a general and multiple sequence alignment-free method that enables zero-shot prediction of mutation effects. A multimodal deep representation learning model embedded in ProMEP was developed to comprehensively learn both sequence and structure contexts from ~160 million proteins. ProMEP achieves state-of-the-art performance in mutational effect prediction and accomplishes a tremendous improvement in speed, enabling efficient and intelligent protein engineering. Specifically, ProMEP accurately forecasts mutational consequences on the gene-editing enzymes TnpB and TadA, and successfully guides the development of high-performance gene-editing tools with their engineered variants. The gene-editing efficiency of a 5-site mutant of TnpB reaches up to 74.04% (vs 24.66% for the wild type); and the base editing tool developed on the basis of a TadA 15-site mutant (in addition to the A106V/D108N double mutation that renders deoxyadenosine deaminase activity to TadA) exhibits an A-to-G conversion frequency of up to 77.27% (vs 69.80% for ABE8e, a previous TadA-based adenine base editor) with significantly reduced bystander and off-target effects compared to ABE8e. ProMEP not only showcases superior performance in predicting mutational effects on proteins but also demonstrates a great capability to guide protein engineering. Therefore, ProMEP enables efficient exploration of the gigantic protein space and facilitates practical design of proteins, thereby advancing studies in biomedicine and synthetic biology.

Multimodal explainable artificial intelligence identifies patients with non-ischaemic cardiomyopathy at risk of lethal ventricular arrhythmias

The efficacy of an implantable cardioverter-defibrillator (ICD) in patients with a non-ischaemic cardiomyopathy for primary prevention of sudden cardiac death is increasingly debated. We developed a multimodal deep learning model for arrhythmic risk prediction that integrated late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI), electrocardiography (ECG) and clinical data. Short-axis LGE-MRI scans and 12-lead ECGs were retrospectively collected from a cohort of 289 patients prior to ICD implantation, across two tertiary hospitals. A residual variational autoencoder was developed to extract physiological features from LGE-MRI and ECG, and used as inputs for a machine learning model (DEEP RISK) to predict malignant ventricular arrhythmia onset. In the validation cohort, the multimodal DEEP RISK model predicted malignant ventricular arrhythmias with an area under the receiver operating characteristic curve (AUROC) of 0.84 (95% confidence interval (CI) 0.71–0.96), a sensitivity of 0.98 (95% CI 0.75–1.00) and a specificity of 0.73 (95% CI 0.58–0.97). The models trained on individual modalities exhibited lower AUROC values compared to DEEP RISK [MRI branch: 0.80 (95% CI 0.65–0.94), ECG branch: 0.54 (95% CI 0.26–0.82), Clinical branch: 0.64 (95% CI 0.39–0.87)]. These results suggest that a multimodal model achieves high prognostic accuracy in predicting ventricular arrhythmias in a cohort of patients with non-ischaemic systolic heart failure, using data collected prior to ICD implantation.

Effectiveness Analysis of Entrepreneurial Legal Risk Prevention Based on Multi-Modal Deep Learning Model

With the emergence of the upsurge of entrepreneurship, entrepreneurs are increasingly concerned about legal risks. In the process of entrepreneurship, legal risk is the biggest hidden danger of entrepreneurial enterprises. The prevention and avoidance of legal risks is a long and arduous process, and not all risks can be identified and avoided in time. The deep learning method has brought great changes to the fields of speech recognition, image recognition and natural language processing. The tasks in these fields only involve single-mode input, but more recent applications need to involve multi-mode intelligence. Multimodal deep learning mainly includes three aspects: multimodal learning representation, multimodal signal fusion and multimodal application. Through the research on the legal risks in recent years, this paper believed that there are many legal problems in the development of entrepreneurial enterprises, including contract disputes, trade secret disputes, trademark infringement disputes, copyright infringement disputes, and so on. This paper aimed to study the problem of Legal Risks of Entrepreneurship (LRE) that entrepreneurs are concerned about, and proposed a new solution from the perspective of image recognition technology. 486 entrepreneurs were investigated by questionnaire. In the process of entrepreneurship, 87.04% of entrepreneurs encountered legal risks, and they would turn to lawyers for help, which is a better way. However, in most cases, entrepreneurs are exposed to LRE because of their insufficient understanding of economic law, so they solve it through lawyers. However, if a lawyer is hired, the cost would be very high, which would bring great economic pressure to the enterprise. Only 17.90% of entrepreneurs would safeguard their legitimate rights and interests through their own knowledge and legal weapons without resorting to lawyers. It can be seen that entrepreneurs have relatively low practical ability in the use of LRE, and their legal practical ability is obviously insufficient.

AI-based prediction of protein–ligand binding affinity and discovery of potential natural product inhibitors against ERK2

Determination of protein–ligand binding affinity (PLA) is a key technological tool in hit discovery and lead optimization, which is critical to the drug development process. PLA can be determined directly by experimental methods, but it is time-consuming and costly. In recent years, deep learning has been widely applied to PLA prediction, the key of which lies in the comprehensive and accurate representation of proteins and ligands. In this study, we proposed a multi-modal deep learning model based on the early fusion strategy, called DeepLIP, to improve PLA prediction by integrating multi-level information, and further used it for virtual screening of extracellular signal-regulated protein kinase 2 (ERK2), an ideal target for cancer treatment. Experimental results from model evaluation showed that DeepLIP achieved superior performance compared to state-of-the-art methods on the widely used benchmark dataset. In addition, by combining previously developed machine learning models and molecular dynamics simulation, we screened three novel hits from a drug-like natural product library. These compounds not only had favorable physicochemical properties, but also bound stably to the target protein. We believe they have the potential to serve as starting molecules for the development of ERK2 inhibitors.

Non-invasive colorectal cancer detection using multimodal deep learning ensemble classifier.

Non-invasive colorectal cancer detection using multimodal deep learning ensemble classifier.

Sarcasm detection in hotel reviews: a multimodal deep learning approach

Purpose This study aims to detect consumer sarcasm through inconsistencies in sentiment features between text and images of hotel reviews. Design/methodology/approach This paper proposes a model for sarcasm detection based on multimodal deep learning using reviews of three hotel brands collected from two travel platforms, which can identify emotional inconsistencies within a modality and across modalities. Text-image interaction information is explored using graph neural networks (GNN) to detect essential clues in sarcasm sentiment. Findings The research results show that the multimodal deep learning model outperforms other baseline models, which can help to understand hotel service evaluation and provide hotel managers with decision-making opinions. Originality/value This research can help hoteliers in two ways: detecting service quality and formulating strategies. By selecting reference hotel brands, hoteliers can better assess their level of service quality (optimal resource allocation ensues); therefore, sarcasm detection research is not only beneficial for hotel managers seeking to improve service quality. The multimodal deep learning method introduced in the present study can be replicated in other industries to help travel platforms optimize their products and services.

Predicting atrial fibrillation ablation success: a novel approach using pre-procedure sinus rhythm ecg and deep learning

Abstract Introduction Pulmonary vein isolation (PVI) has been the corner stone of the treatment of patients with atrial fibrillation (AF), yet its efficacy varies notably between patient groups. Specifically, it shows reduced efficacy in individuals with persistent AF and those with atrial fibrosis. Recent advancements suggest that additional substrate modification could enhance outcomes in specific patient groups. Therefore, predicting the outcomes of lone PVI could help in stratifying patients who would benefit most from this additional substrate modification. Addressing this gap, our study explores the prognostication of PVI outcomes using pre-procedural 12-lead ECG recordings taken in Sinus rhythm, employing advanced deep learning techniques. Methods This retrospective study analyzed 209 patients with paroxysmal and persistent AF who underwent PVI. These patients were followed up for 12 months after the blanking period. Pre-procedural sinus rhythm 12-lead ECGs were processed using an innovative deep learning architecture, integrating Long Short-Term Memory and Convolutional Neural Networks with a cross-attention module. This multimodal deep learning model was trained using ECG and clinical covariates like age, sex, and BMI to predict binary outcomes – AF type (Paroxysmal or Persistent) and AF recurrence. The model's efficacy was evaluated through a 5-fold stratified cross-validation to ensure accuracy and reliability. Model was also tested using only ECG data. Results The multimodal deep learning model performed significantly better as compared to the model trained using ECG data only. The multimodal deep learning model achieved an accuracy of 75.56% and a sensitivity of 82% in prognosticating ablation outcomes. Additionally, the model demonstrated a significant capability in classifying AF into paroxysmal or persistent types with an accuracy of 86.67% and a recall of 89%. Conclusion Our deep learning approach in analyzing sinus rhythm ECG recordings shows promising results in predicting PVI outcomes and identifying AF types. This approach could offer a more accessible and cost-effective tool to stratify AF patients undergoing PVI, identifying those less likely to benefit from standard procedures. Such insights can assist physicians and patients in determining ablation candidacy and selecting the most appropriate ablation strategy and technique.Proposed StrategyDeep learning model approach

Assessing learners’ English public speaking anxiety with multimodal deep learning technologies

Public speaking anxiety (PSA) is a common phenomenon for language learners involving both psychological and physiological symptoms. Timely and effective PSA assessment can help diagnose learners’ speaking anxiety, offer learners feedback to alleviate their anxiety, and improve their public speaking competence. However, it is still a challenging issue to achieve accurate automated assessment of learners’ PSA due to the lack of large-scale and open-source multimodal datasets based on real classroom settings. This study collected the public speaking videos of English as a foreign language (EFL) learners in public speaking courses, and constructed a large-scale multimodal dataset named Speaking Anxiety in Real Classrooms (SARC) with three modalities of acoustic, visual, and textual data (including 1,158 manually-annotated speech videos and corresponding speech drafts of 382 participants). A multimodal deep learning model for automated assessment of learners’ speaking anxiety was proposed, and an online formative assessment platform was then developed to realize the automated assessment of PSA for classroom teaching. A pilot survey study involving 78 participants was then conducted to investigate learners’ acceptance of the platform. Experimental results verified the validity of the deep learning model and the consistency between automated assessment and teacher assessment. Learners’ acceptance data further indicated that collaboration between automated and human assessment provided them with the most satisfactory experience of using the platform to improve their English public speaking.

Heart failure prognosis prediction: Let’s start with the MDL-HFP model

Heart failure, as a critical symptom or terminal stage of assorted heart diseases, is a world-class public health problem. Establishing a prognostic model can help identify high dangerous patients, save their lives promptly, and reduce medical burden. Although integrating structured indicators and unstructured text for complementary information has been proven effective in disease prediction tasks, there are still certain limitations. Firstly, the processing of single branch modes is easily overlooked, which can affect the final fusion result. Secondly, simple fusion will lose complementary information between modalities, limiting the network’s learning ability. Thirdly, incomplete interpretability can affect the practical application and development of the model. To overcome these challenges, this paper proposes the MDL-HFP multimodal model for predicting patient prognosis using the MIMIC-III public database. Firstly, the ADASYN algorithm is used to handle the imbalance of data categories. Then, the proposed improved Deep&Cross Network is used for automatic feature selection to encode structured sparse features, and implicit graph structure information is introduced to encode unstructured clinical notes based on the HR-BGCN model. Finally, the information of the two modalities is fused through a cross-modal dynamic interaction layer. By comparing multiple advanced multimodal deep learning models, the model’s effectiveness is verified, with an average F1 score of 90.42% and an average accuracy of 90.70%. The model proposed in this paper can accurately classify the readmission status of patients, thereby assisting doctors in making judgments and improving the patient’s prognosis. Further visual analysis demonstrates the usability of the model, providing a comprehensive explanation for clinical decision-making.

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning.

The objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data. The research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients' pathological images and clinical data, respectively. A total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS). Our study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.

Predicting rectal cancer tumor budding grading based on MRI and CT with multimodal deep transfer learning: A dual-center study

ObjectiveTo investigate the effectiveness of a multimodal deep learning model in predicting tumor budding (TB) grading in rectal cancer (RC) patients. Materials and methodsA retrospective analysis was conducted on 355 patients with rectal adenocarcinoma from two different hospitals. Among them, 289 patients from our institution were randomly divided into an internal training cohort (n = 202) and an internal validation cohort (n = 87) in a 7:3 ratio, while an additional 66 patients from another hospital constituted an external validation cohort. Various deep learning models were constructed and compared for their performance using T1CE and CT-enhanced images, and the optimal models were selected for the creation of a multimodal fusion model. Based on single and multiple factor logistic regression, clinical N staging and fecal occult blood were identified as independent risk factors and used to construct the clinical model. A decision-level fusion was employed to integrate these two models to create an ensemble model. The predictive performance of each model was evaluated using the area under the curve (AUC), DeLong's test, calibration curve, and decision curve analysis (DCA). Model visualization Gradient-weighted Class Activation Mapping (Grad-CAM) was performed for model interpretation. ResultsThe multimodal fusion model demonstrated superior performance compared to single-modal models, with AUC values of 0.869 (95% CI: 0.761–0.976) for the internal validation cohort and 0.848 (95% CI: 0.721–0.975) for the external validation cohort. N-stage and fecal occult blood were identified as clinically independent risk factors through single and multivariable logistic regression analysis. The final ensemble model exhibited the best performance, with AUC values of 0.898 (95% CI: 0.820–0.975) for the internal validation cohort and 0.868 (95% CI: 0.768–0.968) for the external validation cohort. ConclusionMultimodal deep learning models can effectively and non-invasively provide individualized predictions for TB grading in RC patients, offering valuable guidance for treatment selection and prognosis assessment.

Integrating wearable sensor data and self-reported diaries for personalized affect forecasting

Emotional states, as indicators of affect, are pivotal to overall health, making their accurate prediction before onset crucial. Current studies are primarily centered on immediate short-term affect detection using data from wearable and mobile devices. These studies typically focus on objective sensory measures, often neglecting other forms of self-reported information like diaries and notes. In this paper, we propose a multimodal deep learning model for affect status forecasting. This model combines a transformer encoder with a pre-trained language model, facilitating the integrated analysis of objective metrics and self-reported diaries. To validate our model, we conduct a longitudinal study, enrolling college students and monitoring them over a year, to collect an extensive dataset including physiological, environmental, sleep, metabolic, and physical activity parameters, alongside open-ended textual diaries provided by the participants. Our results demonstrate that the proposed model achieves predictive accuracy of 82.50% for positive affect and 82.76% for negative affect, a full week in advance. The effectiveness of our model is further elevated by its explainability.

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Abstract Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2313.

EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN.

Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training-testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.

Predicting Obstructive Sleep Apnea Based on Computed Tomography Scans Using Deep Learning Models.

Rationale: The incidence of clinically undiagnosed obstructive sleep apnea (OSA) is high among the general population because of limited access to polysomnography. Computed tomography (CT) of craniofacial regions obtained for other purposes can be beneficial in predicting OSA and its severity. Objectives: To predict OSA and its severity based on paranasal CT using a three-dimensional deep learning algorithm. Methods: One internal dataset (N = 798) and two external datasets (N = 135 and N = 85) were used in this study. In the internal dataset, 92 normal participants and 159 with mild, 201 with moderate, and 346 with severe OSA were enrolled to derive the deep learning model. A multimodal deep learning model was elicited from the connection between a three-dimensional convolutional neural network-based part treating unstructured data (CT images) and a multilayer perceptron-based part treating structured data (age, sex, and body mass index) to predict OSA and its severity. Measurements and Main Results: In a four-class classification for predicting the severity of OSA, the AirwayNet-MM-H model (multimodal model with airway-highlighting preprocessing algorithm) showed an average accuracy of 87.6% (95% confidence interval [CI], 86.8-88.6%) in the internal dataset and 84.0% (95% CI, 83.0-85.1%) and 86.3% (95% CI, 85.3-87.3%) in the two external datasets, respectively. In the two-class classification for predicting significant OSA (moderate to severe OSA), the area under the receiver operating characteristic curve, accuracy, sensitivity, specificity, and F1 score were 0.910 (95% CI, 0.899-0.922), 91.0% (95% CI, 90.1-91.9%), 89.9% (95% CI, 88.8-90.9%), 93.5% (95% CI, 92.7-94.3%), and 93.2% (95% CI, 92.5-93.9%), respectively, in the internal dataset. Furthermore, the diagnostic performance of the Airway Net-MM-H model outperformed that of the other six state-of-the-art deep learning models in terms of accuracy for both four- and two-class classifications and area under the receiver operating characteristic curve for two-class classification (P < 0.001). Conclusions: A novel deep learning model, including a multimodal deep learning model and an airway-highlighting preprocessing algorithm from CT images obtained for other purposes, can provide significantly precise outcomes for OSA diagnosis.

Predicting long-term progression of Alzheimer’s disease using a multimodal deep learning model incorporating interaction effects

BackgroundIdentifying individuals with mild cognitive impairment (MCI) at risk of progressing to Alzheimer’s disease (AD) provides a unique opportunity for early interventions. Therefore, accurate and long-term prediction of the conversion from MCI to AD is desired but, to date, remains challenging. Here, we developed an interpretable deep learning model featuring a novel design that incorporates interaction effects and multimodality to improve the prediction accuracy and horizon for MCI-to-AD progression.MethodsThis multi-center, multi-cohort retrospective study collected structural magnetic resonance imaging (sMRI), clinical assessments, and genetic polymorphism data of 252 patients with MCI at baseline from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Our deep learning model was cross-validated on the ADNI-1 and ADNI-2/GO cohorts and further generalized in the ongoing ADNI-3 cohort. We evaluated the model performance using the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and F1 score.ResultsOn the cross-validation set, our model achieved superior results for predicting MCI conversion within 4 years (AUC, 0.962; accuracy, 92.92%; sensitivity, 88.89%; specificity, 95.33%) compared to all existing studies. In the independent test, our model exhibited consistent performance with an AUC of 0.939 and an accuracy of 92.86%. Integrating interaction effects and multimodal data into the model significantly increased prediction accuracy by 4.76% (P = 0.01) and 4.29% (P = 0.03), respectively. Furthermore, our model demonstrated robustness to inter-center and inter-scanner variability, while generating interpretable predictions by quantifying the contribution of multimodal biomarkers.ConclusionsThe proposed deep learning model presents a novel perspective by combining interaction effects and multimodality, leading to more accurate and longer-term predictions of AD progression, which promises to improve pre-dementia patient care.

Multi-modality deep learning model reaches high prediction accuracy in the diagnosis of ovarian cancer

We evaluated the diagnostic performance of a multimodal deep-learning (DL) model for ovarian mass differential diagnosis. This single-center retrospective study included 1,054 ultrasound (US)-detected ovarian tumors (699 benign and 355 malignant). Patients were randomly divided into training (n= 675), validation (n= 169), and testing (n= 210) sets. The model was developed using ResNet-50. Three DL-based models were proposed for benign-malignant classification of these lesions: single-modality model that only utilized US images; dual-modality model that used US images and menopausal status as inputs; and multi-modality model that integrated US images, menopausal status, and serum indicators. After 5-fold cross-validation, 210 lesions were tested. We evaluated the three models using the area under the curve (AUC), accuracy, sensitivity, and specificity. The multimodal model outperformed the single- and dual-modality models with 93.80% accuracy and 0.983 AUC. The Multimodal ResNet-50 DL model outperformed the single- and dual-modality models in identifying benign and malignant ovarian tumors.

Predicting microvascular invasion in hepatocellular carcinoma with a CT- and MRI-based multimodal deep learning model.

To investigate the value of a multimodal deep learning (MDL) model based on computed tomography (CT) and magnetic resonance imaging (MRI) for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC). A total of 287 patients with HCC from our institution and 58 patients from another individual institution were included. Among these, 119 patients with only CT data and 116 patients with only MRI data were selected for single-modality deep learning model development, after which select parameters were migrated for MDL model development with transfer learning (TL). In addition, 110 patients with simultaneous CT and MRI data were divided into a training cohort (n = 66) and a validation cohort (n = 44). We input the features extracted from DenseNet121 into an extreme learning machine (ELM) classifier to construct a classification model. The area under the curve (AUC) of the MDL model was 0.844, which was superior to that of the single-phase CT (AUC = 0.706-0.776, P < 0.05), single-sequence MRI (AUC = 0.706-0.717, P < 0.05), single-modality DL model (AUCall-phase CT = 0.722, AUCall-sequence MRI = 0.731; P < 0.05), clinical (AUC = 0.648, P < 0.05), but not to that of the delay phase (DP) and in-phase (IP) MRI and portal venous phase (PVP) CT models. The MDL model achieved better performance than models described above (P < 0.05). When combined with clinical features, the AUC of the MDL model increased from 0.844 to 0.871. A nomogram, combining deep learning signatures (DLS) and clinical indicators for MDL models, demonstrated a greater overall net gain than the MDL models (P < 0.05). The MDL model is a valuable noninvasive technique for preoperatively predicting MVI in HCC.

Analysing the spatiotemporal variation and influencing factors of Lake Chaohu's CDOM over the past 40 years using machine learning

AbstractChromophoric dissolved organic matter (CDOM) in aquatic environments is an important component of the biogeochemical cycle and carbon cycle. The aim of this study is to investigate the long‐term changes in CDOM in shallow and eutrophic Chaohu Lake, as well as its relationship with climate, environment and social factors. Using long time series Landsat image data and machine learning technology, the spatiotemporal evolution of Chaohu CDOM since 1987 was reconstructed. A total of 180 samples were collected, which were divided into three parts based on regional and hydrological characteristics. The results show that the water quality in different regions were significantly different, and TN may be the key factor driving the change of CDOM in Chaohu Lake. Machine learning algorithms including random forest (RF), support vector regression (SVR), neural network (NN), multimodal deep learning (MDL) model and Extreme Gradient Boosting (XGBoost) were used, among which XGBoost model performed best (R2 = 0.955, mean absolute error [MAE] = 0.024 mg/L, root mean square error [RMSE] = 0.036 mg/L, bias = 1.005) and was used for CDOM spatiotemporal variation retrieval. The change of CDOM was seasonal, highest in August (0.67 m−1) and lowest in December (0.48 m−1), and the western lake is the main source of CDOM. Annual variability of the CDOM indicates that it began to decline after the completion of water pollution control in 2000. Temperature changes were closely related to CDOM (P &lt; 0.01) and agricultural non‐point source pollution plays an important role in Chaohu Lake. This study will provide feasible methods and scientific basis for the long‐term remote sensing supervision of CDOM.

Multimodal Deep Learning for Estimating Lane-Level Urban Traffic by Fusing Closed-Circuit Television and Dedicated Short-Range Communication Data

Seamless, accurate, and reliable traffic information is critical for planning and operational strategies in traffic management. Urban networks are increasingly being equipped with various types of detectors to collect continuous traffic data. However, the widespread installation and management of these detectors at all intersections and road sections are limited by cost constraints and various problems. Moreover, the direct integration and utilization of heterogeneous data sources are challenging because of their distinct and interconnected characteristics. This study introduces a multimodal deep learning model, combining closed-circuit television (CCTV) and dedicated short-range communication (DSRC) data, to estimate lane-level urban traffic volumes. The proposed model employs a multilayer perceptron to extract features from each modality. These features are then fused and used as input into a recurrent neural network model that estimates the lane-level traffic volume. We present the multimodal deep learning model in three forms: (1) fusion of traffic volume, occupancy, and queue length from CCTV data; (2) fusion of traffic volume from CCTV data with travel time from DSRC data; and (3) fusion of different attributes from CCTV and heterogeneous DSRC data. In addition, we develop a single-modality model that solely utilizes CCTV data on traffic volume to compare the performance with the proposed multimodal model and identify scenarios where the multimodal approach is essential. The proposed model demonstrates significant improvements over the single-modality model, providing enhanced accuracy for higher temporal resolutions and lanes that permit left or right turns.