Acute Coronary Syndromes (ACS), including ST- and non-ST-segment elevation myocardial infarction (STEMI, NSTEMI), remain a leading cause of global mortality. Traditional Cardiovascular Risk Scores (CVRS) provide important insights but mainly rely on clinical data, often neglecting environmental factors (e.g.air pollution, climate) that significantly influence cardiovascular health. Integrating complex time-series environmental and clinical datasets also presents substantial challenges. We propose TabulaTime, a multimodal deep learning framework integrating clinical risk factors with environmental data to enhance ACS risk prediction. TabulaTime delivers three innovations: multimodal integration of time-series environmental and clinical data; PatchRWKV for extracting complex temporal patterns with linear computational complexity; and enhanced interpretability through attention mechanisms. TabulaTime improves prediction accuracy by 20.5% over CatBoost, with environmental data contributing a 10.1% gain. PatchRWKV outperforms state-of-the-art models (MLP-, CNN-, RNN- and Transformer-based models). Feature analysis highlights key clinical and environmental predictors. This approach advances personalised prevention and strengthens public health against cardiovascular risks.

Intelligent matching methods for educational resources under a multimodal deep learning framework

A transformer and 3D CNN-based feature fusion network with interpretable ability for Raman spectra analysis: improving the diagnosis of thyroid cancer.

Foodie traps within facebook cannabis promotional posts: Deploying multimodal deep learning AIs to monitor audience engagement.

A multimodal deep learning framework for hemiplegic gait recognition using skeleton and wearable sensor data

VitalGuard-AI: a real-time multi-modal deep learning framework for intelligent health monitoring using wearable IoT devices

Multi-modal deep learning model for predicting recurrence of moderately severe and severe acute pancreatitis.

Pollination optimization in apple orchards faces increasing challenges from climate variability and declining pollinator populations, necessitating precision timing strategies. This study introduces a novel Pollination Importance Index (PII) integrated with a hybrid multi-task deep learning framework (PII-CNN-LSTM) to identify critical pollination windows. The PII dynamically quantifies pollination potential by incorporating flower receptivity, resource availability, biotic stress, and pollinator activity across five apple flower growth stages. The PII-CNN-LSTM architecture simultaneously performs growth stage classification and importance prediction through CNN spatial feature extraction and LSTM temporal modeling, enhanced by attention mechanisms and residual connections. Comparative evaluation against PII-CNN-BiLSTM, PII-CNN-GRU, and PII-CNN-TCN architectures demonstrated superior performance with 97% classification accuracy and minimal prediction error (validation loss: 0.0065, MAE: 0.0505). The model achieved exceptional full-bloom stage identification (99% F1-score), corresponding to its dominant 61.5% contribution to overall pollination importance. Cross-validation using 2024–2025 ground truth data and real-time drone deployment confirmed robust generalizability with temporal correlations exceeding 0.94. The framework successfully identified the critical pollination window from 3rd to 9th days, with optimal intervention timing at 5th to 7th days when importance scores exceeded 0.40. This biologically-grounded temporal precision enables targeted deployment of pollination resources during peak receptivity periods, reducing the need for continuous monitoring and intervention throughout the entire flowering season. The biologically-grounded approach provides scalable, data-driven decision support for precision agriculture, representing a significant advancement in agricultural automation and orchard productivity optimization. • Developed Pollination Importance Index (PII) integrating key pollination factors. • Identified optimal pollination window at days 5–7 with >0.94 temporal correlations. • PII-CNN-LSTM achieved 97% accuracy, outperforming BiLSTM, GRU, and TCN models. • Real-time drone deployment achieved 90% accuracy with YOLOv8s-PII-CNN-LSTM pipeline. • Six-channel fusion combining RGB imagery, PII score, image labels, and temporal sequences.

Whole-slide histopathological images (WSIs) constitute a fundamental approach in disease diagnosis and prognosis. Recently emerging spatial transcriptomics (ST) methods can reveal the spatial gene expression landscape behind the histopathological images, but with much higher cost. Here, therefore, we propose HESpotEx, a dual-stream multimodal deep learning framework to predict the spatial gene expression patterns solely from WSI images. Leveraging graph attention autoencoders, an image encoder and a graph convolution network decoder, HESpotEx is capable of predicting expressions of up to 5,457 genes across individual spatial sampling spots from WSIs. HESpotEx exhibits superior performance and better robustness on ST datasets from various cancer and noncancer samples as well as on a large-scale The Cancer Genome Atlas WSI dataset. Moreover, on our in-house WSI dataset, HESpotEx also underscores diagnosis-associated WSI patches. Finally, HESpotEx shows better cross-sectional consistency in the latest high-resolution ST datasets. Together, our results demonstrate the potential of HESpotEx to decipher the spatial molecular characteristics underlying tissue histological patterns.

Mitigating hallucinations in synthesized clinical texts to improve multimodal deep learning for dermatology.

Lung adenocarcinoma presenting as ground-glass nodules (GGNs) comprises three invasive subtypes (adenocarcinoma in situ [AIS], minimally invasive adenocarcinoma [MIA], invasive adenocarcinoma [IAC]) with distinct prognoses and management strategies. Preoperative discrimination of these subtypes remains challenging for radiologists, and existing deep learning models rarely integrate multi-modal data for reliable prediction. This study aimed to develop and internally validate a multi-modal fusion framework based on the standard ResNet50 architecture, integrating CT images, clinical variables, and tumor markers, to improve the preoperative prediction of ground-glass nodule invasiveness. A retrospective study was conducted including 431 patients with pathologically confirmed ground-glass nodules. All patients underwent standard chest computed tomography before surgery. A multi-modal deep learning model was constructed based on the ResNet50 network, combined with clinical characteristics and laboratory indicators. Model performance was evaluated using accuracy, area under the receiver operating characteristic curve, precision, recall, and F1-score with five-fold cross-validation. The proposed multi-modal model achieved an overall accuracy of 72.2%, precision of 95.6%, negative predictive value of 96.0%, weighted F1-score of 40.0%, and multiclass Matthews correlation coefficient of 73.1% in the three-class classification of AIS, MIA, and IAC. Per-class analysis showed precision of 84.6%, 35.7%, and 84.4% and recall of 57.9%, 29.4%, and 81.8% for AIS, MIA, and IAC, respectively. The fusion model yielded a macro-average AUC of 0.87, which was higher than the CT-only model (0.79) and both the senior (0.67) and junior radiologists (0.57). The model demonstrated superior diagnostic performance compared to human readers, particularly for the challenging MIA subtype. This multi-modal deep learning model combining CT images, clinical variables, and serum tumor markers enables accurate and robust three-class classification of AIS, MIA, and IAC in ground-glass nodules. The proposed model outperforms both human radiologists and the imaging-only model, suggesting its potential as a reliable auxiliary tool to improve preoperative prediction of lung adenocarcinoma invasiveness and assist clinical decision-making.

Accurate prediction of recurrence risk is essential to devise effective and personalized treatment strategies for patients with soft tissue sarcoma (STS). This study aimed to develop and validate a multimodal deep learning framework that integrates clinical features, preoperative MR images, and hematoxylin and eosin-stained whole slide images (WSIs) to predict recurrence in patients with STS. A total of 323 patients with STS were retrospectively enrolled from two hospitals, serving as development and validation sets, respectively. The ShuffleNetV2 network was utilized to develop patch-level and WSI-level signatures. A convolutional neural network fusing the channel and spatial attention mechanisms was used to develop a radiology signature. The combined model was built by integrating clinical features, radiology signature score, and WSI-level signature score with Cox regression analysis. The combined model demonstrated superior performance in the validation set, achieving a C-index of 0.857 and a time-dependent area under the curve of 0.959. Class activation maps facilitated the monitoring of suspected regions to inform recurrence decisions. The recurrence-free survival times of the low- and high-risk cohorts were statistically different (p < 0.05). The proposed multimodal framework offers satisfactory accuracy for predicting recurrence risk in patients with STS and could guide the choice of treatment modality.

RIMGOGraph: integrating AlphaFold-derived residue interaction graphs and protein language embeddings for structure-informed protein function prediction.

Gallbladder cancer (GBC) is a rare gastrointestinal malignancy with a global 5-year survival rate of less than 5%. Early diagnosis is challenging owing to the lack of specific clinical symptoms. Additionally, the high heterogeneity of gallbladder tumors limits the clinical utility of unimodal deep-learning methods for GBC diagnosis. This study aimed to develop a novel multimodal deep-learning model to facilitate the preoperative diagnosis of GBC in more patients. We conducted a retrospective multicenter study using contrast-enhanced arterial phase computed tomography (CT) images and laboratory examination data from 300 patients (150 GBC cases and 150 non-GBC cases) extracted from electronic medical records of two Grade A tertiary hospitals in Shanghai between 2018 and 2020. A novel two-stage multimodal diagnostic model (GBC-DiagNet) was developed: the first stage achieved coarse segmentation of the gallbladder region using a position-constrained 3D Attention U-Net (improved by combined sampling) to avoid over-segmentation; the second stage realized GBC detection via an adaptive feature fusion strategy, which optimizes the weighted integration of handcrafted radiomic, deep radiomic and laboratory examination features to enhance diagnostic performance. On the independent test set, the model achieved an accuracy of 0.933 (95% confidence interval [95% CI]: 0.927-0.94), specificity of 0.912 (95% CI: 0.904-0.922), sensitivity of 0.962 (95% CI: 0.937-0.986), precision of 0.893 (95% CI: 0.875-0.911), an F1-score of 0.926 (95% CI: 0.919-0.932) and AUC (area under the curve) of 0.9706 (95% CI: 0.961-0.981). Compared with the optimal unimodal model, our model improved accuracy, sensitivity, and F1-score by 14.28%, 16.76%, and 16.85%, respectively. Furthermore, compared to state-of-the-art deep-learning architectures (ResNet, DenseNet, MobileNet, ConvNeXt, ViT), our model exhibited absolute improvements of 7.68% in accuracy, 8.03% in F1-score, and 0.0059 in AUC. The proposed multimodal model integrating contrast-enhanced CT and laboratory data achieves stable and clinically meaningful diagnostic performance for gallbladder cancer, supporting its utility as an artificial intelligence-assisted tool for preoperative noninvasive diagnosis.

A mechanism-informed multi-modal deep learning algorithm for PEMFC aging prediction

Objective.Accurate physiological assessment of cardiac function from heart sounds remains challenging due to background noise, variable heart rates, and the need for reliable cardiac-cycle segmentation. This study aimed to develop a fully E2E deep learning framework that extracts diagnostic information directly from raw heart sound recordings for cardiac abnormality detection and classification.Approach.We propose HS-MMNet, an E2E multi-modal deep learning framework designed for physiological heart sound analysis. Recordings are preprocessed (normalization and 25-400 Hz bandpass filtering) and divided into fixed-length 2.5 s segments. A Convolution Head with multi-atrous spatial pyramid and channel-spatial attention extracts fine-grained local temporal patterns from the filtered 1-D waveform. A Transformer Head captures long-range spectro-temporal dependencies from Log-Mel spectrograms. These hypotheses are iteratively fused by a novel multi-hypothesis cross-attention module with cyclic query-key-value assignment and a hypothesis-mixing MLP, enabling rich cross-site interaction and effective suppression of noise and non-informative regions. Recording-level classification is obtained via a fully connected layer.Main results.On the PhysioNet/CinC Challenge 2016 dataset, HS-MMNet achieved 94.80% accuracy, 92.10% sensitivity, 96.85% specificity, 87.50% precision, and 89.74%F1-score, outperforming all previously reported methods. On the balanced five-class Yaseen dataset (normal, aortic stenosis, mitral regurgitation, mitral stenosis, mitral valve prolapse), it attained 99.60% macro-averaged precision, recall, andF1-score with only four misclassifications in 1000 recordings, establishing new state-of-the-art (SOTA) benchmarks.Significance.HS-MMNet represents an advance in automated physiological measurement from heart sounds. By eliminating cardiac cycle detection and multi-channel requirements while achieving SOTA diagnostic performance, it provides a practical, scalable solution for accurate cardiovascular screening with primary-care and low-resource settings.

Isocitrate dehydrogenase (IDH) is a pivotal molecular marker for glioma diagnosis, prognosis, and treatment planning. Multi-modal deep learning methods, which integrate features from multiple magnetic resonance imaging (MRI) sequences, have become a powerful solution for non-invasive IDH genotyping. However, existing methods still have limitations in feature extraction and fusion, which constrains their robustness. In this work, we propose a novel segmentation-guided feature alignment and fusion network (SFAF-Net) for glioma IDH genotyping, with three key innovations: 1) The Segmentation-guided Feature Alignment (SFA) module leverages tumor segmentation supervision to facilitate cross-modal feature alignment; 2) The Redundancy-Attenuated Fusion (RAF) module implements similarity-based selective fusion of modality pairs to reduce feature redundancy; 3) A randomized modality dropout mechanism within RAF enhances model robustness against input variations. Comprehensive experiments conducted on public and private datasets demonstrate that SFAF-Net outperforms state-of-the-art methods across diverse MRI sequences. Moreover, SFAF-Net supports an arbitrary number of input sequences, enabling flexible adaptation to diverse clinical scanning protocols in personalized diagnosis.

Medical image interpretation plays a critical role in lumbar fusion surgery, where accurate analysis of anatomical structures is essential for clinical assessment. However, most existing deep learning approaches rely primarily on visual features and fail to effectively integrate heterogeneous clinical information. This study proposes a multimodal deep learning framework for lumbar spine image interpretation by jointly modeling medical images and associated clinical text. The framework adopts a global-local representation learning strategy to capture both overall anatomical context and fine-grained structural information. A visual encoder extracts hierarchical features from lumbar radiographs and CT scans, while a transformer-based text encoder captures semantic information from clinical reports. These representations are projected into a shared embedding space to enable cross-modal alignment. To enhance feature interaction, a text-guided attention mechanism is introduced to model correspondence between image regions and textual descriptions. The learned multimodal representations are applied to multiple downstream tasks, including cross-modal retrieval, classification, and lumbar structure segmentation. Experimental results show that the proposed framework outperforms image-only baselines and achieves competitive performance compared with existing multimodal approaches. The integration of global and local representations improves feature discrimination and structural modeling. Visualization results provide qualitative evidence that the model focuses on anatomically relevant regions, although such observations should be interpreted with caution. Overall, the proposed framework demonstrates the potential of multimodal representation learning for lumbar spine image analysis and provides a structured approach for integrating heterogeneous clinical data.

To develop and validate a multimodal deep learning model that predicts treatment responses to intravitreal anti-vascular endothelial growth factor (anti-VEGF) injections in patients with diabetic macular oedema (DMO) by combining optical coherence tomography images and clinical data. This study included 107 DMO patients who received three consecutive anti-VEGF treatments. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity. The model's predictions were compared with those of retinal specialists. Among 107 patients, 65 showed good response and 42 showed poor response to treatment. The multimodal model achieved an AUROC of 0.962 (95% CI, 0.945-0.979), accuracy of 0.953 (95% CI, 0.933-0.973), sensitivity of 0.969 (95% CI, 0.951-0.987), and specificity of 0.928 (95% CI, 0.903-0.953) in the internal validation. The model outperformed retinal specialists, who achieved accuracies ranging from 0.571 to 0.857. The multimodal deep learning model demonstrated high accuracy in predicting anti-VEGF treatment responses in DMO patients. This approach could enable more personalised treatment strategies and optimal resource utilisation in ophthalmological care. Further validation with larger, multicentre datasets is warranted to confirm its clinical utility.

Road accidents caused by driver fatigue and cognitive overload remain a significant public safety concern. According to recent traffic safety data, drowsy driving contributes to thousands of fatal accidents each year, emphasizing the urgent need for intelligent driver monitoring systems. To address this, we propose an adaptive multimodal deep learning framework (AML) for real-time cognitive workload assessment and fatigue detection, leveraging the CL-Drive dataset: a multimodal repository of EEG (cognitive load), ECG (cardiac activity), EDA (electrodermal arousal), and gaze tracking (visual attention) captured from 21 participants during simulated driving across nine scenarios of escalating complexity. Our framework integrates a hybrid CNN-BiLSTM architecture to extract spatiotemporal features from raw physiological signals and gaze sequences, capturing localized spatial patterns and long-term temporal dynamics. These features are fused using a transformer-based network with cross-modal attention, which models interactions between modalities (e.g., correlating gaze fixation losses with EEG theta-band surges during distraction) and yields a 3.6 percentage-point absolute accuracy improvement over the strongest conventional fusion baseline under identical evaluation. To address individual variability and privacy, we combine personalized meta-learning-adapting to new drivers with as few as five windowed samples (∼10s of synchronized multimodal data) via episodic fine-tuning-with federated optimization, enabling decentralized model updates and reducing per-client data transfer by 38% through adaptive gradient compression. Experiments on CL-Drive demonstrate state-of-the-art performance under strictly cross-subject evaluation. Under subject-independent 5-fold cross-validation, AML achieves [Formula: see text] accuracy on binary cognitive load classification without personalization, rising to [Formula: see text] with [Formula: see text] calibration samples (∼40s). Under the more rigorous leave-one-subject-out (LOSO) protocol, AML reaches [Formula: see text] without personalization and [Formula: see text] with [Formula: see text] personalization, an improvement of 1.6 percentage points over the strongest published LOSO baseline on this dataset with a further 6.2-11.3 percentage points gained from personalization alone across the LOSO and 5-fold protocols. The framework exhibits robustness to real-world sensor noise (e.g., EEG/EDA motion artifacts) and achieves [Formula: see text] LOSO accuracy with only [Formula: see text] samples (∼10s of calibration per new driver), critical for scalable in-vehicle deployment. By enabling privacy-aware, real-time monitoring of driver states, this work advances intelligent vehicle safety systems and provides a blueprint for adaptive multimodal learning in human-centric AI applications.