Multimodal Data Research Articles

Enhancing EEG data quality and precision for cloud-based clinical applications: an evaluation of the SLOG framework

Automation is revamping our preprocessing pipelines, and accelerating the delivery of personalized digital medicine. It improves efficiency, reduces costs, and allows clinicians to treat patients without significant delays. However, the influx of multimodal data highlights the need to protect sensitive information, such as clinical data, and safeguard data fidelity. One of the neuroimaging modalities that produces large amounts of time-series data is Electroencephalography (EEG). It captures the neural dynamics in a task or resting brain state with high temporal resolution. EEG electrodes placed on the scalp acquire electrical activity from the brain. These electrical potentials attenuate as they cross multiple layers of brain tissue and fluid yielding relatively weaker signals than noise-low signal-to-noise ratio. EEG signals are further distorted by internal physiological artifacts, such as eye movements (EOG) or heartbeat (ECG), and external noise, such as line noise (50 Hz). EOG artifacts, due to their proximity to the frontal brain regions, are particularly challenging to eliminate. Therefore, a widely used EOG rejection method, independent component analysis (ICA), demands manual inspection of the marked EOG components before they are rejected from the EEG data. We underscore the inaccuracy of automatized ICA rejection and provide an auxiliary algorithm-Second Layer Inspection for EOG (SLOG) in the clinical environment. SLOG based on spatial and temporal patterns of eye movements, re-examines the already marked EOG artifacts and confirms no EEG-related activity is mistakenly eliminated in this artifact rejection step. SLOG achieved a 99% precision rate on the simulated dataset while 85% precision on the real EEG dataset. One of the primary considerations for cloud-based applications is operational costs, including computing power. Algorithms like SLOG allow us to maintain data fidelity and precision without overloading the cloud platforms and maxing out our budgets.

Shuffled ECA-Net for stress detection from multimodal wearable sensor data

BackgroundRecently, stress has been recognized as a key factor in the emergence of individual and social issues. Numerous attempts have been made to develop sensor-augmented psychological stress detection techniques, although existing methods are often impractical or overly subjective. To overcome these limitations, we acquired a dataset utilizing both wireless wearable multimodal sensors and salivary cortisol tests for supervised learning. We also developed a novel deep neural network (DNN) model that maximizes the benefits of sensor fusion. MethodWe devised a DNN involving a shuffled efficient channel attention (ECA) module called a shuffled ECA-Net, which achieves advanced feature-level sensor fusion by considering inter-modality relationships. Through an experiment involving salivary cortisol tests on 26 participants, we acquired multiple bio-signals including electrocardiograms, respiratory waveforms, and electrogastrograms in both relaxed and stressed mental states. A training dataset was generated from the obtained data. Using the dataset, our proposed model was optimized and evaluated ten times through five-fold cross-validation, while varying a random seed. ResultsOur proposed model achieved acceptable performance in stress detection, showing 0.916 accuracy, 0.917 sensitivity, 0.916 specificity, 0.914 F1-score, and 0.964 area under the receiver operating characteristic curve (AUROC). Furthermore, we demonstrated that combining multiple bio-signals with a shuffled ECA module can more accurately detect psychological stress. ConclusionsWe believe that our proposed model, coupled with the evidence for the viability of multimodal sensor fusion and a shuffled ECA-Net, would significantly contribute to the resolution of stress-related issues.

Multimodal Dataset Construction and Validation for Driving-Related Anger: A Wearable Physiological Conduction and Vehicle Driving Data Approach

Anger impairs a driver’s control and risk assessment abilities, heightening traffic accident risks. Constructing a multimodal dataset during driving tasks is crucial for accurate anger recognition. This study developed a multimodal physiological -vehicle driving dataset (DPV-MFD) based on drivers’ self-reported anger during simulated driving tasks. In Experiment 1, responses from 624 participants to anger-inducing videos and driving scenarios were collected via questionnaires to select appropriate materials. In Experiments 2 and 3, multimodal dynamic data and self-reported SAM emotion ratings were collected during simulated and real-vehicle tasks, capturing physiological and vehicle responses in neutral and anger states. Spearman’s correlation coefficient analysis validated the DPV-MFD’s effectiveness and explored the relationships between multimodal data and emotional dimensions. The CNN-LSTM deep learning network was used to assess the emotion recognition performance of the DPV-MFD across different time windows, and its applicability in real-world driving scenarios was validated. Compared to using EEG data alone, integrating multimodal data significantly improved anger recognition accuracy, with accuracy and F1 scores rising by 4.49% and 9.14%, respectively. Additionally, real-vehicle data closely matched simulated data, confirming the dataset’s effectiveness for real-world applications. This research is pivotal for advancing emotion-aware human–machine- interaction and intelligent transportation systems.

A Survey of Robot Intelligence with Large Language Models

Since the emergence of ChatGPT, research on large language models (LLMs) has actively progressed across various fields. LLMs, pre-trained on vast text datasets, have exhibited exceptional abilities in understanding natural language and planning tasks. These abilities of LLMs are promising in robotics. In general, traditional supervised learning-based robot intelligence systems have a significant lack of adaptability to dynamically changing environments. However, LLMs help a robot intelligence system to improve its generalization ability in dynamic and complex real-world environments. Indeed, findings from ongoing robotics studies indicate that LLMs can significantly improve robots’ behavior planning and execution capabilities. Additionally, vision-language models (VLMs), trained on extensive visual and linguistic data for the vision question answering (VQA) problem, excel at integrating computer vision with natural language processing. VLMs can comprehend visual contexts and execute actions through natural language. They also provide descriptions of scenes in natural language. Several studies have explored the enhancement of robot intelligence using multimodal data, including object recognition and description by VLMs, along with the execution of language-driven commands integrated with visual information. This review paper thoroughly investigates how foundation models such as LLMs and VLMs have been employed to boost robot intelligence. For clarity, the research areas are categorized into five topics: reward design in reinforcement learning, low-level control, high-level planning, manipulation, and scene understanding. This review also summarizes studies that show how foundation models, such as the Eureka model for automating reward function design in reinforcement learning, RT-2 for integrating visual data, language, and robot actions in vision-language-action models, and AutoRT for generating feasible tasks and executing robot behavior policies via LLMs, have improved robot intelligence.

Developing and Validating a Multimodal Dataset for Neonatal Pain Assessment to Improve AI Algorithms With Clinical Data.

Using Artificial Intelligence (AI) for neonatal pain assessment has great potential, but its effectiveness depends on accurate data labeling. Therefore, precise and reliable neonatal pain datasets are essential for managing neonatal pain. To develop and validate a comprehensive multimodal dataset with accurately labeled clinical data, enhancing AI algorithms for neonatal pain assessment. An assessment team randomly selected healthy neonates for assessment using the Neonatal Pain, Agitation, and Sedation Scale. During painful procedures, 2 cameras recorded neonates' pain reactions on site. After 2weeks, assessors labeled the processed pain data on the EasyDL platform in a single-anonymized setting. The pain scores from the 4 single-modal data types were compared to the total pain scores derived from multimodal data. The On-Site Neonatal Pain Assessment completed using paper quality scales is referred to as OS-NPA, while the modality-data neonatal pain labeling performed using labeling software is MD-NPL. The intraclass correlation coefficient among the 4 single-modal groups ranged from 0.938 to 0.969. The overall pain intraclass correlation coefficient score was 0.99, with a Kappa statistic for pain grade agreement of 0.899. The goodness-of-fit for the linear regression models comparing the OS-NPA and MD-NPL for each assessor was greater than 0.96. MD-NPL represents a productive alternative to OS-NPA for neonatal pain assessment, and the validity of the data labels within the Multimodality Dataset for Neonatal Acute Pain has been validating. These findings offer reliable validation for algorithms designed to assess neonatal pain.

A multimodal learning machine framework for Alzheimer’s disease diagnosis based on neuropsychological and neuroimaging data

Alzheimer’s disease (AD) is the most prevalent form of dementia, with no current cure. Early screening and intervention are vital. In multimodal AD data, besides neuroimaging dimensions, neuropsychological tests based on cognitive domains also provide the clinical information for diagnosing AD. However, previous multimodal methods often fuse these neuropsychological test scores with other data, losing the rich clinical details inherent in each test, including videos, speech, images, and text. To address this, we propose a novel framework with an entropy-based polynomial dimension expansion function that restores this critical information by accurately calculating the optimal polynomial degree. Additionally, the proposed framework offers a series of cognitive-based Extreme Learning Machine (ELM) models to better utilize the detailed clinical insights from neuropsychological tests, reducing diagnostic redundancy and noise. Finally, we design a boosting ensemble strategy that combines diagnostic models from various dimensions and cognitive domains, automatically optimizing weights to enhance diagnostic accuracy. Tested on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset, our approach achieves over 98% accuracy and F1 scores, with no observed bias between mild cognitive impairment (MCI) and AD groups. Therefore, our framework can offer clinicians more logical recommendations for diagnosing and managing the disease.

Informative Data Selection With Uncertainty for Multimodal Object Detection.

Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multimodal data. This is mainly based on two reasons. Multimodal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multimodal data. To tackle this problem, we propose a universal uncertainty-aware multimodal fusion model. It adopts a multipipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2-D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multimodal fusion will provide further insights for future research.

Multimodal data-driven, vertical visualization prediction model for early prediction of atherosclerotic cardiovascular disease in patients with new-onset hypertension.

: Hypertension is an important contributing factor to atherosclerotic cardiovascular disease (ASCVD), and multiple risk factors, many of which are implicated in metabolic disorders, contribute to the cause of hypertension. Despite the promise of multimodal data-driven prediction model, no such prediction model was available to predict the risk of ASCVD in Chinese individuals with new-onset hypertension and no history of ASCVD. : A total of 514 patients were randomly allocated to training and verification cohorts (ratio, 7 : 3). We employed Boruta feature selection and conducted multivariate Cox regression analyses to identify variables associated with ASCVD in these patients, which were subsequently utilized for constructing the predictive model. The performance of prediction model was assessed in terms of discriminatory power (C-index), calibration (calibration curves), and clinical utility [decision curve analysis (DCA)]. : This model was derived from four clinical variables: 24-h SBP coefficient of variation, 24-h DBP coefficient of variation, urea nitrogen and the triglyceride-glucose (TyG) index. Bootstrapping with 500 iterations was conducted to adjust the C-indexes were C-index = 0.731, 95% confidence interval (CI) 0.620-0.794 and C-index: 0.799, 95% CI 0.677-0.892 in the training and verification cohorts, respectively. Calibration plots with 500 bootstrapping iterations exhibited a strong correlation between the predicted and observed occurrences of ASCVD in both the training and verification cohorts. DCA analysis confirmed the clinical utility of this prediction model. The constructed nomogram demonstrated significant additional prognostic utility for ASCVD, as evidenced by improvements in the C-index, net reclassification improvement, integrated discrimination improvement, and DCA compared with the overall ASCVD risk assessment. The developed longitudinal prediction model based on multimodal data can effectively predict ASCVD risk in individuals with an initial diagnosis of hypertension. : The trial was registered in the Chinese Clinical Trial Registry (ChiCTR2300074392).

Translation of MFL and UT data by using generative adversarial networks: A comparative study

Magnetic flux leakage (MFL) and ultrasonic testing (UT) are widely used in-line inspection technologies to detect corrosion defects along pipelines. The integration of MFL and UT data has the potential to provide complementary insights that facilitate a comprehensive assessment of pipeline integrity. However, due to the inherent dissimilarity with their underlying physical principles, these techniques yield notable disparities in signal characteristics, posing challenges in integrating these multimodal data. This study aims to establish a translation mapping between MFL and UT signals to achieve consistent physical interpretations across the two modalities. Thus, this study explored the feasibility of generative adversarial network (GAN) based models encompassing both supervised and unsupervised translation approaches contingent on the availability of aligned data. Furthermore, two translation modes, MFL-UT and UT-MFL, were analyzed, respectively, to understand the effectiveness of the translation direction. The experimental results demonstrate satisfactory performance for both aligned and unaligned data translation, with the UT-MFL translation direction yielding superior results. Overall, the translation approaches pave the way for future applications, especially in subsequent data analysis tasks such as registration, comparison, and fusion of multimodal data.

Optimized polycystic ovarian disease prognosis and classification using AI based computational approaches on multi-modality data

Polycystic Ovarian Disease or Polycystic Ovary Syndrome (PCOS) is becoming increasingly communal among women, owing to poor lifestyle choices. According to the research conducted by National Institutes of Health, it has been observe that PCOS, an endocrine condition common in women of childbearing age, has become a significant contributing factor to infertility. Ovarian abnormalities brought on by PCOS carry a high risk of miscarriage, infertility, cardiac problems, diabetes, uterine cancer, etc. Ovarian cysts, obesity, menstrual irregularities, elevated amounts of male hormones, acne vulgaris, hair loss, and hirsutism are some of the symptoms of PCOS. It is not easy to determine PCOS because of its different combinations of symptoms in different women and various criteria needed for diagnosis. Taking biochemical tests and ovary scanning is a time-consuming process and the financial expenses have become a hardship to the patients. Thus, early prognosis of PCOS is crucial to avoid infertility. The goal of the proposed work is to analyse PCOS symptoms based on clinical data for early diagnosis and to classify into PCOS affected or not. To achieve this objective, clinical features dataset and ultrasound imaging dataset from Kaggle is utilized. Initially 541 instances of 45 clinical features such as testosterone, hirsutism, family history, BMI, fast food, menstrual disorder, risk etc. are considered and correlation-based feature extraction method is applied to this dataset which results in 17 features. The extracted features are applied to various machine learning algorithms such as Logistic Regression, Naïve Bayes and Support Vector Machine. The performance of each method is evaluated based on accuracy, precision, recall, F1-score and the result shows that among three models, Support Vector Machine model achieved high accuracy of 94.44%. In addition to this, 3856 ultrasound images are analysed by CNN based deep learning algorithm and VGG16 transfer learning algorithm. The performance of these models is evaluated using training accuracy, loss and validation accuracy, loss and the result depicts that VGG16 outperforms than CNN model with validation accuracy of 98.29%.

HeteroKGRep: Heterogeneous Knowledge Graph based Drug Repositioning

The process of developing new drugs is both time-consuming and costly, often taking over a decade and billions of dollars to obtain regulatory approval. Additionally, the complexity of patent protection for novel compounds presents challenges for pharmaceutical innovation. Drug repositioning offers an alternative strategy to uncover new therapeutic uses for existing medicines. Previous repositioning models have been limited by their reliance on homogeneous data sources, failing to leverage the rich information available in heterogeneous biomedical knowledge graphs. We propose HeteroKGRep, a novel drug repositioning model that utilizes heterogeneous graphs to address these limitations. HeteroKGRep is a multi-step framework that first generates a similarity graph from hierarchical concept relations. It then applies SMOTE over-sampling to address class imbalance before generating node sequences using a heterogeneous graph neural network. Drug and disease embeddings are extracted from the network and used for prediction. We evaluated HeteroKGRep on a graph containing biomedical concepts and relations from ontologies, pathways and literature. It achieved state-of-the-art performance with 99% accuracy, 95% AUC ROC and 94% average precision on predicting repurposing opportunities. Compared to existing homogeneous approaches, HeteroKGRep leverages diverse knowledge sources to enrich representation learning. Based on heterogeneous graphs, HeteroKGRep can discover new drug-disease associations, leveraging de novo drug development. This work establishes a promising new paradigm for knowledge-guided drug repositioning using multimodal biomedical data.

Toward Explainable Affective Computing: A Review.

Affective computing has an unprecedented potential to change the way humans interact with technology. While the last decades have witnessed vast progress in the field, multimodal affective computing systems are generally black box by design. As affective systems start to be deployed in real-world scenarios, such as education or healthcare, a shift of focus toward improved transparency and interpretability is needed. In this context, how do we explain the output of affective computing models? and how to do so without limiting predictive performance? In this article, we review affective computing work from an explainable AI (XAI) perspective, collecting and synthesizing relevant papers into three major XAI approaches: premodel (applied before training), in-model (applied during training), and postmodel (applied after training). We present and discuss the most fundamental challenges in the field, namely, how to relate explanations back to multimodal and time-dependent data, how to integrate context and inductive biases into explanations using mechanisms such as attention, generative modeling, or graph-based methods, and how to capture intramodal and cross-modal interactions in post hoc explanations. While explainable affective computing is still nascent, existing methods are promising, contributing not only toward improved transparency but, in many cases, surpassing state-of-the-art results. Based on these findings, we explore directions for future research and discuss the importance of data-driven XAI and explanation goals, and explainee needs definition, as well as causability or the extent to which a given method leads to human understanding.

Advances and prospects of multi-modal ophthalmic artificial intelligence based on deep learning: a review.

In recent years, ophthalmology has emerged as a new frontier in medical artificial intelligence (AI) with multi-modal AI in ophthalmology garnering significant attention across interdisciplinary research. This integration of various types and data models holds paramount importance as it enables the provision of detailed and precise information for diagnosing eye and vision diseases. By leveraging multi-modal ophthalmology AI techniques, clinicians can enhance the accuracy and efficiency of diagnoses, and thus reduce the risks associated with misdiagnosis and oversight while also enabling more precise management of eye and vision health. However, the widespread adoption of multi-modal ophthalmology poses significant challenges. In this review, we first summarize comprehensively the concept of modalities in the field of ophthalmology, the forms of fusion between modalities, and the progress of multi-modal ophthalmic AI technology. Finally, we discuss the challenges of current multi-modal AI technology applications in ophthalmology and future feasible research directions. In the field of ophthalmic AI, evidence suggests that when utilizing multi-modal data, deep learning-based multi-modal AI technology exhibits excellent diagnostic efficacy in assisting the diagnosis of various ophthalmic diseases. Particularly, in the current era marked by the proliferation of large-scale models, multi-modal techniques represent the most promising and advantageous solution for addressing the diagnosis of various ophthalmic diseases from a comprehensive perspective. However, it must be acknowledged that there are still numerous challenges associated with the application of multi-modal techniques in ophthalmic AI before they can be effectively employed in the clinical setting.

Detecting Novel Malware Classes with a Foundational Multi-Modality Data Analysis Model

Detecting Novel Malware Classes with a Foundational Multi-Modality Data Analysis Model

IPCT-Net: Parallel information bottleneck modality fusion network for obstructive sleep apnea diagnosis

Obstructive sleep apnea (OSA) is a common sleep breathing disorder and timely diagnosis helps to avoid the serious medical expenses caused by related complications. Existing deep learning (DL)-based methods primarily focus on single-modal models, which cannot fully mine task-related representations. This paper develops a modality fusion representation enhancement (MFRE) framework adaptable to flexible modality fusion types with the objective of improving OSA diagnostic performance, and providing quantitative evidence for clinical diagnostic modality selection. The proposed parallel information bottleneck modality fusion network (IPCT-Net) can extract local-global multi-view representations and eliminate redundant information in modality fusion representations through branch sharing mechanisms. We utilize large-scale real-world home sleep apnea test (HSAT) multimodal data to comprehensively evaluate relevant modality fusion types. Extensive experiments demonstrate that the proposed method significantly outperforms existing methods in terms of participant numbers and OSA diagnostic performance. The proposed MFRE framework delves into modality fusion in OSA diagnosis and contributes to enhancing the screening performance of artificial intelligence (AI)-assisted diagnosis for OSA.

Data Harmonization, Standardization, and Collaboration for Diabetic Retinal Disease (DRD) Research: Report From the 2024 Mary Tyler Moore Vision Initiative Workshop on Data.

The 2024 Mary Tyler Moore Vision Initiative (MTM Vision) Workshop on Data convened to discuss best practices and specific considerations for building a comprehensive, shareable MTM Vision data lake. The workshop aimed to accelerate the development of new indications, therapies, and regulatory pathways for diabetic retinal disease (DRD) by standardizing and harmonizing clinical data and ocular 'omics analyses. Standardization of data collection, the use of common data elements, and data interoperability were emphasized, alongside federated learning approaches to promote data sharing and collaboration while maintaining data privacy and security. The integration of molecular data with other multimodal data types was recognized as a promising strategy for leveraging machine learning and AI approaches to advancing therapeutics development and improving treatment outcomes for DRD patients. Partnerships with entities such as the National Eye Institute, part of the National Institutes of Health, foundations, and industry were deemed vital for the successful implementation of these initiatives.

Machine learning-based prediction of new onset of atrial fibrillation after mitral valve surgery

Abstract Background New-onset postoperative atrial fibrillation (nPOAF) is a common complication after cardiac surgery (30–50%), being associated with unfavorable long-term outcomes. Using the Society of Thoracic Surgeons National Adult Cardiac Database, we used machine learning (ML) to predict nPOAF and related 30-day outcomes following mitral valve (MV) surgery. A total of 27,856 MV operations were performed at 910 centers between 7/1/2017 and 6/30/2020 on patients without AF or a prior permanent pacemaker. The primary endpoint was nPOAF postoperatively. ML techniques utilized included penalized logistic regression, gradient boosting, decision trees, and random forests. Results The overall incidence of nPOAF was 35.4% and that of new pacemaker insertion was 5.6%. Patients who developed nPOAF were older (67 ± 10 vs 60 ± 13 years), had more mitral valve stenosis (14.1% vs 11.7%), and hypertension (72.1% vs 63.3%). They underwent more mitral valve replacement (39.1% vs 32.7%) and coronary artery bypass grafting (23.9% vs 16%). For predicting nPOAF, ML methods offer sensitivity, specificity and precision superior to logistic regression. The accuracy rate was identical with penalized and non-penalized logistic regression (0.672). Conclusions Predicting nPOAF and its short-term sequelae following MV surgery remains highly challenging. Machine learning methods offer a moderate degree of improvement in predicting nPOAF even in large national-level studies, in the absence of multi-modal data, such as real-time wearables data, electrocardiograms, heart rhythm monitoring, or cardiac imaging.

Real-Time Forecasting of Intradialytic Hypotension Using Deep Learning and Multimodal Data Integration

Real-Time Forecasting of Intradialytic Hypotension Using Deep Learning and Multimodal Data Integration

Discriminative Deep Canonical Correlation Analysis for Multi-View Data.

Over the past few years, multimodal data analysis has emerged as an inevitable method for identifying sample categories. In the multi-view data classification problem, it is expected that the joint representation should include the supervised information of sample categories so that the similarity in the latent space implies the similarity in the corresponding concepts. Since each view has different statistical properties, the joint representation should be able to encapsulate the underlying nonlinear data distribution of the given observations. Another important aspect is the coherent knowledge of the multiple views. It is required that the learning objective of the multi-view model efficiently captures the nonlinear correlated structures across different modalities. In this context, this article introduces a novel architecture, termed discriminative deep canonical correlation analysis (D2CCA), for classifying given observations into multiple categories. The learning objective of the proposed architecture includes the merits of generative models to identify the underlying probability distribution of the given observations. In order to improve the discriminative ability of the proposed architecture, the supervised information is incorporated into the learning objective of the proposed model. It also enables the architecture to serve as both a feature extractor as well as a classifier. The theory of CCA is integrated with the objective function so that the joint representation of the multi-view data is learned from maximally correlated subspaces. The proposed framework is consolidated with corresponding convergence analysis. The efficacy of the proposed architecture is studied on different domains of applications, namely, object recognition, document classification, multilingual categorization, face recognition, and cancer subtype identification with reference to several state-of-the-art methods.

Joint self-supervised and supervised contrastive learning for multimodal MRI data: Towards predicting abnormal neurodevelopment

The integration of different imaging modalities, such as structural, diffusion tensor, and functional magnetic resonance imaging, with deep learning models has yielded promising outcomes in discerning phenotypic characteristics and enhancing disease diagnosis. The development of such a technique hinges on the efficient fusion of heterogeneous multimodal features, which initially reside within distinct representation spaces. Naively fusing the multimodal features does not adequately capture the complementary information and could even produce redundancy. In this work, we present a novel joint self-supervised and supervised contrastive learning method to learn the robust latent feature representation from multimodal MRI data, allowing the projection of heterogeneous features into a shared common space, and thereby amalgamating both complementary and analogous information across various modalities and among similar subjects. We performed a comparative analysis between our proposed method and alternative deep multimodal learning approaches. Through extensive experiments on two independent datasets, the results demonstrated that our method is significantly superior to several other deep multimodal learning methods in predicting abnormal neurodevelopment. Our method has the capability to facilitate computer-aided diagnosis within clinical practice, harnessing the power of multimodal data. The source code of the proposed model is publicly accessible on GitHub: https://github.com/leonzyzy/Contrastive-Network.