Single-mode Model Research Articles

Transformer-based fusion model for mild depression recognition with EEG and pupil area signals.

Early detection and treatment are crucial for the prevention and treatment of depression; compared with major depression, current researches pay less attention to mild depression. Meanwhile, analysis of multimodal biosignals such as EEG, eye movement data, and magnetic resonance imaging provides reliable technical means for the quantitative analysis of depression. However, how to effectively capture relevant and complementary information between multimodal data so as to achieve efficient and accurate depression recognition remains a challenge. This paper proposes a novel Transformer-based fusion model using EEG and pupil area signals for mild depression recognition. We first introduce CSP into the Transformer to construct single-modal models of EEG and pupil data and then utilize attention bottleneck to construct a mid-fusion model to facilitate information exchange between the two modalities; this strategy enables the model to learn the most relevant and complementary information for each modality and only share the necessary information, which improves the model accuracy while reducing the computational cost. Experimental results show that the accuracy of the EEG and pupil area signals of single-modal models we constructed is 89.75% and 84.17%, the precision is 92.04% and 95.21%, the recall is 89.5% and 71%, the specificity is 90% and 97.33%, the F1 score is 89.41% and 78.44%, respectively, and the accuracy of mid-fusion model can reach 93.25%. Our study demonstrates that the Transformer model can learn the long-term time-dependent relationship between EEG and pupil area signals, providing an idea for designing a reliable multimodal fusion model for mild depression recognition based on EEG and pupil area signals.

Deep learning radiomics nomogram for preoperatively identifying moderate-to-severe chronic cholangitis in children with pancreaticobiliary maljunction: a multicenter study

BackgroundLong-term severe cholangitis can lead to dense adhesions and increased fragility of the bile duct, complicating surgical procedures and elevating operative risk in children with pancreaticobiliary maljunction (PBM). Consequently, preoperative diagnosis of moderate-to-severe chronic cholangitis is essential for guiding treatment strategies and surgical planning. This study aimed to develop and validate a deep learning radiomics nomogram (DLRN) based on contrast-enhanced CT images and clinical characteristics to preoperatively identify moderate-to-severe chronic cholangitis in children with PBM.MethodsA total of 323 pediatric patients with PBM who underwent surgery were retrospectively enrolled from three centers, and divided into a training cohort (n = 153), an internal validation cohort (IVC, n = 67), and two external test cohorts (ETC1, n = 58; ETC2, n = 45). Chronic cholangitis severity was determined by postoperative pathology. Handcrafted radiomics features and deep learning (DL) radiomics features, extracted using transfer learning with the ResNet50 architecture, were obtained from portal venous-phase CT images. Multivariable logistic regression was used to establish the DLRN, integrating significant clinical factors with handcrafted and DL radiomics signatures. The diagnostic performances were evaluated in terms of discrimination, calibration, and clinical usefulness.ResultsBiliary stones and peribiliary fluid collection were selected as important clinical factors. 5 handcrafted and 5 DL features were retained to build the two radiomics signatures, respectively. The integrated DLRN achieved satisfactory performance, achieving area under the curve (AUC) values of 0.913 (95% CI, 0.834–0.993), 0.916 (95% CI, 0.845–0.987), and 0.895 (95% CI, 0.801–0.989) in the IVC, and two ETCs, respectively. In comparison, the clinical model, handcrafted signature, and DL signature had AUC ranges of 0.654–0.705, 0.823–0.857, and 0.840–0.872 across the same cohorts. The DLRN outperformed single-modality clinical, handcrafted radiomics, and DL radiomics models, with all integrated discrimination improvement values > 0 and P < 0.05. The Hosmer–Lemeshow test and calibration curves showed good consistency of the DLRN (P > 0.05), and the decision curve analysis and clinical impact curve further confirmed its clinical utility.ConclusionsThe integrated DLRN can be a useful and non-invasive tool for preoperatively identifying moderate-to-severe chronic cholangitis in children with PBM, potentially enhancing clinical decision-making and personalized management strategies.

Identification of Parkinson’s disease using MRI and genetic data from the PPMI cohort: an improved machine learning fusion approach

ObjectiveThis study aim to leverage advanced machine learning techniques to develop and validate novel MRI imaging features and single nucleotide polymorphism (SNP) gene data fusion methodologies to enhance the early identification and diagnosis of Parkinson’s disease (PD).MethodsWe leveraged a comprehensive dataset from the Parkinson’s Progression Markers Initiative (PPMI), which includes high-resolution neuroimaging data, genetic single-nucleotide polymorphism (SNP) profiles, and detailed clinical information from individuals with early-stage PD and healthy controls. Two multi-modal fusion strategies were used: feature-level fusion, where we employed a hybrid feature selection algorithm combining Fisher discriminant analysis, an ensemble Lasso (EnLasso) method, and partial least squares (PLS) regression to identify and integrate the most informative features from neuroimaging and genetic data; and decision-level fusion, where we developed an adaptive ensemble stacking (AE_Stacking) model to synergistically integrate the predictions from multiple base classifiers trained on individual modalities.ResultsThe AE_Stacking model achieving the highest average balanced accuracy of 95.36% and an area under the receiver operating characteristic curve (AUC) of 0.974, significantly outperforming feature-level fusion and single-modal models (p &amp;lt; 0.05). Furthermore, by analyzing the features selected across multiple iterations of our models, we identified stable brain region features [lh 6r (FD) and rh 46 (GI)] and key genetic markers (rs356181 and rs2736990 SNPs within the SNCA gene region; rs213202 SNP within the VPS52 gene region), highlighting their potential as reliable early diagnostic indicators for the disease.ConclusionThe AE_Stacking model, trained on MRI and genetic data, demonstrates potential in distinguishing individuals with PD. Our findings enhance understanding of the disease and advance us toward the goal of precision medicine for neurodegenerative disorder.

Optimizing Disease Detection: A Multi-Modal Deep Learning Framework for Medical Imaging and Clinical Data Integration

This study introduces a novel multi-modal deep learning framework that integrates medical imaging data with clinical records for enhanced disease detection. We propose a hybrid architecture combining convolutional neural networks (CNNs) for image analysis and transformer networks for processing clinical data. The framework was evaluated on a dataset of 10,000 patients over 12 months, focusing on detecting early signs of lung cancer and coronary artery disease. Results show our integrated approach achieves significantly higher accuracy compared to single-modality models, with an F1 score of 0.89 (95% CI: 0.87-0.91, p &lt; 0.001). We also introduce a novel interpretability metric for multi-modal models and demonstrate a 30% improvement in model explainability. These findings suggest our approach can enhance diagnostic accuracy while maintaining interpretability in clinical settings.

An explainable transformer model integrating PET and tabular data for histologic grading and prognosis of follicular lymphoma: a multi-institutional digital biopsy study.

Pathological grade is a critical determinant of clinical outcomes and decision-making of follicular lymphoma (FL). This study aimed to develop a deep learning model as a digital biopsy for the non-invasive identification of FL grade. This study retrospectively included 513 FL patients from five independent hospital centers, randomly divided into training, internal validation, and external validation cohorts. A multimodal fusion Transformer model was developed integrating 3D PET tumor images with tabular data to predict FL grade. Additionally, the model is equipped with explainable modules, including Gradient-weighted Class Activation Mapping (Grad-CAM) for PET images, SHapley Additive exPlanations analysis for tabular data, and the calculation of predictive contribution ratios for both modalities, to enhance clinical interpretability and reliability. The predictive performance was evaluated using the area under the receiver operating characteristic curve (AUC) and accuracy, and its prognostic value was also assessed. The Transformer model demonstrated high accuracy in grading FL, with AUCs of 0.964-0.985 and accuracies of 90.2-96.7% in the training cohort, and similar performance in the validation cohorts (AUCs: 0.936-0.971, accuracies: 86.4-97.0%). Ablation studies confirmed that the fusion model outperformed single-modality models (AUCs: 0.974 - 0.956, accuracies: 89.8%-85.8%). Interpretability analysis revealed that PET images contributed 81-89% of the predictive value. Grad-CAM highlighted the tumor and peri-tumor regions. The model also effectively stratified patients by survival risk (P < 0.05), highlighting its prognostic value. Our study developed an explainable multimodal fusion Transformer model for accurate grading and prognosis of FL, with the potential to aid clinical decision-making.

Construction of an Artificially Intelligent Model for Accurate Detection of HCC by Integrating Clinical, Radiological, and Peripheral Immunological Features.

Integrating comprehensive information on hepatocellular carcinoma (HCC) is essential to improve its early detection. We aimed to develop a model with multi-modal features (MMF) using artificial intelligence (AI) approaches to enhance the performance of HCC detection. A total of 1,092 participants were enrolled from 16 centers. These participants were allocated into the training, internal validation, and external validation cohorts. Peripheral blood specimens were collected prospectively and subjected to mass cytometry analysis. Clinical and radiological data were obtained from electrical medical records. Various AI methods were employed to identify pertinent features and construct single-modal models with optimal performance. The XGBoost algorithm was utilized to amalgamate these models, integrating multi-modal information and facilitating the development of a fusion model. Model evaluation and interpretability were demonstrated using the SHapley Additive exPlanations method. We constructed the electronic health record, BioScore, RadiomicScore, and DLScore models based on clinical, radiological, and peripheral immunological features, respectively. Subsequently, these single-modal models were amalgamated to develop an all-in-one MMF model. The MMF model exhibited enhanced performance compared to models comprising only single-modal features in detecting HCC. This superiority in performance was confirmed through the internal and external validation cohorts, yielding area under the curve (AUC) values of 0.985 and 0.915, respectively. Additionally, the MMF model improved the detection ability in subpopulations of HCCs that were negative for alpha-fetoprotein and those with small size, with AUC values of 0.974 and 0.996, respectively. Integrating multi-modal features improved the performance of the model for HCC detection.

Multimodal Pain Recognition in Postoperative Patients: Machine Learning Approach.

Acute pain management is critical in postoperative care, especially in vulnerable patient populations that may be unable to self-report pain levels effectively. Current methods of pain assessment often rely on subjective patient reports or behavioral pain observation tools, which can lead to inconsistencies in pain management. Multimodal pain assessment, integrating physiological and behavioral data, presents an opportunity to create more objective and accurate pain measurement systems. However, most previous work has focused on healthy subjects in controlled environments, with limited attention to real-world postoperative pain scenarios. This gap necessitates the development of robust, multimodal approaches capable of addressing the unique challenges associated with assessing pain in clinical settings, where factors like motion artifacts, imbalanced label distribution, and sparse data further complicate pain monitoring. This study aimed to develop and evaluate a multimodal machine learning-based framework for the objective assessment of pain in postoperative patients in real clinical settings using biosignals such as electrocardiogram, electromyogram, electrodermal activity, and respiration rate (RR) signals. The iHurt study was conducted on 25 postoperative patients at the University of California, Irvine Medical Center. The study captured multimodal biosignals during light physical activities, with concurrent self-reported pain levels using the Numerical Rating Scale. Data preprocessing involved noise filtering, feature extraction, and combining handcrafted and automatic features through convolutional and long-short-term memory autoencoders. Machine learning classifiers, including support vector machine, random forest, adaptive boosting, and k-nearest neighbors, were trained using weak supervision and minority oversampling to handle sparse and imbalanced pain labels. Pain levels were categorized into baseline and 3 levels of pain intensity (1-3). The multimodal pain recognition models achieved an average balanced accuracy of over 80% across the different pain levels. RR models consistently outperformed other single modalities, particularly for lower pain intensities, while facial muscle activity (electromyogram) was most effective for distinguishing higher pain intensities. Although single-modality models, especially RR, generally provided higher performance compared to multimodal approaches, our multimodal framework still delivered results that surpassed most previous works in terms of overall accuracy. This study presents a novel, multimodal machine learning framework for objective pain recognition in postoperative patients. The results highlight the potential of integrating multiple biosignal modalities for more accurate pain assessment, with particular value in real-world clinical settings.

A preoperative predictive model based on multi-modal features to predict pathological complete response after neoadjuvant chemoimmunotherapy in esophageal cancer patients

BackgroundThis study aimed to develop a multi-modality model by incorporating pretreatment computed tomography (CT) radiomics and pathomics features along with clinical variables to predict pathologic complete response (pCR) to neoadjuvant chemoimmunotherapy in patients with locally advanced esophageal cancer (EC).MethodA total of 223 EC patients who underwent neoadjuvant chemoimmunotherapy followed by surgical intervention between August 2021 and December 2023 were included in this study. Radiomics features were extracted from contrast-enhanced CT images using PyrRadiomics, while pathomics features were derived from whole-slide images (WSIs) of pathological specimens using a fine-tuned deep learning model (ResNet-50). After feature selection, three single-modality prediction models and a combined multi-modality model integrating two radiomics features, 11 pathomics features, and two clinicopathological features were constructed using the support vector machine (SVM) algorithm. The performance of the models were evaluated using receiver operating characteristic (ROC) analysis, calibration plots, and decision curve analysis (DCA). Shapley values were also utilized to explain the prediction model.ResultsThe predictive capability of the multi-modality model in predicting pCR yielded an area under the curve (AUC) of 0.89 (95% confidence interval [CI], 0.75-1.00), outperforming the radiomics model (AUC 0.70 [95% CI 0.54-0.85]), pathomics model (AUC 0.77 [95% CI 0.53-1.00]), and clinical model (AUC 0.63 [95% CI 0.46-0.80]). Additionally, both the calibration plot and DCA curves support the clinical utility of the integrated multi-modality model.ConclusionsThe combined multi-modality model we propose can better predict the pCR status of esophageal cancer and help inform clinical treatment decisions.

Multimodal integration using a machine learning approach facilitates risk stratification in HR+/HER2- breast cancer.

Hormone receptor-positive (HR+)/human epidermal growth factor receptor 2-negative (HER2-) breast cancer is the most common type of breast cancer, with continuous recurrence remaining an important clinical issue. Current relapse predictive models in HR+/HER2- breast cancer patients still have limitations. The integration of multidimensional data represents a promising alternative for predicting relapse. In this study, we leverage our multi-omics cohort comprising 579 HR+/HER2- breast cancer patients (200 patients with complete data across 7 modalities) and develop a machine-learning-based model, namely CIMPTGV, which integrates clinical information, immunohistochemistry, metabolomics, pathomics, transcriptomics, genomics, and copy number variations to predict recurrence risk of HR+/HER2- breast cancer. This model achieves concordance indices (C-indices) of 0.871 and 0.869 in the train and test sets, respectively. The risk population predicted by the CIMPTGV model encompasses those identified by single-modality models. Feature analysis reveals that synergistic and complementary effects exist in different modalities. Simultaneously, we develop a simplified model with a mean area under the curve (AUC) of 0.840, presenting a useful approach for clinical applications.

Integrating genetics, metabolites, and clinical characteristics in predicting cardiometabolic health outcomes using machine learning algorithms - A systematic review.

Machine learning (ML) integration of clinical, metabolite, and genetic data reveals variable results in predicting cardiometabolic health (CMH) outcomes. Therefore, we aim to (1) evaluate whether a multi-modal approach incorporating all three data types using ML algorithms can improve CMH outcome prediction compared to single-modal or paired-modal models, and (2) compare the methodologies used in existing prediction models. We systematically searched five databases from 1998 to 2024 for ML predictive modelling studies using the multi-modal approach for CMH outcomes. Risk-of-bias assessment tools were used to assess methodological quality. Study characteristics, ML algorithms, data preprocessing, evaluation methods and metrics, feature selections, and feature importance parameters were synthesized narratively to show methodological heterogeneity. Of the four included studies (3ML algorithms), three were at low risk of bias, and one was at high risk. The multi-modal approach consistently improved T2D and BP prediction compared to single-modal or paired-modal models. Genetics showed the lowest predictive performance in three studies. Logistic regression (n=2 studies) and random forest (n=1) were used in T2D studies, while XGBoost was used in one BP study. One study with missing data and variations in feature selection across all studies hindered a comprehensive comparison of feature importance. Our review emphasizes the potential improvement in T2D and BP prediction using ML algorithms with the multi-modal approach. However, further studies using diverse ML algorithms with optimized methodologies on single-modal, paired-modal, and multi-modal models are needed to gain insights into biomarker selection for predicting CMH outcomes.

A Multimodal Model Based on Transvaginal Ultrasound-Based Radiomics to Predict the Risk of Peritoneal Metastasis in Ovarian Cancer: A Multicenter Study.

This study aimed to develop a predictive model for peritoneal metastasis (PM) in ovarian cancer using a combination radiomics and clinical biomarkers to improve diagnostic accuracy. This retrospective cohort study of 619 ovarian cancer patients involved demographic data, radiomics, O-RADS standardized description, clinical biomarkers, and histological findings. Radiomics features were extracted using 3D Slicer and Pyradiomics, with selective feature extraction using Least Absolute Shrinkage and Selection Operator regression. Model development and validation were carried out using logistic regression and machine learning methods RESULTS: Interobserver agreement was high for radiomics features, with 1049 features initially extracted and 7 features selected through regression analysis. Multi-modal information such as Ascites, Fallopian tube invasion, Greatest diameter, HE4 and D-dimer levels were significant predictors of PM. The developed radiomics nomogram demonstrated strong discriminatory power, with AUC values of 0.912, 0.883, and 0.831 in the training, internal test, and external test sets respectively. The nomogram displayed superior diagnostic performance compared to single-modality models. The integration of multimodal information in a predictive model for PM in ovarian cancer shows promise for enhancing diagnostic accuracy and guiding personalized treatment. This multi-modal approach offers a potential strategy for improving patient outcomes in ovarian cancer management with PM.

Multi-Modal Fusion Techniques for Improved Diagnosis in Medical Imaging

Identifying diverse disease states is crucial for prompt and efficient clinical management. Complementary data from many medical imaging modalities, including MRI, CT, and PET, can be integrated to improve diagnostic performance. This work aims to assess how well multi-modal fusion methods work to enhance medical picture diagnosis. A multicenter study was conducted with 150 patients with different clinical conditions (mean age 58.2 ± 12.4 years, 52% female). After gathering data from MRI, CT, and PET scans, structural, functional, and textural characteristics were removed from each modality. The three fusion strategies studied were fusion through concatenation, fusion through kernels, and fusion through attention. The fused features were used to train classification models such as Convolutional Neural Networks (CNNs), ensemble techniques, and Support Vector Machines (SVMs). ROC analysis was utilized to assess the diagnostic performance. The multi-modal fusion techniques outperformed the single-modality methods in diagnosing performance. Attention-based fusion yielded the top AUCs of 0.92, 0.89, and 0.91 for brain tumors, neurodegenerative diseases, and cardiovascular conditions, respectively. This significantly improved (p&lt;0.05) compared to the AUC of the best single-modality models. Multi-modal fusion methods are powerful for combining data from various imaging modalities to improve diagnostic accuracy for various medical conditions. These findings highlight the advantages of combining information sources to improve clinical judgment and patient care.

A Modified Regular Perturbation Model for the Single-Span Fiber Transmission Using Learnable Methods

In fiber optic communication systems, the dispersion and nonlinear interaction of optical signals are critical to modeling fiber optic communication, and the regular perturbation (RP) model is a simplified modeling method composed of parallel branches, which has obvious advantages in deep learning backpropagation. In this paper, we propose a simplified single-mode fiber signal transmission model based on the RP model, which significantly improves the fitting accuracy of the model for dispersion and nonlinear interactions at the same complexity by adding trainable parameters to the standard RP model. We explain in the paper that this improvement is applicable to dual-polarization systems and still effective under the conditions of large launch power, without dispersion management, and containing amplified spontaneous emission (ASE) noise. The model uses the standard split-step Fourier method (SSFM) to generate labels and updates parameters through gradient descent method. When transmitting a dual-polarization signal with a launch power of 13 dBm, the modified regular perturbation (MRP) model proposed in the paper can reduce the fitting errors by more than 75% compared to the standard RP model after transmitting through a 120 km standard single-mode fiber.

Multi-Modal Machine Learning Approach for COVID-19 Detection Using Biomarkers and X-Ray Imaging.

Background: Accurate and rapid detection of COVID-19 remains critical for clinical management, especially in resource-limited settings. Current diagnostic methods face challenges in terms of speed and reliability, creating a need for complementary AI-based models that integrate diverse data sources. Objectives: This study aimed to develop and evaluate a multi-modal machine learning model that combines clinical biomarkers and chest X-ray images to enhance diagnostic accuracy and provide interpretable insights. Methods: We used a dataset of 250 patients (180 COVID-19 positive and 70 negative cases) collected from clinical settings. Biomarkers such as CRP, ferritin, NLR, and albumin were included alongside chest X-ray images. Random Forest and Gradient Boosting models were used for biomarkers, and ResNet and VGG CNN architectures were applied to imaging data. A late-fusion strategy integrated predictions from these modalities. Stratified k-fold cross-validation ensured robust evaluation while preventing data leakage. Model performance was assessed using AUC-ROC, F1-score, Specificity, Negative Predictive Value (NPV), and Matthews Correlation Coefficient (MCC), with confidence intervals calculated via bootstrap resampling. Results: The Gradient Boosting + VGG fusion model achieved the highest performance, with an AUC-ROC of 0.94, F1-score of 0.93, Specificity of 93%, NPV of 96%, and MCC of 0.91. SHAP and LIME interpretability analyses identified CRP, ferritin, and specific lung regions as key contributors to predictions. Discussion: The proposed multi-modal approach significantly enhances diagnostic accuracy compared to single-modality models. Its interpretability aligns with clinical understanding, supporting its potential for real-world application.

Multi-Level Feature Fusion in CNN-Based Human Action Recognition: A Case Study on EfficientNet-B7.

Accurate human action recognition is becoming increasingly important across various fields, including healthcare and self-driving cars. A simple approach to enhance model performance is incorporating additional data modalities, such as depth frames, point clouds, and skeleton information, while previous studies have predominantly used late fusion techniques to combine these modalities, our research introduces a multi-level fusion approach that combines information at early, intermediate, and late stages together. Furthermore, recognizing the challenges of collecting multiple data types in real-world applications, our approach seeks to exploit multimodal techniques while relying solely on RGB frames as the single data source. In our work, we used RGB frames from the NTU RGB+D dataset as the sole data source. From these frames, we extracted 2D skeleton coordinates and optical flow frames using pre-trained models. We evaluated our multi-level fusion approach with EfficientNet-B7 as a case study, and our methods demonstrated significant improvement, achieving 91.5% in NTU RGB+D 60 dataset accuracy compared to single-modality and single-view models. Despite their simplicity, our methods are also comparable to other state-of-the-art approaches.

ResNet-Vision Transformer based MRI-endoscopy fusion model for predicting treatment response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer: A multicenter study.

Neoadjuvant chemoradiotherapy followed by radical surgery has been a common practice for patients with locally advanced rectal cancer, but the response rate varies among patients. This study aimed to develop a ResNet-Vision Transformer based magnetic resonance imaging (MRI)-endoscopy fusion model to precisely predict treatment response and provide personalized treatment. In this multicenter study, 366 eligible patients who had undergone neoadjuvant chemoradiotherapy followed by radical surgery at eight Chinese tertiary hospitals between January 2017 and June 2024 were recruited, with 2928 pretreatment colonic endoscopic images and 366 pelvic MRI images. An MRI-endoscopy fusion model was constructed based on the ResNet backbone and Transformer network using pretreatment MRI and endoscopic images. Treatment response was defined as good response or non-good response based on the tumor regression grade. The Delong test and the Hanley-McNeil test were utilized to compare prediction performance among different models and different subgroups, respectively. The predictive performance of the MRI-endoscopy fusion model was comprehensively validated in the test sets and was further compared to that of the single-modal MRI model and single-modal endoscopy model. The MRI-endoscopy fusion model demonstrated favorable prediction performance. In the internal validation set, the area under the curve (AUC) and accuracy were 0.852 (95% confidence interval [CI]: 0.744-0.940) and 0.737 (95% CI: 0.712-0.844), respectively. Moreover, the AUC and accuracy reached 0.769 (95% CI: 0.678-0.861) and 0.729 (95% CI: 0.628-0.821), respectively, in the external test set. In addition, the MRI-endoscopy fusion model outperformed the single-modal MRI model (AUC: 0.692 [95% CI: 0.609-0.783], accuracy: 0.659 [95% CI: 0.565-0.775]) and the single-modal endoscopy model (AUC: 0.720 [95% CI: 0.617-0.823], accuracy: 0.713 [95% CI: 0.612-0.809]) in the external test set. The MRI-endoscopy fusion model based on ResNet-Vision Transformer achieved favorable performance in predicting treatment response to neoadjuvant chemoradiotherapy and holds tremendous potential for enabling personalized treatment regimens for locally advanced rectal cancer patients.

Brain-Cognition Fingerprinting via Graph-GCCA with Contrastive Learning.

Many longitudinal neuroimaging studies aim to improve the understanding of brain aging and diseases by studying the dynamic interactions between brain function and cognition. Doing so requires accurate encoding of their multidimensional relationship while accounting for individual variability over time. For this purpose, we propose an unsupervised learning model (called Contrastive Learning-based Graph Generalized Canonical Correlation Analysis (CoGraCa)) that encodes their relationship via Graph Attention Networks and generalized Canonical Correlational Analysis. To create brain-cognition fingerprints reflecting unique neural and cognitive phenotype of each person, the model also relies on individualized and multimodal contrastive learning. We apply CoGraCa to longitudinal dataset of healthy individuals consisting of resting-state functional MRI and cognitive measures acquired at multiple visits for each participant. The generated fingerprints effectively capture significant individual differences and outperform current single-modal and CCA-based multimodal models in identifying sex and age. More importantly, our encoding provides interpretable interactions between those two modalities.

CaLiJD: Camera and LiDAR Joint Contender for 3D Object Detection

Three-dimensional object detection has been a key area of research in recent years because of its rich spatial information and superior performance in addressing occlusion issues. However, the performance of 3D object detection still lags significantly behind that of 2D object detection, owing to challenges such as difficulties in feature extraction and a lack of texture information. To address this issue, this study proposes a 3D object detection network, CaLiJD (Camera and Lidar Joint Contender for 3D object Detection), guided by two-dimensional detection results. CaLiJD creatively integrates advanced channel attention mechanisms with a novel bounding-box filtering method to improve detection accuracy, especially for small and occluded objects. Bounding boxes are detected by the 2D and 3D networks for the same object in the same scene as an associated pair. The detection results that satisfy the criteria are then fed into the fusion layer for training. In this study, a novel fusion network is proposed. It consists of numerous convolutions arranged in both sequential and parallel forms and includes a Grouped Channel Attention Module for extracting interactions among multi-channel information. Moreover, a novel bounding-box filtering mechanism was introduced, incorporating the normalized distance from the object to the radar as a filtering criterion within the process. Experiments were conducted using the KITTI 3D object detection benchmark. The results showed that a substantial improvement in mean Average Precision (mAP) was achieved by CaLiJD compared with the baseline single-modal 3D detection model, with an enhancement of 7.54%. Moreover, the improvement achieved by our method surpasses that of other classical fusion networks by an additional 0.82%. In particular, CaLiJD achieved mAP values of 73.04% and 59.86%, respectively, thus demonstrating state-of-the-art performance for challenging small-object detection tasks such as those involving cyclists and pedestrians.

Radiation Pneumonitis Prediction Using Dual-Modal Data Fusion Based on Med3D Transfer Network.

Radiation pneumonitis (RP) is an inflammatory reaction in the lungs caused by radiation therapy. The etiology of RP is complex and varied, making it challenging to establish accurate predictive models for RP. The aim of this study is to develop a dual-modal prediction model for RP using pre-radiotherapy CT images of the patient's lungs, along with clinical and dose data. In this paper, an RP prediction model utilizing dual-modal data is proposed. Firstly, for CT image data, the Med3D transfer network, trained on a large-scale medical dataset, is employed to extract CT image features. To adapt the transfer network for the RP task, the last convolutional block, which incorporates the 3D channel attention mechanism, is trained and saved as the optimal model for extracting deep features once training stabilizes. Subsequently, an autoencoder (AE) is employed to compress the deep features to reduce their dimensionality. Secondly, for clinical and dose features, univariate analysis and lasso regression are used to screen the features. Finally, the two groups of features are multiplied by their respective adaptive rates to achieve data fusion, and then input into the binary classification model for training and prediction. The KAN classifier with adaptive feature fusion demonstrates superior performance, achieving precision rates of 74% and 76%, a recall rate of 73%, and an AUC of 86%. These results surpass those obtained from single-modality approaches. The experimental results on a dataset of 117 chest cancer patients receiving radiotherapy show that the dual-modal data fusion model can predict RP more effectively than single-modality models.

Development of an AI model for predicting hypoxia status and prognosis in non-small cell lung cancer using multi-modal data.

Prognosis prediction is crucial for non-small cell lung cancer (NSCLC) treatment planning. While tumor hypoxia significantly impacts patient outcomes, identifying hypoxic genomic markers remains challenging. This study sought to identify hypoxic computed tomography (CT) radiomic features and create an artificial intelligence (AI) model for NSCLC through the integration of multi-modal data. In total, 452 NSCLC patients were enrolled in this study, including patients from The Second Affiliated Hospital of Soochow University (SC, n=112), The Cancer Genome Atlas (TCGA)-NSCLC dataset (n=74), the radiogenomics dataset (n=130), and the Gene Expression Omnibus (GEO) datasets (GSE19188: n=82, and GSE87340: n=54). Hypoxia status was classified using optimized cut-off values of hypoxia enrichment scores, which were calculated through single-sample gene set enrichment analysis (ssGSEA) of hypoxic genes. Radiomic features were extracted using three-dimensional (3D)-Slicer software. The least absolute shrinkage and selection operator (LASSO) algorithm was used to identify hypoxic CT radiomic features. A model named ssuBERT (semantic structured unit embedded in Bidirectional Encoder Representations from Transformers) was developed to analyze electronic health records (EHRs). An AI model for overall survival prediction was constructed by integrating CT radiomic features, ssuBERT features, and clinical data, and evaluated using five-fold cross-validation. Higher hypoxia levels were correlated with worse survival outcomes. Twenty-eight radiomic features showed significant discriminatory power in detecting hypoxia status with an area under the curve (AUC) of 0.8295. The ssuBERT model achieved a weighted accuracy of 0.945 in recognizing semantic structured units in EHRs. The EHR model exhibited superior predictive performance among the single-modal models with an AUC of 0.7662. However, the multi-modal AI model had the highest average AUC of 0.8449 and an F1 score of 0.7557. The AI model demonstrated potential in predicting NSCLC patient prognosis through multi-modal data integration, warranting further validation.