Multimodal Prediction Research Articles

Radiomics-based prediction of HCC response to atezolizumab/bevacizumab.

636 Background: Advanced hepatocellular carcinoma (HCC) treatment has evolved with the introduction of atezolizumab/bevacizumab, showing improved outcomes over sorafenib. However, the response varies among patients, particularly between viral and non-viral etiologies. This study aimed to develop and evaluate multimodal prediction models combining quantitative imaging and clinical markers to predict treatment response in HCC patients. Methods: From March 2020 to May 2023, patients with advanced HCC treated with atezolizumab/bevacizumab were retrospectively identified from six centers in Germany and Austria. Patients underwent baseline contrast-enhanced liver MRI and follow-up imaging to assess therapy response. Machine learning models, including RandomForestClassifier, were developed for radiomics, clinical, and combined datasets. Hyperparameter tuning was performed using RandomizedSearchCV, followed by cross-validation to evaluate model performance. Results: The study included 103 patients, with 70 achieving disease control (DC) and 33 experiencing disease progression (PD). Key findings included significant differences in treatment response and progression-free survival between DC and PD groups. The radiomics model, using 14 selected features, achieved 73.1% accuracy and a ROC AUC of 0.635 on the test set. The clinical model, with 4 selected features, achieved 73% accuracy and a ROC AUC of 0.649 on the test set. The combined model showed improved performance with 69% accuracy and a ROC AUC of 0.753 on the test set. Hyperparameter tuning further enhanced the combined model's accuracy to 80.1% and ROC AUC to 0.771 on the test set. Conclusions: The hybrid model combining clinical and radiological data outperformed individual models, providing better predictions of response to atezolizumab/bevacizumab in HCC patients.

Development of a universal multimodal prediction method to optimise process parameters for improving densification during laser powder bed fusion

ABSTRACT This work develops a universal machine learning method that enhances the densification of metallic materials fabricated by laser powder bed fusion (LPBF) by optimising process parameters using multimodal data. By integrating ResNet-50 neural network and LightGBM model, this method combines process parameters with surface morphology of LPBF-fabricated samples at the feature scale, enabling extraction of multidimensional feature information and improving prediction accuracy of relative density. Statistical analysis demonstrates exceptional performance of the model on both training and testing datasets, yielding coefficient of determination (R²) values of 0.9821 and 0.8169, and mean absolute error values of 0.0178 and 0.0108, respectively. Bayesian optimisation was employed using the multimodal model as a surrogate model to identify potential optimal combinations within the parameter space. These results validate the effectiveness of establishing a multimodal machine learning method based on multimodal data sources to optimise LPBF process parameters and improve relative density of fabricated new materials.

Multi-modal prediction of extracorporeal support-a resource intensive therapy, utilizing a large national database.

Extracorporeal membrane oxygenation (ECMO) is among the most resource-intensive therapies in critical care. The COVID-19 pandemic highlighted the lack of ECMO resource allocation tools. We aimed to develop a continuous ECMO risk prediction model to enhance patient triage and resource allocation. We leveraged multimodal data from the National COVID Cohort Collaborative (N3C) to develop a hierarchical deep learning model, labeled "PreEMPT-ECMO" (Prediction, Early Monitoring, and Proactive Triage for ECMO) which integrates static and multi-granularity time series features to generate continuous predictions of ECMO utilization. Model performance was assessed across time points ranging from 0 to 96hours prior to ECMO initiation, using both accuracy and precision metrics. Between January 2020 and May 2023, 101400 patients were included, with 1298 (1.28%) supported on ECMO. PreEMPT-ECMO outperformed established predictive models, including Logistic Regression, Support Vector Machine, Random Forest, and Extreme Gradient Boosting Tree, in both accuracy and precision at all time points. Model interpretation analysis also highlighted variations in feature contributions through each patient's clinical course. We developed a hierarchical model for continuous ECMO use prediction, utilizing a large multicenter dataset incorporating both static and time series variables of various granularities. This novel approach reflects the nuanced decision-making process inherent in ECMO initiation and has the potential to be used as an early alert tool to guide patient triage and ECMO resource allocation. Future directions include prospective validation and generalizability on non-COVID-19 refractory respiratory failure, aiming to improve patient outcomes.

Accurate and transferable drug-target interaction prediction with DrugLAMP.

Accurate prediction of drug-target interactions (DTIs), especially for novel targets or drugs, is crucial for accelerating drug discovery. Recent advances in pretrained language models (PLMs) and multi-modal learning present new opportunities to enhance DTI prediction by leveraging vast unlabeled molecular data and integrating complementary information from multiple modalities. We introduce DrugLAMP (PLM-assisted multi-modal prediction), a PLM-based multi-modal framework for accurate and transferable DTI prediction. DrugLAMP integrates molecular graph and protein sequence features extracted by PLMs and traditional feature extractors. We introduce two novel multi-modal fusion modules: (i) pocket-guided co-attention (PGCA), which uses protein pocket information to guide the attention mechanism on drug features, and (ii) paired multi-modal attention (PMMA), which enables effective cross-modal interactions between drug and protein features. These modules work together to enhance the model's ability to capture complex drug-protein interactions. Moreover, the contrastive compound-protein pre-training (2C2P) module enhances the model's generalization to real-world scenarios by aligning features across modalities and conditions. Comprehensive experiments demonstrate DrugLAMP's state-of-the-art performance on both standard benchmarks and challenging settings simulating real-world drug discovery, where test drugs/targets are unseen during training. Visualizations of attention maps and application to predict cryptic pockets and drug side effects further showcase DrugLAMP's strong interpretability and generalizability. Ablation studies confirm the contributions of the proposed modules. Source code and datasets are freely available at https://github.com/Lzcstan/DrugLAMP. All data originate from public sources.

Gut-Brain Nexus: Mapping Multi-Modal Links to Neurodegeneration at Biobank Scale.

Alzheimer's disease (AD) and Parkinson's disease (PD) are influenced by genetic and environmental factors. Using data from UK Biobank, SAIL Biobank, and FinnGen, we conducted an unbiased, population-scale study to: 1) Investigate how 155 endocrine, nutritional, metabolic, and digestive system disorders are associated with AD and PD risk prior to their diagnosis, considering known genetic influences; 2) Assess plasma biomarkers' specificity for AD or PD in individuals with these conditions; 3) Develop a multi-modal classification model integrating genetics, proteomics, and clinical data relevant to conditions affecting the gut-brain axis. Our findings show that certain disorders elevate AD and PD risk before AD and PD diagnosis including: insulin and non-insulin dependent diabetes mellitus, noninfective gastro-enteritis and colitis, functional intestinal disorders, and bacterial intestinal infections, among others. Polygenic risk scores revealed lower genetic predisposition to AD and PD in individuals with co-occurring disorders in the study categories, underscoring the importance of regulating the gut-brain axis to potentially prevent or delay the onset of neurodegenerative diseases. The proteomic profile of AD/PD cases was influenced by comorbid endocrine, nutritional, metabolic, and digestive systems conditions. Importantly, we developed multi-modal prediction models integrating clinical, genetic, proteomic and demographic data, the combination of which performs better than any single paradigm approach in disease classification. This work aims to illuminate the intricate interplay between various physiological factors involved in the gut-brain axis and the development of AD and PD, providing a multifactorial systemic understanding that goes beyond traditional approaches. Further, we have developed an interactive resource for the scientific community [https://gut-brain-nexus.streamlit.app/] where researchers can investigate components of the predictive model and can investigate feature effects on a sample level.

Radiomics-based multimodality prediction of left atrial appendage thrombus using coronary computed tomography angiography

Abstract Background Timely detection of thrombus in the left atrial appendage (LAA) may help guide decision-making. However, existing models have challenges in extracting comprehensive image information. It has been shown that radiomics-based prediction models can be used for accurate identification. This study investigates potential radiomic markers and constructs a highly efficient multimodality prediction model for LAA thrombus in patients undergoing coronary computed tomography angiography (CCTA). Methods We retrospectively enrolled patients with nonvalvular atrial fibrillation (NVAF) who underwent CCTA and transesophageal echocardiography (TEE) from May 2015 to May 2020. These patients were classified into thrombus or non-thrombus groups based on TEE findings. The data were balanced by resampling and assigned to training or test sets. Radiomics was conducted on the extracted features, and a random forest algorithm was employed for feature selection and importance ranking. We constructed different multimodality thrombus prediction models and compared their performance using the area under the receiver operating characteristic curve (AUC) and other efficacy parameters. Results Among 670 patients (60.17±10.98 years, 70.1% male), 5.2% (n=35) had LAA thrombi. A total of 1232 radiomics features were extracted, with 25 features selected using a random forest for modeling. Two radiomics features (Ibp_3D_m1_firstorder Skewness and original_shape_Flatness) ranked the highest. The radiomics-based multimodality prediction model achieved an AUC of 0.964 (95% confidence interval [CI]: 0.947-0.982), an accuracy of 0.926 (95% CI: 0.925-0.926), and an F1 score of 0.924, outperforming other traditional prediction models. Decision curve analysis also indicated that this model provided the best net clinical benefit. Conclusions Radiomics facilitates a comprehensive exploration of predictors associated with LAA thrombi. A radiomics-based multimodality prediction model can significantly enhance the efficiency of non-invasively detecting thrombi.Receiver operating characteristic curveDecision curve analysis

Incident atrial fibrillation prediction using ECG-based deep learning at a specialized tertiary cardiac care center

Abstract Background Deep learning (DL) applied to electrocardiograms (ECG) is an emerging modality in the prediction of incident atrial fibrillation or atrial flutter (AF). The generalizability of such methods to a tertiary heart center (THC) population has not yet been formally explored. Purpose Evaluate the performance of DL in predicting 5-year incident AF using a resting 12-lead ECG in sinus rhythm acquired at a THC. Methods In a retrospective study, we examined 1.4 million ECGs acquired from over 250,000 adults at a THC between 2004 and 2022. ECGs were excluded if they were performed within 30 days of cardiac surgery, captured a rhythm other than sinus rhythm, or were acquired in patients with pre-existing AF. Incident AF at 5 years was modelled as a binary outcome and was determined on the basis of available outpatient and inpatient clinical databases and ECG diagnoses. Included ECGs were randomly split by distributing patients into training (70%), validation (10%), and test (20%) sets. A ResNet-50 model was trained using the training set. Hyperparameters were optimized using the validation set. The tuned model's performance is reported on the test set. The results at the patient level were derived by averaging the model's probability outputs for ECGs grouped according to both their AF outcome and the patient's identity. Bootstrapping was used to report confidence intervals (CI). Sensitivity and specificity are reported at a classification threshold based on the Matthews correlation coefficient. Saliency maps were used to enhance the model’s explainability. Results A total of 669,782 ECGs (47% of the screened ECGs) were included among 145,323 patients. Mean age was 63±15 years and 62% were male. The 5-year incident AF outcome was observed in 12% of ECGs and 16% of patients. The performance of the tuned model was first evaluated at the ECG level on the test set demonstrating an area under the receiver operating curve (AUC) of 0.75 (95% CI: 0.745-0.753) and an integrated calibration index of 0.009 (95% CI: 0.008-0.011). When testing the model at the patient level to simulate a deployment scenario, the AUC improved to 0.78 (95% CI: 0.768-0.783) with a sensitivity of 50% (95% CI: 49-52) and specificity of 87% (95% CI: 86.5-87.3). Stratified results by sex showed a better AUC in females, 0.81 (95% CI: 0.80-0.82), compared to males, 0.75 (95% CI: 0.74-0.76). Subgroup analyses revealed a trend of improved performance at the patient level with an increasing number of ECGs. Saliency maps highlighted the P-wave area as having the highest influence on the model’s prediction, thereby enhancing the model’s plausibility and explainability. Conclusion A unimodal ECG-based deep learning model showed promising 5-year incident AF prediction performance in a cohort of all-comer patients at a tertiary heart center. Further studies could explore the use of multimodal prediction models that integrate ECGs with other clinical and imaging data.

Multimodal Prediction of Obsessive-Compulsive Disorder and Comorbid Depression Severity and Energy Delivered by Deep Brain Electrodes.

To develop reliable, valid, and efficient measures of obsessive-compulsive disorder (OCD) severity, comorbid depression severity, and total electrical energy delivered (TEED) by deep brain stimulation (DBS), we trained and compared random forests regression models in a clinical trial of participants receiving DBS for refractory OCD. Six participants were recorded during open-ended interviews at pre- and post-surgery baselines and then at 3-month intervals following DBS activation. Ground-truth severity was assessed by clinical interview and self-report. Visual and auditory modalities included facial action units, head and facial landmarks, speech behavior and content, and voice acoustics. Mixed-effects random forest regression with Shapley feature reduction strongly predicted severity of OCD, comorbid depression, and total electrical energy delivered by the DBS electrodes (intraclass correlation, ICC, = 0.83, 0.87, and 0.81, respectively. When random effects were omitted from the regression, predictive power decreased to moderate for severity of OCD and comorbid depression and remained comparable for total electrical energy delivered (ICC = 0.60, 0.68, and 0.83, respectively). Multimodal measures of behavior outperformed ones from single modalities. Feature selection achieved large decreases in features and corresponding increases in prediction. The approach could contribute to closed-loop DBS that would automatically titrate DBS based on affect measures.

GLNNMDA: a multimodal prediction model for microbe-drug associations based on global and local features

Microbes have been demonstrated to be closely linked to diseases that pose a major threat to human health. Computing technologies can help researchers find potential microbe-drug associations more quickly and precisely. In this study, we introduced a novel computational prediction model called GLNNMDA based on global and local features of microbes and drugs to infer possible microbe-drug correlations. In GLNNMDA, we first constructed a heterogeneous network based on known microbe-drug relationships by integrating multiple similarity metrics of drugs and microbes. Subsequently, low-dimensional features will be extracted for nodes in the heterogeneous network by adopting the graph attention encoder. Next, based on combining these low-dimensional features with multiple properties of microbes and drugs to form a new comprehensive feature matrix, we would utilize the GLF module to extract the global and local features for microbes and drugs respectively, and then, we would further fuse these global and local features to come up with predictions of possible microbe-drug associations. Moreover, in order to evaluate the prediction performance of GLNNMDA, under the framework of fivefold cross-validation, intensive comparative experiments and case studies were done on different well-known public databases. The results showed that GLNNMDA obtained the highest AUC values as well as AUPR values of 0.9802 ± 0.0011, 0.9773 ± 0.0021 and 0.8586 ± 0.0004, 0.8008 ± 0.0031 in the two databases, MDAD and aBiofilm, respectively, compared to the state-of-the-art competing prediction methods. In addition, case studies of well-known microorganisms and drugs have demonstrated the effectiveness of GLNNMDA in inferring potential microbial drug associations, which implies that GLNNMDA may be a useful tool for microbe-drug association prediction in the future. The source code is available at: “https://github.com/KuangHaiYue/GLNNMDA.git”.

Artificial intelligence techniques in liver cancer.

Hepatocellular Carcinoma (HCC), the most common primary liver cancer, is a significant contributor to worldwide cancer-related deaths. Various medical imaging techniques, including computed tomography, magnetic resonance imaging, and ultrasound, play a crucial role in accurately evaluating HCC and formulating effective treatment plans. Artificial Intelligence (AI) technologies have demonstrated potential in supporting physicians by providing more accurate and consistent medical diagnoses. Recent advancements have led to the development of AI-based multi-modal prediction systems. These systems integrate medical imaging with other modalities, such as electronic health record reports and clinical parameters, to enhance the accuracy of predicting biological characteristics and prognosis, including those associated with HCC. These multi-modal prediction systems pave the way for predicting the response to transarterial chemoembolization and microvascular invasion treatments and can assist clinicians in identifying the optimal patients with HCC who could benefit from interventional therapy. This paper provides an overview of the latest AI-based medical imaging models developed for diagnosing and predicting HCC. It also explores the challenges and potential future directions related to the clinical application of AI techniques.

Multimodal prediction of the need of clozapine in treatment resistant schizophrenia; a pilot study in first-episode psychosis

As many as one third of the patients diagnosed with schizophrenia do not respond to first-line antipsychotic medication. This group may benefit from the atypical antipsychotic medication clozapine, but initiation of treatment is often delayed, which may worsen prognosis. Predicting which patients do not respond to traditional antipsychotic medication at the onset of symptoms would provide fast-tracked treatment for this group of patients. We collected data from patient records of 38 first-episode psychosis patients, of whom seven did not respond to traditional antipsychotic medications. We used clinical data including medical records, voxel-based morphometry MRI data and inter-subject correlation fMRI data, obtained during movie viewing, to predict future treatment resistance. Using a neural network model, we correctly predicted future treatment resistance in six of the seven treatment resistance patients and 25 of 31 patients who did not require clozapine treatment. Prediction improved significantly when using imaging data in tandem with clinical data. The accuracy of the neural network model was significantly higher than the accuracy of a support vector machine algorithm. These results support the notion that treatment resistant schizophrenia could represent a separate entity of psychotic disorders, characterized by morphological and functional changes in the brain which could represent biomarkers detectable at early onset of symptoms.

Resting state EEG relates to short- and long-term cognitive functioning after cardiac arrest

BackgroundApproximately half of cardiac arrest survivors have persistent cognitive impairment. Guidelines recommend early screening to identify patients at risk for cognitive impairment, but there is no consensus on the best screening method. We aimed to identify quantitative EEG measures relating with short- and long-term cognitive function after cardiac arrest for potential to cognitive outcome prediction. MethodsWe analyzed data from a prospective longitudinal multicenter cohort study designed to develop a prediction model for cognitive outcome after cardiac arrest. For the current analysis, we used twenty-minute EEG registrations from 80 patients around one week after cardiac arrest. We calculated power spectral density, normalized alpha-to-theta ratio (nATR), peak frequency, and center of gravity (CoG) of this peak frequency. We related these with global cognitive functioning (scores on the Montreal Cognitive Assessment (MoCA)) at one week, three and twelve months follow-up with multivariate mixed effect models, and with performance on standard neuropsychological examination at twelve months using Pearson correlation coefficients. ResultsEach individual EEG parameter related to MoCA at one week (βnATR = 7.36; P < 0.01; βpeak frequency = 1.73, P < 0.01; βCoG = -9.88, P < 0.01). The nATR also related with the MoCA at three months ((βnATR = 2.49; P 0.01). No EEG metrics significantly related to the MoCA score at twelve months. nATR and peak frequency related with memory performance at twelve months. Results were consistent in sensitivity analyses. ConclusionEarly resting-state EEG parameters relate with short-term global cognitive functioning and with memory function at one year after cardiac arrest. Additional predictive values in multimodal prediction models need further study.

1512: multi-modality prediction of radiation induced hypothyroidism for nasopharyngeal carcinoma patients

1512: multi-modality prediction of radiation induced hypothyroidism for nasopharyngeal carcinoma patients

Prediction of immunochemotherapy response for diffuse large B-cell lymphoma using artificial intelligence digital pathology.

Diffuse large B-cell lymphoma (DLBCL) is a heterogeneous and prevalent subtype of aggressive non-Hodgkin lymphoma that poses diagnostic and prognostic challenges, particularly in predicting drug responsiveness. In this study, we used digital pathology and deep learning to predict responses to immunochemotherapy in patients with DLBCL. We retrospectively collected 251 slide images from 216 DLBCL patients treated with rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisone (R-CHOP), with their immunochemotherapy response labels. The digital pathology images were processed using contrastive learning for feature extraction. A multi-modal prediction model was developed by integrating clinical data and pathology image features. Knowledge distillation was employed to mitigate overfitting on gigapixel histopathology images to create a model that predicts responses based solely on pathology images. Based on the importance derived from the attention mechanism of the model, we extracted histological features that were considered key textures associated with drug responsiveness. The multi-modal prediction model achieved an impressive area under the ROC curve of 0.856, demonstrating significant associations with clinical variables such as Ann Arbor stage, International Prognostic Index, and bulky disease. Survival analyses indicated their effectiveness in predicting relapse-free survival. External validation using TCGA datasets supported the model's ability to predict survival differences. Additionally, pathology-based predictions show promise as independent prognostic indicators. Histopathological analysis identified centroblastic and immunoblastic features to be associated with treatment response, aligning with previous morphological classifications and highlighting the objectivity and reproducibility of artificial intelligence-based diagnosis. This study introduces a novel approach that combines digital pathology and clinical data to predict the response to immunochemotherapy in patients with DLBCL. This model shows great promise as a diagnostic and prognostic tool for clinical management of DLBCL. Further research and genomic data integration hold the potential to enhance its impact on clinical practice, ultimately improving patient outcomes.

Asymmetric Influences of ENSO Phases on the Predictability of North Pacific Sea Surface Temperature

AbstractThe North Pacific sea surface temperature (SST) has a profound climatic influence. The El Niño‐Southern Oscillation (ENSO) significantly impacts the North Pacific SST; however, the influence of the distinct phases of ENSO on SST predictability remains unclear. To overcome the model limitations, this study assessed SST predictability under diverse ENSO phases using reanalysis. The predictability limit of the North Pacific SST under La Niña (8.4 months) is longer than that under Neutral (5.9 months) and El Niño (5.5 months) conditions, which unveils asymmetry. This asymmetry mirrors contemporary multimodal prediction skills. Error growth dynamics reveal La Niña's robust signal strength with a slow error growth rate, in contrast to El Niño's weaker signal and faster error growth. There exhibits intermediate signal strength and elevated error growth in Neutral condition. Physically, predictability signal strength aligns with SST variability, whereas the error growth rate correlates with atmospheric‐ocean heating anomalies. La Niña, which induces positive heating anomalies, minimizes the impact of atmospheric noise, resulting in lower error growth. The result is beneficial for improving North Pacific SST predictions.

Multistep validation of a post-ERCP pancreatitis prediction system integrating multimodal data: a multicenter study

Background and study aimsThe impact of various categories of information on the prediction of Post Endoscopic Retrograde Cholangiopancreatography Pancreatitis (PEP) remains uncertain. We aimed to comprehensively investigate the risk factors associated with PEP by constructing and validating a model incorporating multi-modal data through multiple steps. Patients and MethodsA total of 1,916 cases underwent ERCP were retrospectively collected from multiple centers for model construction. Through literature research, 49 electronic health record (EHR) features and one image feature related to PEP were identified. The EHR features were categorized into baseline, diagnosis, technique, and prevent strategies, covering pre-ERCP, intra-ERCP, and peri-ERCP phases. We first incrementally constructed models 1-4 incorporating these four feature categories, then added the image feature into models 1-4 and developed models 5-8. All models underwent testing and comparison using both internal and external test sets. Once the optimal model was selected, we conducted comparison among multiple machine learning algorithms. ResultsCompared with model 2 incorporating baseline and diagnosis features, adding technique and prevent strategies (model 4) greatly improved the sensitivity (63.89% vs 83.33%, p<0.05) and specificity (75.00% vs 85.92%, p<0.001). Similar tendency was observed in internal and external tests. In model 4, the top three features ranked by weight were previous pancreatitis, NSAIDS, and difficult cannulation. The image-based feature has the highest weight in model 5-8. Lastly, model 8 employed Random Forest algorithm showed the best performance. ConclusionsWe firstly developed a multi-modal prediction model for identifying PEP with clinical-acceptable performance. The image and technique features are crucial for PEP prediction.

Literature review and protocol for a prospective multicentre cohort study on multimodal prediction of seizure recurrence after unprovoked first seizure

IntroductionEpilepsy is a common neurological disorder characterised by recurrent seizures. Almost half of patients who have an unprovoked first seizure (UFS) have additional seizures and develop epilepsy. No current predictive...

Goal-Guided Graph Attention Network with Interactive State Refinement for Multi-Agent Trajectory Prediction.

The accurate prediction of the future trajectories of traffic participants is crucial for enhancing the safety and decision-making capabilities of autonomous vehicles. Modeling social interactions among agents and revealing the inherent relationships is crucial for accurate trajectory prediction. In this context, we propose a goal-guided and interaction-aware state refinement graph attention network (SRGAT) for multi-agent trajectory prediction. This model effectively integrates high-precision map data and dynamic traffic states and captures long-term temporal dependencies through the Transformer network. Based on these dependencies, it generates multiple potential goals and Points of Interest (POIs). Through its dual-branch, multimodal prediction approach, the model not only proposes various plausible future trajectories associated with these POIs, but also rigorously assesses the confidence levels of each trajectory. This goal-oriented strategy enables SRGAT to accurately predict the future movement trajectories of other vehicles in complex traffic scenarios. Tested on the Argoverse and nuScenes datasets, SRGAT surpasses existing algorithms in key performance metrics by adeptly integrating past trajectories and current context. This goal-guided approach not only enhances long-term prediction accuracy, but also ensures its reliability, demonstrating a significant advancement in trajectory forecasting.

Multimodal prediction of student performance: A fusion of signed graph neural networks and large language models

In online education platforms, accurately predicting student performance is essential for timely dropout prevention and interventions for at-risk students. This task is made difficult by the prevalent use of Multiple-Choice Questions (MCQs) in learnersourcing platforms, where noise in student-generated content and the limitations of existing unsigned graph-based models, specifically their inability to distinguish the semantic meaning between correct and incorrect responses, hinder accurate performance predictions. To address these issues, we introduce the Large Language Model enhanced Signed Bipartite graph Contrastive Learning (LLM-SBCL) model—a novel Multimodal Model utilizing Signed Graph Neural Networks (SGNNs) and a Large Language Model (LLM). Our model uses a signed bipartite graph to represent students’ answers, with positive and negative edges denoting correct and incorrect responses, respectively. To mitigate noise impact, we apply contrastive learning to the signed graphs, combined with knowledge point embeddings from the LLM to further enhance our model’s predictive performance. Upon evaluating our model on five real-world datasets, it demonstrates superior accuracy and stability, exhibiting an average F1 improvement of 3.7% over the best baseline models.

SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival

Accurately predicting the survival rate of cancer patients is crucial for aiding clinicians in planning appropriate treatment, reducing cancer-related medical expenses, and significantly enhancing patients’ quality of life. Multimodal prediction of cancer patient survival offers a more comprehensive and precise approach. However, existing methods still grapple with challenges related to missing multimodal data and information interaction within modalities. This paper introduces SELECTOR, a heterogeneous graph-aware network based on convolutional mask encoders for robust multimodal prediction of cancer patient survival. SELECTOR comprises feature edge reconstruction, convolutional mask encoder, feature cross-fusion, and multimodal survival prediction modules. Initially, we construct a multimodal heterogeneous graph and employ the meta-path method for feature edge reconstruction, ensuring comprehensive incorporation of feature information from graph edges and effective embedding of nodes. To mitigate the impact of missing features within the modality on prediction accuracy, we devised a convolutional masked autoencoder (CMAE) to process the heterogeneous graph post-feature reconstruction. Subsequently, the feature cross-fusion module facilitates communication between modalities, ensuring that output features encompass all features of the modality and relevant information from other modalities. Extensive experiments and analysis on six cancer datasets from TCGA demonstrate that our method significantly outperforms state-of-the-art methods in both modality-missing and intra-modality information-confirmed cases. Our codes are made available at https://github.com/panliangrui/Selector.