Multimodal Deep Learning Research Articles

Prediction of recurrence risk in endometrial cancer with multimodal deep learning

Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. Here we developed HECTOR (histopathology-based endometrial cancer tailored outcome risk), a multimodal deep learning prognostic model using hematoxylin and eosin-stained, whole-slide images and tumor stage as input, on 2,072 patients from eight EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n = 353) and two external (n = 160 and n = 151) test sets of 0.789, 0.828 and 0.815, respectively, outperforming the current gold standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low-, intermediate- and high-risk groups, respectively, by Kaplan–Meier analysis). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold standard and may help delivery of personalized treatment in EC.

Predicting atrial fibrillation ablation success: a novel approach using pre-procedure sinus rhythm ecg and deep learning

Abstract Introduction Pulmonary vein isolation (PVI) has been the corner stone of the treatment of patients with atrial fibrillation (AF), yet its efficacy varies notably between patient groups. Specifically, it shows reduced efficacy in individuals with persistent AF and those with atrial fibrosis. Recent advancements suggest that additional substrate modification could enhance outcomes in specific patient groups. Therefore, predicting the outcomes of lone PVI could help in stratifying patients who would benefit most from this additional substrate modification. Addressing this gap, our study explores the prognostication of PVI outcomes using pre-procedural 12-lead ECG recordings taken in Sinus rhythm, employing advanced deep learning techniques. Methods This retrospective study analyzed 209 patients with paroxysmal and persistent AF who underwent PVI. These patients were followed up for 12 months after the blanking period. Pre-procedural sinus rhythm 12-lead ECGs were processed using an innovative deep learning architecture, integrating Long Short-Term Memory and Convolutional Neural Networks with a cross-attention module. This multimodal deep learning model was trained using ECG and clinical covariates like age, sex, and BMI to predict binary outcomes – AF type (Paroxysmal or Persistent) and AF recurrence. The model's efficacy was evaluated through a 5-fold stratified cross-validation to ensure accuracy and reliability. Model was also tested using only ECG data. Results The multimodal deep learning model performed significantly better as compared to the model trained using ECG data only. The multimodal deep learning model achieved an accuracy of 75.56% and a sensitivity of 82% in prognosticating ablation outcomes. Additionally, the model demonstrated a significant capability in classifying AF into paroxysmal or persistent types with an accuracy of 86.67% and a recall of 89%. Conclusion Our deep learning approach in analyzing sinus rhythm ECG recordings shows promising results in predicting PVI outcomes and identifying AF types. This approach could offer a more accessible and cost-effective tool to stratify AF patients undergoing PVI, identifying those less likely to benefit from standard procedures. Such insights can assist physicians and patients in determining ablation candidacy and selecting the most appropriate ablation strategy and technique.Proposed StrategyDeep learning model approach

An automatic inspection system for the detection of tire surface defects and their severity classification through a two-stage multimodal deep learning approach

AbstractIn the tire manufacturing field, the pursuit of uncompromised product quality stands as a cornerstone. This paper introduces an innovative multimodal approach aimed at automating the tire quality control process through the use of deep learning on data obtained from stereo-photometric cameras meticulously integrated into a purpose-built, sophisticated tire acquisition system capable of comprehensive data capture across all tire zones. The defects sought exhibit significant variations in size (ranging from a few millimeters to several tens of centimeters) and type (including abnormal stains during processing, marks resulting from demolding issues, foreign particles, air bubbles, deformations, etc.). Our proposed methodology comprises two distinct stages: an initial instance segmentation phase for defect detection and localization, followed by a classification stage based on severity levels, integrating features extracted from the detection network of the first stage alongside tire metadata. Experimental validation demonstrates that the proposed approach achieves automation objectives, attaining satisfactory results in terms of defect detection and classification according to severity, with a F1 score between 0.7 and 0.89 depending on the tire zone. In addition, this study presents a novel method applicable to all tire areas, addressing a wide variety of defects within the domain.

The EarSAVAS Dataset

Subject-aware vocal activity sensing on wearables, which specifically recognizes and monitors the wearer's distinct vocal activities, is essential in advancing personal health monitoring and enabling context-aware applications. While recent advancements in earables present new opportunities, the absence of relevant datasets and effective methods remains a significant challenge. In this paper, we introduce EarSAVAS, the first publicly available dataset constructed specifically for subject-aware human vocal activity sensing on earables. EarSAVAS encompasses eight distinct vocal activities from both the earphone wearer and bystanders, including synchronous two-channel audio and motion data collected from 42 participants totaling 44.5 hours. Further, we propose EarVAS, a lightweight multi-modal deep learning architecture that enables efficient subject-aware vocal activity recognition on earables. To validate the reliability of EarSAVAS and the efficiency of EarVAS, we implemented two advanced benchmark models. Evaluation results on EarSAVAS reveal EarVAS's effectiveness with an accuracy of 90.84% and a Macro-AUC of 89.03%. Comprehensive ablation experiments were conducted on benchmark models and demonstrated the effectiveness of feedback microphone audio and highlighted the potential value of sensor fusion in subject-aware vocal activity sensing on earables. We hope that the proposed EarSAVAS and benchmark models can inspire other researchers to further explore efficient subject-aware human vocal activity sensing on earables.

EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion

Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradictory conclusions in the literature. Some authors assert that eye gaze data can enhance prediction accuracy, while others consider eye tracking irrelevant for predictive tasks. We argue that this disagreement lies in how researchers process eye-tracking data as most remain agnostic to the human component and apply the data directly to DL models without proper preprocessing. We present EyeXNet, a multimodal DL architecture that combines images and radiologists’ fixation masks to predict abnormality locations in chest X-rays. We focus on fixation maps during reporting moments as radiologists are more likely to focus on regions with abnormalities and provide more targeted regions to the predictive models. Our analysis compares radiologist fixations in both silent and reporting moments, revealing that more targeted and focused fixations occur during reporting. Our results show that integrating the fixation masks in a multimodal DL architecture outperformed the baseline model in five out of eight experiments regarding average Recall and six out of eight regarding average Precision. Incorporating fixation masks representing radiologists’ classification patterns in a multimodal DL architecture benefits lesion detection in chest X-ray (CXR) images, particularly when there is a strong correlation between fixation masks and generated proposal regions. This highlights the potential of leveraging fixation masks to enhance multimodal DL architectures for CXR image analysis. This work represents a first step towards human-centered DL, moving away from traditional data-driven and human-agnostic approaches.

MULTIMODAL SYSTEM FOR FACIAL EMOTION RECOGNITION BASED ON DEEP LEARNING

Emotions are one of the main ways of communication between people and of expressing attitudes towards objects, products, services, etc. They are divided to verbal and non-verbal classes. Human speech and intonation belong to the first class, and to the second (non-verbal) facial and body emotions, known as body language. The subject of this report is the development of multimodal deep learning system intended to recognize facial and body emotions and their relationship with the scene (weather) in which they occur. It is based on three deep learning neural networks (DNN) each one for recognition of facial emotion, body emotion and weather. Combining their results, we improve significantly the final facial emotion recognition (FER) results.

Urban flood susceptibility mapping using remote sensing, social sensing and an ensemble machine learning model

Flood susceptibility mapping is crucial for urban disaster management. However, the heterogeneity of urban land use and the complexity of terrain pose challenges to the accuracy and generalizability of flood models. In this paper, we propose a framework named the EMF model for urban flood mapping. Specifically, the social sensing and remote sensing were used for flooding information collection. The XGBoost, Support Vector Classifier (SVC), Multilayer Perceptron (MLP), and Multimodal Deep Learning (MDL), were used as the predictive models, and their results were ensembled using a Random Forest model to produce the final outcome. Results show that the EMF model outperforms the standalone models, with an accuracy of 0.942 on the training set and 0.940 on the testing set. The accuracy of the five models is ranked as EMF > MDL > XGBoost > SVC > MLP. The flood map indicates that flooding has a greater impact on the outskirts and suburbs of the city compared to its central urban areas. Farmland is the most affected land type, making up 54.8 % of the flooded area. Overall, the proposed framework enables rapid and accurate identification of flood-prone areas, providing technical support for managers in formulating effective flood prevention strategies.

Improving the Prognostic Evaluation Precision of Hospital Outcomes for Heart Failure Using Admission Notes and Clinical Tabular Data: Multimodal Deep Learning Model.

Clinical notes contain contextualized information beyond structured data related to patients' past and current health status. This study aimed to design a multimodal deep learning approach to improve the evaluation precision of hospital outcomes for heart failure (HF) using admission clinical notes and easily collected tabular data. Data for the development and validation of the multimodal model were retrospectively derived from 3 open-access US databases, including the Medical Information Mart for Intensive Care III v1.4 (MIMIC-III) and MIMIC-IV v1.0, collected from a teaching hospital from 2001 to 2019, and the eICU Collaborative Research Database v1.2, collected from 208 hospitals from 2014 to 2015. The study cohorts consisted of all patients with critical HF. The clinical notes, including chief complaint, history of present illness, physical examination, medical history, and admission medication, as well as clinical variables recorded in electronic health records, were analyzed. We developed a deep learning mortality prediction model for in-hospital patients, which underwent complete internal, prospective, and external evaluation. The Integrated Gradients and SHapley Additive exPlanations (SHAP) methods were used to analyze the importance of risk factors. The study included 9989 (16.4%) patients in the development set, 2497 (14.1%) patients in the internal validation set, 1896 (18.3%) in the prospective validation set, and 7432 (15%) patients in the external validation set. The area under the receiver operating characteristic curve of the models was 0.838 (95% CI 0.827-0.851), 0.849 (95% CI 0.841-0.856), and 0.767 (95% CI 0.762-0.772), for the internal, prospective, and external validation sets, respectively. The area under the receiver operating characteristic curve of the multimodal model outperformed that of the unimodal models in all test sets, and tabular data contributed to higher discrimination. The medical history and physical examination were more useful than other factors in early assessments. The multimodal deep learning model for combining admission notes and clinical tabular data showed promising efficacy as a potentially novel method in evaluating the risk of mortality in patients with HF, providing more accurate and timely decision support.

Assessing learners’ English public speaking anxiety with multimodal deep learning technologies

Public speaking anxiety (PSA) is a common phenomenon for language learners involving both psychological and physiological symptoms. Timely and effective PSA assessment can help diagnose learners’ speaking anxiety, offer learners feedback to alleviate their anxiety, and improve their public speaking competence. However, it is still a challenging issue to achieve accurate automated assessment of learners’ PSA due to the lack of large-scale and open-source multimodal datasets based on real classroom settings. This study collected the public speaking videos of English as a foreign language (EFL) learners in public speaking courses, and constructed a large-scale multimodal dataset named Speaking Anxiety in Real Classrooms (SARC) with three modalities of acoustic, visual, and textual data (including 1,158 manually-annotated speech videos and corresponding speech drafts of 382 participants). A multimodal deep learning model for automated assessment of learners’ speaking anxiety was proposed, and an online formative assessment platform was then developed to realize the automated assessment of PSA for classroom teaching. A pilot survey study involving 78 participants was then conducted to investigate learners’ acceptance of the platform. Experimental results verified the validity of the deep learning model and the consistency between automated assessment and teacher assessment. Learners’ acceptance data further indicated that collaboration between automated and human assessment provided them with the most satisfactory experience of using the platform to improve their English public speaking.

A multimodal deep learning tool for detection of junctional ectopic tachycardia in children with congenital heart disease

Junctional ectopic tachycardia (JET) is a prevalent life-threatening arrhythmia in children with congenital heart disease. It has a marked resemblance to normal sinus rhythm, often leading to delay in diagnosis and management. The study sought to develop a novel multimodal automated arrhythmia detection tool that outperforms existing JET detection tools. This is a cohort study performed on 40 patients with congenital heart disease at Texas Children's Hospital. Electrocardiogram and central venous pressure waveform data produced by bedside monitors are captured by the Sickbay platform. Convolutional neural networks (CNNs) were trained to classify each heartbeat as either normal sinus rhythm or JET based only on raw electrocardiogram signals. Our best model improved the area under the curve from 0.948 to 0.952 and the true positive rate at 5% false positive rate from 71.8% to 80.6%. Using a 3-model ensemble further improved the area under the curve to 0.953 and the true positive rate at 5% false positive rate to 85.2%. Results on a subset of data show that adding central venous pressure can significantly improve area under the receiver-operating characteristic curve from 0.646 to 0.825. This study validates the efficacy of deep neural networks to notably improve JET detection accuracy. We have built a performant and reliable model that can be used to create a bedside alarm that diagnoses JET, allowing for precise diagnosis of this life-threatening postoperative arrhythmia and prompt intervention. Future validation of the model in a larger cohort is needed.

Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival prediction.

The prognosis of breast cancer is often unfavorable, emphasizing the need for early metastasis risk detection and accurate treatment predictions. This study aimed to develop a novel multi-modal deep learning model using preoperative data to predict disease-free survival (DFS). We retrospectively collected pathology imaging, molecular and clinical data from The Cancer Genome Atlas and one independent institution in China. We developed a novel Deep Learning Clinical Medicine Based Pathological Gene Multi-modal (DeepClinMed-PGM) model for DFS prediction, integrating clinicopathological data with molecular insights. The patients included the training cohort (n=741), internal validation cohort (n=184), and external testing cohort (n=95). Integrating multi-modal data into the DeepClinMed-PGM model significantly improved area under the receiver operating characteristic curve (AUC) values. In the training cohort, AUC values for 1-, 3-, and 5-year DFS predictions increased to 0.979, 0.957, and 0.871, while in the external testing cohort, the values reached 0.851, 0.878, and 0.938 for 1-, 2-, and 3-year DFS predictions, respectively. The DeepClinMed-PGM's robust discriminative capabilities were consistently evident across various cohorts, including the training cohort [hazard ratio (HR) 0.027, 95% confidence interval (CI) 0.0016-0.046, P<0.0001], the internal validation cohort (HR 0.117, 95% CI 0.041-0.334, P<0.0001), and the external cohort (HR 0.061, 95% CI 0.017-0.218, P<0.0001). Additionally, the DeepClinMed-PGM model demonstrated C-index values of 0.925, 0.823, and 0.864 within the three cohorts, respectively. This study introduces an approach to breast cancer prognosis, integrating imaging and molecular and clinical data for enhanced predictive accuracy, offering promise for personalized treatment strategies.

Heart failure prognosis prediction: Let’s start with the MDL-HFP model

Heart failure, as a critical symptom or terminal stage of assorted heart diseases, is a world-class public health problem. Establishing a prognostic model can help identify high dangerous patients, save their lives promptly, and reduce medical burden. Although integrating structured indicators and unstructured text for complementary information has been proven effective in disease prediction tasks, there are still certain limitations. Firstly, the processing of single branch modes is easily overlooked, which can affect the final fusion result. Secondly, simple fusion will lose complementary information between modalities, limiting the network’s learning ability. Thirdly, incomplete interpretability can affect the practical application and development of the model. To overcome these challenges, this paper proposes the MDL-HFP multimodal model for predicting patient prognosis using the MIMIC-III public database. Firstly, the ADASYN algorithm is used to handle the imbalance of data categories. Then, the proposed improved Deep&Cross Network is used for automatic feature selection to encode structured sparse features, and implicit graph structure information is introduced to encode unstructured clinical notes based on the HR-BGCN model. Finally, the information of the two modalities is fused through a cross-modal dynamic interaction layer. By comparing multiple advanced multimodal deep learning models, the model’s effectiveness is verified, with an average F1 score of 90.42% and an average accuracy of 90.70%. The model proposed in this paper can accurately classify the readmission status of patients, thereby assisting doctors in making judgments and improving the patient’s prognosis. Further visual analysis demonstrates the usability of the model, providing a comprehensive explanation for clinical decision-making.

Research on Intelligent System of Multimodal Deep Learning in Image Recognition

In this paper, a multi-scale image estimation method based on wavelet transform is proposed, which can effectively remove motion features from multiple videos. Then the autoencoder with sparsity limit is used to adjust the input signal to achieve effective compression. The effective features are extracted and the optimal unique vector is learned. The improved convolutional neural network is used to recognize weak moving objects. Experiments show that the algorithm can achieve high accuracy without large-scale learning samples, and the highest recognition rate is 99.36%. This algorithm has a great improvement over conventional algorithm.

Robust Intelligent Malware Detection Using Deep Learning

Abstract: In the modern era of technology, malicious software, or malware, holds a serious security hazard as computer users, businesses, and governments see an uptick in malware attacks. In attempts to identify unknown malware, current malware detection solutions use dynamic as well as static examination of malware signatures and behavior patterns, which takes time and is unsuccessful. Modern malware employs evasive strategies such as metamorphosis and polymorphism to rapidly alter its actions and produce a multitude of variants. Machine learning algorithms (MLAs) are being used more and more to do an efficient malware analysis because new malware is primarily versions of current malware. Extensive feature engineering, feature learning, and feature representation are needed for this. It is likely to fully eliminate the feature engineering stage by utilizing sophisticated MLAs like deep learning. Even though there have been a few fresh investigations in the field, the algorithms' performance is skewed by the training set. It is a prerequisite to reduce bias and figure out these techniques holistically in order to develop new, improved techniques for successful zero-day malware detection. This paper fills a vacuum in the literature by comparing and contrasting deep learning architectures with standard MLAs for malware detection, classification, and categorization using public and private datasets. The public and private dataset’s train and test splits, which were gathered during distinctly different periods, are not connected to one another in the experimental study. Furthermore, we provide a new method of image processing with ideal parameters for deep learning architectures and MLAs. In response to a thorough scientific assessment of these methodologies, deep learning architectures perform more efficiently than traditional MLAs. All in all, our work suggests a scalable and multimodal deep learning system for real-time malware detection through visual means. An improved technique for successful zero-day malware detection is the visualization and deep learning architectures for static, dynamic, and image processing based blended methods in a big data environment.

Formalizing Multimedia Recommendation through Multimodal Deep Learning

Recommender systems (RSs) provide customers with a personalized navigation experience within the vast catalogs of products and services offered on popular online platforms. Despite the substantial success of traditional RSs, recommendation remains a highly challenging task, especially in specific scenarios and domains. For example, human affinity for items described through multimedia content (e.g., images, audio, and text), such as fashion products, movies, and music, is multi-faceted and primarily driven by their diverse characteristics. Therefore, by leveraging all available signals in such scenarios, multimodality enables us to tap into richer information sources and construct more refined user/item profiles for recommendations. Despite the growing number of multimodal techniques proposed for multimedia recommendation, the existing literature lacks a shared and universal schema for modeling and solving the recommendation problem through the lens of multimodality. Given the recent advances in multimodal deep learning for other tasks and scenarios where precise theoretical and applicative procedures exist, we also consider it imperative to formalize a general multimodal schema for multimedia recommendation. In this work, we first provide a comprehensive literature review of multimodal approaches for multimedia recommendation from the last eight years. Second, we outline the theoretical foundations of a multimodal pipeline for multimedia recommendation by identifying and formally organizing recurring solutions/patterns; at the same time, we demonstrate its rationale by conceptually applying it to selected state-of-the-art approaches in multimedia recommendation. Third, we conduct a benchmarking analysis of recent algorithms for multimedia recommendation within Elliot, a rigorous framework for evaluating recommender systems, where we re-implement such multimedia recommendation approaches. Finally, we highlight the significant unresolved challenges in multimodal deep learning for multimedia recommendation and suggest possible avenues for addressing them. The primary aim of this work is to provide guidelines for designing and implementing the next generation of multimodal approaches in multimedia recommendation.

Research on Online Review Information Classification Based on Multimodal Deep Learning

The incessant evolution of online platforms has ushered in a multitude of shopping modalities. Within the food industry, however, assessing the delectability of meals can only be tentatively determined based on consumer feedback encompassing aspects such as taste, pricing, packaging, service quality, delivery timeliness, hygiene standards, and environmental considerations. Traditional text data mining techniques primarily focus on consumers’ emotional traits, disregarding pertinent information pertaining to the online products themselves. In light of these aforementioned issues in current research methodologies, this paper introduces the Bert BiGRU Softmax model combined with multimodal features to enhance the efficacy of sentiment classification in data analysis. Comparative experiments conducted using existing data demonstrate that the accuracy rate of the model employed in this study reaches 90.9%. In comparison to single models or combinations of three models with the highest accuracy rate of 7.7%, the proposed model exhibits superior accuracy and proves to be highly applicable to online reviews.

MMSyn: A New Multimodal Deep Learning Framework for Enhanced Prediction of Synergistic Drug Combinations.

Combination therapy is a promising strategy for the successful treatment of cancer. The large number of possible combinations, however, mean that it is laborious and expensive to screen for synergistic drug combinations in vitro. Nevertheless, because of the availability of high-throughput screening data and advances in computational techniques, deep learning (DL) can be a useful tool for the prediction of synergistic drug combinations. In this study, we proposed a multimodal DL framework, MMSyn, for the prediction of synergistic drug combinations. First, features embedded in the drug molecules were extracted: structure, fingerprint, and string encoding. Then, gene expression data, DNA copy number, and pathway activity were used to describe cancer cell lines. Finally, these processed features were integrated using an attention mechanism and an interaction module and then input into a multilayer perceptron to predict drug synergy. Experimental results showed that our method outperformed five state-of-the-art DL methods and three traditional machine learning models for drug combination prediction. We verified that MMSyn achieved superior performance in stratified cross-validation settings using both the drug combination and cell line data. Moreover, we performed a set of ablation experiments to illustrate the effectiveness of each component and the efficacy of our model. In addition, our visual representation and case studies further confirmed the effectiveness of our model. All results showed that MMSyn can be used as a powerful tool for the prediction of synergistic drug combinations.

A multimodal deep learning algorithm for polyphonic music applied to music sentiment analysis and generation

A multimodal deep learning algorithm for polyphonic music applied to music sentiment analysis and generation

Exploratory drug discovery in breast cancer patients: A multimodal deep learning approach to identify novel drug candidates targeting RTK signaling

Breast cancer, a highly formidable and diverse malignancy predominantly affecting women globally, poses a significant threat due to its intricate genetic variability, rendering it challenging to diagnose accurately. Various therapies such as immunotherapy, radiotherapy, and diverse chemotherapy approaches like drug repurposing and combination therapy are widely used depending on cancer subtype and metastasis severity. Our study revolves around an innovative drug discovery strategy targeting potential drug candidates specific to RTK signalling, a prominently targeted receptor class in cancer. To accomplish this, we have developed a multimodal deep neural network (MM-DNN) based QSAR model integrating omics datasets to elucidate genomic, proteomic expression data, and drug responses, validated rigorously. The results showcase an R2 value of 0.917 and an RMSE value of 0.312, affirming the model's commendable predictive capabilities. Structural analogs of drug molecules specific to RTK signalling were sourced from the PubChem database, followed by meticulous screening to eliminate dissimilar compounds. Leveraging the MM-DNN-based QSAR model, we predicted the biological activity of these molecules, subsequently clustering them into three distinct groups. Feature importance analysis was performed. Consequently, we successfully identified prime drug candidates tailored for each potential downstream regulatory protein within the RTK signalling pathway. This method makes the early stages of drug development faster by removing inactive compounds, providing a hopeful path in combating breast cancer.

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning.

The objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data. The research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients' pathological images and clinical data, respectively. A total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS). Our study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.