Deep Multimodal Fusion Research Articles

A deep multimodal fusion method for personality traits prediction

A deep multimodal fusion method for personality traits prediction

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis.

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

Deep Multimodal Fusion of Visual and Auditory Features for Robust Material Recognition

This paper presents a deep neural network incorporating visual and auditory data fusion to enhance material recognition performance. Traditional recognition techniques relying on single data modalities face accuracy and robustness limitations, especially in complex real-world environments. To address these challenges, we develop a multimodal fusion-based model. The proposed approach first extracts features from input images and sounds separately using CNNs and spectral analysis. A concatenation layer then integrates the visual and auditory features. Extensive experiments demonstrate superior material classification over uni-modal methods, with 100% test accuracy across seven material types. The multi-modal fusion model also demonstrates stronger resilience to noise and illumination variations. This research provides a valuable foundation for robust material perception in intelligent systems.

A Deep Multimodal Fusion and Multitasking Trajectory Prediction Model for Typhoon Trajectory Prediction to Reduce Flight Scheduling Cancellation

A Deep Multimodal Fusion and Multitasking Trajectory Prediction Model for Typhoon Trajectory Prediction to Reduce Flight Scheduling Cancellation

Deep Multi-Modal Fusion Model for Identification of Eight Different Particles in Urinary Sediment

Abstract Urine sediment examination (USE) is an essential aspect in detecting urinary system diseases, and it is a prerequisite for diagnostic procedures. Urine images are complex, containing numerous particles, which makes a detailed analysis and interpretation challenging. It is crucial for both patients and medical professionals to conduct urine analysis automatically, quickly and inexpensively, without compromising reliability. In this paper, we present a deep multi-modal fusion system, commonly employed in artificial intelligence, capable of automatically distinguishing particles in urine sediment. To achieve this objective, we first created a new dataset comprising erythrocytes, leukocytes, yeast, epithelium, bacteria, crystals, cylinders, and other particles (such as sperm). The data were gathered from urinalysis requests made between July 2022 and September 2022 at the biochemistry laboratory of Fethi Sekin Medical Center Hospital. A dataset containing 8509 images was compiled using the Optika B293PLi microscope with trinocular brightfield. We propose a 5-step process for detecting particles in the dataset using a multi-modal fusion deep learning model: i) The obtained images were augmented by applying affine transformation. ii) To distinguish images, we opted for ResNet18 and ResNet50 models, which yielded high performance in medical data. iii) Feature vectors from both models were fused to generate more consistent, accurate, and useful particle features. iv) We employed ReliefF, Neighborhood Component Analysis (NCA), and Minimum-Redundancy Maximum-Relevancy (mRMR) feature selection methods, widely used to determine features that maximise particle discrimination success. v) In the final step, Support Vector Machine (SVM) was utilised to distinguish the particles. The results demonstrate that the highest accuracy value achieved is 98.54 % when employing the ReliefF algorithm. Contributions of the study include eliminating standardisation differences in manual microscopy, achieving high accuracy in particle discrimination, offering an artificial intelligence-based system applicable in laboratory environments, and providing the dataset as educational and practical material for biochemistry professionals.

Enhancing Cardiovascular Disease Detection through Deep Multimodal Fusion: Integrating Radiology and Electrocardiogram via Convolutional Neural Network

Cardiovascular Diseases (CVDs), also known as heart diseases are related to a process of atherosclerosis, which is when a plaque builds up in the arteries and blocks blood flow. CVDs are an ever-increasing health risk for the general population, and according to the American Heart Association, the number of deaths related to CVDs has surpassed the previous 910,000 all-time record from 2003. To solve this problem, numerous novel approaches have been presented for a solution to decrease such anomalies with traditional methods utilizing computed tomography scans (CT scans) to calculate coronary artery calcium. However, countless problems arise from using CT scans as CT scans are not accessible to the general public, and early diagnosis is especially difficult. So, this research paper aims to explore the potential of AI-driven biomarkers in enhancing the efficiency of CVD diagnosis. Moreover, this study presents a novel approach that uses artificial intelligence to enhance the accuracy of CVD diagnosis. Specifically, this paper suggests an innovative methodology that utilizes the classification of x-ray images that is additionally supplemented by classifying electrocardiograms. These techniques allow for analysis of multi-modal data sources allowing latent information on CVDs to be extracted. Throughout the experiments, the proposed method has proven that it is superior compared to other state-of-the-art methods, as it has achieved an accuracy of 90.49%.

Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks.

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D+2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

MFU-Net: a deep multimodal fusion network for breast cancer segmentation with dual-layer spectral detector CT

MFU-Net: a deep multimodal fusion network for breast cancer segmentation with dual-layer spectral detector CT

Hybrid optimized multimodal spatiotemporal feature fusion for vision-based sports activity recognition

Vision-based Human Activity Recognition (HAR) is a challenging research task in sports. This paper aims to track the player’s movements and recognize the different types of sports activities in videos. The proposed work aims in developing Hybrid Optimized Multimodal SpatioTemporal Feature Fusion (HOM-STFF) model using skeletal information for vision-based sports activity recognition. The proposed HOM-STFF model presents a deep multimodal feature fusion approach that combines the features that are generated from the multichannel-1DCNN and 2D-CNN network model using a concatenative feature fusion process. The fused features are fed into the 2-GRU model that generates temporal features for activity recognition. Nature-inspired Bald Eagle Search Optimizer (BESO) is applied to optimize the network weights during training. Finally, performance of the classification model is evaluated and compared for identifying different activities in sports videos. Experimentation was carried out with the three vision-based sports datasets namely, Sports Videos in the Wild (SVW), UCF50 sports action and Self-build dataset, which achieved accuracy rate of 0.9813, 0.9506 and 0.9733, respectively. The results indicate that the proposed HOM-STFF model outperforms the other state-of-the-art methods in terms of activity detection capability.

Retracted: Automatic Detection and Classification of Epileptic Seizures in Patients with Liver Cirrhosis and Overlapping Hev Infection Based on Deep Multimodal Fusion Technology

Retracted: Automatic Detection and Classification of Epileptic Seizures in Patients with Liver Cirrhosis and Overlapping Hev Infection Based on Deep Multimodal Fusion Technology

DMFDDI: deep multimodal fusion for drug-drug interaction prediction.

Drug combination therapy has gradually become a promising treatment strategy for complex or co-existing diseases. As drug-drug interactions (DDIs) may cause unexpected adverse drug reactions, DDI prediction is an important task in pharmacology and clinical applications. Recently, researchers have proposed several deep learning methods to predict DDIs. However, these methods mainly exploit the chemical or biological features of drugs, which is insufficient and limits the performances of DDI prediction. Here, we propose a new deep multimodal feature fusion framework for DDI prediction, DMFDDI, which fuses drug molecular graph, DDI network and the biochemical similarity features of drugs to predict DDIs. To fully extract drug molecular structure, we introduce an attention-gated graph neural network for capturing the global features of the molecular graph and the local features of each atom. A sparse graph convolution network is introduced to learn the topological structure information of the DDI network. In the multimodal feature fusion module, an attention mechanism is used to efficiently fuse different features. To validate the performance of DMFDDI, we compare it with 10 state-of-the-art methods. The comparison results demonstrate that DMFDDI achieves better performance in DDI prediction. Our method DMFDDI is implemented in Python using the Pytorch machine-learning library, and it is freely available at https://github.com/DHUDEBLab/DMFDDI.git.

Android malware detection method based on graph attention networks and deep fusion of multimodal features

Currently, Android malware detection methods always focus on one kind of app feature, such as structural, semantic, or other statistical features. This paper proposes a novel Android malware detection method that integrates multiple features of Android applications. First, to effectively extract the structural and semantic features, we propose a new type of call graph named the class-set call graph (CSCG) that uses the sets of Java classes as nodes and the call relationships between class sets as edges, and we design a dynamic adaptive CSCG construction method that can automatically determine the node size for applications with different scales. The topic model is used to mine the source code semantics from the class sets as the node features. Then, we use a graph attention network (GAT) with max pooling to extract the CSCG feature that encompass both the semantic and structural features of the Android application. Furthermore, we construct a deep multimodal feature fusion network to fuse the CSCG features with permission features. Experimental results show that our method achieves a detection accuracy of 97.28%–99.54% on the three constructed datasets, which is better than the existing methods.

Multimodal sentiment analysis for social media contents during public emergencies

Abstract Purpose Nowadays, public opinions during public emergencies involve not only textual contents but also contain images. However, the existing works mainly focus on textual contents and they do not provide a satisfactory accuracy of sentiment analysis, lacking the combination of multimodal contents. In this paper, we propose to combine texts and images generated in the social media to perform sentiment analysis. Design/methodology/approach We propose a Deep Multimodal Fusion Model (DMFM), which combines textual and visual sentiment analysis. We first train word2vec model on a large-scale public emergency corpus to obtain semantic-rich word vectors as the input of textual sentiment analysis. BiLSTM is employed to generate encoded textual embeddings. To fully excavate visual information from images, a modified pretrained VGG16-based sentiment analysis network is used with the best-performed fine-tuning strategy. A multimodal fusion method is implemented to fuse textual and visual embeddings completely, producing predicted labels. Findings We performed extensive experiments on Weibo and Twitter public emergency datasets, to evaluate the performance of our proposed model. Experimental results demonstrate that the DMFM provides higher accuracy compared with baseline models. The introduction of images can boost the performance of sentiment analysis during public emergencies. Research limitations In the future, we will test our model in a wider dataset. We will also consider a better way to learn the multimodal fusion information. Practical implications We build an efficient multimodal sentiment analysis model for the social media contents during public emergencies. Originality/value We consider the images posted by online users during public emergencies on social platforms. The proposed method can present a novel scope for sentiment analysis during public emergencies and provide the decision support for the government when formulating policies in public emergencies.

Deep multi-modal fusion network with gated unit for breast cancer survival prediction

Accurate survival prediction is a critical goal in the prognosis of breast cancer patients because it can help physicians make more patient-friendly decisions and further guide appropriate treatment. Breast cancer is often caused by genetic abnormalities, which prompts researchers to consider information such as gene expression and copy number variation in addition to clinical data in their studies. The integration of these multi-modal data can improve the predictive power of models. However, with the highly unbalanced information of breast cancer patient data, it becomes a new challenge for breast cancer patient survival prediction to fully extract the characteristic information of these multi-modal data and to consider the complementarity of this information. To this end, we propose a deep multi-modal fusion network (DMMFN) to predict the five-year survival of breast cancer patients by integrating clinical data, copy number variation data, and gene expression data. The imbalanced dataset is first processed using the oversampling method SMOTE-NC. Then the abstract modal features of the multi-modal data are extracted by the two-layer one-dimensional convolutional neural network and the bi-directional long short-term memory network. Next, the weight coefficients of each modal data are dynamically adjusted using gated multimodal units to obtain fusion features. Finally, the fusion features are fed into the MaxoutMLP classifier to obtain the final prediction results. We conducted experiments on the METABRIC dataset to verify the validity of the multi-modal data and compared it with other methods. The comprehensive performance evaluation shows that DMMFN has better prediction performance.

A novel digital twin approach based on deep multimodal information fusion for aero-engine fault diagnosis

Condition monitoring and fault diagnosis play an important role in the safety and reliability of aero-engine. Digital twin (DT) technology, which can realize the fusion of physical space and virtual space, has significant advantages over previous researches that only focus on physical mechanisms or big data. In this paper, a novel DT approach based on deep multimodal information fusion (MIF) is proposed, which integrates information from the physical-based model (PBM) and the data-driven model. Two deep Boltzmann machines (DBMs) are constructed for feature extraction from sensor data and nonlinear component-level model simulation data, respectively. Whereby information from these two modalities is mapped into a high-dimensional space and forms a joint representation, and then combined with a multi-layer feedforward neural network to form the MIF model for real-time fault detection and isolation. In addition, an adaptive correction model for performance degradation is constructed by additionally analyzing the probability distribution of engine operation data. Compared with the traditional single-modality method, the proposed DT approach fuses the information of two key modalities and realizes the adaptive updating of the PBM model. The experimental results indicate that the proposed DT approach improves the accuracy of fault diagnosis and reduces the error of parameter prediction.

Deep Multi-Modal U-Net Fusion Methodology of Thermal and Ultrasonic Images for Porosity Detection in Additive Manufacturing

AbstractWe developed a deep fusion methodology of nondestructive in-situ thermal and ex-situ ultrasonic images for porosity detection in laser-based additive manufacturing (LBAM). A core challenge with the LBAM is the lack of fusion between successive layers of printed metal. Ultrasonic imaging can capture structural abnormalities by passing waves through successive layers. Alternatively, in-situ thermal images track the thermal history during fabrication. The proposed sensor fusion U-Net methodology fills the gap in fusing in-situ images with ex-situ images by employing a two-branch convolutional neural network (CNN) for feature extraction and segmentation to produce a 2D image of porosity. We modify the U-Net framework with the inception and long short term memory (LSTM) blocks. We validate the models by comparing our single modality models and fusion models with ground truth X-ray computed tomography (XCT) images. The inception U-Net fusion model achieved the highest mean intersection over union score of 0.93.

MMHFNet: Multi-modal and multi-layer hybrid fusion network for voice pathology detection

Automatic voice pathology detection using non-invasive techniques that utilize patients’ speech and electroglottograph (EGG) signals play a vital role in diagnosis and early medical intervention. In this paper, a novel deep Multi-Modal and Multi-Layer Hybrid Fusion Network (MMHFNet) is proposed to improve the performance of non-invasive voice pathology detection systems. MMHFNet simultaneously incorporates complementary information of different modalities (speech and EGG signals). It also vertically combines the low-level features, extracted from shallow layers, and high-level features, extracted from deep layers, to take the full advantage of spatio-spectral information of different layers for multi-layer fusion. The features extracted by MMHFNet are then fed into an LSTM classification network to diagnose the voice pathology. Comprehensive experiments are conducted on the publicly available Saarbruecken Voice Database (SVD) to evaluate the performance of the proposed MMHFNet. This dataset is used in two manners; one using its all samples and the other with selected samples to form the largest balanced SVD dataset. Experimental results demonstrated that the proposed MMHFNet achieves accuracy rates of 91% and 96.05% for datasets with all and balanced samples, respectively.

Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction.

In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients' time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient's clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient's time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline.

Deep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation

Multimodal sequence learning aims to utilize information from different modalities to enhance overall performance. Mainstream works often follow an intermediate-fusion pipeline, which explores both modality-specific and modality-supplementary information for fusion. However, the unaligned and heterogeneously distributed multimodal sequences pose significant challenges to the fusion task: 1) to extract both effective unimodal and crossmodal representations and 2) to overcome the overfitting issue in joint multimodal sequence optimization. In this work, we propose regularized expressive representation distillation (RERD) that aims to seek effective multimodal representations and to enhance the generalization of fusion. First, to improve unimodal representation learning, unimodal representations are assigned to multi-head distillation encoders, where the unimodal representations are iteratively updated through distillation attention layers. Second, to alleviate the overfitting issue in joint crossmodal optimization, a multimodal sinkhorn distance regularizer is proposed to reinforce the expressive representation extraction and to reduce the modality gap before fusion adaptively. These representations produce a comprehensive view of the multimodal sequences, which are utilized for downstream fusion tasks. Experimental results on several popular benchmarks demonstrate that the proposed method achieves state-of-the-art performance, compared with widely used baselines for deep multimodal sequence fusion. Source code will be shared publicly.

Deep Multimodal Fusion Model for Building Structural Type Recognition Using Multisource Remote Sensing Images and Building-Related Knowledge

Deep Multimodal Fusion Model for Building Structural Type Recognition Using Multisource Remote Sensing Images and Building-Related Knowledge