Multimodal Deep Learning Method Research Articles

Sarcasm detection in hotel reviews: a multimodal deep learning approach

Purpose This study aims to detect consumer sarcasm through inconsistencies in sentiment features between text and images of hotel reviews. Design/methodology/approach This paper proposes a model for sarcasm detection based on multimodal deep learning using reviews of three hotel brands collected from two travel platforms, which can identify emotional inconsistencies within a modality and across modalities. Text-image interaction information is explored using graph neural networks (GNN) to detect essential clues in sarcasm sentiment. Findings The research results show that the multimodal deep learning model outperforms other baseline models, which can help to understand hotel service evaluation and provide hotel managers with decision-making opinions. Originality/value This research can help hoteliers in two ways: detecting service quality and formulating strategies. By selecting reference hotel brands, hoteliers can better assess their level of service quality (optimal resource allocation ensues); therefore, sarcasm detection research is not only beneficial for hotel managers seeking to improve service quality. The multimodal deep learning method introduced in the present study can be replicated in other industries to help travel platforms optimize their products and services.

Multimodal deep learning-based diagnostic model for BPPV

BackgroundBenign paroxysmal positional vertigo (BPPV) is a prevalent form of vertigo that necessitates a skilled physician to diagnose by observing the nystagmus and vertigo resulting from specific changes in the patient’s position. In this study, we aim to explore the integration of eye movement video and position information for BPPV diagnosis and apply artificial intelligence (AI) methods to improve the accuracy of BPPV diagnosis.MethodsWe collected eye movement video and diagnostic data from 518 patients with BPPV who visited the hospital for examination from January to March 2021 and developed a BPPV dataset. Based on the characteristics of the dataset, we propose a multimodal deep learning diagnostic model, which combines a video understanding model, self-encoder, and cross-attention mechanism structure.ResultOur validation test on the test set showed that the average accuracy of the model reached 81.7%, demonstrating the effectiveness of the proposed multimodal deep learning method for BPPV diagnosis. Furthermore, our study highlights the significance of combining head position information and eye movement information in BPPV diagnosis. We also found that postural and eye movement information plays a critical role in the diagnosis of BPPV, as demonstrated by exploring the necessity of postural information for the diagnostic model and the contribution of cross-attention mechanisms to the fusion of postural and oculomotor information. Our results underscore the potential of AI-based methods for improving the accuracy of BPPV diagnosis and the importance of considering both postural and oculomotor information in BPPV diagnosis.

Automatic classification of immune-mediated glomerular diseases based on multi-modal multi-instance learning

To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy (OM), immunofluorescence microscopy (IM), and transmission electron microscopy (TEM). We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi- instance model for classification of 3 immune-mediated glomerular diseases, namely immunoglobulin A nephropathy (IgAN), membranous nephropathy (MN), and lupus nephritis (LN). This model adopts an instance-level multi-instance learning (I-MIL) method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient. By comparing this model with unimodal and bimodal models, we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion. The multi-modal multi-instance model combining OM, IM, and TEM images had a disease classification accuracy of (88.34±2.12)%, superior to that of the optimal unimodal model [(87.08±4.25)%] and that of the optimal bimodal model [(87.92±3.06)%]. This multi- modal multi- instance model based on OM, IM, and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.

Multi-modal deep learning methods for classification of chest diseases using different medical imaging and cough sounds.

Chest disease refers to a wide range of conditions affecting the lungs, such as COVID-19, lung cancer (LC), consolidation lung (COL), and many more. When diagnosing chest disorders medical professionals may be thrown off by the overlapping symptoms (such as fever, cough, sore throat, etc.). Additionally, researchers and medical professionals make use of chest X-rays (CXR), cough sounds, and computed tomography (CT) scans to diagnose chest disorders. The present study aims to classify the nine different conditions of chest disorders, including COVID-19, LC, COL, atelectasis (ATE), tuberculosis (TB), pneumothorax (PNEUTH), edema (EDE), pneumonia (PNEU). Thus, we suggested four novel convolutional neural network (CNN) models that train distinct image-level representations for nine different chest disease classifications by extracting features from images. Furthermore, the proposed CNN employed several new approaches such as a max-pooling layer, batch normalization layers (BANL), dropout, rank-based average pooling (RBAP), and multiple-way data generation (MWDG). The scalogram method is utilized to transform the sounds of coughing into a visual representation. Before beginning to train the model that has been developed, the SMOTE approach is used to calibrate the CXR and CT scans as well as the cough sound images (CSI) of nine different chest disorders. The CXR, CT scan, and CSI used for training and evaluating the proposed model come from 24 publicly available benchmark chest illness datasets. The classification performance of the proposed model is compared with that of seven baseline models, namely Vgg-19, ResNet-101, ResNet-50, DenseNet-121, EfficientNetB0, DenseNet-201, and Inception-V3, in addition to state-of-the-art (SOTA) classifiers. The effectiveness of the proposed model is further demonstrated by the results of the ablation experiments. The proposed model was successful in achieving an accuracy of 99.01%, making it superior to both the baseline models and the SOTA classifiers. As a result, the proposed approach is capable of offering significant support to radiologists and other medical professionals.

Pattern recognition study of musical rhythm perception based on multimodal deep learning

Abstract Rhythm perception is becoming more and more important in the field of music information processing and music understanding. The study first adopts signal processing methods to extract musical features, then uses feature fusion techniques to integrate features of different modalities into a single feature vector. Based on this model, the study identifies the rhythmic activation function of music and combines it with the hidden Markov model to infer the rhythm of the music. One of the key points of the study is to perform rhythm recognition on music containing drums, to explore the recognition effect. One of the focuses of the study is to recognize the rhythm of music containing drums to explore the recognition effect.In addition, the study also analyzes the Softmax output values of the music and compares the recognition effect of different models. The results show that the rhythm recognition of music using the multimodal deep learning method performs the best in terms of the F-Measure value, the Cemgil value, the Goto value, and the P-score value, with the respective 65.65%, 66.76%, 36.75%, and 36.75%. 66.76%, 36.75%, and 75.68%.Especially in the drum music recognition, the position of each drum music is accurately recognized, proving the model’s effectiveness in this paper. The research provides a new feasible method for the recognition and understanding of music rhythms and a valuable reference for the research in this field.

A High-Performance Multimodal Deep Learning Model for Detecting Minority Class Sample Attacks

A large amount of sensitive information is generated in today’s evolving network environment. Some hackers utilize low-frequency attacks to steal sensitive information from users. This generates minority attack samples in real network traffic. As a result, the data distribution in real network traffic is asymmetric, with a large number of normal traffic and a rare number of attack traffic. To address the data imbalance problem, intrusion detection systems mainly rely on machine-learning-based methods to detect minority attacks. Although this approach can detect minority attacks, the performance is not satisfactory. To solve the above-mentioned problems, this paper proposes a novel high-performance multimodal deep learning method. The method is based on deep learning. It captures the features of minority class attacks based on generative adversarial networks, which in turn generate high-quality minority class sample attacks. Meanwhile, it uses the designed multimodal deep learning model to learn the features of minority attacks. The integrated classifier then utilizes the learned features for multi-class classification. This approach achieves 99.55% and 99.95% F-measure, 99.56% and 99.96% accuracy on the CICIDS2017 dataset and the NSL-KDD dataset, respectively, with the highest false positives at only 3.4%. This exceeds the performance of current state-of-the-art methods.

Predicting stroke outcome: A case for multimodal deep learning methods with tabular and CT Perfusion data

Motivation:Acute ischemic stroke is one of the leading causes of morbidity and disability worldwide, often followed by a long rehabilitation period. To improve and personalize stroke rehabilitation, it is essential to provide a reliable prognosis to caregivers and patients. Deep learning techniques might improve the predictions by incorporating different data modalities. We present a multimodal approach to predict the functional status of acute ischemic stroke patients after their discharge based on tabular data and CT perfusion imaging. Methods:We conducted experiments on tabular, imaging, and multimodal deep learning architectures to predict dichotomized mRS scores 3 months after the event. The dataset was collected from a Dutch hospital and includes 98 CVA patients with a visible occlusion on their CT perfusion scan. Tabular data is based on the Dutch Acute Stroke Audit data, and imaging data consists of summed-up CT perfusion maps. Results:On the tabular data, TabNet outperformed our baselines with an AUC of 0.71, while ResNet-10 on the imaging data performed comparably with an AUC of 0.70. Our implementation of the multimodal DAFT architecture outperforms baselines as well as comparable studies by achieving an 0.75 AUC, and 0.80 F1 score. This was achieved with a final model of less than a hundred thousand optimizable parameters, and a dataset less than half the size of reference papers. Conclusion:Overall, we demonstrate the feasibility of predicting the functional outcome for ischemic stroke patients and the usability of multimodal deep learning architectures for this purpose.

On the consensus of synchronous temporal and spatial views: A novel multimodal deep learning method for social video prediction

The blowout development of video social platforms has spawned a wide range of social video prediction (SVP) tasks, such as video attractiveness prediction and video sentiment classification. In this paper, we propose to enhance SVP by making synchronous predictions based on temporal and spatial data perspectives and reconciling them to form a consistent predictive view. To this end, we develop a novel multimodal deep learning method named MATSC (modality-awareness- and temporal-spatial-consistency-based neural network). Specifically, MATSC first constructs the temporal predictive view by capturing valuable fine-grained data patterns and generating diverse multimodal representations via the modality-awareness learning strategy. Then, MATSC constructs the spatial predictive view by exploiting diverse modality-wise interactive patterns in fine-grained video clips. Third, MATSC reconciles the heterogeneous temporal and spatial predictive capabilities via a temporal-spatial-consistency learning objective. Empirical results based on three SVP datasets show the outperformance of MATSC over state-of-the-art benchmarks, demonstrating the enhancement effect of synergizing temporal and spatial data views for SVP tasks.

Multi-Modal Deep Learning for Effective Malicious Webpage Detection

The pervasive threat of malicious webpages, which can lead to financial loss, data breaches, and malware infections, underscores the need for effective detection methods. Conventional techniques for detecting malicious web content primarily rely on URL-based features or features extracted from various webpage components, employing a single feature vector input into a machine learning model for classifying webpages as benign or malicious. However, these approaches insufficiently address the complexities inherent in malicious webpages. To overcome this limitation, a novel Multi-Modal Deep Learning method for malicious webpage detection is proposed in this study. Three types of automatically extracted features, specifically those derived from the URL, the JavaScript code, and the webpage text, are leveraged. Each feature type is processed by a distinct deep learning model, facilitating a comprehensive analysis of the webpage. The proposed method demonstrates a high degree of effectiveness, achieving an accuracy rate of 97.90% and a false negative rate of a mere 2%. The results highlight the advantages of utilizing multi-modal features and deep learning techniques for detecting malicious webpages. By considering various aspects of web content, the proposed method offers improved accuracy and a more comprehensive understanding of malicious activities, thereby enhancing web user security and effectively mitigating the risks associated with malicious webpages.

DeepCIP: A multimodal deep learning method for the prediction of internal ribosome entry sites of circRNAs

Circular RNAs (circRNAs) have been found to have the ability to encode proteins through internal ribosome entry sites (IRESs), which are essential RNA regulatory elements for cap-independent translation. Identification of IRES elements in circRNA is crucial for understanding its function. Previous studies have presented IRES predictors based on machine learning techniques, but they were mainly designed for linear RNA IRES. In this study, we proposed DeepCIP (Deep learning method for CircRNA IRES Prediction), a multimodal deep learning approach that employs both sequence and structural information for circRNA IRES prediction. Our results demonstrate the effectiveness of the sequence and structure models used by DeepCIP in feature extraction and suggest that integrating sequence and structural information efficiently improves the accuracy of prediction. The comparison studies indicate that DeepCIP outperforms other comparative methods on the test set and real circRNA IRES dataset. Furthermore, through the integration of an interpretable analysis mechanism, we elucidate the sequence patterns learned by our model, which align with the previous discovery of motifs that facilitate circRNA translation. Thus, DeepCIP has the potential to enhance the study of the coding potential of circRNAs and contribute to the design of circRNA-based drugs. DeepCIP as a standalone program is freely available at https://github.org/zjupgx/DeepCIP.

Multimodal Deep Learning Methods on Image and Textual Data to Predict Radiotherapy Structure Names

Physicians often label anatomical structure sets in Digital Imaging and Communications in Medicine (DICOM) images with nonstandard random names. Hence, the standardization of these names for the Organs at Risk (OARs), Planning Target Volumes (PTVs), and ‘Other’ organs is a vital problem. This paper presents novel deep learning methods on structure sets by integrating multimodal data compiled from the radiotherapy centers of the US Veterans Health Administration (VHA) and Virginia Commonwealth University (VCU). These de-identified data comprise 16,290 prostate structures. Our method integrates the multimodal textual and imaging data with Convolutional Neural Network (CNN)-based deep learning approaches such as CNN, Visual Geometry Group (VGG) network, and Residual Network (ResNet) and shows improved results in prostate radiotherapy structure name standardization. Evaluation with macro-averaged F1 score shows that our model with single-modal textual data usually performs better than previous studies. The models perform well on textual data alone, while the addition of imaging data shows that deep neural networks achieve better performance using information present in other modalities. Additionally, using masked images and masked doses along with text leads to an overall performance improvement with the CNN-based architectures than using all the modalities together. Undersampling the majority class leads to further performance enhancement. The VGG network on the masked image-dose data combined with CNNs on the text data performs the best and presents the state-of-the-art in this domain.

A Deep Feature Fusion Method for Complex Ground Object Classification in the Land Cover Ecosystem Using ZY1-02D and Sentinel-1A

Despite the successful application of multimodal deep learning (MDL) methods for land use/land cover (LULC) classification tasks, their fusion capacity has not yet been substantially examined for hyperspectral and synthetic aperture radar (SAR) data. Hyperspectral and SAR data have recently been widely used in land cover classification. However, the speckle noise of SAR and the heterogeneity with the imaging mechanism of hyperspectral data have hindered the application of MDL methods for integrating hyperspectral and SAR data. Accordingly, we proposed a deep feature fusion method called Refine-EndNet that combines a dynamic filter network (DFN), an attention mechanism (AM), and an encoder–decoder framework (EndNet). The proposed method is specifically designed for hyperspectral and SAR data and adopts an intra-group and inter-group feature fusion strategy. In intra-group feature fusion, the spectral information of hyperspectral data is integrated by fully connected neural networks in the feature dimension. The fusion filter generation network (FFGN) suppresses the presence of speckle noise and the influence of heterogeneity between multimodal data. In inter-group feature fusion, the fusion weight generation network (FWGN) further optimizes complementary information and improves fusion capacity. Experimental results from ZY1-02D satellite hyperspectral data and Sentinel-1A dual-polarimetric SAR data illustrate that the proposed method outperforms the conventional feature-level image fusion (FLIF) and MDL methods, such as S2ENet, FusAtNet, and EndNets, both visually and numerically. We first attempt to investigate the potentials of ZY1-02D satellite hyperspectral data affected by thick clouds, combined with SAR data for complex ground object classification in the land cover ecosystem.

Semantic-Aligned Cross-Modal Visual Grounding Network with Transformers

Multi-modal deep learning methods have achieved great improvements in visual grounding; their objective is to localize text-specified objects in images. Most of the existing methods can localize and classify objects with significant appearance differences but suffer from the misclassification problem for extremely similar objects, due to inadequate exploration of multi-modal features. To address this problem, we propose a novel semantic-aligned cross-modal visual grounding network with transformers (SAC-VGNet). SAC-VGNet integrates visual and textual features with semantic alignment to highlight important feature cues for capturing tiny differences between similar objects. Technically, SAC-VGNet incorporates a multi-modal fusion module to effectively fuse visual and textual descriptions. It also introduces contrastive learning to align linguistic and visual features on the text-to-pixel level, enabling the capture of subtle differences between objects. The overall architecture is end-to-end without the need for extra parameter settings. To evaluate our approach, we manually annotate text descriptions for images in two fine-grained visual grounding datasets. The experimental results demonstrate that SAC-VGNet significantly improves performance in fine-grained visual grounding.

Forecasting hourly attraction tourist volume with search engine and social media data for decision support

Developing a tourism forecasting function in decision support systems has become critical for businesses and governments. The existing forecasting models considering spatial relations contain insufficient information, and the spatial aggregation of simple tourist volume series limits the forecasting accuracy. Using human-generated search engines and social media data has the potential to address this issue. In this paper, a spatial aggregation-based multimodal deep learning method for hourly attraction tourist volume forecasting is developed. The model first extracts the daily features of attractions from search engine data; then mines the spatial aggregation relationships in social media data and multi-attraction tourist volume data. Finally, the model fuses hourly features with daily features to make forecasting. The model is tested using a dataset containing several attractions with real-time tourist volume at 15-minute intervals from November 27, 2018, to March 18, 2019, in Beijing. And the empirical and Diebold-Mariano test results demonstrate that the proposed framework can outperform state-of-the-art baseline models with statistically significant improvements at the 1% level. Compared with the best baseline model, the MAPE values are reduced by 50.0% and 27.3% in 4A attractions and 5A attractions, respectively; and the RMSE values are reduced by 48.3% and 26.1%, respectively. The method in this paper can be used as a function embedded in the decision support system to help multi-department collaboration.

Correction to: Multimodal deep learning methods enhance genomic prediction of wheat breeding

Correction to: Multimodal deep learning methods enhance genomic prediction of wheat breeding

Multidimensional heterogeneous data clustering algorithm for power transmission and transformation equipment

Considering that the operation of power transmission and transformation equipment is not timely discovered due to the untimely data integration, a multi-dimensional heterogeneous data clustering algorithm for power transmission and transformation equipment based on multimodal deep learning is proposed. The multi-modal deep learning method is used to mine relevant data and measure the similarity between the data, which can improve the accuracy of subsequent multi-bit heterogeneous data clustering of power transmission and transformation equipment. Set up a clustering center and process data clustering to complete multi-dimensional heterogeneous data clustering of power transmission and transformation equipment. The experimental results show that the method has high clustering accuracy in the clustering of voltage deviation overrun times, voltage harmonic total distortion rate overrun times, and voltage flicker overrun times.

Multimodal deep learning methods enhance genomic prediction of wheat breeding.

While several statistical machine learning methods have been developed and studied for assessing the genomic prediction (GP) accuracy of unobserved phenotypes in plant breeding research, few methods have linked genomics and phenomics (imaging). Deep learning (DL) neural networks have been developed to increase the GP accuracy of unobserved phenotypes while simultaneously accounting for the complexity of genotype-environment interaction (GE); however, unlike conventional GP models, DL has not been investigated for when genomics is linked with phenomics. In this study we used 2 wheat data sets (DS1 and DS2) to compare a novel DL method with conventional GP models. Models fitted for DS1 were GBLUP, gradient boosting machine (GBM), support vector regression (SVR) and the DL method. Results indicated that for 1 year, DL provided better GP accuracy than results obtained by the other models. However, GP accuracy obtained for other years indicated that the GBLUP model was slightly superior to the DL. DS2 is comprised only of genomic data from wheat lines tested for 3 years, 2 environments (drought and irrigated) and 2-4 traits. DS2 results showed that when predicting the irrigated environment with the drought environment, DL had higher accuracy than the GBLUP model in all analyzed traits and years. When predicting drought environment with information on the irrigated environment, the DL model and GBLUP model had similar accuracy. The DL method used in this study is novel and presents a strong degree of generalization as several modules can potentially be incorporated and concatenated to produce an output for a multi-input data structure.

A Review on Methods and Applications in Multimodal Deep Learning

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This article focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, physiological signals, flow, RGB, pose, depth, mesh, and point cloud. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the past five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Last, main issues are highlighted separately for each domain, along with their possible future research directions.

Attentive statement fraud detection: Distinguishing multimodal financial data with fine-grained attention

Financial statement fraud caused by listed companies directly jeopardizes the reliability the financial reporting process. Leveraging multimodal information for financial statement fraud detection (FSFD) has recently become of great interest to academic research and industrial applications. Unfortunately, the predictive ability of multimodal information in FSFD remains largely underexplored, particularly the fusion ambiguity embedded in and among multi-modalities. In this study, we propose a novel attention-based multimodal deep learning method, named RCMA, toward an accurate FSFD. RCMA synthesizes a fine-grained attention mechanism including three innovative attention modules, i.e., ratio-aware attention, chapter-aware attention, and modality-aware attention mechanism. The first two attention mechanisms help to liberate the extraordinary predictive power of the financial modality and the textual modality on FSFD, respectively. Moreover, the proposed modality-aware attention mechanism enables better coordination between the two modalities. Furthermore, to ensure effective learning on the attention-based multimodal embedding, we design a novel loss function named Focal and Consistency Loss, or FCL. It considers class-imbalance and modality-consistency simultaneously, to specialize the optimization of FSFD. The experimental results on the real-world dataset show that the proposed RCMA on FSFD task outperformed the state-of-the-art benchmarks. Furthermore, interpretation analysis visualizes the attention weights of different ratio groups, chapters, and modalities from RCMA, and illustrates how these interpretations influence stakeholders' decision process for FSFD.

A Multimodal Deep Learning Approach to Predicting Systemic Diseases from Oral Conditions.

Background: It is known that oral diseases such as periodontal (gum) disease are closely linked to various systemic diseases and disorders. Deep learning advances have the potential to make major contributions to healthcare, particularly in the domains that rely on medical imaging. Incorporating non-imaging information based on clinical and laboratory data may allow clinicians to make more comprehensive and accurate decisions. Methods: Here, we developed a multimodal deep learning method to predict systemic diseases and disorders from oral health conditions. A dual-loss autoencoder was used in the first phase to extract periodontal disease-related features from 1188 panoramic radiographs. Then, in the second phase, we fused the image features with the demographic data and clinical information taken from electronic health records (EHR) to predict systemic diseases. We used receiver operation characteristics (ROC) and accuracy to evaluate our model. The model was further validated by an unseen test dataset. Findings: According to our findings, the top three most accurately predicted chapters, in order, are the Chapters III, VI and IX. The results indicated that the proposed model could predict systemic diseases belonging to Chapters III, VI and IX, with AUC values of 0.92 (95% CI, 0.90-94), 0.87 (95% CI, 0.84-89) and 0.78 (95% CI, 0.75-81), respectively. To assess the robustness of the models, we performed the evaluation on the unseen test dataset for these chapters and the results showed an accuracy of 0.88, 0.82 and 0.72 for Chapters III, VI and IX, respectively. Interpretation: The present study shows that the combination of panoramic radiograph and clinical oral features could be considered to train a fusion deep learning model for predicting systemic diseases and disorders.