Single Modality Data Research Articles

Explainable multimodal learning for predictive maintenance of steam generators

Prognostics and Health Management (PHM) is identified as an important lever for enhancing the development of predictive maintenance to ensure the reliability, availability, and safety of industrial systems. However, the efficiency of data- driven PHM approaches is dependent on the quality and quantity of data. Therefore, exploiting multiple data sources can provide additional, useful information than single-modal data. For instance, by incorporating multiple data sources, including condition monitoring data, images from cameras, and texts from maintenance technicians’ reports, multi-modal learning can provide a more comprehensive and accurate understanding of the system’s health. However, multi-modal deep learning is complex to understand. To address this complexity, it is crucial to incorporate explainable artificial intelligent techniques to provide clear and interpretable insights into how the model makes decisions. In this light, this paper proposes the application of the model-agnostic-explanation approach, i.e., SHAP, to explain the working mechanism of multimodal learning for the prediction of industrial steam generator degradation. Particularly, we determine the important features of each data modality and investigate how multimodal learning can overcome the issues of low-quality data from a single modality due to the additional information from other data modalities.

A Survey of Deep Learning-based Facial Expression Recognition Research

Facial expression is one of the ways to convey emotional expression. Deep learning is used to analyze facial expression to understand people's true feelings, and human-computer interaction is integrated into it. However, in the natural real environment and various interference (such as lighting, age and ethnicity), facial expression recognition will face many challenges. In recent years, with the development of artificial intelligence, scholars have studied more and more facial expression recognition in the case of interference, which not only promotes the theoretical research, but also makes it popularized in the application. Facial expression recognition is to identify facial expressions to carry out emotion analysis, and emotion analysis can be analyzed with the help of facial expressions, speech, text, video and other signals. Therefore, facial expression recognition can be regarded as a research direction of emotion analysis. This paper focuses on the perspective of facial expression recognition to summarize. In the process of facial expression recognition, researchers usually try to combine multiple modal information such as voice, text, picture and video for analysis. Due to the differences between single-modal data set and multi-modal data set, this paper will analyze static facial expression recognition, dynamic facial expression recognition and multi-modal fusion. This research has a wide range of applications, such as: smart elderly care, medical research, detection of fatigue driving and other fields.

SymmetricNet: end-to-end mesoscale eddy detection with multi-modal data fusion

Mesoscale eddies play a significant role in marine energy and matter transportation. Due to their huge impact on the ocean, mesoscale eddy detection has been studied for many years. However, existing methods mainly use single-modal data, such as the sea surface height (SSH), to detect mesoscale eddies, resulting in inaccurate detection results. In this paper, we propose an end-to-end mesoscale eddy detection method based upon multi-modal data fusion. Particularly, we don’t only use SSH, but also add data of other two modals, i.e., the sea surface temperature (SST) and the velocity of flow, which are closely related to mesoscale eddy detection. Moreover, we design a novel network named SymmetricNet, which is able to achieve multi-modal data fusion in mesoscale eddy detection. The proposed SymmetricNet mainly contains a downsampling pathway and an upsampling pathway, where the low-level feature maps from the downsampling pathway and the high-level feature maps from the upsampling pathway are merged through lateral connections. In addition, we apply dilated convolutions to the network structure to increase the receptive field without sacrificing resolution. Experiments on multi-modal mesoscale eddy dataset demonstrate the advantages of the proposed method over previous approaches for mesoscale eddy detection.

Classification of multi-modal remote sensing images based on knowledge graph

ABSTRACT With the development of remote sensing (RS) technology, single-modal data alone has gradually become difficult to meet the requirement for high accuracy of RS image classification. As a result, multi-modal RS image classification has become a hot research topic. However, multi-modal RS image classification faces challenges in effectively utilizing advanced semantic information regarding the relationships between land cover classes and extracting discriminative features from the data. Based on this, a multi-modal RS image classification method based on knowledge graph (KG) is proposed. Firstly, graph topology is used to align the hyperspectral image (HSI) and multispectral image (MSI) features extracted by graph convolution networks. Constraint alignment is performed on different feature graphs to reduce the difficulty of fusion and the false recognition rate. Then, we use self-attention and cross-attention to purposefully fuse HSI and MSI to obtain discriminative features rich in two modal information and achieve feature weighted fusion. Finally, a KG based on object spatial relationships is constructed to obtain spatial relationships between different classes to assist in multi-modal RS image classification. The experimental results on the Houston and Ausburg datasets demonstrate that the proposed method achieves overall accuracy of 90.40% and 90.85%, respectively, both of which are more than 3% higher than existing classification methods. The results indicate that our method has better classification performance and can provide a useful reference for RS image classification research.

Multimodal integration for data-driven classification of mental fatigue during construction equipment operations: Incorporating electroencephalography, electrodermal activity, and video signals

Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.

Inversion of Leaf Area Index in Citrus Trees Based on Multi-Modal Data Fusion from UAV Platform

The leaf area index (LAI) is an important growth indicator used to assess the health status and growth of citrus trees. Although LAI estimation based on unmanned aerial vehicle (UAV) platforms has been widely used for field crops, mainly focusing on food crops, less research has been reported on the application to fruit trees, especially citrus trees. In addition, most studies have used single-modal data for modeling, but some studies have shown that multi-modal data can be effective in improving experimental results. This study utilizes data collected from a UAV platform, including RGB images and point cloud data, to construct single-modal regression models named VoVNet (using RGB data) and PCNet (using point cloud data), as well as a multi-modal regression model called VPNet (using both RGB data and point cloud data). The LAI of citrus trees was estimated using deep neural networks, and the results of two experimental hyperparameters (loss function and learning rate) were compared under different parameters. The results of the study showed that VoVNet had Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-Squared (R2) of 0.129, 0.028, and 0.647, respectively. In comparison, PCNet decreased by 0.051 and 0.014 to 0.078 and 0.014 for MAE and MSE, respectively, while R2 increased by 0.168 to 0.815. VPNet decreased by 0% and 42.9% relative to PCNet in terms of MAE and MSE to 0.078 and 0.008, respectively, while R2 increased by 5.6% to 0.861. In addition, the use of loss function L1 gave better results than L2, while a lower learning rate gave better results. It is concluded that the fusion of RGB data and point cloud data collected by the UAV platform for LAI estimation is capable of monitoring citrus trees’ growth process, which can help farmers to track the growth condition of citrus trees and improve the efficiency and quality of orchard management.

FL-FD: Federated learning-based fall detection with multimodal data fusion

Multimodal data fusion is a critical element of fall detection systems, as it provides more comprehensive information than single-modal data. Yet, data heterogeneity between sources has posed a challenge for the effective fusion of such data. This paper proposes a novel multimodal data fusion method under a federated learning (FL) framework that addresses the privacy concerns of users while exploiting the complementarity of such data. Specifically, we fuse time-series data from wearable sensors and visual data from cameras at the input level, where the data is first transformed into images using the Gramian Angular Field (GAF) method. Moreover, each user is treated as a private client in the FL system whereby the fall detection model is trained without requiring the sharing of user data. The proposed method is evaluated using the UP-Fall dataset, where we perform different fall detection tasks: binary classification for fall and non-fall detection yields a remarkable accuracy of 99.927%, while multi-classification for different fall activity recognition attains an accurate result of 89.769%.

Multimodal learning on graphs for disease relation extraction

Disease knowledge graphs have emerged as a powerful tool for artificial intelligence to connect, organize, and access diverse information about diseases. Relations between disease concepts are often distributed across multiple datasets, including unstructured plain text datasets and incomplete disease knowledge graphs. Extracting disease relations from multimodal data sources is thus crucial for constructing accurate and comprehensive disease knowledge graphs. We introduce REMAP, a multimodal approach for disease relation extraction. The REMAP machine learning approach jointly embeds a partial, incomplete knowledge graph and a medical language dataset into a compact latent vector space, aligning the multimodal embeddings for optimal disease relation extraction. Additionally, REMAP utilizes a decoupled model structure to enable inference in single-modal data, which can be applied under missing modality scenarios. We apply the REMAP approach to a disease knowledge graph with 96,913 relations and a text dataset of 1.24 million sentences. On a dataset annotated by human experts, REMAP improves language-based disease relation extraction by 10.0% (accuracy) and 17.2% (F1-score) by fusing disease knowledge graphs with language information. Furthermore, REMAP leverages text information to recommend new relationships in the knowledge graph, outperforming graph-based methods by 8.4% (accuracy) and 10.4% (F1-score). REMAP is a flexible multimodal approach for extracting disease relations by fusing structured knowledge and language information. This approach provides a powerful model to easily find, access, and evaluate relations between disease concepts.

RGBT tracking based on prior least absolute shrinkage and selection operator and quality aware fusion of deep and handcrafted features

Object tracking has always been one of the most challenging topics in machine vision. In recent years, trackers have used RGB-T datasets to overcome the limitations of single-modality data. This paper introduces a Prior Least Absolute Shrinkage and Selection Operator (PLASSO)-based cost function optimized by the Alternating Direction Method of Multipliers (ADMM) for a Discriminative Correlation Filter (DCF)-based tracker. Detailed closed-form solutions for these optimization problems are also given in this paper. The proposed PLASSO-based approach adaptively determines the scale, parameters, and features of the PLASSO-ADSPF tracker. PLASSO- ADSPF first finds the response map in the Fourier domain for the input images and then uses an efficient adaptive function to fuse the results. Based on the response maps, the proposed tracker adaptively and consciously extracts features (handcraft and Deep Features (DF)) to improve the speed of the tracking algorithm. PLASSO-ADSPF extracts DF from the convolution layer output of a modified pre-trained deep network via SoftMax Tsallis Entropy (TsEn) as a proposed cost function in the last layer. Also, a graph is proposed to determine longitudinal and transverse scales independently and accurately. Moreover, a new bounding box regression is proposed based on PLASSO and Cholesky factorization to refine the target box. Extensive experiments have been performed for the PLASSO-ADSPF tracker on five popular datasets. The results show the superiority of the proposed approach compared to the state-of-the-art trackers. Our code is available at PLASSO-ADSPF-tracker.

In Your Eyes: Modality Disentangling for Personality Analysis in Short Video

With the dramatic growth of various short video platforms, users are more likely to share their social stream online and make their social connections stronger. To better understand their preferences, personality analysis has attracted more attention. Unlike single modal data such as text or images, which is hard to comprehensively uncover one's personal traits, personality analysis on short video is verified to be much more accurate but also more challenging because of the huge gap between incompatible data modalities. We have noticed that the key problem is how to disentangle the complexity from multimodal data to find their consistency and uniqueness. In this article, we propose a novel video analysis framework for personality detection with visual, acoustic, and textual neural networks. Specifically, to enhance our model's sensitivity to personality detection, we first propose three deep learning channels to learn modal features. The framework can not only extract each modal feature but also learn time-varying pattern via a temporal alignment network. To identify the consistency and uniqueness across multiple modalities, we creatively propose to maximize the similarity of common information learned by a shared neural network across multiple modalities and extend the distance of exclusive information learned by private networks of different modalities. Extensive experiments on the real-world dataset demonstrate that our model can outperform existing baselines.

CNTSeg: A multimodal deep-learning-based network for cranial nerves tract segmentation.

The segmentation of cranial nerves (CNs) tracts based on diffusion magnetic resonance imaging (dMRI) provides a valuable quantitative tool for the analysis of the morphology and course of individual CNs. Tractography-based approaches can describe and analyze the anatomical area of CNs by selecting the reference streamlines in combination with ROIs-based (regions-of-interests) or clustering-based. However, due to the slender structure of CNs and the complex anatomical environment, single-modality data based on dMRI cannot provide a complete and accurate description, resulting in low accuracy or even failure of current algorithms in performing individualized CNs segmentation. In this work, we propose a novel multimodal deep-learning-based multi-class network for automated cranial nerves tract segmentation without using tractography, ROI placement or clustering, called CNTSeg. Specifically, we introduced T1w images, fractional anisotropy (FA) images, and fiber orientation distribution function (fODF) peaks into the training data set, and design the back-end fusion module which uses the complementary information of the interphase feature fusion to improve the segmentation performance. CNTSeg has achieved the segmentation of 5 pairs of CNs (i.e. optic nerve CN II, oculomotor nerve CN III, trigeminal nerve CN V, and facial-vestibulocochlear nerve CN VII/VIII). Extensive comparisons and ablation experiments show promising results and are anatomically convincing even for difficult tracts. The code will be openly available athttps://github.com/IPIS-XieLei/CNTSeg.

A Spatio-Temporal Siamese Neural Network for Multimodal Handwriting Abnormality Screening of Parkinson’s Disease

Currently, hand motion recognition of single-modality data has been extensively explored for the analysis of various contact and noncontact sensors, and it is recognized that all the existing technologies have both strengths and limitations. As a significant motor symptom, hand tremor is usually utilized for the diagnosis and evaluation of Parkinson’s disease; furthermore, a multimodal analysis of the handwriting pattern of the patient has made up for the one-sided way of learning the hand movement in a single measurement dimension. Especially, considering a variety of measurement resources, it shows promising performance in recognizing handwriting patterns of Parkinson’s disease. In this work, a novel Spatio-temporal Siamese neural network (ST-SiamNN) is proposed to learn the handwriting differences between healthy individuals and patients with Parkinson’s disease, process data onto multiple sensors, and enhance the characteristics of handwriting in Parkinson’s disease. Uniquely, it is a discriminative model of multilabel and multinetwork constructed by a Siamese network, which consists of four modules: a preprocessor for handwritten data enhancement, a Siamese bidirectional memory neural network (SiamBiMNN) for temporal and texture feature extraction and difference enhancement, a Siamese octave convolutional neural network (SiamOctCNN) for spatial feature extraction and difference enhancement, and a decision-making layer to rejudge the output features of the Siamese networks to obtain more accurate auxiliary diagnosis results. The framework proposed in this article is verified on two handwritten datasets of multiple modalities, i.e., images, smart pen signals, and graphics tablet signals, which are compared with several state-of-the-art studies.

A joint multi-modal learning method for early-stage knee osteoarthritis disease classification

Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Furthermore, a visual analysis of the important features in the multimodal data verified the relations among the modalities when classifying the grade of knee OA disease.

Emma: An accurate, efficient, and multi-modality strategy for autonomous vehicle angle prediction

Autonomous driving and self-driving vehicles have become the most popular selection for customers for their convenience. Vehicle angle prediction is one of the most prevalent topics in the autonomous driving industry, that is, realizing real-time vehicle angle prediction. However, existing methods of vehicle angle prediction utilize only single-modal data to achieve model prediction, such as images captured by the camera, which limits the performance and efficiency of the prediction system. In this paper, we present Emma, a novel vehicle angle prediction strategy that achieves multi-modal prediction and is more efficient. Specifically, Emma exploits both images and inertial measurement unit (IMU) signals with a fusion network for multi-modal data fusion and vehicle angle prediction. Moreover, we design and implement a few-shot learning module in Emma for fast domain adaptation to varied scenarios (e.g., different vehicle models). Evaluation results demonstrate that Emma achieves overall 97.5% accuracy in predicting three vehicle angle parameters (yaw, pitch, and roll), which outperforms traditional single-modalities by approximately 16.7%–36.8%. Additionally, the few-shot learning module presents promising adaptive ability and shows overall 79.8% and 88.3% accuracy in 5-shot and 10-shot settings, respectively. Finally, empirical results show that Emma reduces energy consumption by 39.7% when running on the Arduino UNO board.

A Pre-Seismic Anomaly Detection Approach Based on Earthquake Cross Partial Multi-View Data Fusion

It is a challenge to detect pre-seismic anomalies by using only one dataset due to the complexity of earthquakes. Therefore, it is a promising direction to use multiparameteric data. The earthquake cross partial multi-view data fusion approach (EQ-CPM) is proposed in this paper. By using this method, electromagnetic data and seismicity indicators are fused. This approach tolerates the absence of data and complements the missing part in fusion. First, the effectiveness of seismicity indicators and electromagnetic data was validated through two earthquake case studies. Then, four machine learning algorithms were applied to detect pre-seismic anomalies by using the fused data and two original datasets. The results show that the fused data provided better performance than the single-modal data. In the Matthews correlation coefficient index, the results of our method showed an 8% improvement compared with the latest study.

Pancancer survival prediction using a deep learning architecture with multimodal representation and integration.

Use of multi-omics data carrying comprehensive signals about the disease is strongly desirable for understanding and predicting disease progression, cancer particularly as a serious disease with a high mortality rate. However, recent methods currently fail to effectively utilize the multi-omics data for cancer survival prediction and thus significantly limiting the accuracy of survival prediction using omics data. In this work, we constructed a deep learning model with multimodal representation and integration to predict the survival of patients using multi-omics data. We first developed an unsupervised learning part to extract high-level feature representations from omics data of different modalities. Then, we used an attention-based method to integrate feature representations, produced by the unsupervised learning part, into a single compact vector and finally we fed the vector into fully connected layers for survival prediction. We used multimodal data to train the model and predict pancancer survival, and the results show that using multimodal data can lead to higher prediction accuracy compared to using single modal data. Furthermore, we used the concordance index and the 5-fold cross-validation method for comparing our proposed method with current state-of-the-art methods and our results show that our model achieves better performance on the majority of cancer types in our testing datasets. https://github.com/ZhangqiJiang07/MultimodalSurvivalPrediction. Supplementary data are available at Bioinformatics online.

FSFM: A Feature Square Tower Fusion Module for Multimodal Object Detection

With the increasing social needs, single-modal data have been unable to provide sufficient information for object detection. Reasonably processing multimodal information and fusing specific information of different modal data is one of the research hotspots in the field of data processing. To this end, this article proposes a feature square tower fusion module called FSFM, which is able to realize multimodal feature fusion by aggregating multilevel feature information and is used for object detection. First, the feature square tower strategy is put forward and embedded into the multimodal feature fusion framework. The multilevel features of the two modalities are fused by top-down feature aggregation. Second, by minimizing the foreground and background classification losses, a feature constraint module is constructed to constrain the infrared features to make them more salient. Third, a weighted feature fusion strategy is proposed based on second-order statistics (SOS) to guarantee strong discrimination of the fusion features in different scenarios. Finally, faster R-CNN is applied to detect the fused features. To illustrate the effectiveness of the method, object detection experiments are conducted on the multimodal datasets <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{M}^{ \boldsymbol {{3}}}$ </tex-math></inline-formula> FD and multispectral. The results show that the proposed network can achieve better fusion detection effects.

Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation.

Semi-supervised learning has a great potential in medical image segmentation tasks with a few labeled data, but most of them only consider single-modal data. The excellent characteristics of multi-modal data can improve the performance of semi-supervised segmentation for each image modality. However, a shortcoming for most existing multi-modal solutions is that as the corresponding processing models of the multi-modal data are highly coupled, multi-modal data are required not only in the training but also in the inference stages, which thus limits its usage in clinical practice. Consequently, we propose a semi-supervised contrastive mutual learning (Semi-CML) segmentation framework, where a novel area-similarity contrastive (ASC) loss leverages the cross-modal information and prediction consistency between different modalities to conduct contrastive mutual learning. Although Semi-CML can improve the segmentation performance of both modalities simultaneously, there is a performance gap between two modalities, i.e., there exists a modality whose segmentation performance is usually better than that of the other. Therefore, we further develop a soft pseudo-label re-learning (PReL) scheme to remedy this gap. We conducted experiments on two public multi-modal datasets. The results show that Semi-CML with PReL greatly outperforms the state-of-the-art semi-supervised segmentation methods and achieves a similar (and sometimes even better) performance as fully supervised segmentation methods with 100% labeled data, while reducing the cost of data annotation by 90%. We also conducted ablation studies to evaluate the effectiveness of the ASC loss and the PReL module.

Deep-learning-enabled multimodal data fusion for lung disease classification

The recent pandemic has revealed the urgent need for lung disease diagnosis at early stages in humans. Deep learning-based automatic diagnosis methods typically rely on single-modality data such as medical imaging. However, analysis of single modality data is not so reliable to diagnose the disease at its early stages. Clinical data, blood tests together with imaging methods are very powerful and reliable sources to detect the presence of disease in the human body. This study attempts to use medical imaging data with clinical information to develop a multimodal fusion approach to detect lung disease. Two architectures of multimodal network based on late and intermediate fusion is proposed. Besides, an approach of adapting batch size is also introduced. Experiments show that the performance of intermediate fusion is better than the late fusion model with both direct and adaptive batch size approach.

Predicting 30-day all-cause hospital readmission using multimodal spatiotemporal graph neural networks.

Reduction in 30-day readmission rate is an important quality factor for hospitals as it can reduce the overall cost of care and improve patient post-discharge outcomes. While deep-learning-based studies have shown promising empirical results, several limitations exist in prior models for hospital readmission prediction, such as: (a) only patients with certain conditions are considered, (b) do not leverage data temporality, (c) individual admissions are assumed independent of each other, which ignores patient similarity, (d) limited to single modality or single center data. In this study, we propose a multimodal, spatiotemporal graph neural network (MM-STGNN) for prediction of 30-day all-cause hospital readmission, which fuses in-patient multimodal, longitudinal data and models patient similarity using a graph. Using longitudinal chest radiographs and electronic health records from two independent centers, we show that MM-STGNN achieved an area under the receiver operating characteristic curve (AUROC) of 0.79 on both datasets. Furthermore, MM-STGNN significantly outperformed the current clinical reference standard, LACE+ (AUROC=0.61), on the internal dataset. For subset populations of patients with heart disease, our model significantly outperformed baselines, such as gradient-boosting and Long Short-Term Memory models (e.g., AUROC improved by 3.7 points in patients with heart disease). Qualitative interpretability analysis indicated that while patients' primary diagnoses were not explicitly used to train the model, features crucial for model prediction may reflect patients' diagnoses. Our model could be utilized as an additional clinical decision aid during discharge disposition and triaging high-risk patients for closer post-discharge follow-up for potential preventive measures.