Multimodal Data Fusion Research Articles

OptEF-BCI: An Optimization-Based Hybrid EEG and fNIRS-Brain Computer Interface.

Multimodal data fusion (electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS)) has been developed as an important neuroimaging research field in order to circumvent the inherent limitations of individual modalities by combining complementary information from other modalities. This study employed an optimization-based feature selection algorithm to systematically investigate the complementary nature of multimodal fused features. After preprocessing the acquired data of both modalities (i.e., EEG and fNIRS), the temporal statistical features were computed separately with a 10 s interval for each modality. The computed features were fused to create a training vector. A wrapper-based binary enhanced whale optimization algorithm (E-WOA) was used to select the optimal/efficient fused feature subset using the support-vector-machine-based cost function. An online dataset of 29 healthy individuals was used to evaluate the performance of the proposed methodology. The findings suggest that the proposed approach enhances the classification performance by evaluating the degree of complementarity between characteristics and selecting the most efficient fused subset. The binary E-WOA feature selection approach showed a high classification rate (94.22 ± 5.39%). The classification performance exhibited a 3.85% increase compared with the conventional whale optimization algorithm. The proposed hybrid classification framework outperformed both the individual modalities and traditional feature selection classification (p < 0.01). These findings indicate the potential efficacy of the proposed framework for several neuroclinical applications.

TransCDR: a deep learning model for enhancing the generalizability of cancer drug response prediction through transfer learning and multimodal data fusion for drug representation

TransCDR: a deep learning model for enhancing the generalizability of cancer drug response prediction through transfer learning and multimodal data fusion for drug representation

An adaptive reinforcement learning-based multimodal data fusion framework for human–robot confrontation gaming

Playing games between humans and robots have become a widespread human–robot confrontation (HRC) application. Although many approaches were proposed to enhance the tracking accuracy by combining different information, the problems of the intelligence degree of the robot and the anti-interference ability of the motion capture system still need to be solved. In this paper, we present an adaptive reinforcement learning (RL) based multimodal data fusion (AdaRL-MDF) framework teaching the robot hand to play Rock–Paper–Scissors (RPS) game with humans. It includes an adaptive learning mechanism to update the ensemble classifier, an RL model providing intellectual wisdom to the robot, and a multimodal data fusion structure offering resistance to interference. The corresponding experiments prove the mentioned functions of the AdaRL-MDF model. The comparison accuracy and computational time show the high performance of the ensemble model by combining k-nearest neighbor (k-NN) and deep convolutional neural network (DCNN). In addition, the depth vision-based k-NN classifier obtains a 100% identification accuracy so that the predicted gestures can be regarded as the real value. The demonstration illustrates the real possibility of HRC application. The theory involved in this model provides the possibility of developing HRC intelligence.

Seizure detection with reduced electroencephalogram channels: research trends and outlook.

Epilepsy is a prevalent condition characterized by recurrent, unpredictable seizures. Monitoring with surface electroencephalography (EEG) is the gold standard for diagnosing epilepsy, but a time-consuming, uncomfortable and sometimes ineffective process for patients. Further, using EEG over a brief monitoring period has variable success, dependent on patient tolerance and seizure frequency. The availability of hospital resources and hardware and software specifications inherently restrict the options for comfortable, long-term data collection, resulting in limited data for training machine-learning models. This mini-review examines the current patient journey, providing an overview of the current state of EEG monitoring with reduced electrodes and automated channel reduction methods. Opportunities for improving data reliability through multi-modal data fusion are suggested. We assert the need for further research in electrode reduction to advance brain monitoring solutions towards portable, reliable devices that simultaneously offer patient comfort, perform ultra-long-term monitoring and expedite the diagnosis process.

Task Recommendation Method Combining Multimodal Cognition and Collaboration in Mobile Crowdsensing Systems

A multimodal data fusion optimization and task recommendation method is proposed in mobile crowd sensing for the complex data redundancy generated by sensing users, which leads to low task recommendation accuracy and high sensing cost. The method extracts the modal features using BERT and Faster-RCNN. It obtains self-attention and cross-guided features by the attention mechanism to achieve intra-modal and inter-modal information sharing and reduce the risk of fusing unrelated modal features. These obtained modal features are then used for hierarchical fusion. Extracting fusion features of different granularity by capturing implicit features within a single modality and complementary features between multiple modalities, then jointly optimizing them, thus making the fusion results more focused on information of interest to users in multimodal historical task data. The cross-guided self-attention mechanism is designed to improve the accuracy of multimodal data fusion by fully fusing modal data and the joint optimization of their different fusion features. Thus it increases the sensing user's interest in completing the task, which in turn increases the motivation of the sensing user to participate in the sensing task and improves the sensing quality. Finally, the similarity between the new task and the historical task is calculated to decide whether to recommend the new task to the sensing user or not. Experiments based on the Flicker8k and Pascal Sentence datasets show that our proposed method can effectively fuse multimodal data and improve the accuracy of task recommendations.

Self-supervision assisted multimodal remote sensing image classification with coupled self-looping convolution networks

Recently, remote sensing community has seen a surge in the use of multimodal data for different tasks such as land cover classification, change detection and many more. However, handling multimodal data requires synergistically using the information from different sources. Currently, deep learning (DL) techniques are being religiously used in multimodal data fusion owing to their superior feature extraction capabilities. But, DL techniques have their share of challenges. Firstly, DL models are mostly constructed in the forward fashion limiting their feature extraction capability. Secondly, multimodal learning is generally addressed in a supervised setting, which leads to high labelled data requirement. Thirdly, the models generally handle each modality separately, thus preventing any cross-modal interaction. Hence, we propose a novel self-supervision oriented method of multimodal remote sensing data fusion. For effective cross-modal learning, our model solves a self-supervised auxiliary task to reconstruct input features of one modality from the extracted features of another modality, thus enabling more representative pre-fusion features. To counter the forward architecture, our model is composed of convolutions both in backward and forward directions, thus creating self-looping connections, leading to a self-correcting framework. To facilitate cross-modal communication, we have incorporated coupling across modality-specific extractors using shared parameters. We evaluate our approach on three remote sensing datasets, namely Houston 2013 and Houston 2018, which are HSI-LiDAR datasets and TU Berlin, which is an HSI-SAR dataset, where we achieve the respective accuracy of 93.08%, 84.59% and 73.21%, thus beating the state of the art by a minimum of 3.02%, 2.23% and 2.84%.

External multi-modal imaging sensor calibration for sensor fusion: A review

Multi-modal data fusion has gained popularity due to its diverse applications, leading to an increased demand for external sensor calibration. Despite several proven calibration solutions, they fail to fully satisfy all the evaluation criteria, including accuracy, automation, and robustness. Thus, this review aims to contribute to this growing field by examining recent research on multi-modal imaging sensor calibration and proposing future research directions. The literature review comprehensively explains the various characteristics and conditions of different multi-modal external calibration methods, including traditional motion-based calibration and feature-based calibration. Target-based calibration and targetless calibration are two types of feature-based calibration, which are discussed in detail. Furthermore, the paper highlights systematic calibration as an emerging research direction. Finally, this review concludes crucial factors for evaluating calibration methods and provides a comprehensive discussion on their applications, with the aim of providing valuable insights to guide future research directions. Future research should focus primarily on the capability of online targetless calibration and systematic multi-modal sensor calibration.

Multimodal data fusion for cancer biomarker discovery with deep learning.

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

An improved MPGA-ACO-BP algorithm and comprehensive evaluation system for intelligence workshop multi-modal data fusion

The digital economy is a new economic form taking data as an important production factor and digital and intelligent technology as a driving force for transformation. The core idea is to extract and fuse the knowledge implicit in data and transform it into intelligence to drive the transformation of traditional manufacturing industries, and one of its key technologies is multi-modal data fusion. In this paper, an improved MPGA-ACO-BP algorithm is proposed, and combined with an improved entropy-weighted TOPSIS method comprehensive evaluation system, which effectively solves the problem of “data scale inconsistency” between modal data leading to difficult model fusion and fusion accuracy. Finally, the validity of the theory and methods of this paper are verified using the example of multi-modal data fusion tool wear prediction in an intelligence workshop. By distilling the corresponding evaluation metrics inductively, the improved comprehensive evaluation system in this paper can also be extended to different production control scenarios to provide them with the corresponding integration information, which has a certain practical value.

AI-enabled and multimodal data driven smart health monitoring of wind power systems: A case study

The development of AI has enabled the fault detection of industrial components to be achieved through the combination with deep learning. A detection method combined with deep learning has also emerged for the fault detection of fan blades, such as models based on neural networks using the appearance or sound of the blades. However, the detection model obtained from a single data type often has limitations, such as low accuracy and overfitting. This is also the problem with fan blade detection. In contrast, multimodal data fusion detection models are often more stable. The modality diversity of blade diagnosis is strong, and it can be achieved from multiple modalities such as image, sound, and vibration. To improve the accuracy of fault diagnosis of fan blades, this article proposes a multimodal double-layer detection system (MTDS) based on decision-level and feature-level fusion. The system includes a wind turbine simulation platform and a multimodal detection system. It mainly obtains different modal data of the simulated wind turbine from the image, sound, and vibration signals, including blade images through unmanned aerial vehicle photography, blade vibration signals through electronic vibrators, and blade sound signals through microphones. The highly correlated sound and vibration modal data are fused at the feature level, and a detection model based on the sound and vibration mixed mode is implemented using a sound-vibration-CNN (SV-CNN) proposed in this case. Then, a detection model of the image mode is trained based on the blade image using a Convolution Block Attention Module ResNet (CBAM-ResNet) network. Finally, the detection input of the two modal models is fed into a perceptron to obtain the final prediction result, and the decision-level fusion is implemented to achieve fan blade detection based on multimodal, namely the implementation of MTDS.

Multimodal deep learning to predict prognosis in adult and pediatric brain tumors

BackgroundThe introduction of deep learning in both imaging and genomics has significantly advanced the analysis of biomedical data. For complex diseases such as cancer, different data modalities may reveal different disease characteristics, and the integration of imaging with genomic data has the potential to unravel additional information than when using these data sources in isolation. Here, we propose a DL framework that combines these two modalities with the aim to predict brain tumor prognosis.MethodsUsing two separate glioma cohorts of 783 adults and 305 pediatric patients we developed a DL framework that can fuse histopathology images with gene expression profiles. Three strategies for data fusion were implemented and compared: early, late, and joint fusion. Additional validation of the adult glioma models was done on an independent cohort of 97 adult patients.ResultsHere we show that the developed multimodal data models achieve better prediction results compared to the single data models, but also lead to the identification of more relevant biological pathways. When testing our adult models on a third brain tumor dataset, we show our multimodal framework is able to generalize and performs better on new data from different cohorts. Leveraging the concept of transfer learning, we demonstrate how our pediatric multimodal models can be used to predict prognosis for two more rare (less available samples) pediatric brain tumors.ConclusionsOur study illustrates that a multimodal data fusion approach can be successfully implemented and customized to model clinical outcome of adult and pediatric brain tumors.

Patient-GAT: Sarcopenia Prediction using Multi-modal Data Fusion and Weighted Graph Attention Networks.

Graph Attention Networks (GAT) have been extensively used to perform node-level classification on data that can be represented as a graph. However, few papers have investigated the effectiveness of using GAT on graph representations of patient similarity networks. This paper proposes Patient-GAT, a novel method to predict chronic health conditions by first integrating multi-modal data fusion to generate patient vector representations using imputed lab variables with other structured data. This data representation is then used to construct a patient network by measuring patient similarity, finally applying GAT to the patient network for disease prediction. We demonstrated our framework by predicting sarcopenia using real-world EHRs obtained from the Indiana Network for Patient Care. We evaluated the performance of our system by comparing it to other baseline models, showing that our model outperforms other methods. In addition, we studied the contribution of the temporal representation of the lab data and discussed the interpretability of this model by analyzing the attention coefficients of the trained Patient-GAT model. Our code can be found on Github.

Multimodal Emotion Recognition and Fluctuation: A Study on Sentiment Analysis of Online Public Opinion

Emotion is an important way for individuals to express their views on the Internet and an important variable that shapes public opinion. Considering the multimodality of data, such as text, picture and video, and the subtlety of emotional expression, a multimodal sentiment analysis model that addresses content involving difference senses, such as sight, hearing and touch at the same time is very necessary. This study outlines the basic steps, classification strategies and research methods of sentimental analysis and acknowledges the differences between sentimental analyses on text, picture and video. As multimodal sentiment recognition is still in its initial stage, there’s still room for improvement in cross-disciplinary research on multimodal data of text, picture, audio, video in terms of weighted scoring, complex emotion and intensity recognition. It’s concluded that future studies should focus on the intensity of different emotions, multimodal data fusion and how weighted scoring influences an emotion recognition model and explore application possibilities.

Concordance between focal brain temperature elevations and focal edema in temporal lobe epilepsy.

Neuroinflammation (NI) is a pathophysiological factor in many neurological disorders, including epilepsy. Because NI causes microstructural tissue damage that worsens with increasing brain temperature, abnormally elevated brain temperatures may be a surrogate measure of the biochemical consequences of NI. This study investigated whether patients with temporal lobe epilepsy (TLE) have abnormal brain temperature elevations (TCRE ) in seizure-producing regions that show evidence of edema and/or microstructural damage. Twenty adults with TLE and 20 healthy controls (HCs) were scanned at 3-Tesla. TCRE in each voxel was calculated (TCRE =-102.61(ΔH20-CRE) + 206.1°C) by non-invasive volumetric magnetic resonance spectroscopic imaging and thermometry (MRSI-t). Multi-shell diffusion images were processed by neurite orientation and density imaging (NODDI). Voxel wise two-sample t tests computed group differences in imaging data. Multimodal data fusion (joint independent component analysis [ICA]) determined the spatial coupling of TCRE with NODDI. TCRE analyses showed that, compared to HCs, TLEs had higher TCRE (p < .001). NODDI analyses showed increased extracellular free water (pFWE < 0.05) in the medial temporal lobes, with the most pronounced increases ipsilateral to seizure onset. TLEs also had increased angular dispersion of neurites (p < .001) and decreased neurite density (pFWE <0.05) in the ictal-onset medial temporal lobe, as well as more widespread, bilateral patterns of abnormalities. Focal increases in TCRE were spatially concordant with increased free water in the left inferior and middle temporal gyri and the associated cortex. In TLE, ICA loadings extracted from this region of overlap were associated with greater mood disturbance (r=.34, p=.02) and higher depression scores (r=.37, p=.009). The spatial concordance between focal TCRE elevations and edema in TLE supports the notion that brain thermometry visualizes the correlates of focal NI. MRSI-t-based TCRE elevations may, therefore, be a useful biomarker for identifying seizure-producing tissue in patients with focal epilepsy caused by brain damage.

PhytoOracle: Scalable, modular phenomics data processing pipelines.

As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to (i) improve data processing efficiency; (ii) provide an extensible, reproducible computing framework; and (iii) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area).

Remote sensing for field pea yield estimation: A study of multi-scale data fusion approaches in phenomics.

Remote sensing using unmanned aerial systems (UAS) are prevalent for phenomics and precision agricultural applications. The high-resolution data for these applications can provide useful spectral characteristics of crops associated with performance traits such as seed yield. With the recent availability of high-resolution satellite imagery, there has been growing interest in using this technology for plot-scale remote sensing applications, particularly those related to breeding programs. This study compared the features extracted from high-resolution satellite and UAS multispectral imagery (visible and near-infrared) to predict the seed yield from two diverse plot-scale field pea yield trials (advanced breeding and variety testing) using the random forest model. The multi-modal (spectral and textural features) and multi-scale (satellite and UAS) data fusion approaches were evaluated to improve seed yield prediction accuracy across trials and time points. These approaches included both image fusion, such as pan-sharpening of satellite imagery with UAS imagery using intensity-hue-saturation transformation and additive wavelet luminance proportional approaches, and feature fusion, which involved integrating extracted spectral features. In addition, we also compared the image fusion approach to high-definition satellite data with a resolution of 0.15 m/pixel. The effectiveness of each approach was evaluated with data at both individual and combined time points. The major findings can be summarized as follows: (1) the inclusion of the texture features did not improve the model performance, (2) the performance of the model using spectral features from satellite imagery at its original resolution can provide similar results as UAS imagery, with variation depending on the field pea yield trial under study and the growth stage, (3) the model performance improved after applying multi-scale, multiple time point feature fusion, (4) the features extracted from the pan-sharpened satellite imagery using intensity-hue-saturation transformation (image fusion) showed better model performance than those with original satellite imagery or high definition imagery, and (5) the green normalized difference vegetation index and transformed triangular vegetation index were identified as key features contributing to high model performance across trials and time points. These findings demonstrate the potential of high-resolution satellite imagery and data fusion approaches for plot-scale phenomics applications.

Emma: An accurate, efficient, and multi-modality strategy for autonomous vehicle angle prediction

Autonomous driving and self-driving vehicles have become the most popular selection for customers for their convenience. Vehicle angle prediction is one of the most prevalent topics in the autonomous driving industry, that is, realizing real-time vehicle angle prediction. However, existing methods of vehicle angle prediction utilize only single-modal data to achieve model prediction, such as images captured by the camera, which limits the performance and efficiency of the prediction system. In this paper, we present Emma, a novel vehicle angle prediction strategy that achieves multi-modal prediction and is more efficient. Specifically, Emma exploits both images and inertial measurement unit (IMU) signals with a fusion network for multi-modal data fusion and vehicle angle prediction. Moreover, we design and implement a few-shot learning module in Emma for fast domain adaptation to varied scenarios (e.g., different vehicle models). Evaluation results demonstrate that Emma achieves overall 97.5% accuracy in predicting three vehicle angle parameters (yaw, pitch, and roll), which outperforms traditional single-modalities by approximately 16.7%–36.8%. Additionally, the few-shot learning module presents promising adaptive ability and shows overall 79.8% and 88.3% accuracy in 5-shot and 10-shot settings, respectively. Finally, empirical results show that Emma reduces energy consumption by 39.7% when running on the Arduino UNO board.

Slice2Volume: Fusion of multimodal medical imaging and light microscopy data of irradiation-injured brain tissue in 3D

Comprehending cellular changes of radiation-induced brain injury is crucial to prevent and treat the pathology. We provide a unique open dataset of proton-irradiated mouse brains consisting of medical imaging, radiation dose simulations, and large-scale microscopy images, all registered into a common coordinate system. This allows dose-dependent analyses on single-cell level.

Understanding Flow Experience in Video Learning by Multimodal Data

Video-based learning has successfully become an effective alternative to face-to-face instruction. In such situations, modeling or predicting learners’ flow experience during video learning is critical for enhancing the learning experience and advancing learning technologies. In this study, we set up an instructional scenario for video learning according to flow theory. Different learning states, i.e., boredom, fit (flow), and anxiety, were successfully induced by varying the difficulty levels of the learning task. We collected learners’ electrocardiogram (ECG) signals as well as facial video, upper body posture and speech data during the learning process. We proposed classification models of the learning state and regression models to predict flow experience by utilizing different combinations of the data from the four modalities. The results showed that the model performance of learning state recognition was significantly improved by the decision-level fusion of multimodal data. By using the selected important features from all data sources, such as the standard deviation of normal to normal R-R intervals (SDNN), high-frequency (HF) heart rate variability and mel-frequency cepstral coefficients (MFCC), the multilayer perceptron (MLP) classifier gave the best recognition result of learning states (i.e., mean AUC of 0.780). The recognition accuracy of boredom, fit (flow) and anxiety reached 47.48%, 80.89% and 47.41%, respectively. For flow experience prediction, the MLP regressor based on the fusion of two modalities (i.e., ECG and posture) achieved the optimal prediction (i.e., mean RMSE of 0.717). This study demonstrates the feasibility of modeling and predicting the flow experience in video learning by combining multimodal data.

Effective Techniques for Multimodal Data Fusion: A Comparative Analysis.

Data processing in robotics is currently challenged by the effective building of multimodal and common representations. Tremendous volumes of raw data are available and their smart management is the core concept of multimodal learning in a new paradigm for data fusion. Although several techniques for building multimodal representations have been proven successful, they have not yet been analyzed and compared in a given production setting. This paper explored three of the most common techniques, (1) the late fusion, (2) the early fusion, and (3) the sketch, and compared them in classification tasks. Our paper explored different types of data (modalities) that could be gathered by sensors serving a wide range of sensor applications. Our experiments were conducted on Amazon Reviews, MovieLens25M, and Movie-Lens1M datasets. Their outcomes allowed us to confirm that the choice of fusion technique for building multimodal representation is crucial to obtain the highest possible model performance resulting from the proper modality combination. Consequently, we designed criteria for choosing this optimal data fusion technique.