Multimodal Information Fusion Research Articles

Equipping with Human Cognition: Driver Intention Recognition with Multimodal Information Fusion

Equipping with Human Cognition: Driver Intention Recognition with Multimodal Information Fusion

A multimodal educational robots driven via dynamic attention.

With the development of artificial intelligence and robotics technology, the application of educational robots in teaching is becoming increasingly popular. However, effectively evaluating and optimizing multimodal educational robots remains a challenge. This study introduces Res-ALBEF, a multimodal educational robot framework driven by dynamic attention. Res-ALBEF enhances the ALBEF (Align Before Fuse) method by incorporating residual connections to align visual and textual data more effectively before fusion. In addition, the model integrates a VGG19-based convolutional network for image feature extraction and utilizes a dynamic attention mechanism to dynamically focus on relevant parts of multimodal inputs. Our model was trained using a diverse dataset consisting of 50,000 multimodal educational instances, covering a variety of subjects and instructional content. The evaluation on an independent validation set of 10,000 samples demonstrated significant performance improvements: the model achieved an overall accuracy of 97.38% in educational content recognition. These results highlight the model's ability to improve alignment and fusion of multimodal information, making it a robust solution for multimodal educational robots.

Spatiotemporal Sensitive Network for Non-Contact Heart Rate Prediction from Facial Videos

Heart rate (HR) is an important indicator reflecting the overall physical and mental health of the human body, playing a crucial role in diagnosing cardiovascular and neurological diseases. Recent research has revealed that variations in the light absorption of human skin captured through facial video over the cardiac cycle, due to changes in blood volume, can be utilized for non-contact HR estimation. However, most existing methods rely on single-modal video sources (such as RGB or NIR), which often yield suboptimal results due to noise and the limitations of a single information source. To overcome these challenges, this paper proposes a multimodal information fusion architecture named the spatiotemporal sensitive network (SS-Net) for non-contact heart rate estimation. Firstly, spatiotemporal feature maps are utilized to extract physiological signals from RGB and NIR videos effectively. Next, a spatiotemporal sensitive (SS) module is introduced to extract useful physiological signal information from both RGB and NIR spatiotemporal maps. Finally, a multi-level spatiotemporal context fusion (MLSC) module is designed to fuse and complement information between the visible light and infrared modalities. Then, different levels of fused features are refined in task-specific branches to predict both remote photoplethysmography (rPPG) signals and heart rate (HR) signals. Experiments conducted on three datasets demonstrate that the proposed SS-Net achieves superior performance compared to existing methods.

Multimodal Information Fusion and Data Generation for Evaluation of Second Language Emotional Expression

This study aims to develop an emotion evaluation method for second language learners, utilizing multimodal information to comprehensively evaluate students’ emotional expressions. Addressing the limitations of existing emotion evaluation methods, which primarily focus on the acoustic features of speech (e.g., pronunciation, frequency, and rhythm) and often neglect the emotional expressions conveyed through voice and facial videos, this paper proposes an emotion evaluation method based on multimodal information. The method includes the following three main parts: (1) generating virtual data using a Large Language Model (LLM) and audio-driven facial video synthesis, as well as integrating the IEMOCAP dataset with self-recorded student videos and audios containing teacher ratings to construct a multimodal emotion evaluation dataset; (2) a graph convolution-based emotion feature encoding network to extract emotion features from multimodal information; and (3) an emotion evaluation network based on Kolmogorov–Arnold Networks (KAN) to compare students’ emotion features with standard synthetic data for precise evaluation. The emotion recognition method achieves an unweighted accuracy (UA) of 68.02% and an F1 score of 67.11% in experiments with the IEMOCAP dataset and TTS data. The emotion evaluation model, using the KAN network, outperforms the MLP network, with a mean squared error (MSE) of 0.811 compared to 0.943, providing a reliable tool for evaluating language learners’ emotional expressions.

Opportunities and challenges in the development of artificial intelligence research in spinal surgery

Artificial intelligence has emerged as a game-changer in the field of spine surgery, offering transformative diagnostic and therapeutic approaches for spinal conditions. The application of AI in spine research encompasses a diverse range of diseases, with usage scenarios becoming increasingly widespread and technological integration going deeper. AI technology shows immense promise and value in the diagnosis of spinal diseases, the formulation of treatment strategies, surgical navigation, prognostic evaluation, and postoperative rehabilitation. Through deep learning and machine learning, AI can aid doctors in enhancing diagnostic accuracy, creating personalized treatment plans, and executing precise maneuvers during surgery, thus improving operational safety. Moreover, AI can provide intelligent support for patients' postoperative recovery, facilitating the restoration of their functions. However, current research is still in its nascent stage and confronts several challenges, such as the lack of standardized databases, the simplicity of algorithmic learning models, the inadequate fusion of multi-modal clinical information, and the limited clinical applicability. By developing open-source, standardized spine databases, incorporating innovative intelligent core algorithms, and establishing normalized screening, diagnostic, and predictive models for spinal conditions, we trust that we can drive the innovation and refinement of diagnostic and treatment technologies in spine surgery. This will realize an effective conjunction between technological provision and clinical demands, continuously elevating the intelligence level of spine surgery and offering safer, more effective medical services to a vast array of patients.

LQR-based control strategy for improving human–robot companionship and natural obstacle avoidance

In the dynamic and unstructured environment of human–robot symbiosis, companion robots require natural human–robot interaction and autonomous intelligence through multimodal information fusion to achieve effective collaboration. Nevertheless, the control precision and coordination of the accompanying actions are not satisfactory in practical applications. This is primarily attributed to the difficulties in the motion coordination between the accompanying target and the mobile robot. This paper proposes a companion control strategy based on the Linear Quadratic Regulator (LQR) to enhance the coordination and precision of robot companion tasks. This method enables the robot to adapt to sudden changes in the companion target’s motion. Besides, the robot could smoothly avoid obstacles during the companion process. Firstly, a human–robot companion interaction model based on nonholonomic constraints is developed to determine the relative position and orientation between the robot and the companion target. Then, an LQR-based companion controller incorporating behavioral dynamics is introduced to simultaneously avoid obstacles and track the companion target’s direction and velocity. Finally, various simulations and real-world human–robot companion experiments are conducted to regulate the relative position, orientation, and velocity between the target object and the robot platform. Experimental results demonstrate the superiority of this approach over conventional control algorithms in terms of control distance and directional errors throughout system operation. The proposed LQR-based control strategy ensures coordinated and consistent motion with target persons in social companion scenarios.

Building a Multimodal Dataset of Academic Paper for Keyword Extraction

ABSTRACTUp to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio modalities leads to deficiencies in information richness and overlooks potential correlations, thereby constraining the model's ability to learn representations of the data and the accuracy of model predictions. Furthermore, the currently available multimodal datasets for keyword extraction task are particularly scarce, further hindering the progress of research on multimodal keyword extraction task. Therefore, this study constructs a multimodal dataset of academic paper consisting of 1,000 samples, with each sample containing paper text, images, audios and keywords. Based on unsupervised and supervised methods of keyword extraction, experiments are conducted using textual data from papers, as well as text extracted from images and audio. The aim is to investigate the differences in performance in keyword extraction task with respect to different modal information and the fusion of multimodal information. The experimental results indicate that text from different modalities exhibits distinct characteristics in the model. The concatenation of paper text, image text and audio text can effectively enhance the keyword extraction performance of academic papers.

WCNN-RSN: a novel fault diagnosis method for rolling bearing using multimodal feature fusion

Abstract Fault diagnosis of rolling bearings is significant for the safe operation of engineering equipment. Many intelligent diagnostic methods have been successfully developed. However, it is often susceptible to noisy environments. Therefore, the paper proposes a rolling bearing fault diagnosis method based on multimodal information fusion in time and time–frequency domains by combining an improved 1D-convolutional neural network (CNN) with ResNet50 wavelet improved CNN-ResNet (WCNN-RSN). The algorithm employs the Multi-Head Attention (MHA) mechanism to complementarily fuse fault features in different scales, achieving fault diagnosis by fully extracting fault features. The experimental results show that the diagnostic effect of WCNN-RSN is better than that of the comparison methods under noise interference, which proves that the proposed method possesses good anti-noise and generalization ability.

Dual-stream fusion network with multi-head self-attention for multi-modal fake news detection

With the rapid advancement of social media platforms like Weibo and WeChat, alongside the emergence of deepfake technologies, tackling fake information has become a major challenge for society and government institutions. To address this, developing efficient and intelligent methods for fake news detection is crucial. This paper introduces a dual-stream fusion network model (DSF-MHSA) based on deep learning, designed to detect fake news across web pages, images, and text. The model tackles issues such as cross-lingual discrepancies, data imbalance, and multimodal information fusion by integrating deep learning models like ERNIE-M, AlexNet, and ShuffleNet, along with three multi-head self-attention mechanisms. It processes textual and image data separately to capture long-range dependencies and global information, enhancing understanding and recognition. A unified multi-head self-attention mechanism then merges these insights to strengthen cross-modal correlation detection. The model is tested on datasets from Twitter, Weibo, and IKCEST-2023, which includes real online news from various social media sources. Results show that the DSF-MHSA model achieves over 90 % accuracy, surpassing traditional models in news detection tasks. This research offers significant practical value for the identification and understanding of news content.

Position query-guided cross-modal flow field prediction model of a transonic compressor cascade

The gradient of flow parameters in a transonic compressor cascade flow field varies significantly, especially in the region of shock waves, which causes a significant challenge to its high-precision flow field prediction. In this study, the position query-guided cross-modal flow field prediction model (PGCM) is proposed to effectively predict the flow field parameter distribution of a transonic compressor cascade. The PGCM utilizes the self-attention mechanism for the global and deep geometric feature extraction of configurations, which contributes to an in-depth understanding of the spatial relationships between coordinate points within the flow field, accurately capturing and analyzing the structural complexity of a compressor cascade flow. In addition, the PGCM integrates the cross-attention mechanism that establishes correlations between different input sequences, which enhances the performance of the model in querying and interpreting flow parameters at specific coordinates. The flow field prediction models are developed to predict the flow parameter distributions of different cascade geometries at Mach numbers of 0.78 and 0.93, respectively. The validation results indicate that the PGCM performs significantly better than the existing convolutional neural network and vision transformer, especially in the prediction of the pressure coefficient Cp distribution. The PGCM is adaptable to the variation of flow conditions and geometrical configurations efficiently and accurate in predicting the flow field of a compressor cascade. This paper demonstrates the promising potential of conducting the multi-modal information fusion to enhance the capability of flow field prediction.

ACDF-YOLO: Attentive and Cross-Differential Fusion Network for Multimodal Remote Sensing Object Detection

Object detection in remote sensing images has received significant attention for a wide range of applications. However, traditional unimodal remote sensing images, whether based on visible light or infrared, have limitations that cannot be ignored. Visible light images are susceptible to ambient lighting conditions, and their detection accuracy can be greatly reduced. Infrared images often lack rich texture information, resulting in a high false-detection rate during target identification and classification. To address these challenges, we propose a novel multimodal fusion network detection model, named ACDF-YOLO, basedon the lightweight and efficient YOLOv5 structure, which aims to amalgamate synergistic data from both visible and infrared imagery, thereby enhancing the efficiency of target identification in remote sensing imagery. Firstly, a novel efficient shuffle attention module is designed to assist in extracting the features of various modalities. Secondly, deeper multimodal information fusion is achieved by introducing a new cross-modal difference module to fuse the features that have been acquired. Finally, we combine the two modules mentioned above in an effective manner to achieve ACDF. The ACDF not only enhances the characterization ability for the fused features but also further refines the capture and reinforcement of important channel features. Experimental validation was performed using several publicly available multimodal real-world and remote sensing datasets. Compared with other advanced unimodal and multimodal methods, ACDF-YOLO separately achieved a 95.87% and 78.10% mAP0.5 on the LLVIP and VEDAI datasets, demonstrating that the deep fusion of different modal information can effectively improve the accuracy of object detection.

3D Visual Grounding-Audio: 3D scene object detection based on audio

3D Visual Grounding (3DVG) is a prevalent multi-modal information fusion task capable of accurately localizing target objects referenced in natural language descriptions within a point cloud scene. Nevertheless, the stringent demands for input and output devices present substantial hurdles for the application and integration of 3DVG in fields like remote robotic control and telemedicine. To address this challenge, we introduce several innovative approaches. Firstly, we have initiated a novel multi-modal task, termed 3D Visual Grounding-Audio (3DVG-Audio), which is based on the fusion of audio and point cloud. To the best of our knowledge, this represents the first instance of an Audio-Point Cloud multi-modal task. 3DVG-Audio achieves precise localization of audio-mentioned objects within the point cloud by utilizing the point cloud in conjunction with the corresponding audio input. Secondly, building upon the ScanRefer, we have developed a novel dataset named 3DVG-AudioSet, specifically designed for the training and evaluation of the 3DVG-Audio method. Finally, we have crafted a tailored loss function to further enhance the performance of 3DVG-Audio and introduced a method named AP-Refer, which serves as a benchmark for the task. Extensive experimental results demonstrate the potential for deep integration of audio and point cloud to tackle complex real-world challenges. AP-Refer has successfully addressed the 3DVG-Audio, circumventing the limitations of conventional 3DVG methods, and exhibits significant application potential.

MIDF-DMAP: Multimodal information dynamic fusion for drug molecule activity prediction

BackgroundComputer-aided drug development can alleviate limitations in drug research and development processes, such as long cycles, high costs, and limited targeting. Deep learning models can be used to predict the activity of drug compound molecules. However, existing models still need to adequately address semantic redundancy, conflicts, and noise between different data modalities. It results in low accuracy in predicting molecular activity and many false positive results that need further resolution. ObjectiveThis study aims to design a parallel deep learning model for processing multimodal molecular information fusion, dynamically adjusting the weights of different modal features during the fusion process. This approach ensures that high-quality modal information is more significant in molecular activity prediction, improving the model’s predictive accuracy. MethodsFirst, three modal features of drug molecules are extracted: one-dimensional fingerprints, two-dimensional topological structures, and three-dimensional geometric structures. Then, a dynamic weighting mechanism is proposed to fuse these three modal features organically. Finally, the fused features are input into the model classifier to obtain molecular activity prediction results. ResultsExperimental results show that the proposed model improved the Receiver Operating Characteristic-Area Under the Curve (ROC-AUC) value by 29.96% compared to the Weave model and 26.38% compared to the GC model on the BBBP dataset. Compared to the GraphMVP model and the multi-channel substructure graph model MSGG, the ROC-AUC value increased by 20.44% and 15.80%, respectively. This method also achieved good results on imbalanced samples and small molecule datasets. ConclusionThis study provides an effective multimodal dynamic fusion method for virtual drug screening, enriching computer-aided drug development theory and improving drug research and development efficiency.

A Brain-Inspired Decision-Making method for upper limb exoskeleton based on Multi-Brain-Region structure and multimodal information fusion

For the challenges faced by upper limb exoskeleton in meeting human motion intention, this article proposes a novel brain-inspired decision-making model based on multi-brain-region information transmission mechanism and multimodal information fusion method to provide accurate prediction of motion trajectory. Firstly, a multimodal information perception and fusion method is proposed. Based on surface electromyographic (sEMG) signals and angle signals collected by sensors. Then a random forest algorithm is applied to screen the key features of sEMG signal for trajectory prediction. A multimodal information hierarchical fusion method based on dual-period mechanism is designed to solve the problem of fusion defect of information with different frequency. Finally, a multi-brain-region decision-making model based on hybrid hierarchical liquid state machine (HHLSM) is established to predict motion trajectory for the exoskeleton to meet human motion intention. Experiments show that the established model has a high computational efficiency and improves the effectiveness of exoskeleton assistance. The brain-inspired decision-making model based on HHLSM established in this paper is of great significance for further research on brain-inspired control and improving the human-robot interaction effect of upper limb exoskeleton.

Early warning system for nocardiosis in largemouth bass (Micropterus salmoides) based on multimodal information fusion

The high-density culture of the recirculating aquaculture system (RAS) makes bacterial and parasitic diseases in fish likely to spread within these systems. Therefore, preventing and controlling the occurrence of fish diseases in RAS is crucial for the future development of aquaculture. However, the current circulating water system lacks early warning measures to prevent and control diseases, resulting in poor accuracy in early warning and detection of diseases. To tackle this challenge, this study introduces an early warning system for largemouth bass (Micropterus salmoides) disease, which is based on You Only Look Once vision 8 (YOLOv8), ByteTrack, Long-Short-Term-Memory(LSTM), and Fuzzy Inference System (FIS). The system utilizes the water quality, surface characteristics, and behavioral traits of diseased fish to predict and prevent disease outbreaks. The system achieved an accuracy of 79.33% for identifying infected body surface features, 80.65% for identifying diseased water quality, and 81.08% for predicting diseased behavior. The experimental results indicate that the early warning system is highly reliable and effective, achieving integrated disease identification accuracy as high as 94.08%. This study enhances the accuracy of early disease warning in fish disease conditions, achieving early warning of nocardiosis in largemouth bass. The study provides crucial technical support for the sustainable and high-quality development of the aquaculture industry.

Integrating Multi-Organ Imaging-Derived Phenotypes and Genomic Information for Predicting the Occurrence of Common Diseases.

As medical imaging technologies advance, these tools are playing a more and more important role in assisting clinical disease diagnosis. The fusion of biomedical imaging and multi-modal information is profound, as it significantly enhances diagnostic precision and comprehensiveness. Integrating multi-organ imaging with genomic information can significantly enhance the accuracy of disease prediction because many diseases involve both environmental and genetic determinants. In the present study, we focused on the fusion of imaging-derived phenotypes (IDPs) and polygenic risk score (PRS) of diseases from different organs including the brain, heart, lung, liver, spleen, pancreas, and kidney for the prediction of the occurrence of nine common diseases, namely atrial fibrillation, heart failure (HF), hypertension, myocardial infarction, asthma, type 2 diabetes, chronic kidney disease, coronary artery disease (CAD), and chronic obstructive pulmonary disease, in the UK Biobank (UKBB) dataset. For each disease, three prediction models were developed utilizing imaging features, genomic data, and a fusion of both, respectively, and their performances were compared. The results indicated that for seven diseases, the model integrating both imaging and genomic data achieved superior predictive performance compared to models that used only imaging features or only genomic data. For instance, the Area Under Curve (AUC) of HF risk prediction was increased from 0.68 ± 0.15 to 0.79 ± 0.12, and the AUC of CAD diagnosis was increased from 0.76 ± 0.05 to 0.81 ± 0.06.

Dynamic graph topology generating mechanism: Framework for feature-level multimodal information fusion applied to lower-limb activity recognition

Lower-limb activity recognition (LLAR), which is driven by wearable sensors, has recently become a popular research topic in human assistive devices. Although multimodal information fusion is crucial for improving the accuracy of LLAR, traditional feature-level fusion strategies cannot directly model the relationship between multimodal features according to the lower-limb activity patterns and they realize fusion based on the relationship. To address these limitations, a dynamic graph topology generation (DGTG) mechanism that uses a graph topology to model the relationship between multimodal features mined by a convolutional neural network (CNN) is proposed in this study. A weighted graph isomorphism network (WGIN) that can realize multimodal feature fusion based on the relationship (i.e., graph topology) is also proposed, resulting in a novel feature-level multimodal information fusion framework called CNN-DGTG-WGIN. This proposed method achieved the best average accuracies of 96.3 and 99.7% for two publicly available datasets. This method integrates the determination of the graph topology into an end-to-end learning process without requiring any prior knowledge or original graph structures. Further, CNN-DGTG-WGIN effectively incorporates both linear and nonlinear correlations between features during the DGTG process. Analysis results revealed that the graph topology (multimodal feature relationship) learned by CNN-DGTG-WGIN was closely related to lower-limb activity patterns, encoding relevant information about these patterns, which is the essence of its superior performance.

Evaluation on algorithms and models for multi-modal information fusion and evaluation in new media art and film and television cultural creation

Evaluation on algorithms and models for multi-modal information fusion and evaluation in new media art and film and television cultural creation

Laboratory Abnormal Behavior Detection Based on Multimodal Information Fusion

The traditional laboratory anomaly detection methods mainly focus on the hidden dangers caused by chemical leaks and other items, ignoring the impact of abnormal behaviors such as incorrect operations and improper behavior on safety in the laboratory. This paper proposes a laboratory abnormal behavior detection method based on multimodal information fusion. The method generates a dense optical flow field of RGB image sequences based on optical flow theory and global smoothing constraints, and mines motion mode information. Meanwhile, the contour modal information of behavior is captured through convolution and adjacency matrix operations. Using decision level and proximity functions to integrate student behavior motion mode information and contour mode information, and using the maximum value as the behavior detection result. The experimental results show that the method can effectively detect abnormal behavior in the laboratory environment, with small detection errors and a specificity close to 1.00, effectively ensuring the safety of the laboratory environment.

An effective multi-modal adaptive contextual feature information fusion method for Chinese long text classification

Chinese long text classification plays a vital role in Natural Language Processing. Compared to Chinese short texts, Chinese long texts contain more complex semantic feature information. Furthermore, the distribution of these semantic features is uneven due to the varying lengths of the texts. Current research on Chinese long text classification models primarily focuses on enhancing text semantic features and representing Chinese long texts as graph-structured data. Nonetheless, these methods are still susceptible to noise information and tend to overlook the deep semantic information in long texts. To address the above challenges, this study proposes a novel and effective method called MACFM, which introduces a deep feature information mining method and an adaptive modal feature information fusion strategy to learn the semantic features of Chinese long texts thoroughly. First, we present the DCAM module to capture complex semantic features in Chinese long texts, allowing the model to learn detailed high-level representation features. Then, we explore the relationships between word vectors and text graphs, enabling the model to capture abundant semantic information and text positional information from the graph. Finally, we develop the AMFM module to effectively combine different modal feature representations and eliminate the unrelated noise information. The experimental results on five Chinese long text datasets show that our method significantly improves the accuracy of Chinese long text classification tasks. Furthermore, the generalization experiments on five English datasets and the visualized results demonstrate the effectiveness and interpretability of the MACFM model.