Multimodal Fusion Algorithm Research Articles

An Adaptive Multimodal Fusion Network Based on Multilinear Gradients for Visual Question Answering

As an interdisciplinary field of natural language processing and computer vision, Visual Question Answering (VQA) has emerged as a prominent research focus in artificial intelligence. The core of the VQA task is to combine natural language understanding and image analysis to infer answers by extracting meaningful features from textual and visual inputs. However, most current models struggle to fully capture the deep semantic relationships between images and text owing to their limited capacity to comprehend feature interactions, which constrains their performance. To address these challenges, this paper proposes an innovative Trilinear Multigranularity and Multimodal Adaptive Fusion algorithm (TriMMF) that is designed to improve the efficiency of multimodal feature extraction and fusion in VQA tasks. Specifically, the TriMMF consists of three key modules: (1) an Answer Generation Module, which generates candidate answers by extracting fused features and leveraging question features to focus on critical regions within the image; (2) a Fine-grained and Coarse-grained Interaction Module, which achieves multimodal interaction between question and image features at different granularities and incorporates implicit answer information to capture complex multimodal correlations; and (3) an Adaptive Weight Fusion Module, which selectively integrates coarse-grained and fine-grained interaction features based on task requirements, thereby enhancing the model’s robustness and generalization capability. Experimental results demonstrate that the proposed TriMMF significantly outperforms existing methods on the VQA v1.0 and VQA v2.0 datasets, achieving state-of-the-art performance in question–answer accuracy. These findings indicate that the TriMMF effectively captures the deep semantic associations between images and text. The proposed approach provides new insights into multimodal interaction and fusion research, combining domain adaptation techniques to address a broader range of cross-domain visual question answering tasks.

Performance of multi-modality and multi-classifier fusion models for predicting radiation-induced oral mucositis in patients with nasopharyngeal carcinoma

To evaluate the performance of different multi-modality fusion models for predicting radiation-induced oral mucositis (RIOM) following radiotherapy in patients with nasopharyngeal carcinoma (NPC). We retrospectively collected the data from 198 patients with locally advanced NPC who experienced RIOM following radiotherapy at the Affiliated Tumor Hospital of Guangzhou Medical University from September, 2022 to February, 2023. Based on oral radiation dose-volume parameters and clinical features of NPC, basic classification models were developed using different combinations of feature selection algorithms and classifiers and integrated using a multi-criterion decision-making (MCDM)-based classifier fusion (MCF) strategy and its variant, the H-MCF model. The basic classification models, MCF model, the H-MCF model with a single modality or multiple modalities and other ensemble classifiers were compared for performances for predicting RIOM by assessing the area under the ROC curve (AUC), accuracy, sensitivity, and specificity. The H-MCF model, which integrated multi-modality features, achieved the highest accuracy for predicting severe RIOM with an AUC of 0.883, accuracy of 0.850, sensitivity of 0.933, and specificity of 0.800. Compared with each of the individual classifiers, the multimodal multi-classifier fusion algorithm combining clinical and dosimetric modalities demonstrates superior performance in predicting the incidence of severe RIOM in NPC patients following radiotherapy.

Aggressiveness classification of clear cell renal cell carcinoma using registration-independent radiology-pathology correlation learning.

Renal cell carcinoma (RCC) is a common cancer that varies in clinical behavior. Clear cell RCC (ccRCC) is the most common RCC subtype, with both aggressive and indolent manifestations. Indolent ccRCC is often low-grade without necrosis and can be monitored without treatment. Aggressive ccRCC is often high-grade and can cause metastasis and death if not promptly detected and treated. While most RCCs are detected on computed tomography (CT) scans, aggressiveness classification is based on pathology images acquired from invasive biopsy orsurgery. CT imaging-based aggressiveness classification would be an important clinical advance, as it would facilitate non-invasive risk stratification and treatment planning. Here, we present a novel machine learning method, Correlated Feature Aggregation By Region (CorrFABR), for CT-based aggressiveness classification ofccRCC. CorrFABR is a multimodal fusion algorithm that learns from radiology and pathology images, and clinical variables in a clinically-relevant manner. CorrFABR leverages registration-independent radiology (CT) and pathology image correlations using features from vision transformer-based foundation models to facilitate aggressiveness assessment on CT images. CorrFABR consists of three main steps: (a) Feature aggregation where region-level features are extracted from radiology and pathology images at widely varying image resolutions, (b) Fusion where radiology features correlated with pathology features (pathology-informed CT biomarkers) are learned, and (c) Classification where the learned pathology-informed CT biomarkers, together with clinical variables of tumor diameter, gender, and age, are used to distinguish aggressive from indolent ccRCC using multi-layer perceptron-based classifiers. Pathology images are only required in the first two steps of CorrFABR, and are not required in the prediction module. Therefore, CorrFABR integrates information from CT images, pathology images, and clinical variables during training, but for inference, it relies solely on CT images and clinical variables, ensuring its clinical applicability. CorrFABR was trained with heterogenous, publicly-available data from 298 ccRCC tumors (136 indolent tumors, 162 aggressive tumors) in a five-fold cross-validation setup and evaluated on an independent test set of 74 tumors with a balanced distribution of aggressive and indolent tumors. Ablation studies were performed to test the utility of each component ofCorrFABR. CorrFABR outperformed the other classification methods, achieving an ROC-AUC (area under the curve) of 0.855±0.0005 (95% confidence interval: 0.775, 0.947), F1-score of 0.793±0.029, sensitivity of 0.741±0.058, and specificity of 0.876±0.032 in classifying ccRCC as aggressive or indolent subtypes. It was found that pathology-informed CT biomarkers learned through registration-independent correlation learning improves classification performance over using CT features alone, irrespective of the kind of features or the classification model used. Tumor diameter, gender, and age provide complementary clinical information, and integrating pathology-informed CT biomarkers with these clinical variables further improvesperformance. CorrFABR provides a novel method for CT-based aggressiveness classification of ccRCC by enabling the identification of pathology-informed CT biomarkers, and integrating them with clinical variables. CorrFABR enables learning of these pathology-informed CT biomarkers through a novel registration-independent correlation learning module that considers unaligned radiology and pathology images at widely varying imageresolutions.

Application and Research of Intelligent Multi-source Information Fusion Technology in Firefighting Equipment

As modern firefighting environments become increasingly complex, traditional firefighter equipment and operational methods are no longer adequate to address the high-risk challenges of fire rescue operations. Fire scenes are characterized by complex and hazardous environmental factors, with firefighters facing multiple threats such as extreme temperatures, dense smoke, and toxic gases. Therefore, ensuring the safety and efficiency of firefighters is of paramount importance. This study integrates multiple sensor technologies to enable real-time monitoring of firefighters' physiological status, posture changes, and fire scene conditions. The system incorporates ErgoLAB wireless ECG and PPG sensors, the BWT61CL posture sensor, and the MAG14mini infrared camera, which are capable of simultaneously capturing key physiological data and heat distribution in the fire scene. To ensure data accuracy and real-time performance, the study employs efficient signal preprocessing techniques, including noise reduction, baseline correction, and time synchronization. In addition, wireless transmission technology and multimodal data fusion algorithms are utilized to comprehensively analyze the firefighters' status and fire scene conditions. This approach significantly enhances the precision of firefighter safety monitoring and operational efficiency, demonstrating substantial innovation and practical value.

Adherent Peanut Image Segmentation Based on Multi-Modal Fusion.

Aiming at the problem of the difficult segmentation of adherent images due to the not fully convex shape of peanut pods, their complex surface texture, and their diverse structures, a multimodal fusion algorithm is proposed to achieve a 2D segmentation of adherent peanut images with the assistance of 3D point clouds. Firstly, the point cloud of a running peanut is captured line by line using a line structured light imaging system, and its three-dimensional shape is obtained through splicing and combining it with a local surface-fitting algorithm to calculate a normal vector and curvature. Seed points are selected based on the principle of minimum curvature, and neighboring points are searched using the KD-Tree algorithm. The point cloud is filtered and segmented according to the normal angle and the curvature threshold until achieving the completion of the point cloud segmentation of the individual peanut, and then the two-dimensional contour of the individual peanut model is extracted by using the rolling method. The search template is established, multiscale feature matching is implemented on the adherent image to achieve the region localization, and finally, the segmentation region is optimized by an opening operation. The experimental results show that the algorithm improves the segmentation accuracy, and the segmentation accuracy reaches 96.8%.

Research and comparison of algorithms based on multi-modal fusion

Abstract A multi-modal fusion algorithm is an important method for information fusion on multi-modal data provided by different sensors. It can make full use of the advantages of multiple sensors and improve the accuracy and robustness of data processing and decision-making. This paper aims to study the performance difference between the extended Kalman filter (EKF) algorithm and other algorithms in multi-modal fusion and explore a method to fuse multiple algorithms further to improve the accuracy of fusion results. The author uses the classic test set data set for experiments to evaluate the performance of different algorithms. By comparing and analyzing the performance of each algorithm in data fusion tasks, we can get their advantages and disadvantages in different scenarios. Among them, the extended Kalman filter algorithm is a classical algorithm based on Bayesian filtering, which can estimate the system state through recursive state estimation and covariance update. In addition, the author also uses linear regression, random forest, and other algorithms to compare. Then, the author uses several test sets to evaluate the performance of each algorithm. Through a comprehensive analysis of the MSE and MAE errors output by the algorithm, the applicability, advantages, and disadvantages of each algorithm in different scenarios are obtained.

Multimodal LLM Driven Computer Interface

Abstract: This paper introduces the Multimodal-driven Computer Interface, a framework that enables multimodal models to interact with and control a computer. The framework receives user input through various modalities (e.g., speech, text, gestures), processes it through a multimodal fusion algorithm, and generates appropriate actions using a decision-making module. These actions are then executed on the computer through platform-specific APIs. Currently, it is integrated with the multimodal LLM and operates on Windows, Mac, and Linux systems. While the framework demonstrates promising results, challenges remain in improving the accuracy of mouse click location prediction and adapting to diverse user needs and preferences. The Multimodaldriven Computer Interface has the potential to revolutionize human-computer interaction, opening up new possibilities for accessibility, productivity, and entertainment.

Synchronous Imaging and Multimodal Fusion of Optical and Electromagnetic Measurements for Overlapping Defects Inspection

It is a challenging problem to accurately detect overlapping defects. This paper proposes a system that can obtain optical and electromagnetic signals synchronously. It contains an industrial camera, a chromatic confocal displacement sensor (CCDS) and an eddy current testing (ECT) probe with high-resolution array tunneling magnetoresistance (TMR) sensors. A multimodal fusion algorithm is proposed to combine the advantages of the optical testing and ECT. The fusion of the visual image and CCDS measurement results in morphology of the surface defects. To recognize the overlapping buried defect, a finite element method model is developed to estimate the surface defect signal using parameters from the optical measurements, and then the surface defect signal is subtracted from the experimental image, based on which the overlapping defects are precisely detected. This method can be widely used in industry to comprehensively assess the health and integrity of structures.

Multimodal fusion and human-robot interaction control of an intelligent robot.

Introduction: Small-scaled robotic walkers play an increasingly important role in Activity of Daily Living (ADL) assistance in the face of ever-increasing rehab requirements and existing equipment drawbacks. This paper proposes a Rehabilitation Robotic Walker (RRW) for walking assistance and body weight support (BWS) during gait rehabilitation. Methods: The walker provides the patients with weight offloading and guiding force to mimic a series of the physiotherapist's (PT's) movements, and creates a natural, comfortable, and safe environment. This system consists of an omnidirectional mobile platform, a BWS mechanism, and a pelvic brace to smooth the motions of the pelvis. To recognize the human intentions, four force sensors, two joysticks, and one depth-sensing camera were used to monitor the human-machine information, and a multimodal fusion algorithm for intention recognition was proposed to improve the accuracy. Then the system obtained the heading angle E, the pelvic pose F, and the motion vector H via the camera, the force sensors, and the joysticks respectively, classified the intentions with feature extraction and information fusion, and finally outputted the motor speed control through the robot's kinematics. Results: To validate the validity of the algorithm above, a preliminary test with three volunteers was conducted to study the motion control. The results showed that the average error of the integral square error (ISE) was 2.90 and the minimum error was 1.96. Discussion: The results demonstrated the efficiency of the proposed method, and that the system is capable of providing walking assistance.

Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) for Depression Recognition with Facial Expression and Pupil Diameter.

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.

Precision Teaching of Civics Integration in Universities and Primary Schools with Individual Differences in the Age of Artificial Intelligence

Abstract Leveraging the unique capabilities of intelligent technologies, this study presents an advanced approach to Civics and Politics education, emphasizing personalized and multimodal teaching strategies suitable for various educational stages. We establish an Integrated Civics and Political Teaching Platform using multimodal feature fusion algorithms coupled with personalized recommendation techniques. This platform incorporates genetic algorithm-based analysis of students’ learning behaviors, facilitating a Precision Teaching Service Model that tailors education to individual needs. A comparative study of semester scores highlights the model’s effectiveness: notably, a substantial 19.89 point increase in declarative knowledge scores and a significant reduction in the variability of strategic knowledge scores by 5.28 points. These results affirm the benefits of personalized teaching in enhancing student outcomes in Civics education.

Quantum Fuzzy Neural Network for multimodal sentiment and sarcasm detection

Sentiment and sarcasm detection in social media contribute to assessing social opinion trends. Over the years, most artificial intelligence (AI) methods have relied on real values to characterize the sentimental and sarcastic features in language. These methods often overlook the complexity and uncertainty of sentimental and sarcastic elements in human language. Therefore, this paper proposes the Quantum Fuzzy Neural Network (QFNN), a multimodal fusion and multitask learning algorithm with a Seq2Seq structure that combines Classical and Quantum Neural Networks (QNN), and fuzzy logic. Complex numbers are used in the Fuzzifier to capture sentiment and sarcasm features, and QNN are used in the Defuzzifier to obtain the prediction. The experiments are conducted on classical computers by constructing quantum circuits in a simulated noisy environment. The results show that QFNN can outperform several recent methods in sarcasm and sentiment detection task on two datasets (Mustard and Memotion). Moreover, by assessing the fidelity of quantum circuits in a noisy environment, QFNN was found to have excellent robustness. The QFNN circuit also possesses expressible and entanglement capabilities, proving effective in various settings. Our code is available at https://github.com/prayagtiwari/QFNN.

Analysis on the innovative education mode of the integration of information technology and traditional teaching in the era of big data

Abstract In order to develop students’ concentration in the classroom, this paper proposes to analyze students’ learning concentration from three dimensions, where the superiority of the random forest classification algorithm is found in the dimension of head posture estimation so that students’ attention ranges can be analyzed. For data acquisition, the OpenPose platform is used for real-time detection of body, foot, hand, and face key points. A comprehensive evaluation study of students’ concentration levels in the classroom was achieved through an effective algorithm. Finally, the multimodal information fusion algorithm was investigated, and it was concluded that the weights could be calculated by hierarchical analysis to achieve a comprehensive evaluation of students’ learning concentration. By analyzing the course data of 12 students, the distribution of SFR values, Yaw angle and Pitch angle distribution, PERCLOS value distribution, and correct/error rate distribution of answers were counted, and the final scores of respective learning engagement were obtained as 0.91, 0.62, 0.80, 0.36, 0.82, 0.73, 0.81, 0.63, 0.81, 0.81 The model scores were compared with the expert scores, and the accuracy rate reached 98.6%, which proved that the model proposed in this paper is effective and can correctly reflect the real state of students’ learning in the classroom.

Temperature monitoring of vehicle brake drum based on dual light fusion and deep learning

The braking system of a vehicle is the most crucial component for ensuring the safety on the road. Moreover, the extraction of temperature field data from a vehicle's tires plays a crucial role in vehicle condition monitoring. Nevertheless, the conventional infrared thermal imaging detection method necessitates manual recognition of the tire components and marking of each temperature point one by one in the infrared image. To address this issue, a method combination with dual-light fusion and deep learning is proposed to implement automatic monitoring for vehicle brake drum temperature. Firstly, we propose a late-end multimodal fusion algorithm based on the YOLOv5 network to integrate visible light and infrared thermal imaging This algorithm allows for feature extraction from both modalities and performing decision-level image fusion, thereby improving the robustness of automatic vehicle and tire detection. Additionally, we employ a semantic segmentation network to accurately segment the tires, extract the relevant visible pixel information, and obtain the temperature field matrix from the infrared images while eliminating extraneous background details. Subsequently, a three-dimensional model of tire’s temperature field is constructed obtaining the dynamics of tire temperature during the vehicle operation. Through experimentation, we demonstrate the efficacy of our proposed method in achieving all-weather roadside vehicle classification and tire temperature monitoring, which provides technical and theoretical support for the active risk prevention and control in road transportation.

MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments

The current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the experimental practice. However, many traditional multimodal fusion algorithms rely solely on individual modalities for the analysis of users’ experimental intentions, failing to fully utilize the intention information for each modality. To rectify these issues, we present a new multimodal fusion algorithm, MFIRA, which intersects and blends intention probabilities between channels by executing parallel processing of multimodal information at the intention layer. Additionally, we developed an augmented reality (AR) virtual experiment platform based on the Hololens 2, which enables students to conduct experiments using speech, gestures, and vision. Employing the MFIRA algorithm, the system captures users’ experimental intent and navigates or rectifies errors to guide students through their experiments. The experimental results indicate that the MFIRA algorithm boasts a 97.3% accuracy rate in terms of interpreting users’ experimental intent. Compared to existing experimental platforms, this system is considerably more interactive and immersive for students and is highly applicable in secondary school experimental chemistry classrooms.

Emotion-Recognition Algorithm Based on Weight-Adaptive Thought of Audio and Video

Emotion recognition commonly relies on single-modal recognition methods, such as voice and video signals, which demonstrate a good practicability and universality in some scenarios. Nevertheless, as emotion-recognition application scenarios continue to expand and the data volume surges, single-modal emotion recognition proves insufficient to meet people’s needs for accuracy and comprehensiveness when the amount of data reaches a certain scale. Thus, this paper proposes the application of multimodal thought to enhance emotion-recognition accuracy and conducts corresponding data preprocessing on the selected dataset. Appropriate models are constructed for both audio and video modalities: for the audio-modality emotion-recognition task, this paper adopts the “time-distributed CNNs + LSTMs” model construction scheme; for the video-modality emotion-recognition task, the “DeepID V3 + Xception architecture” model construction scheme is selected. Furthermore, each model construction scheme undergoes experimental verification and comparison with existing emotion-recognition algorithms. Finally, this paper attempts late fusion and proposes and implements a late-fusion method based on the idea of weight adaptation. The experimental results demonstrate the superiority of the multimodal fusion algorithm proposed in this paper. When compared to the single-modal emotion-recognition algorithm, the accuracy of recognition is increased by almost 4%, reaching 84.33%.

A multimodal domestic service robot interaction system for people with declined abilities to express themselves

Driven by the shortage of qualified nurses and the increasing average age of the population, the ambient assisted living style using intelligent service robots and smart home systems has become an excellent choice to free up caregiver time and energy and provide users with a sense of independence. However, users’ unique environments and differences in abilities to express themselves through different interaction modalities make intention recognition and interaction between user and service system very difficult, limiting the use of these new nursing technologies. This paper presents a multimodal domestic service robot interaction system and proposes a multimodal fusion algorithm for intention recognition to deal with these problems. The impacts of short-term and long-term changes were taken into account. Implemented interaction modalities include touch, voice, myoelectricity gesture, visual gesture, and haptics. Users could freely choose one or more modalities through which to express themselves. Virtual games and virtual activities of independent living were designed for pre-training and evaluating users’ abilities to use different interaction modalities in their unique environments. A domestic service robot interaction system was built, on which a set of experiments were carried out to test the system’s stability and intention recognition ability in different scenarios. The experiment results show that the system is stable and effective and can adapt to different scenarios. In addition, the intention recognition rate in the experiments was 93.62%. Older adults could master the system quickly and use it to provide some assistance for their independent living.

A multimodal fusion method for Alzheimer's disease based on DCT convolutional sparse representation.

The medical information contained in magnetic resonance imaging (MRI) and positron emission tomography (PET) has driven the development of intelligent diagnosis of Alzheimer's disease (AD) and multimodal medical imaging. To solve the problems of severe energy loss, low contrast of fused images and spatial inconsistency in the traditional multimodal medical image fusion methods based on sparse representation. A multimodal fusion algorithm for Alzheimer' s disease based on the discrete cosine transform (DCT) convolutional sparse representation is proposed. The algorithm first performs a multi-scale DCT decomposition of the source medical images and uses the sub-images of different scales as training images, respectively. Different sparse coefficients are obtained by optimally solving the sub-dictionaries at different scales using alternating directional multiplication method (ADMM). Secondly, the coefficients of high-frequency and low-frequency subimages are inverse DCTed using an improved L1 parametric rule combined with improved spatial frequency novel sum-modified SF (NMSF) to obtain the final fused images. Through extensive experimental results, we show that our proposed method has good performance in contrast enhancement, texture and contour information retention.

Multimodal Sensors and ML‐Based Data Fusion for Advanced Robots

Nature has proved that multiple sense and processing capabilities are critical for target recognition and race survival. As such, advanced robots that perform missions in an unstructured environment highly need organism‐similar multimodal sensing and processing systems for sophisticated unstructured environmental stimuli. Herein, recent progress in multimodal sensing and processing systems for advanced robotics is reviewed. Multimodal sensors including tactile sensors that capture surface properties of objects (i.e., thermal conductivity, temperature, softness, and electron affinity), visual sensors that capture the color and size of objects, and gas sensors that capture object smell are summarized. The multimodal data fusion algorithms that process multimodal signals from multimodal sensors to achieve object recognition and decision making are also presented. The challenges and future development of multimodal sensors and data fusion algorithms are further discussed. Advances in these areas open new avenues for advanced robotics applications in human–robot collaboration, rescue missions, garbage sorting, and intelligent prosthetics.

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

With the development of electronic technology, intelligent cars can gradually realize more complex artificial intelligence algorithms. The video caption algorithm is one of them. However, current video caption algorithms only consider single-visual information when applied to urban traffic scenes, which leads to the failure to generate accurate captions of complex sets. The multimodal fusion algorithm based on Transformer is one of the solutions to this problem. However, the existing algorithms have the difficulties of a low fusion performance and high computational complexity. We propose a new video caption Transformer-based model, the MFVC (Multimodal Fusion for Video Caption), to solve these issues. We introduce audio modal data and the attention bottleneck module to increase the available information to describe the generative model and improve the model effect with less operation costs through the attention bottleneck module. Finally, the experiment is conducted on the available datasets, MSR-VTT and MSVD. Meanwhile, to verify the effect of the model on the urban traffic scene, the experiment is carried out on the self-built traffic caption dataset BUUISE, and the evaluation index confirms the model. This model can achieve good results on both available datasets and urban traffic datasets and has excellent application prospects in the intelligent driving industry.