Multimodal Information Fusion Research Articles

HAZMAT Vehicle Reidentification in Road Tunnels Based on the Fusion of Appearance and Spatiotemporal Information.

Vehicles transporting hazardous material (HAZMAT) pose a severe threat to highway safety, especially in road tunnels. Vehicle reidentification is essential for identifying and warning abnormal states of HAZMAT vehicles in road tunnels. However, there is still no public dataset for benchmarking this task. To this end, this work releases a real-world tunnel HAZMAT vehicle reidentification dataset, VisInt-THV-ReID, including 10,048 images with 865 HAZMAT vehicles and their spatiotemporal information. A method based on multimodal information fusion is proposed to realize vehicle reidentification by fusing vehicle appearance and spatiotemporal information. We design a spatiotemporal similarity determination method for vehicles based on the spatiotemporal law of vehicles in tunnels. Compared with other reidentification methods based on multimodal information fusion, i.e., PROVID, Visual + ST, and Siamese-CNN, experimental results show that our approach significantly improves the vehicle reidentification recognition precision.

A survey of artificial intelligence in tongue image for disease diagnosis and syndrome differentiation.

The rapid development of artificial intelligence technology has gradually extended from the general field to all walks of life, and intelligent tongue diagnosis is the product of a miraculous connection between this new discipline and traditional disciplines. We reviewed the deep learning methods and machine learning applied in tongue image analysis that have been studied in the last 5 years, focusing on tongue image calibration, detection, segmentation, and classification of diseases, syndromes, and symptoms/signs. Introducing technical evolutions or emerging technologies were applied in tongue image analysis; as we have noticed, attention mechanism, multiscale features, and prior knowledge were successfully applied in it, and we emphasized the value of combining deep learning with traditional methods. We also pointed out two major problems concerned with data set construction and the low reliability of performance evaluation that exist in this field based on the basic essence of tongue diagnosis in traditional Chinese medicine. Finally, a perspective on the future of intelligent tongue diagnosis was presented; we believe that the self-supervised method, multimodal information fusion, and the study of tongue pathology will have great research significance.

Edge Detection Guide Network for Semantic Segmentation of Remote-Sensing Images

The acquisition of high-resolution satellite and airborne remote sensing images has been significantly simplified due to the rapid development of sensor technology. Several practical applications of high-resolution remote sensing images (HRRSIs) are based on semantic segmentation. However, single-modal HRRSIs are difficult to classify accurately in the complex situation of some scene objects; therefore, the semantic segmentation of multi-source information fusion is gaining popularity. The inherent difference between multimodal features and the semantic gap between multi-level features typically affect the performance of existing multi-mode fusion methods. We propose a multimodal fusion network based on edge detection to address these issues. This method aids multimodal information fusion by utilizing spatial information contained in the boundary. An edge detection guide module is included in the feature extraction stage to realize the boundary information through the fusion of details and semantics between high-level and low-level features. The boundary information is extended into the well-designed multimodal adaptive fusion block (MAFB) to obtain the multimodal fusion features. Furthermore, a residual adaptive fusion block (RAFB) and a spatial position module (SPM) in the feature decoding stage have been designed to fuse multi-level features from the standpoint of local and global dependence. We compared our method to several state-of-the-art (SOTA) methods using the International Society for Photogrammetry and Remote Sensing’s (ISPRS) Vaihingen and Potsdam datasets. The final results demonstrate that our method achieves excellent performance.

CPG3D: Cross-Modal Priors Guided 3D Object Reconstruction

Three-dimensional reconstruction is a multimedia technology widely used in computer-aided modeling and 3D animation. Nevertheless, it is still hard for reconstruction methods to overcome the 3D geometry missing and the object occlusion in the single-view images. In this paper, we propose a novel method (CPG3D) for reconstructing high-quality 3D shapes from a single image under the guidance of prior knowledge. Using the single-view image as the query, prior knowledge is collected from public 3D datasets, which can compensate for missing 3D geometries and assist the 3D reconstruction network to high fidelity results. Our method consists of three parts: 1) Cross-modal 3D shape retrieval module: This part retrieves related 3D shapes based on 2D images. Here, we apply the pre-trained model to guarantee the correlation between the retrieved 3D shape and the input image. 2) Multimodal information fusion module: We propose a multimodal attention mechanism to handle the information fusing of 2D visual and 3D structural information; 3) Three-dimensional reconstruction module: We propose a novel encoder-decoder network for 3D shape reconstruction. Specifically, we employ the skip connection operation to link the target image's visual information with the 3D model's structural information to enhance the prediction of 3D details. During training, we employ two carefully designed loss functions to lead the multimodal learning to obtain proper modal features. On the ShapeNet and Pix3D datasets, the final experimental results reveal that our method notably increases reconstruction quality and outperforms SOTA methods.

A Pose Estimation Algorithm for Multimodal Data Fusion

In response to the problem that the previous pose detection systems are not effective under conditions such as severe occlusion or uneven illumination, this paper focuses on the multimodal information fusion pose estimation problem. The main work is to design a multimodal data fusion pose estimation algorithm for the problem of pose estimation in complex scenes such as low-texture targets and poor lighting conditions. The network takes images and point clouds as input and extracts local color and spatial features of the target object using the improved DenseNet and PointNet++ networks, which are combined with a microscopic bit-pose iterative network to achieve end-to-end bit-pose estimation. Excellent detection accuracy was obtained on two benchmark datasets of LineMOD (97.8%) and YCB-Video (95.3%) for pose estimation. The algorithm is able to obtain accurate poses of target objects from complex scenes, providing accurate, real-time and robust relative poses for object tracking in motion and wave compensation.

AGGN: Attention-based glioma grading network with multi-scale feature extraction and multi-modal information fusion

In this paper, a magnetic resonance imaging (MRI) oriented novel attention-based glioma grading network (AGGN) is proposed. By applying the dual-domain attention mechanism, both channel and spatial information can be considered to assign weights, which benefits highlighting the key modalities and locations in the feature maps. Multi-branch convolution and pooling operations are applied in a multi-scale feature extraction module to separately obtain shallow and deep features on each modality, and a multi-modal information fusion module is adopted to sufficiently merge low-level detailed and high-level semantic features, which promotes the synergistic interaction among different modality information. The proposed AGGN is comprehensively evaluated through extensive experiments, and the results have demonstrated the effectiveness and superiority of the proposed AGGN in comparison to other advanced models, which also presents high generalization ability and strong robustness. In addition, even without the manually labeled tumor masks, AGGN can present considerable performance as other state-of-the-art algorithms, which alleviates the excessive reliance on supervised information in the end-to-end learning paradigm.

Fusion of tactile and visual information in deep learning models for object recognition

Humans use multimodal sensory information to understand the physical properties of their environment. Intelligent decision-making systems such as the ones used in robotic applications could also utilize the fusion of multimodal information to improve their performance and reliability. In recent years, machine learning and deep learning methods are used at the heart of such intelligent systems. Developing visuo-tactile models is a challenging task due to various problems such as performance, limited datasets, reliability, and computational efficiency. In this research, we propose four efficient models based on dynamic neural network architectures for unimodal and multimodal object recognition. For unimodal object recognition, TactileNet and VisionNet are proposed. For multimodal object recognition, the FusionNet-A and the FusionNet-B are designed to implement early and late fusion strategies, respectively. The proposed models have a flexible structure and are able to change at the train or test phase to accommodate the amount of available information. Model confidence calibration is employed to enhance the reliability and generalization of the models. The proposed models are evaluated on MIT CSAIL large-scale multimodal dataset. Our results demonstrate accurate performance in both unimodal and multimodal scenarios. It has been illustrated that by using different fusion strategies and augmenting the tactile-based models with visual information, the top-1 error rate of the single-frame tactile model was reduced by 78% and the mean average precision was increased by 2.19 times. Although the focus has been on the fusion of tactile and visual modalities, the proposed design methodology can be generalized to include more modalities.

Forecasting power demand in China with a CNN-LSTM model including multimodal information

Accurate forecasting of social power demand is the country's primary task in making decisions on power overall planning, coal power withdrawal, and renewable energy investment. The integration of text data based and traditional time series data may improve the power demand forecasting ability. Therefore, based on the idea of multimodal information fusion, we construct a novel comprehensive power demand prediction model CNN-LSTM (Convolution Neural Network, Long Short-term Memory) in a multi-heterogeneous data environment. Empirical results show that the proposed prediction model is effective, and it proves that the organic fusion of time series data and text data can effectively improve forecasting performance. And China's power demand growth will gradually slow down or even show a downward trend in the next two years, which provides an important decision-making reference for the low-carbon transformation of China's power system.

Dual-Channel Convolutional Network-Based Fault Cause Identification for Active Distribution System Using Realistic Waveform Measurements

Accurate and rapid identification of distribution system fault causes is essential for power system reliability enhancement. Manual fault cause identification requires extensive human resources that leads to extended power outage time. To this end, this paper proposes a dual-channel convolutional neural network (DC-CNN)-based method for distribution system fault cause identification using realistic data from waveform measurement units. The fault mechanism and waveform characteristics of different fault causes are investigated by analyzing large amounts of field waveform data. The short-time Fourier transform (STFT) is advocated to extract the frequency-domain features, which are used together with the time-domain data for constructing the time-frequency feature images. This leads to improved feature extraction via the proposed DC-CNN-enabled multimodal information fusion. A fully connected layer with a maxout unit (FCM layer) is constructed to enhance the mapping ability of high-level features and improve classification accuracy. Extensive test results using field data demonstrate the superiority of the proposed method over other methods.

CrossGAN-Detection: A generative adversarial network with directly controllable fusion for target detection

Most of the deep learning object detection methods based on multi-modal information fusion cannot directly control the quality of the fused images at present, because the fusion only depends on the detection results. The indirectness of control is not conducive to the target detection of the network in principle. For the sake of the problem, we propose a multimodal information cross-fusion detection method based on a generative adversarial network (CrossGAN-Detection), which is composed of GAN and a target detection network. And the target detection network acts as the second discriminator of GAN during training. Through the content loss function and dual discriminator, directly controllable guidance is provided for the generator, which is designed to learn the relationship between different modes adaptively through cross fusion. We conduct abundant experiments on the KITTI dataset, which is the prevalent dataset in the fusion-detection field. The experimental results show that the AP of the novel method for vehicle detection achieves 96.66%, 87.15%, and 78.46% in easy, moderate, and hard categories respectively, which is improved about 7% compared to the state-of-art methods.

Robot gaining robust pouring skills through fusing vision and audio

In the pouring task of service robots, the robust and accurate estimate of liquid height is a crucial step. However, neither vision nor audio alone can achieve better liquid height estimation. We instead propose a visual–audio information fusion network to enable robots with good pouring skills. Visual and audio information are used as information sources. Firstly, visual features are extracted by residual network based on attention model. Secondly, the Fourier characteristic matrix of audio information is obtained by fast Fourier transform, and then the audio feature is extracted by long–short term memory. Thirdly, visual features and audio features are fused by fully connected network to output the liquid height and state of the cup. Finally, a sinusoidal and transient fusion control method is proposed, which takes the liquid height and cup state as inputs, outputs the angle of the gripper, and provides an implementation method for the pouring task. Experiments are carried out to evaluate the performance of multimodal information fusion method and verify the effectiveness of the algorithm for pouring tasks of service robots.

Automatic construction site hazard identification integrating construction scene graphs with BERT based domain knowledge

Hazards often arise from interactions of different parties and can cost great loss. Proactively identification can prevent it from happening. Computer vision currently can identify the entities and attributes inside the construction scenes but fail to generate interaction-level scene descriptions and integrate them with domain knowledge for hazards inference. This paper proposed an automatic hazard inference method using construction scene graphs and C-BERT network. First, computer vision was utilized to build the construction scene graphs with interaction-level scene descriptions including entities, attributes, and their interactions. Second, C-BERT network was designed to infer hazards by integrating scene graphs with domain knowledge like construction regulations. 5 different working scenes were used to demonstrate the validity of the proposed approach and reached 97.82% of hazard identification accuracy. It provided an efficient method for combining visual information and domain knowledge for automated safety monitoring, and path for the industry's massive multimodal information fusion.

Optimized Piano Music Education Model Based on Multimodal Information Fusion for Emotion Recognition in Multimedia Video Networks

Emotion is the important information that people transmit in the process of communication, and the change of emotional state affects people’s perception and decision-making, which introduces the emotional dimension into human-computer interaction. The modes of emotional expression include facial expressions, speech, posture, physiological signals, text, and so on. Emotion recognition is essentially a multimodal fusion problem. This paper investigates the different teaching modes of the teachers and students of our school, designs the load capacity through the K-means algorithm, builds a multimedia network sharing classroom, and creates a piano music situation to stimulate students’ learning interest, using audiovisual and other tools to mobilize students’ emotions, using multimedia guidance to extend students’ piano music knowledge, and comprehensively improve students’ aesthetic ability and autonomous learning ability. Comparing the changes of students after 3 months of teaching, the results of the study found that multimedia sharing classrooms can be up to 50% ahead of traditional teaching methods in enhancing students’ interest, and teachers’ acceptance of multimedia network sharing classrooms is also high.

Multidimensional CNN-Based Deep Segmentation Method for Tumor Identification.

Weighted MR images of 421 patients with nasopharyngeal cancer were obtained at the head and neck level, and the tumors in the images were assessed by two expert doctors. 346 patients' multimodal pictures and labels served as training sets, whereas the remaining 75 patients' multimodal images and labels served as independent test sets. Convolutional neural network (CNN) for modal multidimensional information fusion and multimodal multidimensional information fusion (MMMDF) was used. The three models' performance is compared, and the findings reveal that the multimodal multidimensional fusion model performs best, while the two-modal multidimensional information fusion model performs second. The single-modal multidimensional information fusion model has the poorest performance. In MR images of nasopharyngeal cancer, a convolutional network can precisely and efficiently segment tumors.

A Survey on the Progress of Computer-Assisted Vascular Intervention

Vascular diseases seriously threaten human health. As a commonly used minimally invasive surgery, vascular interventional diagnosis and treatment faces a variety of clinical burdens, so the support of computer-aided technology is urgently needed. In recent years, the rapid development of extravascular and intravascular medical imaging technology and artificial intelligence methods has promoted the related research of computer-assisted vascular intervention technology. The organic relationship between the basic content and key technologies of computer-assisted vascular intervention and its applications are reviewed and systematically analyzed, including multi-modal medical angiography image analysis, catheter guide wire position tracking, vascular associated lesion characteristics, morphological recognition, and other hot issues, and the relationships between the above contents and surgical planning and surgical navigation are analyzed. It is concluded that the clinical application trend in this field is to combine the integration of diagnosis and treatment with vascular interventional robot technology. Through the analysis of research status and development trends, it is concluded that the challenges faced by various research content are: the accuracy of the segmentation model to quantify the vascular structure and the completeness of topology extraction, intervention path vectorization and accuracy of multimodal information fusion, correlation between vascular anatomical labeling and surgical path navigation, and the level and evaluation method of human-computer interaction in vascular intervention. Analysis shows that in the process of solving these problems, the artificial intelligence diagnosis and treatment technology will continue to be promoted, especially the rapid development and clinical application of robot-assisted vascular intervention.

Low-Carbon Awareness Information Technology of Enterprise Executives Based on Big Data and Multimodal Information Fusion

The so-called multimodal information refers to the information from different information sources on different or the same side of the same description target. These pieces of information are different in terms of storage structure, representation, semantic connotation, credibility, and emphasis, but there is a certain inevitable connection between them. This paper aims to study how to analyze and study the low carbon of enterprises with the help of multimodal information fusion based on the background of big data and construct the evolutionary neural network of the improved adaptive genetic algorithm. This paper puts forward the problem of low carbon in enterprises, which is based on multimodal information fusion, and then elaborates on the concept and related algorithms of multimodal information fusion. Information fusion has carried out case design and analysis. Through the research and analysis of enterprise low carbon and self-adaptive algorithm, it can be obtained that the neural network has reached the threshold of 3.95 after iterating for nearly 60 generations, and stopped iterating to obtain the best individual. The evolutionary neural network in this paper reaches stability after a small number of iterations and can basically achieve a certain low carbonization.

A Multimodal Information Fusion Model for Robot Action Recognition with Time Series

The current robotics field, led by a new generation of information technology, is moving into a new stage of human-machine collaborative operation. Unlike traditional robots that need to use isolation rails to maintain a certain safety distance from people, the new generation of human-machine collaboration systems can work side by side with humans without spatial obstruction, giving full play to the expertise of people and machines through an intelligent assignment of operational tasks and improving work patterns to achieve increased efficiency. The robot’s efficient and accurate recognition of human movements has become a key factor in measuring robot performance. Usually, the data for action recognition is video data, and video data is time-series data. Time series describe the response results of a certain system at different times. Therefore, the study of time series can be used to recognize the structural characteristics of the system and reveal its operation law. As a result, this paper proposes a time series-based action recognition model with multimodal information fusion and applies it to a robot to realize friendly human-robot interaction. Multifeatures can characterize data information comprehensively, and in this study, the spatial flow and motion flow features of the dataset are extracted separately, and each feature is input into a bidirectional long and short-term memory network (BiLSTM). A confidence fusion method was used to obtain the final action recognition results. Experiment results on the publicly available datasets NTU-RGB + D and MSR Action 3D show that the method proposed in this paper can improve action recognition accuracy.

High-accuracy wearable detection of freezing of gait in Parkinson's disease based on pseudo-multimodal features

ObjectiveFreezing of gait (FoG) is a serious symptom of Parkinson's disease and prompt detection of FoG is crucial for fall prevention. Although multimodal data combining electroencephalography (EEG) benefit accurate FoG detection, the preparation, acquisition, and analysis of EEG signals are time-consuming and costly, which impedes the application of multimodal information in FoG detection. This work proposes a wearable FoG detection method that merges multimodal information from acceleration and EEG while avoiding the acquisition of real EEG data. MethodsA proxy measurement (PM) model based on long-short-term-memory (LSTM) network was proposed to measure EEG features from accelerations, and pseudo-multimodal features, i.e., pseudo-EEG and acceleration, could be extracted using a highly wearable inertial sensor for FoG detection. ResultsBased on a self-collected FoG dataset, the performance of different feature combinations were compared in terms of subject-dependent and cross-subject settings. In both settings, pseudo-multimodal features achieved the most promising performance, with a geometric mean of 91.0 ± 5.0% in subject-dependent setting and 91.0 ± 3.5% in cross-subject setting. ConclusionOur study suggests that wearable FoG detection can be enhanced through leveraging cross-modal information fusion. SignificanceThe new method provides a promising path for multimodal information fusion and the long-term monitoring of FoG in living environments.

Design of Multimodal Neural Network Control System for Mechanically Driven Reconfigurable Robot.

According to the characteristics and division rules of the modules, this paper divides the rotary module, the swing module, and the mobile module. In order to realize the rapid identification of modules, according to the basic principles of module design, the above modules are put into the module library established by Access. According to the modular modeling method, kinematic models are established, respectively. In order to automatically establish the kinematic model of the robot, a unified expression of modules is established. According to the unified expression of the modules, the kinematics of the reconfigurable robot is analyzed. According to the characteristics of the configuration plane, the configuration plane is divided, and the expression form of the position and attitude of the configuration plane is given. Combined with the principle of neural network and multimodal information fusion, a multimodal information fusion model based on long- and short-term memory neural network is established. Aiming at the control problem of mechanically driven reconfigurable robots, a specific long- and short-term memory neural network model is designed, and the long- and short-term memory neural network algorithm is applied to the robot control problem based on multimodal information fusion. The design of the controller and driver of each joint is the basis of the distributed control system. This paper discusses the hardware design of the joint controller and driver and the realization of the position control system and discusses the method of realizing distributed control based on the Modbus protocol of RS485 communication. Through the comprehensive experiment of the configuration, the point control and the continuous path control are carried out to verify the correctness of the theoretical analysis of the system and the reliability of the system hardware and software operation.

Noise-tolerant RGB-D feature fusion network for outdoor fruit detection

In the process of farm automation, fruit detection is the basis and guarantee for yield prediction, automatic picking, and other orchard operations. RGB images can only obtain the two-dimensional information of the scene, which is not sufficient to effectively distinguish fruits that are dense growth and occlusion by branches and leaves. With the development of depth sensors, using RGB-D images with more complementary information can boost the performance of fruit detection. However, due to the nature of sensors and scene configurations, the quality of outdoor depth images is poor, posing a challenge when fusing RGB-D features. Therefore, this paper proposes an end-to-end RGB-D object detection network, termed as noise-tolerant feature fusion network (NT-FFN), to utilize the outdoor multi-modal data properly and improve the detection accuracy. Specifically, the NT-FFN first uses two structurally identical feature extractors to extract single-modal (color and depth) features, which is the base of the subsequent feature fusion. Then, to avoid introducing too much depth noise and focus the perception on the important part of the features, an attention-based fusion module is designed to adaptively fuse the multi-modal features. Finally, multi-scale features from the color images and the fusion modules are used to predict object position, which not only improves the network's ability to detect multi-scale objects but also further enhances the noise immunity of the network. In addition, this paper constructs an RGB-D citrus fruit dataset, which contributes to comprehensively evaluating the proposed network. Evaluation metrics on the dataset show that the NT-FFN achieves an AP50 of 95.4% with a real-time speed, which outperforms single-modal methods, common multi-modal fusion strategies, and advanced multi-modal detection methods. The proposed NT-FFN also achieves excellent detection results in other fruit detection tasks, which verifies its generalization ability. This study provides the possibility and foundation for performing multi-modal information fusion in outdoor fruit detection.