Multimodal Fusion Network Research Articles

Robust multimodal fusion network employing novel Empirical Riglit Wavelet Transform for brain images

Machine learning is useful for pattern recognition, if allowed access to patient data, it can notice patterns that would be missed by human doctors, which could be used to predict if a person is at risk for a disease that would not have been anticipated by a doctor. In this paper, the authors have proposed an Empirical Riglit Wavelet Transform algorithm. In this algorithm, the authors have fused the filter banks of CT and MR images obtained from Ridgelet and Little wood Empirical Wavelet Transform. Four possible combinations were used for the fusion. Image boundaries were evaluated as performance parameters. With that parameters helps in understanding the small elements and details from given CT and MR images. The objective of this paper is to classify and extract specific patterns in the images using different combinations of CT and MR by fusing them. The proposed algorithm is validated via filter banks obtained for fused CT-MT images using the same techniques.

Multimodal fusion recognition for digital twin

The digital twin is the concept of transcending reality, which is the reverse feedback from the real physical space to the virtual digital space. People hold great prospects for this emerging technology. In order to realize the upgrading of the digital twin industrial chain, it is urgent to introduce more modalities, such as vision, haptics, hearing and smell, into the virtual digital space, which assists physical entities and virtual objects in creating a closer connection. Therefore, perceptual understanding and object recognition have become an urgent hot topic in the digital twin. Existing surface material classification schemes often achieve recognition through machine learning or deep learning in a single modality, ignoring the complementarity between multiple modalities. In order to overcome this dilemma, we propose a multimodal fusion network in our article that combines two modalities, visual and haptic, for surface material recognition. On the one hand, the network makes full use of the potential correlations between multiple modalities to deeply mine the modal semantics and complete the data mapping. On the other hand, the network is extensible and can be used as a universal architecture to include more modalities. Experiments show that the constructed multimodal fusion network can achieve 99.42% classification accuracy while reducing complexity.

Self-supervised multi-modal fusion network for multi-modal thyroid ultrasound image diagnosis

Ultrasound is a typical non-invasive diagnostic method often used to detect thyroid cancer lesions. However, due to the limitations of the information provided by ultrasound images, shear wave elastography (SWE) and color doppler ultrasound (CDUS) are also used clinically to assist in diagnosis, which makes the diagnosis time-consuming, labor-intensive, and highly subjective process. Therefore, automatic diagnosis of benign and malignant thyroid nodules is beneficial for the clinical diagnosis of the thyroid. To this end, based on three modalities of gray-scale ultrasound images(US), SWE, and CDUS, we propose a deep learning-based multi-modal feature fusion network for the automatic diagnosis of thyroid disease based on the ultrasound images. First, three ResNet18s initialized by self-supervised learning are used as branches to extract the image information of each modality, respectively. Then, a multi-modal multi-head attention branch is used to remove the common information of three modalities, and the knowledge of each modal is combined for thyroid diagnosis. At the same time, to better integrate the features between modalities, a multi-modal feature guidance module is also proposed to guide the feature extraction of each branch and reduce the difference between each-modal feature. We verify the multi-modal thyroid ultrasound image diagnosis method on the self-collected dataset, and the results prove that this method could provide fast and accurate assistance for sonographers in diagnosing thyroid nodules.

Multi-Modal Feature Fusion Network with Adaptive Center Point Detector for Building Instance Extraction

Building information extraction utilizing remote sensing technology has vital applications in many domains, such as urban planning, cadastral mapping, geographic information censuses, and land-cover change analysis. In recent years, deep learning algorithms with strong feature construction ability have been widely used in automatic building extraction. However, most methods using semantic segmentation networks cannot obtain object-level building information. Some instance segmentation networks rely on predefined detectors and have weak detection ability for buildings with complex shapes and multiple scales. In addition, the advantages of multi-modal remote sensing data have not been effectively exploited to improve model performance with limited training samples. To address the above problems, we proposed a CNN framework with an adaptive center point detector for the object-level extraction of buildings. The proposed framework combines object detection and semantic segmentation with multi-modal data, including high-resolution aerial images and LiDAR data, as inputs. Meanwhile, we developed novel modules to optimize and fuse multi-modal features. Specifically, the local spatial–spectral perceptron can mutually compensate for semantic information and spatial features. The cross-level global context module can enhance long-range feature dependence. The adaptive center point detector explicitly models deformable convolution to improve detection accuracy, especially for buildings with complex shapes. Furthermore, we constructed a building instance segmentation dataset using multi-modal data for model training and evaluation. Quantitative analysis and visualized results verified that the proposed network can improve the accuracy and efficiency of building instance segmentation.

A Multi-Modal Fusion Network Guided by Feature Co-Occurrence for Urban Region Function Recognition

A Multi-Modal Fusion Network Guided by Feature Co-Occurrence for Urban Region Function Recognition

Robust multimodal fusion network using adversarial learning for brain tumor grading

Background and ObjectiveGliomas are graded using multimodal magnetic resonance imaging, which provides important information for treatment and prognosis. When modalities are missing, the grading is degraded. We propose a robust brain tumor grading model that can handle missing modalities. MethodsOur method was developed and tested on Brain Tumor Segmentation Challenge 2017 dataset (n = 285) via nested five-fold cross-validation. Our method adopts adversarial learning to generate the features of missing modalities relative to the features obtained from a full set of modalities in the latent space. An attention-based fusion block across modalities fuses the features of each available modality into a shared representation. Our method's results are compared to those of two other models where 15 missing-modality scenarios are explicitly considered and a joint training approach with random dropouts is used. ResultsOur method outperforms the two competing methods in classifying high-grade gliomas (HGGs) and low-grade gliomas (LGGs), achieving an area under the curve of 87.76% on average for all missing-modality scenarios. The activation maps derived with our method confirm that it focuses on the enhancing portion of the tumor in HGGs and on the edema and non-enhancing portions of the tumor in LGGs, which is consistent with prior expertise. An ablation study shows the added benefits of a fusion block and adversarial learning for handling missing modalities. ConclusionOur method shows robust grading of gliomas in all cases of missing modalities. Our proposed network might have positive implications in glioma care by learning features robust to missing modalities.

Semantic‐enhanced multimodal fusion network for fake news detection

The increasing popularity of social media facilitates the propagation of fake news, posing a major threat to the government and journalism, and thereby making how to detect fake news from social media an urgent requirement. In general, multimodal-based methods can achieve better performance because of the complementation among different modalities. However, the majority of them simply concatenate features from different modalities, failing to well preserve the mutual information in common features. To address this issue, a novel framework named semantic-enhanced multimodal fusion network is proposed for fake news detection, which can better capture mutual features among events and thus benefit the detection of fake news. This model consists of three subnetworks, namely multimodal fusion and event domain adaptation networks as well as the fake news detector. Specifically, the multimodal fusion network aims to extract deep features from texts and images and fuse them into a common semantic feature known as a snapshot. Then, the fake news detector can learn the representation of posts. Finally, the event domain adaptation network can single out and remove the peculiar features of each event, and keep shared features among events. The experimental results show that the proposed model outperforms some state-of-the-art approaches on two real-world multimedia data sets.

Research on 24-Hour Dense Crowd Counting and Object Detection System Based on Multimodal Image Optimization Feature Fusion

Motivation. In the environment of day and night video surveillance, in order to improve the accuracy of machine vision dense crowd counting and target detection, this paper designs a day and night dual-purpose crowd counting and crowd detection network based on multimode image fusion. Methods. Two sub-models, RGBD-Net and RGBT-Net, are designed in this paper. The depth image features and thermal imaging features are effectively fused with the features of visible light images, so that the model has stronger anti-interference characteristics and robustness to the light noise interference caused by the sudden fall of light at night. The above models use density map regression-guided detection method to complete population counting and detection. Results. The model completed daytime training and testing on MICC dataset. Through verification, the average absolute error of the model was 1.025, the mean square error was 1.521, and the recall rate of target detection was 97.11%. Night vision training and testing were completed on the RGBT-CC dataset. After verification, the average absolute error of the network was 18.16, the mean square error was 32.14, and the recall rate of target detection was 97.65%. By verifying the effectiveness of the multimode medium-term fusion network, it is found to exceed the current most advanced bimodal fusion method. Conclusion. The experimental results show that the proposed multimodal fusion network can solve the counting and detection problem in the video surveillance environment during day and night. The ablation experiment further proves the effectiveness of the parameters of the two models.

AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model

Multimodal sentiment analysis is an essential task in natural language processing which refers to the fact that machines can analyze and recognize emotions through logical reasoning and mathematical operations after learning multimodal emotional features. For the problem of how to consider the effective fusion of multimodal data and the relevance of multimodal data in multimodal sentiment analysis, we propose an attention-based mechanism feature relevance fusion multimodal sentiment analysis model (AFR-BERT). In the data pre-processing stage, text features are extracted using the pre-trained language model BERT (Bi-directional Encoder Representation from Transformers), and the BiLSTM (Bi-directional Long Short-Term Memory) is used to obtain the internal information of the audio. In the data fusion phase, the multimodal data fusion network effectively fuses multimodal features through the interaction of text and audio information. During the data analysis phase, the multimodal data association network analyzes the data by exploring the correlation of fused information between text and audio. In the data output phase, the model outputs the results of multimodal sentiment analysis. We conducted extensive comparative experiments on the publicly available sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experimental results show that AFR-BERT improves on the classical multimodal sentiment analysis model in terms of relevant performance metrics. In addition, ablation experiments and example analysis show that the multimodal data analysis network in AFR-BERT can effectively capture and analyze the sentiment features in text and audio.

Multimodal ultrasound fusion network for differentiating between benign and malignant solid renal tumors.

Objective: We aim to establish a deep learning model called multimodal ultrasound fusion network (MUF-Net) based on gray-scale and contrast-enhanced ultrasound (CEUS) images for classifying benign and malignant solid renal tumors automatically and to compare the model’s performance with the assessments by radiologists with different levels of experience. Methods: A retrospective study included the CEUS videos of 181 patients with solid renal tumors (81 benign and 100 malignant tumors) from June 2012 to June 2021. A total of 9794 B-mode and CEUS-mode images were cropped from the CEUS videos. The MUF-Net was proposed to combine gray-scale and CEUS images to differentiate benign and malignant solid renal tumors. In this network, two independent branches were designed to extract features from each of the two modalities, and the features were fused using adaptive weights. Finally, the network output a classification score based on the fused features. The model’s performance was evaluated using five-fold cross-validation and compared with the assessments of the two groups of radiologists with different levels of experience. Results: For the discrimination between benign and malignant solid renal tumors, the junior radiologist group, senior radiologist group, and MUF-Net achieved accuracy of 70.6%, 75.7%, and 80.0%, sensitivity of 89.3%, 95.9%, and 80.4%, specificity of 58.7%, 62.9%, and 79.1%, and area under the receiver operating characteristic curve of 0.740 (95% confidence internal (CI): 0.70–0.75), 0.794 (95% CI: 0.72–0.83), and 0.877 (95% CI: 0.83–0.93), respectively. Conclusion: The MUF-Net model can accurately classify benign and malignant solid renal tumors and achieve better performance than senior radiologists. Key points: The CEUS video data contain the entire tumor microcirculation perfusion characteristics. The proposed MUF-Net based on B-mode and CEUS-mode images can accurately distinguish between benign and malignant solid renal tumors with an area under the receiver operating characteristic curve of 0.877, which surpasses senior radiologists’ assessments by a large margin.

Swin-MFA: A Multi-Modal Fusion Attention Network Based on Swin-Transformer for Low-Light Image Human Segmentation.

In recent years, image segmentation based on deep learning has been widely used in medical imaging, automatic driving, monitoring and security. In the fields of monitoring and security, the specific location of a person is detected by image segmentation, and it is segmented from the background to analyze the specific actions of the person. However, in low-illumination conditions, it is a great challenge to the traditional image-segmentation algorithms. Unfortunately, a scene with low light or even no light at night is often encountered in monitoring and security. Given this background, this paper proposes a multi-modal fusion network based on the encoder and decoder structure. The encoder, which contains a two-branch swin-transformer backbone instead of the traditional convolutional neural network, fuses the RGB and depth features with a multiscale fusion attention block. The decoder is also made up of the swin-transformer backbone and is finally connected via the encoder with several residual connections, which are proven to be beneficial in improving the accuracy of the network. Furthermore, this paper first proposes the low light–human segmentation (LLHS) dataset of portrait segmentation, with aligned depth and RGB images with fine annotation under low illuminance, by combining the traditional monocular camera and a depth camera with active structured light. The network is also tested in different levels of illumination. Experimental results show that the proposed network has good robustness in the scene of human segmentation in a low-light environment with varying illumination. The mean Intersection over Union (mIoU), which is often used to evaluate the performance of image segmentation model, of the Swin-MFA in the LLHS dataset is 81.0, is better than those of ACNet, 3DGNN, ESANet, RedNet and RFNet at the same level of depth in a mixed multi-modal network and is far ahead of the segmentation algorithm that only uses RGB features, so it has important practical significance.

A multi-stage dynamical fusion network for multimodal emotion recognition.

In recent years, emotion recognition using physiological signals has become a popular research topic. Physiological signal can reflect the real emotional state for individual which is widely applied to emotion recognition. Multimodal signals provide more discriminative information compared with single modal which arose the interest of related researchers. However, current studies on multimodal emotion recognition normally adopt one-stage fusion method which results in the overlook of cross-modal interaction. To solve this problem, we proposed a multi-stage multimodal dynamical fusion network (MSMDFN). Through the MSMDFN, the joint representation based on cross-modal correlation is obtained. Initially, the latent and essential interactions among various features extracted independently from multiple modalities are explored based on specific manner. Subsequently, the multi-stage fusion network is designed to split the fusion procedure into multi-stages using the correlation observed before. This allows us to exploit much more fine-grained unimodal, bimodal and trimodal intercorrelations. For evaluation, the MSMDFN was verified on multimodal benchmark DEAP. The experiments indicate that our method outperforms the related one-stage multi-modal emotion recognition works.

Infrared Small-Target Detection Based on Radiation Characteristics with a Multimodal Feature Fusion Network

Infrared small-target detection has widespread influences on anti-missile warning, precise weapon guidance, infrared stealth and anti-stealth, military reconnaissance, and other national defense fields. However, small targets are easily submerged in background clutter noise and have fewer pixels and shape features. Furthermore, random target positions and irregular motion can lead to target detection being carried out in the whole space–time domain. This could result in a large amount of calculation, and the accuracy and real-time performance are difficult to be guaranteed. Therefore, infrared small-target detection is still a challenging and far-reaching research hotspot. To solve the above problem, a novel multimodal feature fusion network (MFFN) is proposed, based on morphological characteristics, infrared radiation, and motion characteristics, which could compensate for the deficiency in the description of single modal characteristics of small targets and improve the recognition precision. Our innovations introduced in the paper are addressed in the following three aspects: Firstly, in the morphological domain, we propose a network with the skip-connected feature pyramid network (SCFPN) and dilated convolutional block attention module integrated with Resblock (DAMR) introduced to the backbone, which is designed to improve the feature extraction ability for infrared small targets. Secondly, in the radiation characteristic domain, we propose a prediction model of atmospheric transmittance based on deep neural networks (DNNs), which predicts the atmospheric transmittance effectively without being limited by the complex environment to improve the measurement accuracy of radiation characteristics. Finally, the dilated convolutional-network-based bidirectional encoder representation from a transformers (DC-BERT) structure combined with an attention mechanism is proposed for the feature extraction of radiation and motion characteristics. Finally, experiments on our self-established optoelectronic equipment detected dataset (OEDD) show that our method is superior to eight state-of-the-art algorithms in terms of the accuracy and robustness of infrared small-target detection. The comparative experimental results of four kinds of target sequences indicate that the average recognition rate Pavg is 92.64%, the mean average precision (mAP) is 92.01%, and the F1 score is 90.52%.

Model application to quantitatively evaluate placental features from ultrasound images with gestational diabetes.

The goal of this study was to introduce PFCnet (placental features classification network), an multimodel model for evaluating and classifying placental features in gestational diabetes mellitus (GDM) and normal late pregnancy. Deep learning algorithms could be utilized to fully automate the examination of alterations in the placenta caused by hyperglycemia. A total of 718 placental ultrasound images, including 139 cases of GDM, were collected, including gray-scale images (GSIs) and microflow images (MFIs). Ultrasonic assessment parameters and perinatal features were recorded. We divided gestational age into two categories for analysis (37 weeks and 37 weeks) based on the cut-off value level of placental maturity. The PFCnet model was introduced for identifying placental characteristics from normal and GDM pregnancies after extensive training and optimization. The model was scored using metrics such as sensitivity, specificity, accuracy, and the area under the curve (AUC). In view of multimodal fusion (GSIs and MFIs) and deep network optimization training, the overall diagnostic performance of the PFCnet model depending on the region of interest (ROI) was excellent (AUC: 93%), with a sensitivity of 89%, a specificity of 92%, and an accuracy of 92% in the independent test set. The fusion features of GSIs and MFIs in the placenta showed a higher discriminative power than single-mode features (accuracy: Fusion 92% vs. GSIs 84% vs. MFIs 82%). The independent test set at 37 weeks exhibited a better specificity (75% vs. 69%) but a lower sensitivity(95% vs. 100%). With its dual channel identification of placental parenchymal and vascular lesions in obstetric complications, the PFCnet classification model has the potential to be a useful tool for detecting placental tissue abnormalities caused by hyperglycemia.

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection.

Violence detection aims to locate violent content in video frames. Improving the accuracy of violence detection is of great importance for security. However, the current methods do not make full use of the multi-modal vision and audio information, which affects the accuracy of violence detection. We found that the violence detection accuracy of different kinds of videos is related to the change of optical flow. With this in mind, we propose an optical flow-aware-based multi-modal fusion network (OAMFN) for violence detection. Specifically, we use three different fusion strategies to fully integrate multi-modal features. First, the main branch concatenates RGB features and audio features and the optical flow branch concatenates optical flow features with RGB features and audio features, respectively. Then, the cross-modal information fusion module integrates the features of different combinations and applies weights to them to capture cross-modal information in audio and video. After that, the channel attention module extracts valuable information by weighting the integration features. Furthermore, an optical flow-aware-based score fusion strategy is introduced to fuse features of different modalities from two branches. Compared with methods on the XD-Violence dataset, our multi-modal fusion network yields APs that are 83.09% and 1.4% higher than those of the state-of-the-art methods in offline detection, and 78.09% and 4.42% higher than those of the state-of-the-art methods in online detection.

College Student Retention Risk Analysis from Educational Database Using Multi-Task Multi-Modal Neural Fusion

We develop a Multimodal Spatiotemporal Neural Fusion network for MTL (MSNF-MTCL) to predict 5 important students' retention risks: future dropout, next semester dropout, type of dropout, duration of dropout and cause of dropout. First, we develop a general purpose multi-modal neural fusion network model MSNF for learning students' academic information representation by fusing spatial and temporal unstructured advising notes with spatiotemporal structured data. MSNF combines a Bidirectional Encoder Representations from Transformers (BERT)-based document embedding framework to represent each advising note, Long-Short Term Memory (LSTM) network to model temporal advising note embeddings, LSTM network to model students' temporal performance variables and students' static demographics altogether. The final fused representation from MSNF has been utilized on a Multi-Task Cascade Learning (MTCL) model towards building MSNF-MTCL for predicting 5 student retention risks. We evaluate MSNF-MTCL on a large educational database consists of 36,445 college students over 18 years period of time that provides promising performances comparing with the nearest state-of-art models. Additionally, we test the fairness of such model given the existence of biases.

Hierarchical attention network for multivariate time series long-term forecasting.

Multivariate time series long-term forecasting has always been the subject of research in various fields such as economics, finance, and traffic. In recent years, attention-based recurrent neural networks (RNNs) have received attention due to their ability of reducing error accumulation. However, the existing attention-based RNNs fail to eliminate the negative influence of irrelevant factors on prediction, and ignore the conflict between exogenous factors and target factor. To tackle these problems, we propose a novel Hierarchical Attention Network (HANet) for multivariate time series long-term forecasting. At first, HANet designs a factor-aware attention network (FAN) and uses it as the first component of the encoder. FAN weakens the negative impact of irrelevant exogenous factors on predictions by assigning small weights to them. Then HANet proposes a multi-modal fusion network (MFN) as the second component of the encoder. MFN employs a specially designed multi-modal fusion gate to adaptively select how much information about the expression of current time come from target and exogenous factors. Experiments on two real-world datasets reveal that HANet not only outperforms state-of-the-art methods, but also provides interpretability for prediction.

Series-Constellation Feature Based Blind Modulation Recognition for Beyond 5G MIMO-OFDM Systems With Channel Fading

Due to the shortage of radio spectrum in the current 5G and upcoming 6G systems, the cognitive radio (CR) technique is indispensable for spectrum management and can put the unutilized spectrum to good use. As the core technology of CR, blind modulation recognition (BMR) plays a pivotal role in improving spectral efficiency. However, the BMR research on MIMO-OFDM systems still lacks enough attention. Given the prosperity of deep learning, we propose a series-constellation multi-modal feature network (SC-MFNet) to recognize the modulation types of MIMO-OFDM subcarriers. Without any prior information, a blind signal separation algorithm is employed to reconstruct the impaired transmitted signal. Considering the insufficient features of signal series, we propose a segment accumulated constellation diagram (SACD) strategy to produce the striking constellation features. Moreover, the proposed multi-modal feature fusion network is employed to collect the advantages of series and SACD features, which are extracted by one-dimensional convolution (Conv1DNet) branch and improved EfficientNet branch, respectively. Experimental results demonstrate that in a realistic non-cooperative cognitive communication scenario where prior information is exempted, the proposed SC-MFNet outperforms the traditional feature-based methods and the state-of-the-art neural networks which are based on either constellation features or series features.

MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph

Entity Alignment (EA) is a crucial task in knowledge fusion, which aims to link entities with the same real-world identity from different Knowledge Graphs (KGs). Existing methods have achieved satisfactory performance, however, they mainly focus on single modal KG, which is difficult to be effectively applied to multi-modal scenes. In this paper, we propose a Multi-modal Joint entity Alignment Framework (MultiJAF), which can effectively utilize the knowledge of various modalities. Concretely, we first learn the embeddings of different modalities, i.e., structure, attribute and image modalities. Next, we adopt an attention-based multi-modal fusion network to integrate these embeddings and use obtained joint embeddings to compute a joint embedding-based similarity matrix SJ. Moreover, we design a Numerical Process Module (NPM) to infer a similarity matrix SN according to the numerical information of entities. In the end, we utilize a simple late fusion method to ensemble two similarity matrices for the final alignment. In addition, to reduce the cost of labeling data, we propose a novel NPM-based unsupervised multi-modal EA method. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed MultiJAF.

Multi-modal affine fusion network for social media rumor detection.

With the rapid development of the Internet, people obtain much information from social media such as Twitter and Weibo every day. However, due to the complex structure of social media, many rumors with corresponding images are mixed in factual information to be widely spread, which misleads readers and exerts adverse effects on society. Automatically detecting social media rumors has become a challenge faced by contemporary society. To overcome this challenge, we proposed the multimodal affine fusion network (MAFN) combined with entity recognition, a new end-to-end framework that fuses multimodal features to detect rumors effectively. The MAFN mainly consists of four parts: the entity recognition enhanced textual feature extractor, the visual feature extractor, the multimodal affine fuser, and the rumor detector. The entity recognition enhanced textual feature extractor is responsible for extracting textual features that enhance semantics with entity recognition from posts. The visual feature extractor extracts visual features. The multimodal affine fuser extracts the three types of modal features and fuses them by the affine method. It cooperates with the rumor detector to learn the representations for rumor detection to produce reliable fusion detection. Extensive experiments were conducted on the MAFN based on real Weibo and Twitter multimodal datasets, which verified the effectiveness of the proposed multimodal fusion neural network in rumor detection.