Multimodal Feature Fusion Research Articles

Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion.

With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis.

Infrared Small-Target Detection Based on Radiation Characteristics with a Multimodal Feature Fusion Network

Infrared small-target detection has widespread influences on anti-missile warning, precise weapon guidance, infrared stealth and anti-stealth, military reconnaissance, and other national defense fields. However, small targets are easily submerged in background clutter noise and have fewer pixels and shape features. Furthermore, random target positions and irregular motion can lead to target detection being carried out in the whole space–time domain. This could result in a large amount of calculation, and the accuracy and real-time performance are difficult to be guaranteed. Therefore, infrared small-target detection is still a challenging and far-reaching research hotspot. To solve the above problem, a novel multimodal feature fusion network (MFFN) is proposed, based on morphological characteristics, infrared radiation, and motion characteristics, which could compensate for the deficiency in the description of single modal characteristics of small targets and improve the recognition precision. Our innovations introduced in the paper are addressed in the following three aspects: Firstly, in the morphological domain, we propose a network with the skip-connected feature pyramid network (SCFPN) and dilated convolutional block attention module integrated with Resblock (DAMR) introduced to the backbone, which is designed to improve the feature extraction ability for infrared small targets. Secondly, in the radiation characteristic domain, we propose a prediction model of atmospheric transmittance based on deep neural networks (DNNs), which predicts the atmospheric transmittance effectively without being limited by the complex environment to improve the measurement accuracy of radiation characteristics. Finally, the dilated convolutional-network-based bidirectional encoder representation from a transformers (DC-BERT) structure combined with an attention mechanism is proposed for the feature extraction of radiation and motion characteristics. Finally, experiments on our self-established optoelectronic equipment detected dataset (OEDD) show that our method is superior to eight state-of-the-art algorithms in terms of the accuracy and robustness of infrared small-target detection. The comparative experimental results of four kinds of target sequences indicate that the average recognition rate Pavg is 92.64%, the mean average precision (mAP) is 92.01%, and the F1 score is 90.52%.

Alzheimer’s disease diagnosis via multimodal feature fusion

Alzheimer’s disease (AD) is the most common neurodegenerative disorder in the elderly. Early diagnosis of AD plays a vital role in slowing down the progress of AD because there is no effective drug to treat the disease. Some deep learning models have recently been presented for AD diagnosis and have more satisfactory performance than classic machine learning methods. Nevertheless, most of the existing computer-aided diagnostic models used neuroimaging features for diagnosis, ignoring patients’ clinical and biological information. This makes the AD diagnosis inaccurate. In this study, we propose a novel multimodal feature transformation and fusion model for AD diagnosis. The feature transformation aims to avoid the difference in feature dimensions between different modal data and further mine the significant features for AD diagnosis. A geometric algebra-based feature extension method is proposed to obtain different levels of high-dimensional features from patients’ clinical and personal biological data. Then, an influence degree-based feature filtration algorithm is proposed to filtrate those features that have no apparent guiding significance for AD diagnosis. Finally, an ANN (Artificial Neural Network)-based framework is designed to fuse transformed features with neuroimaging features extracted by CNN (Convolutional Neural Network) for AD diagnosis. The more in-depth feature mining of patients’ clinical information and biological information can significantly improve the performance of computer-aided AD diagnosis. The experiments are obtained on the ADNI dataset. Our proposed model can converge faster and achieves 96.2% accuracy in AD diagnostic task and 87.4% accuracy in MCI (Mild Cognitive Impairment) diagnostic task. Compared with other methods, our proposed approach has an excellent performance in AD diagnosis and surpasses SOTA (state-of-the-art) methods. Therefore, our model can provide more reasonable suggestions for clinicians to diagnose and treat disease.

Deep learning and multimodal feature fusion for the aided diagnosis of Alzheimer's disease

Deep learning and multimodal feature fusion for the aided diagnosis of Alzheimer's disease

Intelligent Matching System of Clauses in International Investment Arbitration Cases Based on Big Data Statistical Model

Abstract The international investment arbitration system between investors and host countries originated from the international commercial arbitration system. International investment disputes are usually disputes caused by the host country’s policies to protect public interests and the dissatisfaction of private investors caused by regulations. There are few methods for intelligently recommending judgment documents based on factual texts in international investment arbitration cases. Because of this, this paper proposes a recommendation algorithm for international investment arbitration adjudication documents based on multimodal feature fusion. This paper teaches the structural information of paragraph tags and the relationship between paragraph contexts and incomplete judgment documents. The algorithm achieves a good structuring effect of international investment arbitration documents.

Multimodal feature fusion and exploitation with dual learning and reinforcement learning for recipe generation

Recipes belong to long paragraphs with a cooking logic. To recipes from images and food names is more challenging in VQA (Visual Question Answering) due to the gap between images and texts. Although multimodal feature fusion, as a typical solver in VQA, is adopted in most situations for enhancing the accuracy, fused features obtained in this way can hardly provide guidance for keeping logic in produced texts. In this paper, ingredients are introduced to enhancing the relationship between food images and recipes, since they can reflect the cooking logic to a great extent, and dual learning is adopted to provide a complementary view by reconstructing ingredients from produced recipes. In order to make a full exploitation of ingredients for producing effective recipes, ingredients are fused into images and food names with an attention mechanism in the forward flow, and in the backward flow, a reconstructor is designed to reproduce ingredients from recipes. In addition, reinforcement learning is employed to guide ingredient reconstruction for preserving effective features in fused information explicitly. Extensive experiments demonstrate that more attention is allocated for producing effective recipes, and ablative study shows the reasonability of different components in the proposed method.

Intelligent Analysis of Exercise Health Big Data Based on Deep Convolutional Neural Network.

In this paper, the algorithm of the deep convolutional neural network is used to conduct in-depth research and analysis of sports health big data, and an intelligent analysis system is designed for the practical process. A convolutional neural network is one of the most popular methods of deep learning today. The convolutional neural network has the feature of local perception, which allows a complete image to be divided into several small parts, by learning the characteristic features of each local part and then merging the local information at the high level to get the full representation information. In this paper, we first apply a convolutional neural network for four classifications of brainwave data and analyze the accuracy and recall of the model. The model is then further optimized to improve its accuracy and is compared with other models to confirm its effectiveness. A demonstration platform of emotional fatigue detection with multimodal data feature fusion was established to realize data acquisition, emotional fatigue detection, and emotion feedback functions. The emotional fatigue detection platform was tested to verify that the proposed model can be used for time-series data feature learning. According to the platform requirement analysis and detailed functional design, the development of each functional module of the platform was completed and system testing was conducted. The big data platform constructed in this study can meet the basic needs of health monitoring for data analysis, which is conducive to the formation of a good situation of orderly and effective interaction among multiple subjects, thus improving the information service level of health monitoring and promoting comprehensive health development.

BM-NAS: Bilevel Multimodal Neural Architecture Search

Deep neural networks (DNNs) have shown superior performances on various multimodal learning problems. However, it often requires huge efforts to adapt DNNs to individual multimodal tasks by manually engineering unimodal features and designing multimodal feature fusion strategies. This paper proposes Bilevel Multimodal Neural Architecture Search (BM-NAS) framework, which makes the architecture of multimodal fusion models fully searchable via a bilevel searching scheme. At the upper level, BM-NAS selects the inter/intra-modal feature pairs from the pretrained unimodal backbones. At the lower level, BM-NAS learns the fusion strategy for each feature pair, which is a combination of predefined primitive operations. The primitive operations are elaborately designed and they can be flexibly combined to accommodate various effective feature fusion modules such as multi-head attention (Transformer) and Attention on Attention (AoA). Experimental results on three multimodal tasks demonstrate the effectiveness and efficiency of the proposed BM-NAS framework. BM-NAS achieves competitive performances with much less search time and fewer model parameters in comparison with the existing generalized multimodal NAS methods. Our code is available at https://github.com/Somedaywilldo/BM-NAS.

Vehicle Detection for Unmanned Systems Based on Multimodal Feature Fusion

This paper proposes a 3D vehicle-detection algorithm based on multimodal feature fusion to address the problem of low vehicle-detection accuracy in unmanned system environment awareness. The algorithm matches the coordinate relationships between the two sensors and reduces sampling errors by combining the millimeter-wave radar and camera calibration. Statistical filtering is used to remove redundant points from the millimeter-wave radar data to reduce outlier interference; a multimodal feature fusion module is constructed to fuse the point cloud and image information using pixel-by-pixel averaging. Moreover, feature pyramids are added to extract fused high-level feature information, which is used to improve detection accuracy in complex road scenarios. A feature fusion region proposal structure was established to generate region proposals based on the high-level feature information. The vehicle detection results were obtained by matching the detection frames in their vertices after removal of the redundant detection frames using non-maximum suppression. Experimental results from the KITTI dataset show that the proposed method improved the efficiency and accuracy of vehicle detection with the corresponding average of 0.14 s and 84.71%.

Automatic pavement crack detection using multimodal features fusion deep neural network

ABSTRACT The existing pavement crack detection algorithms are all dealing with detection and segmentation separately, ignoring the feature correlation between the bounding box coordinates and mask information, and there are still problems such as low crack detection accuracy, incomplete detection, and segmentation fracture in practical application. In view of the above situation, this paper took the detection bounding box coordinates and mask information as multimodal features of the same pavement crack area, and proposed a One-Stage MFFNet (Multimodal Feature Fusion Network), which improved the pavement crack detection accuracy and segmentation integrity obviously. And experimental results of different models were compared on self-collected datasets and two public datasets (CFD and CRACK500) respectively. Compared with Mask R-CNN, the average detection accuracy and average segmentation accuracy were improved by 2.6% and 4.7%, respectively. Compared with the optimised model of RDSNet, the detection accuracy and processing speed were improved by 1.8% and 2FPS, respectively. In addition, MFFNet had significantly improved the integrity of pavement crack segmentation results. The results showed that the proposed MFFNet model achieved the best detection and segmentation accuracy, and was an effective and high-precision pavement crack detection model.

Boosting RGB-D salient object detection with adaptively cooperative dynamic fusion network

The suitable employment of RGB and depth data shows great significance in promoting the development of computer vision tasks and robot-environment interactions. However, there are different advantages and disadvantages in the early and late fusion of the two types of data. In addition, due to the diversity of the object information, using a single type of data in a specific scenario results in being semantically misleading. Based on the above considerations, we propose a transformer-based adaptively cooperative dynamic fusion network (ACDNet) with a dynamic composite structure (DCS) for salient object detection. This structure is designed to flexibly utilize the advantages of feature fusion in different stages. Second, an adaptively cooperative semantic guidance (ACG) scheme is designed to suppress inaccurate features in multilevel multimodal feature fusion. Furthermore, we proposed a perceptual aggregation module (PAM) to optimize the network from the perspectives of spatial perception and scale perception, which strengthens the network’s ability to perceive multiscale objects. Extensive experiments conducted on 8 RGB-D SOD datasets illustrate that the proposed network outperforms 24 state-of-the-art algorithms.

Series-Constellation Feature Based Blind Modulation Recognition for Beyond 5G MIMO-OFDM Systems With Channel Fading

Due to the shortage of radio spectrum in the current 5G and upcoming 6G systems, the cognitive radio (CR) technique is indispensable for spectrum management and can put the unutilized spectrum to good use. As the core technology of CR, blind modulation recognition (BMR) plays a pivotal role in improving spectral efficiency. However, the BMR research on MIMO-OFDM systems still lacks enough attention. Given the prosperity of deep learning, we propose a series-constellation multi-modal feature network (SC-MFNet) to recognize the modulation types of MIMO-OFDM subcarriers. Without any prior information, a blind signal separation algorithm is employed to reconstruct the impaired transmitted signal. Considering the insufficient features of signal series, we propose a segment accumulated constellation diagram (SACD) strategy to produce the striking constellation features. Moreover, the proposed multi-modal feature fusion network is employed to collect the advantages of series and SACD features, which are extracted by one-dimensional convolution (Conv1DNet) branch and improved EfficientNet branch, respectively. Experimental results demonstrate that in a realistic non-cooperative cognitive communication scenario where prior information is exempted, the proposed SC-MFNet outperforms the traditional feature-based methods and the state-of-the-art neural networks which are based on either constellation features or series features.

Popularity prediction for marketer-generated content: A text-guided attention neural network for multi-modal feature fusion

In this paper, we focus on the popularity prediction for marketer-generated content (MGC), which has not been investigated by current studies. To address this problem, we propose a novel deep learning approach, namely the text-guided attention neural network (TGANN) model, to make full use of heterogeneous and multi-modal data related to MGCs (e.g., text descriptions and images). In the proposed model, we first design a filter-based topic model to filter out the noise words and extract topic features from textual descriptions. To lessen the influence of irrelevant information in images, we then propose a text-guided attention mechanism to use the text's topic features to guide the image region representations. Lastly, to determine the contribution of each topic and each image, the TGANN model introduces the attention computation for each visual modality and textual modality. Our experiments are conducted on two real-world datasets of MGC. The quantitative results show that the proposed model outperforms several state-of-the-art methods. The qualitative experiments demonstrate that our model can accurately capture topic attentions, image attentions, and image region attentions. The proposed model provides important practical implications for marketers and online platforms, e.g., estimating the success of online advertising campaigns, creating more attractive marketing contents, and improving recommendation systems.

Multispectral water leakage detection based on a one-stage anchor-free modality fusion network for metro tunnels

Water leakage detection has remained one of the high-priority research topics for metro tunnel inspection. Previous research mainly uses a single visible light camera as a sensor to detect the leakage regions in the image, which is extremely sensitive to inadequate illumination inside the tunnel. To robustly and accurately detect these water leakage defects in metro tunnels, we propose a novel detection method based on multispectral modality fusion that combines the advantages of visual-optical (VIS) and thermal infrared (IR) sensors. First, a multispectral data collection system is designed, and a four-dimensional water leakage dataset containing 1840 pictures is collected and labeled. Second, a simple and efficient one-stage anchor-free multispectral modality fusion network is proposed for water leakage detection. The proposed method consists of single-modality feature extraction and multimodality feature fusion based on a feature pyramid network (FPN). Finally, the VIS/IR single modality verification experiment proves that the VIS/IR based detector has unavoidable disadvantages in water leakage detection. The multispectral detection experiment proves that our proposed modality fusion detector achieves an approximately 3.35% lower average miss rate than the state-of-the-art method and can accurately and robustly detect leakage defects not affected by light conditions.

Multimodal Feature Fusion Based Hypergraph Learning Model.

Hypergraph learning is a new research hotspot in the machine learning field. The performance of the hypergraph learning model depends on the quality of the hypergraph structure built by different feature extraction methods as well as its incidence matrix. However, the existing models are all hypergraph structures built based on one feature extraction method, with limited feature extraction and abstract expression ability. This paper proposed a multimodal feature fusion method, which firstly built a single modal hypergraph structure based on different feature extraction methods, and then extended the hypergraph incidence matrix and weight matrix of different modals. The extended matrices fuse the multimodal abstract feature and an expanded Markov random walk range during model learning, with stronger feature expression ability. However, the extended multimodal incidence matrix has a high scale and high computational cost. Therefore, the Laplacian matrix fusion method was proposed, which performed Laplacian matrix transformation on the incidence matrix and weight matrix of every model, respectively, and then conducted a weighted superposition on these Laplacian matrices for subsequent model training. The tests on four different types of datasets indicate that the hypergraph learning model obtained after multimodal feature fusion has a better classification performance than the single modal model. After Laplace matrix fusion, the average time can be reduced by about 40% compared with the extended incidence matrix, the classification performance can be further improved, and the index F1 can be improved by 8.4%.

An Enhanced Term Weighted Question Embedding for Visual Question Answering

Visual Question Answering (VQA) is a multi-modal AI-complete task of answering natural language questions about images. Literature solved VQA with a three-phase pipeline: image and question featurisation, multi-modal feature fusion and answer generation or prediction. Most of the works have given attention to the second phase, where multi-modal features get combined ignoring the effect of individual input features. This work investigates VQA’s natural language question embedding phase by proposing a new question featurisation framework based on Supervised Term Weighting (STW) schemes. In addition, two new STW schemes integrating text semantics, qf.cos and tf.rf.sim, have been introduced to boost the framework’s performance. A series of tests on the DAQUAR VQA dataset is used to compare the new system to conventional pre-trained word embedding. Over the past few years, STW schemes have been commonly used in text classification research. In light of this, tests are carried out to verify the effectiveness of the two newly proposed STW schemes in the general text classification task.

CIRNet: An improved RGBT tracking via cross-modality interaction and re-identification

RGBT tracking is receiving more and more attention because of its huge tracking potential in an all-weather environment. RGB and thermal source data contain different levels of information about the object. Utilizing the complementary advantage of different levels of information can effectively improve the tracking performance. Existing work focuses on the extraction and fusion of multi-modal features. Although these methods effectively deploy the fusion of information among multiple modalities, they ignore the potential value of multi-level shared clues in different modalities. In addition, these works cannot provide effective candidate boxes after tracking drift, resulting in limited tracker performance. In this paper, we propose a cross-modality interaction and re-identification network that performs multi-level modality-shared, modality-specific and object probability prediction learning. We designed two feature extraction sub-networks, namely, a multi-level modality-shared fusion network and modality complementary sub-network. Specifically, the two sub-networks extract and fuse multi-level modality shared information and modality specific information, respectively. To optimize tracking drift, object-aware branches that predict the object-centered state are designed. Our object-aware branching is simple, neat and efficient. Moreover, to achieve the visual tracking real-time requirement, we designed the object regression branch that does not require repeated region suggestion input. By extensive experiments and comparisons with state-of-the-art trackers on the RGBT tracking benchmark dataset, our tracker achieves leading performance and essentially real-time tracking speeds. Tracking drift caused by occlusion, fast motion and camera moving is significantly optimized.

English Pronunciation Calibration Model Based on Multimodal Acoustic Sensor

In recent years, with the increasing frequency of international exchanges, people have gradually realized that language is a tool of communication and communication, and language learning should attach importance to oral teaching. However, in traditional classrooms, one of the problems faced by oral teaching is the mismatch of the teacher-student ratio: a teacher has to deal with dozens of students, one-on-one oral teaching and pronunciation guidance is impossible, and it is also affected by the teachers and the environment constraints. Therefore, the research on how to efficiently automate pronunciation training is becoming more and more popular. Many phonemes in English have different facial visual features, especially vowels. Almost all of them can be distinguished by the roundness and tightness of the lips in appearance. In order to give full play to the role of lip features in oral pronunciation error detection, this paper proposes a multimodal feature fusion model based on lip angle features. The model interpolates the lip features constructed based on the opening and closing angles and combines audio and video in time series. Feature alignment and fusion and feature learning and classification are realized through the two-way LSTM SOFTMAX layer, and finally, end-to-end pronunciation error detection is realized through CTC. It is verified on the GRID audio and video corpus after phoneme conversion and the self-built multimodal test set. The experimental results show that the model has a higher false pronunciation recognition rate than the traditional single-modal acoustic error detection model. The increase in error detection rate is more obvious. Verification by the audio and video corpus with white noise was added, and the proposed model has better noise immunity than the traditional acoustic model.

Attention guided discriminative feature learning and adaptive fusion for grading hepatocellular carcinoma with Contrast-enhanced MR.

Multimodality medical imaging has played a significant role in lesion diagnosis and characterization. However, there are remaining challenges in the procedure of multimodality feature fusion based lesion characterization. First, large inter-modality variations make it difficult to harness the complementary information between modalities for better characterization. Subsequently, large intra-class and small inter-class variations due to the heterogeneity of neoplasm makes the classification more challenging. Finally, the relative importance of modalities for the characterization has not been thoroughly investigated, easily resulting in non-optimal fusion performance. In this study, we propose an attention guided discriminative and adaptive fusion (AGDAF) method based on deep learning architecture to address above three problems. Specifically, we first design a novel cross-modal intra- and inter-attention module to focus on learning both the intra-modality relations and inter-modality relations. Then, we introduce a discriminative feature learning loss to reduce the distance of features in the same class and increase the distance of features in different classes of neoplasm in single modalities. Finally, we design an adaptive weighting strategy to increase the contribution of modalities with relatively lower loss values and reduce the impact of modalities with large loss values for the final loss function. Experimental results of grading clinical hepatocellular carcinoma demonstrate that the proposed method significantly outperforms the previously reported multimodality feature fusion methods. In addition, ablation study also demonstrates the effectiveness of the proposed cross-modal intra- and inter-attention module, discriminative module, and adaptive weight adjustment module for multimodality feature fusion in lesion characterization.

False Information Detection via Multimodal Feature Fusion and Multi-Classifier Hybrid Prediction

In the existing false information detection methods, the quality of the extracted single-modality features is low, the information between different modalities cannot be fully fused, and the original information will be lost when the information of different modalities is fused. This paper proposes a false information detection via multimodal feature fusion and multi-classifier hybrid prediction. In this method, first, bidirectional encoder representations for transformers are used to extract the text features, and S win-transformer is used to extract the picture features, and then, the trained deep autoencoder is used as an early fusion method of multimodal features to fuse text features and visual features, and the low-dimensional features are taken as the joint features of the multimodalities. The original features of each modality are concatenated into the joint features to reduce the loss of original information. Finally, the text features, image features and joint features are processed by three classifiers to obtain three probability distributions, and the three probability distributions are added proportionally to obtain the final prediction result. Compared with the attention-based multimodal factorized bilinear pooling, the model achieves 4.3% and 1.2% improvement in accuracy on Weibo dataset and Twitter dataset. The experimental results show that the proposed model can effectively integrate multimodal information and improve the accuracy of false information detection.