Multimodal Feature Fusion Research Articles

Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Graph convolutional networks are widely used in skeleton-based action recognition because of their good fitting ability to non-Euclidean data. While conventional multi-scale temporal convolution uses several fixed-size convolution kernels or dilation rates at each layer of the network, we argue that different layers and datasets require different receptive fields. We use multi-scale adaptive convolution kernels and dilation rates to optimize traditional multi-scale temporal convolution with a simple and effective self attention mechanism, allowing different network layers to adaptively select convolution kernels of different sizes and dilation rates instead of being fixed and unchanged. Besides, the effective receptive field of the simple residual connection is not large, and there is a great deal of redundancy in the deep residual network, which will lead to the loss of context when aggregating spatio-temporal information. This article introduces a feature fusion mechanism that replaces the residual connection between initial features and temporal module outputs, effectively solving the problems of context aggregation and initial feature fusion. We propose a multi-modality adaptive feature fusion framework (MMAFF) to simultaneously increase the receptive field in both spatial and temporal dimensions. Concretely, we input the features extracted by the spatial module into the adaptive temporal fusion module to simultaneously extract multi-scale skeleton features in both spatial and temporal parts. In addition, based on the current multi-stream approach, we use the limb stream to uniformly process correlated data from multiple modalities. Extensive experiments show that our model obtains competitive results with state-of-the-art methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.

DXVNet-ViT-Huge (JFT) Multimode Classification Network Based on Vision Transformer

Aiming at the problem that traditional CNN network is not good at extracting global features of images, Based on DXVNet network, Conditional Random Fields (CRF) component and pre-trained ViT-Huge (Vision Transformer) are adopted in this paper Transformer model expands and builds a brand new DXVNet-ViT-Huge (JFT) network. CRF component can help the network learn the constraint conditions of each word corresponding prediction label, improve the D-GRU method based word label prediction errors, and improve the accuracy of sequence annotation. The Transformer architecture of the ViT (Huge) model can extract the global feature information of the image, while CNN is better at extracting the local features of the image. Therefore, the ViT (Huge) Huge pre-training model and CNN pre-training model adopt the multi-modal feature fusion technology. Two complementary image feature information is fused by Bi-GRU to improve the performance of network classification. The experimental results show that the newly constructed Dxvnet-Vit-Huge (JFT) model achieves good performance, and the F1 values in the two real public data sets are 6.03% and 7.11% higher than the original DXVNet model, respectively.

Detection of autism spectrum disorder from changing of pupil diameter using multi-modal feature fusion based hybrid CNN model

Detection of autism spectrum disorder from changing of pupil diameter using multi-modal feature fusion based hybrid CNN model

A novel method for image captioning using multimodal feature fusion employing mask RNN and LSTM models

A novel method for image captioning using multimodal feature fusion employing mask RNN and LSTM models

RMFNet: Redetection Multimodal Fusion Network for RGBT Tracking

The development of single-modality target tracking based on visible light has been limited in recent years because visual light images are highly susceptible to environmental and lighting influences. Thermal infrared images can well compensate for this defect, so RGBT tracking has attracted increasing attention. However, existing studies are limited to the aggregation of multimodal information using feature fusion, ignoring the role of decision-level fusion in tracking, and the original re-detection algorithm in the used baseline model is prone to the accumulation of failures. To deal with these problems, we propose the Redetection Multimodal Fusion Network (RMFNet). The network is divided into three branches, the visible light branch, the thermal infrared branch, and the fusion branch. The three-branch structure can plainly utilize the complementary advantages of multimodal information and the commonalities and specific characteristics of the two modalities. We propose a multimodal feature fusion module (EFM), which can adaptively calculate the reliability of the modality and perform a weighted fusion of the two-modality features features. The existing redetection algorithm is improved, and the re-detection mechanism of global search in the current frame is added to reduce the accumulation of failures. We have conducted extensive comparative validation on two widely used benchmark datasets, GTOT and RGBT234. The outcomes of the experiments suggest that RMFNet outperforms other tracking methods.

Multi-modal multi-task feature fusion for RGBT tracking

RGBT tracking has received more and more attention in recent years, and in this paper, we propose a multi-task auxiliary learning framework for RGBT tracking. Specifically, we simplify the tracking task to an instance classification task and make it the primary task of the framework. We designed three auxiliary tasks and used a hard-parameter sharing approach to jointly train multiple tasks, hoping that the primary task would benefit from them. The three auxiliary tasks are contrastive instance discrimination, one-shot instance segmentation, and instance semantic matching. The contrastive instance discrimination method promotes the classification process of the primary task by constraining the features in the representation space. One-shot instance segmentation trains the network in a weakly supervised way to focus on more fine-grained features. In addition, in order to make the network pay more attention to the invariant features of instance target during tracking, we introduce a semantic matching task to alleviate the model drift problem caused by time change. Based on the results on three RGBT tracking benchmarks, the proposed framework is not inferior to the state-of-the-art trackers.

Compensated Attention Feature Fusion and Hierarchical Multiplication Decoder Network for RGB-D Salient Object Detection

Multi-modal feature fusion and effectively exploiting high-level semantic information are critical in salient object detection (SOD). However, the depth maps complementing RGB image fusion strategies cannot supply effective semantic information when the object is not salient in the depth maps. Furthermore, most existing (UNet-based) methods cannot fully exploit high-level abstract features to guide low-level features in a coarse-to-fine fashion. In this paper, we propose a compensated attention feature fusion and hierarchical multiplication decoder network (CAF-HMNet) for RGB-D SOD. Specifically, we first propose a compensated attention feature fusion module to fuse multi-modal features based on the complementarity between depth and RGB features. Then, we propose a hierarchical multiplication decoder to refine the multi-level features from top down. Additionally, a contour-aware module is applied to enhance object contour. Experimental results show that our model achieves satisfactory performance on five challenging SOD datasets, including NJU2K, NLPR, STERE, DES, and SIP, which verifies the effectiveness of the proposed CAF-HMNet.

Feature fusion and latent feature learning guided brain tumor segmentation and missing modality recovery network

Accurate brain tumor segmentation is an essential step for clinical diagnosis and surgical treatment. Multimodal brain tumor segmentation strongly relies on an effective fusion method and an excellent segmentation network. However, it is common to have some missing MR modalities in clinical scenarios due to image corruption, acquisition protocol, scanner availability and scanning cost, which can heavily decrease the tumor segmentation accuracy, and also cause information loss for down-streaming disease analysis. To address this issue, I propose a novel multimodal feature fusion and latent feature learning guided deep neural network. On the one hand, the proposed network can help to segment brain tumors when one or more modalities are missing. On the other hand, it can retrieve the missing modalities to compensate for incomplete data. The proposed network consists of three key components. First, a Multimodal Feature Fusion Module (MFFM) is proposed to effectively fuse the complementary information from different modalities, consisting of a Cross-Modality Fusion Module (CMFM) and a Multi-Scale Fusion Module (MSFM). Second, a Spatial Consistency-based Latent Feature Learning Module (SC-LFLM) is presented to exploit multimodal latent correlation and extract the relevant features to benefit segmentation. Third, the Multi-Task Learning (MTL) paths are integrated to supervise the segmentation and recover the missing modalities. The proposed method is evaluated on BraTS 2018 dataset, and it can achieve superior segmentation results when one or more modalities are missing, compared with the state-of-the-art methods. Furthermore, the proposed modules can be easily adapted to other multimodal network architectures and research fields.

FusionTC: Encrypted App Traffic Classification Using Decision-Level Multimodal Fusion Learning of Flow Sequence

In recent years, an increasing number of mobile platforms and applications have adopted traffic encryption protocol technology to ensure privacy and security. Existing researches on encrypted traffic identification approaches often rely on a single-modal feature pattern (such as packet sequence and statistical features), which cannot fully represent the detail information of complex traffic features, and so their predictions are susceptible to anomalies. In order to improve the effect of classification on encrypted app traffic, we propose FusionTC, a novel app traffic classification framework based on feature fusion of flow sequence. FusionTC consists of two-level subclassifiers, which are used to perform decision-level fusion of multimodal features by an upgraded stacking method. The comprehensive capture and fusion of multimodal traffic details, coupled with the refined processing and segmentation of traffic, enables FusionTC to significantly promote classification accuracy and enhance robustness in challenging situations. Based on our self-built app traffic dataset, FusionTC improves the accuracy by at least 3.2% over the state-of-the-art approaches.

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions.

Multimodal emotion recognition has gained much traction in the field of affective computing, human-computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.

Automatic Classification Method of Arrhythmias Based on 12-Lead Electrocardiogram.

Cardiovascular disease is one of the main causes of death worldwide. Arrhythmias are an important group of cardiovascular diseases. The standard 12-lead electrocardiogram signals are an important tool for diagnosing arrhythmias. Although 12-lead electrocardiogram signals provide more comprehensive arrhythmia information than single-lead electrocardiogram signals, it is difficult to effectively fuse information between different leads. In addition, most of the current researches working on automatic diagnosis of cardiac arrhythmias are based on modeling and analysis of single-mode features extracted from one-dimensional electrocardiogram sequences, ignoring the frequency domain features of electrocardiogram signals. Therefore, developing an automatic arrhythmia detection algorithm based on 12-lead electrocardiogram with high accuracy and strong generalization ability is still challenging. In this paper, a multimodal feature fusion model based on the mechanism is developed. This model utilizes a dual channel deep neural network to extract different dimensional features from one-dimensional and two-dimensional electrocardiogram time-frequency maps, and combines attention mechanism to effectively fuse the important features of 12-lead, thereby obtaining richer arrhythmia information and ultimately achieving accurate classification of nine types of arrhythmia signals. This study used electrocardiogram signals from a mixed dataset to train, validate, and evaluate the model, with an average of F1 score and average accuracy reached 0.85 and 0.97, respectively. Experimental results show that our algorithm has stable and reliable performance, so it is expected to have good practical application potential.

Cross-Modal Image Registration via Rasterized Parameter Prediction for Object Tracking

Object tracking requires heterogeneous images that are well registered in advance, with cross-modal image registration used to transform images of the same scene generated by different sensors into the same coordinate system. Infrared and visible light sensors are the most widely used in environmental perception; however, misaligned pixel coordinates in cross-modal images remain a challenge in practical applications of the object tracking task. Traditional feature-based approaches can only be applied in single-mode scenarios, and cannot be well extended to cross-modal scenarios. Recent deep learning technology employs neural networks with large parameter scales for prediction of feature points for image registration. However, supervised learning methods require numerous manually aligned images for model training, leading to the scalability and adaptivity problems. The Unsupervised Deep Homography Network (UDHN) applies Mean Absolute Error (MAE) metrics for cost function computation without labelled images; however, it is currently inapplicable for cross-modal image registration. In this paper, we propose aligning infrared and visible images using a rasterized parameter prediction algorithm with similarity measurement evaluation. Specifically, we use Cost Volume (CV) to predict registration parameters from coarse-grained to fine-grained layers with a raster constraint for multimodal feature fusion. In addition, motivated by the utilization of mutual information in contrastive learning, we apply a cross-modal similarity measurement algorithm for semi-supervised image registration. Our proposed method achieves state-of-the-art performance on the MS-COCO and FLIR datasets.

Speech emotion recognition using multimodal feature fusion with machine learning approach

Speech emotion recognition using multimodal feature fusion with machine learning approach

A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents.

Sleep-deprived fatigued person is likely to commit more errors that may even prove to be fatal. Thus, it is necessary to recognize this fatigue. The novelty of the proposed research work for the detection of this fatigue is that it is nonintrusive and based on multimodal feature fusion. In the proposed methodology, fatigue is detected by obtaining features from four domains: visual images, thermal images, keystroke dynamics, and voice features. In the proposed methodology, the samples of a volunteer (subject) are obtained from all four domains for feature extraction, and empirical weights are assigned to the four different domains. Young, healthy volunteers (n = 60) between the age group of 20 to 30 years participated in the experimental study. Further, they abstained from the consumption of alcohol, caffeine, or other drugs impacting their sleep pattern during the study. Through this multimodal technique, appropriate weights are given to the features obtained from the four domains. The results are compared with k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers. The proposed nonintrusive technique has obtained an average detection accuracy of 93.33% in 3-fold cross-validation.

A Cross-Modal Feature Fusion Model Based on ConvNeXt for RGB-D Semantic Segmentation

Semantic segmentation, as the pixel level classification with dividing an image into multiple blocks based on the similarities and differences of categories (i.e., assigning each pixel in the image to a class label), is an important task in computer vision. Combining RGB and Depth information can improve the performance of semantic segmentation. However, there is still a problem of the way to deeply integrate RGB and Depth. In this paper, we propose a cross-modal feature fusion RGB-D semantic segmentation model based on ConvNeXt, which uses ConvNeXt as the skeleton network and embeds a cross-modal feature fusion module (CMFFM). The CMFFM designs feature channel-wise and spectral-wise fusion, which can realize the deeply feature fusion of RGB and Depth. The in-depth multi-modal feature fusion in multiple stages improves the performance of the model. Experiments are performed on the public dataset of SUN-RGBD, showing the best segmentation by our proposed model ConvNeXt-CMFFM with the highest mIoU score of 53.5% among the nine comparative models. The outstanding performance of ConvNeXt-CMFFM is also achieved on our self-built dataset of RICE-RGBD with the highest mIoU score and pixel accuracy among the three comparative datasets. The ablation experiment on our rice dataset shows that compared with ConvNeXt (without CMFFM), the mIoU score of ConvNext-CMFFM is increased from 71.5% to 74.8% and its pixel accuracy is increased from 86.2% to 88.3%, indicating the effectiveness of the added feature fusion module in improving segmentation performance. This study shows the feasibility of the practical application of the proposed model in agriculture.

A progressive phased attention model fused histopathology image features and gene features for lung cancer staging prediction.

Identifying the stage of lung cancer accurately from histopathology images and gene is very important for the diagnosis and treatment of lung cancer. Despite the substantial progress achieved by existing methods, it remains challenging due to large intra-class variances, and a high degree of inter-class similarities. In this paper, we propose a phased Multimodal Multi-scale Attention Model (MMAM) that predicts lung cancer stages using histopathology image data and gene data. The model consists of two phases. In Phase1, we propose a Staining Difference Elimination Network (SDEN) to eliminate staining differences between different histopathology images, In Phase2, it utilizes the image feature extractor provided by Phase1 to extract image features, and sends the multi-scale image features together with gene features into our Adaptive Enhanced Attention Fusion (AEAF) module for multimodal multi-scale features fusion to enable prediction of lung cancer staging. We evaluated the proposed MMAM on the TCGA lung cancer dataset, and achieved 88.51% AUC and 88.17% accuracy on classification prediction of lung cancer stages I, II, III, and IV. The method can help doctors diagnose the stage of lung cancer patients and can benefit from multimodal data.

Intelligent antepartum fetal monitoring via deep learning and fusion of cardiotocographic signals and clinical data.

Cardiotocography (CTG), which measures uterine contraction (UC) and fetal heart rate (FHR), is a crucial tool for assessing fetal health during pregnancy. However, traditional computerized cardiotocography (cCTG) approaches have non-negligible calibration errors in feature extraction and heavily rely on the expertise and prior experience to define diagnostic features from CTG or FHR signals. Although previous works have studied deep learning methods for extracting CTG or FHR features, these methods still neglect the clinical information of pregnant women. In this paper, we proposed a multimodal deep learning architecture (MMDLA) for intelligent antepartum fetal monitoring that is capable of performing automatic CTG feature extraction, fusion with clinical data and classification. The multimodal feature fusion was achieved by concatenating high-level CTG features, which were extracted from preprocessed CTG signals via a convolution neural network (CNN) with six convolution layers and five fully connected layers, and the clinical data of pregnant women. Eventually, light gradient boosting machine (LGBM) was implemented as fetal status assessment classifier. The effectiveness of MMDLA was evaluated using a dataset of 16,355 cases, each of which includes FHR signal, UC signal and pertinent clinical data like maternal age and gestational age. With an accuracy of 90.77% and an area under the curve (AUC) value of 0.9201, the multimodal features performed admirably. The data imbalance issue was also effectively resolved by the LGBM classifier, with a normal-F1 value of 0.9376 and an abnormal-F1 value of 0.8223. In summary, the proposed MMDLA is conducive to the realization of intelligent antepartum fetal monitoring.

Common Mental Disorders in Smart City Settings and Use of Multimodal Medical Sensor Fusion to Detect Them

Cities have undergone numerous permanent transformations at times of severe disruption. The Lisbon earthquake of 1755, for example, sparked the development of seismic construction rules. In 1848, when cholera spread through London, the first health law in the United Kingdom was passed. The Chicago fire of 1871 led to stricter building rules, which led to taller skyscrapers that were less likely to catch fire. Along similar lines, the COVID-19 epidemic may have a lasting effect, having pushed the global shift towards greener, more digital, and more inclusive cities. The pandemic highlighted the significance of smart/remote healthcare. Specifically, the elderly delayed seeking medical help for fear of contracting the infection. As a result, remote medical services were seen as a key way to keep healthcare services running smoothly. When it comes to both human and environmental health, cities play a critical role. By concentrating people and resources in a single location, the urban environment generates both health risks and opportunities to improve health. In this manuscript, we have identified the most common mental disorders and their prevalence rates in cities. We have also identified the factors that contribute to the development of mental health issues in urban spaces. Through careful analysis, we have found that multimodal feature fusion is the best method for measuring and analysing multiple signal types in real time. However, when utilizing multimodal signals, the most important issue is how we might combine them; this is an area of burgeoning research interest. To this end, we have highlighted ways to combine multimodal features for detecting and predicting mental issues such as anxiety, mood state recognition, suicidal tendencies, and substance abuse.

BioscoreNet: Traumatic Brain Injury (TBI) detection using a multimodal self-attention fusion neural network and a passive bioscore monitoring framework from smartphone sensor data

Traumatic Brain Injury (TBI) affects millions of individuals globally and can cause motor, cognitive and emotional deficits. Emerging research in the field of mobile health has demonstrated that smartphone sensor data can be used to monitor various chronic diseases, which can facilitate early ailment detection and continuous monitoring of recovering patients. The goal of the paper’s contributions is two-fold: Proposing (1) A Deep Learning framework that uses smartphone sensor data mobility data (accelerometer, magnetometer, gyroscope, pedometer, pressure, altitude, accessibility) to detect whether a subject has TBI after a head injury. (2) The concept of a TBI Bioscore, a number that quantifies the certainty (0–1) that a subject has TBI. The TBI deep learning detection model uses a self-attention mechanism for multimodal feature fusion and a stacked LSTM for prediction. This model achieves a Balanced Accuracy (BA) of 90.2% and a True Positive Rate (TPR) of 83.3% in correctly identifying TBI instances. The Bioscore is generated using Monte-Carlo Dropout uncertainty estimation. When a users’ TBI Bioscores are visualized throughout, it steadily decreases for users who have normal recoveries. The proposed TBI detection and Bioscore framework facilitates population-level passive, continuous and remote TBI screening and monitoring. A glanceable display of color-coded Bioscores can enable a small team of medical staff to monitor a large pool of patients and intervene preemptively. • This paper proposed a deep learning framework that uses smartphone sensor data (accelerometer, magnetometer, gyroscope, pedometer, pressure, altitude, accessibility) to detect whether a subject has Traumatic Brain Injury (TBI) after a head injury. • This paper also proposed the concept of a TBI Bioscore, a number that quantifies the certainty (0–1) that a subject has TBI, generated using Monte-Carlo Dropout uncertainty estimation that facilitates populationlevel monitoring. • The TBI deep learning detection model uses a self-attention mechanism for multimodal feature fusion and a stacked LSTM for prediction. • The TBI detection model achieves a Balanced Accuracy (BA) of 90.2% and a True Positive Rate (TPR) of 83.3% in correctly identifying TBI instances.

MFCNet: A multi-modal fusion and calibration networks for 3D pancreas tumor segmentation on PET-CT images

In clinical diagnosis, positron emission tomography and computed tomography (PET-CT) images containing complementary information are fused. Tumor segmentation based on multi-modal PET-CT images is an important part of clinical diagnosis and treatment. However, the existing current PET-CT tumor segmentation methods mainly focus on positron emission tomography (PET) and computed tomography (CT) feature fusion, which weakens the specificity of the modality. In addition, the information interaction between different modal images is usually completed by simple addition or concatenation operations, but this has the disadvantage of introducing irrelevant information during the multi-modal semantic feature fusion, so effective features cannot be highlighted. To overcome this problem, this paper propose a novel Multi-modal Fusion and Calibration Networks (MFCNet) for tumor segmentation based on three-dimensional PET-CT images. First, a Multi-modal Fusion Down-sampling Block (MFDB) with a residual structure is developed. The proposed MFDB can fuse complementary features of multi-modal images while retaining the unique features of different modal images. Second, a Multi-modal Mutual Calibration Block (MMCB) based on the inception structure is designed. The MMCB can guide the network to focus on a tumor region by combining different branch decoding features using the attention mechanism and extracting multi-scale pathological features using a convolution kernel of different sizes. The proposed MFCNet is verified on both the public dataset (Head and Neck cancer) and the in-house dataset (pancreas cancer). The experimental results indicate that on the public and in-house datasets, the average Dice values of the proposed multi-modal segmentation network are 74.14% and 76.20%, while the average Hausdorff distances are 6.41 and 6.84, respectively. In addition, the experimental results show that the proposed MFCNet outperforms the state-of-the-art methods on the two datasets.