Multimodal Feature Fusion Research Articles

Physics-Inspired Multimodal Feature Fusion Cascaded Networks for Data-Driven Magnetic Core Loss Modeling

Physics-Inspired Multimodal Feature Fusion Cascaded Networks for Data-Driven Magnetic Core Loss Modeling

Graphic association learning: Multimodal feature extraction and fusion of image and text using artificial intelligence techniques

With the advancement of technology in recent years, the application of artificial intelligence in real life has become more extensive. Graphic recognition is a hot spot in the current research of related technologies. It involves machines extracting key information from pictures and combining it with natural language processing for in-depth understanding. Existing methods still have obvious deficiencies in fine-grained recognition and deep understanding of contextual context. Addressing these issues to achieve high-quality image-text recognition is crucial for various application scenarios, such as accessibility technologies, content creation, and virtual assistants. To tackle this challenge, a novel approach is proposed that combines the Mask R-CNN, DCGAN, and ALBERT models. Specifically, the Mask R-CNN specializes in high-precision image recognition and segmentation, the DCGAN captures and generates nuanced features from images, and the ALBERT model is responsible for deep natural language processing and semantic understanding of this visual information. Experimental results clearly validate the superiority of this method. Compared to traditional image-text recognition techniques, the recognition accuracy is improved from 85.3% to 92.5%, and performance in contextual and situational understanding is enhanced. The advancement of this technology has far-reaching implications for research in machine vision and natural language processing and open new possibilities for practical applications.

Multi-scale hybrid attention aggregation networks for multi-modal monitoring in laser-induced thermal-crack processing

In recent years, the rapid development of deep learning has led to an extensive amount of research into machining condition monitoring technologies. However, the complexity of multi-modal signals present in actual machining scenarios poses a significant challenge to signal fusion, and most models find it difficult to balance the relationship between performance and deployment cost. In this research, a multi-scale hybrid attention aggregation network (MHAAN) is proposed for multi-modal monitoring in laser-induced thermal-crack processing. Based on the feature distribution of visual signals and acoustic emission (AE) signals, the multi spatial scale aggregation module (MSSAM) and Multi frequency domain aggregation module (MFDAM) are designed. A cross-modal dot-product interaction module is employed to facilitate multilevel interactive fusion of the extracted feature vectors. This module enables feature transfer across different modalities, and then extracts the multi-model fusion feature. The practicality of MHAAN has been validated using a dataset obtained from the process of glass cutting with laser-induced thermal-crack propagation (LITP). Compared with commonly used models, the superiority of MHAAN in multi-modal feature extraction and fusion is demonstrated. Finally, the effectiveness of the module design based on prior knowledge is proven through ablation experiments, improving the interpretability of the model.

Dynamic graph topology generating mechanism: Framework for feature-level multimodal information fusion applied to lower-limb activity recognition

Lower-limb activity recognition (LLAR), which is driven by wearable sensors, has recently become a popular research topic in human assistive devices. Although multimodal information fusion is crucial for improving the accuracy of LLAR, traditional feature-level fusion strategies cannot directly model the relationship between multimodal features according to the lower-limb activity patterns and they realize fusion based on the relationship. To address these limitations, a dynamic graph topology generation (DGTG) mechanism that uses a graph topology to model the relationship between multimodal features mined by a convolutional neural network (CNN) is proposed in this study. A weighted graph isomorphism network (WGIN) that can realize multimodal feature fusion based on the relationship (i.e., graph topology) is also proposed, resulting in a novel feature-level multimodal information fusion framework called CNN-DGTG-WGIN. This proposed method achieved the best average accuracies of 96.3 and 99.7% for two publicly available datasets. This method integrates the determination of the graph topology into an end-to-end learning process without requiring any prior knowledge or original graph structures. Further, CNN-DGTG-WGIN effectively incorporates both linear and nonlinear correlations between features during the DGTG process. Analysis results revealed that the graph topology (multimodal feature relationship) learned by CNN-DGTG-WGIN was closely related to lower-limb activity patterns, encoding relevant information about these patterns, which is the essence of its superior performance.

Occlusion-guided multi-modal fusion for vehicle-infrastructure cooperative 3D object detection

In autonomous driving, leveraging sensor data (e.g. camera, LiDAR data) from both the vehicle and the infrastructure significantly improves perception capabilities. However, this integration traditionally results in increased demands on communication bandwidth. To address these challenges, we introduce Fusion2comm, an occlusion-guided feature fusion approach designed to optimize vehicle-infrastructure cooperative 3D object detection. Our innovative strategy employs an intelligent fusion of camera and LiDAR data to enhance the expressiveness of features. Subsequently, it leverages a segmentation model to extract foreground features and utilizes an occlusion-based selection of communication content, effectively easing bandwidth constraints. We propose a multimodal foreground feature fusion architecture that selectively processes and transmits critical information, substantially reducing irrelevant background data transfer. An innovative occlusion confidence-aware communication technique dynamically adjusts communication regions based on occlusion levels, ensuring efficient data exchange. Fusion2comm sets a new benchmark in the DAIR-V2X dataset, achieving an average precision of 71.25% with minimal bandwidth usage of 221.04 bytes. Our comprehensive experimental evaluations confirm that Fusion2comm substantially advances detection precision while simultaneously improving communication efficiency.

MMDG-DTI: Drug–target interaction prediction via multimodal feature fusion and domain generalization

Recently, deep learning has become the essential methodology for Drug–Target Interaction (DTI) prediction. However, the existing learning-based representation methods embed the prior knowledge encapsulated by the training data and, as such, acquire redundant domain information irrelevant to the DTI prediction, compromising the generalization capability of a deep network to unfamiliar instances. To tackle this limitation, we propose a novel DTI prediction method using Multi-Modal feature fusion and Domain Generalization (MMDG-DTI). Specifically, we first use pre-trained Large Language Models (LLMs) to obtain generalized textual features covering nearly the whole biological text vocabulary, with the capacity to handle unseen samples and extract robust and discriminative features. In parallel, we propose a unified structural feature extractor based on a hybrid Graph Neural Network (GNN). The extractor improves the performance of MMDG-DTI by extracting features from another modality and discarding the representation of domain-specific prior substructures. Then, we integrate the complementary information of LLM and GNN features using an innovative classifier, while enhancing the generalization ability with the help of Domain Adversarial Training (DAT) and contrastive learning. Last, we propose a novel cross-domain evaluation protocol to enhance the predication fidelity in practical applications. The quantitative and visualization results obtained on several benchmarking datasets demonstrate that MMDG-DTI achieves state-of-the-art performance under both single-domain and cross-domain settings, confirming the superiority of the proposed method. The datasets and codes will be available on the project homepage: https://github.com/JU-HuaY/MMDG-DTI.

Multi-functional scar tissue discrimination platform construction and exploration of molecular mechanism for scar formation

Scars that form after skin injury can cause structural and functional skin damage. Currently, scar tissue determination relies mainly on doctors’ subjective observations and judgments and lacks objectivity. However, current deep learning models can only achieve specific discrimination using unimodal data, which limits the comprehensive understanding of scar tissue and may reduce accuracy and stability. To solve these problems, in this study, a skin scar recognition platform based on advanced deep learning and a weighted aggregation network fusion method is proposed. It is implemented using a residual network-based CNN model and a logistic regression model with L1 regularization and is suitable for both unimodal and multimodal data. The experimental results showed that the proposed platform achieved a satisfactory accuracy of 98.26% for image discrimination. In the gene discrimination model test performed on a test dataset containing 17 gene expression samples, all samples were accurately discriminated. In addition, the proposed multimodal discrimination model achieved a discrimination accuracy of 98.23%. These results validate the effectiveness of deep feature extraction and multimodal feature fusion techniques for image discrimination tasks. On this basis, to deeply explore the pathogenesis of scar formation, a method with the ability to integrate regularization, sparsity, and orthogonality constraints, multiconstraint joint non-negative matrix factorization (MCJNMF), was used to explore the genetic correlation between collagen micrographic image features and gene expression data. In this study, we confirmed the association between the calcium signaling pathway, MAPK signaling pathway, and collagen fiber repair, and successfully identified 11 potential therapeutic targets, including TRIM59 and TBC1D9, which provide important clues for future scar treatment and prevention strategies.

A 3D hierarchical cross-modality interaction network using transformers and convolutions for brain glioma segmentation in MR images.

Precise glioma segmentation from multi-parametric magnetic resonance (MR) images is essential for brain glioma diagnosis. However, due to the indistinct boundaries between tumor sub-regions and the heterogeneous appearances of gliomas in volumetric MR scans, designing a reliable and automated glioma segmentation method is still challenging. Although existing 3D Transformer-based or convolution-based segmentation networks have obtained promising results via multi-modal feature fusion strategies or contextual learning methods, they widely lack the capability of hierarchical interactions between different modalities and cannot effectively learn comprehensive feature representations related to all glioma sub-regions. To overcome these problems, in this paper, we propose a 3D hierarchical cross-modality interaction network (HCMINet) using Transformers and convolutions for accurate multi-modal glioma segmentation, which leverages an effective hierarchical cross-modality interaction strategy to sufficiently learn modality-specific and modality-shared knowledge correlated to glioma sub-region segmentation from multi-parametric MR images. In the HCMINet, we first design a hierarchical cross-modality interaction Transformer (HCMITrans) encoder to hierarchically encode and fuse heterogeneous multi-modal features by Transformer-based intra-modal embeddings and inter-modal interactions in multiple encoding stages, which effectively captures complex cross-modality correlations while modeling global contexts. Then, we collaborate an HCMITrans encoder with a modality-shared convolutional encoder to construct the dual-encoder architecture in the encoding stage, which can learn the abundant contextual information from global and local perspectives. Finally, in the decoding stage, we present a progressive hybrid context fusion (PHCF) decoder to progressively fuse local and global features extracted by the dual-encoder architecture, which utilizes the local-global context fusion (LGCF) module to efficiently alleviate the contextual discrepancy among the decoding features. Extensive experiments are conducted on two public and competitive glioma benchmark datasets, including the BraTS2020 dataset with 494 patients and the BraTS2021 dataset with 1251 patients. Results show that our proposed method outperforms existing Transformer-based and CNN-based methods using other multi-modal fusion strategies in our experiments. Specifically, the proposed HCMINet achieves state-of-the-art mean DSC values of 85.33% and 91.09% on the BraTS2020 online validation dataset and the BraTS2021 local testing dataset, respectively. Our proposed method can accurately and automatically segment glioma regions from multi-parametric MR images, which is beneficial for the quantitative analysis of brain gliomas and helpful for reducing the annotation burden of neuroradiologists.

RTMFusion: An enhanced dual-stream architecture algorithm fusing RGB and depth features for instance segmentation of tomato organs

Accurate and rapid acquisition of semantically annotated point clouds from tomato cultivation regions is crucial for autonomous tomato harvesting robots. This entails the robot’s ability to perceive tomato point clouds from various locations and identify the stems and pedicels connecting the fruits. However, amidst background interference from similar stem and pedicel features, the heterogeneous growth postures of tomato clusters and the slender and short nature of pedicels pose challenges in semantic point cloud segmentation and matching of different tomato fruits, pedicels, and stems. Leveraging the salient feature representation of depth images in fine contours and distance dimensions, this paper introduces RTMFusion, a lightweight dual-stream backbone instance segmentation model that integrates color images and depth images. A Depth Enhanced Feature Fusion module is designed for multi-modal feature fusion. Fruit-bearing organ matching is facilitated by a method based on distance constraints and conditional search. Key characteristics are extracted from semantic point clouds, and an optimal condition search is conducted on candidates meeting constraint conditions. On the test dataset, our model improved the mask mAP by 4.6 % compared to the baseline, achieving 72.1 % with a speed of 17.1 ms. Our matching method achieved 99.2 % pedicel matching accuracy and 97.3 % stem matching accuracy on 150 greenhouse samples. These results demonstrate the effectiveness of accurately identifying and matching tomato organs in real-world scenarios, addressing the challenge from complex greenhouse environments. This integrated visual system enhances the success rate of visual identification and the precision of cutting position localization for tomato harvesting robots.

Multi-modality hierarchical attention networks for defect identification in pipeline MFL detection

Abstract Magnetic flux leakage (MFL) testing is widely used for acquiring MFL signals to detect pipeline defects, and data-driven approaches have been effectively investigated for MFL defect identification. However, with the increasing complexity of pipeline defects, current methods are constrained by the incomplete information from single modal data, which fail to meet detection requirements. Moreover, the incorporation of multimodal MFL data results in feature redundancy. Therefore, the multi-modality hierarchical attention networks (MMHAN) are proposed for defect identification. Firstly, stacked residual blocks with cross-level attention module (CLAM) and multiscale 1D-CNNs with multiscale attention module are utilized to extract multiscale defect features. Secondly, the multi-modality feature enhancement attention module (MMFEAM) is developed to enhance critical defect features by leveraging correlations among multimodal features. Lastly, the multi-modality feature fusion attention module (MMFFAM) is designed to dynamically integrate multimodal features deeply, utilizing the consistency and complementarity of multimodal information. Extensive experiments were conducted on multimodal pipeline datasets to assess the proposed MMHAN. The experimental results demonstrate that MMHAN achieves a higher identification accuracy, validating its exceptional performance.

FuseNet: a multi-modal feature fusion network for 3D shape classification

FuseNet: a multi-modal feature fusion network for 3D shape classification

Efficient detection of driver fatigue state based on all-weather illumination scenarios

Among the causes of the annually traffic accidents, driving fatigue is the main culprit. In consequence, it is of great practical significance to carry out the research of driving fatigue detection and early warning system. However, there are still two problems in the latest methods of driving fatigue detection: one is that a single information cannot precisely reflect the actual state of the driver in different fatigue phases, another one is the detection effect is not very well or even difficult to detect under abnormal illumination. In this paper, the multi-task cascaded convolutional networks (MTCNN) and infrared-based remote photo-plethysmography (rPPG) theory are used to extract the driver’s facial and physiological information, and the multi-modal specific fatigue information is deeply excavated, and the multi-modal feature fusion model is constructed to comprehensively analyze the driver’s fatigue variation tendency. Aiming at the matter of low detection accuracy under abnormal illumination, the multi-modal features extracted from visible light images and infrared images are fused by multi-loss reconstruction (MLR) module, and the driving fatigue detection module is established which is based on Bi-LSTM model by utilizing fatigue timing. The experiments were validated under all-weather illumination scenarios and were carried out on the datasets NTHU-DDD, UTA-RLDDD and FAHD. The results show that the multi-modal driving fatigue detection model has better performance than the single-modal model, and the accuracy is improved by 8.1%. In the abnormal illumination such as strong and weak light, the accuracy of the method can reach 91.7% at the highest and 83.6% at the lowest. Meanwhile, in the normal illumination, it can reach 93.2%.

Image-Acceleration Multimodal Danger Detection Model on Mobile Phone for Phone Addicts.

With the popularity of smartphones, a large number of "phubbers" have emerged who are engrossed in their phones regardless of the situation. In response to the potential dangers that phubbers face while traveling, this paper proposes a multimodal danger perception network model and early warning system for phubbers, designed for mobile devices. This proposed model consists of surrounding environment feature extraction, user behavior feature extraction, and multimodal feature fusion and recognition modules. The environmental feature module utilizes MobileNet as the backbone network to extract environmental description features from the rear-view image of the mobile phone. The behavior feature module uses acceleration time series as observation data, maps the acceleration observation data to a two-dimensional image space through GADFs (Gramian Angular Difference Fields), and extracts behavior description features through MobileNet, while utilizing statistical feature vectors to enhance the representation capability of behavioral features. Finally, in the recognition module, the environmental and behavioral characteristics are fused to output the type of hazardous state. Experiments indicate that the accuracy of the proposed model surpasses existing methods, and it possesses the advantages of compact model size (28.36 Mb) and fast execution speed (0.08 s), making it more suitable for deployment on mobile devices. Moreover, the developed image-acceleration multimodal phubber hazard recognition network combines the behavior of mobile phone users with surrounding environmental information, effectively identifying potential hazards for phubbers.

Multi-modal fusion and feature enhancement U-Net coupling with stem cell niches proximity estimation for voxel-wise GBM recurrence prediction.

Objective.We aim to develop a Multi-modal Fusion and Feature Enhancement U-Net (MFFE U-Net) coupling with stem cell niche proximity estimation to improve voxel-wise Glioblastoma (GBM) recurrence prediction.Approach.57 patients with pre- and post-surgery magnetic resonance (MR) scans were retrospectively solicited from 4 databases. Post-surgery MR scans included two months before the clinical diagnosis of recurrence and the day of the radiologicaly confirmed recurrence. The recurrences were manually annotated on the T1ce. The high-risk recurrence region was first determined. Then, a sparse multi-modal feature fusion U-Net was developed. The 50 patients from 3 databases were divided into 70% training, 10% validation, and 20% testing. 7 patients from the 4th institution were used as external testing with transfer learning. Model performance was evaluated by recall, precision, F1-score, and Hausdorff Distance at the 95% percentile (HD95). The proposed MFFE U-Net was compared to the support vector machine (SVM) model and two state-of-the-art neural networks. An ablation study was performed.Main results.The MFFE U-Net achieved a precision of 0.79 ± 0.08, a recall of 0.85 ± 0.11, and an F1-score of 0.82 ± 0.09. Statistically significant improvement was observed when comparing MFFE U-Net with proximity estimation couple SVM (SVMPE), mU-Net, and Deeplabv3. The HD95 was 2.75 ± 0.44 mm and 3.91 ± 0.83 mm for the 10 patients used in the model construction and 7 patients used for external testing, respectively. The ablation test showed that all five MR sequences contributed to the performance of the final model, with T1ce contributing the most. Convergence analysis, time efficiency analysis, and visualization of the intermediate results further discovered the characteristics of the proposed method.Significance. We present an advanced MFFE learning framework, MFFE U-Net, for effective voxel-wise GBM recurrence prediction. MFFE U-Net performs significantly better than the state-of-the-art networks and can potentially guide early RT intervention of the disease recurrence.

MFFnet: Multimodal Feature Fusion Network for Synthetic Aperture Radar and Optical Image Land Cover Classification

MFFnet: Multimodal Feature Fusion Network for Synthetic Aperture Radar and Optical Image Land Cover Classification

An Overview of Behavioral Recognition

Human behavior recognition has become a popular research topic in the field of computer vision. With the introduction of deep learning and attention mechanisms, this field has been further promoted. However, issues such as dataset acquisition and preprocessing operations on multimodal datasets, modeling of long time information in videos, and fusion of temporal and spatial information still exist. In this paper, we first outline some video action recognition datasets and related preprocessing techniques, including frame extraction, optical flow extraction, and skeletal feature acquisition. Then, the relevant models are classified and parsed according to their characteristics and the types of input data modalities. In addition, we evaluate the performance of the models on several benchmark datasets to gain a deeper understanding of the model development process. Finally, we summarized the current challenges faced in the field of video behavior recognition, including model timeliness, data set subjectivity and effective fusion of multi-modal features, and proposed possible future improvement directions in order to provide more ideas and methods for subsequent research.

Multi-grained contrastive representation learning for label-efficient lesion segmentation and onset time classification of acute ischemic stroke

Ischemic lesion segmentation and the time since stroke (TSS) onset classification from paired multi-modal MRI imaging of unwitnessed acute ischemic stroke (AIS) patients is crucial, which supports tissue plasminogen activator (tPA) thrombolysis decision-making. Deep learning methods demonstrate superiority in TSS classification. However, they often overfit task-irrelevant features due to insufficient paired labeled data, resulting in poor generalization. We observed that unpaired data are readily available and inherently carry task-relevant cues, but are less often considered and explored. Based on this, in this paper, we propose to fully excavate the potential of unpaired unlabeled data and use them to facilitate the downstream AIS analysis task. We first analyze the utility of features at the varied grain and propose a multi-grained contrastive learning (MGCL) framework to learn task-related prior representations from both coarse-grained and fine-grained levels. The former can learn global prior representations to enhance the location ability for the ischemic lesions and perceive the healthy surroundings, while the latter can learn local prior representations to enhance the perception ability for semantic relation between the ischemic lesion and other health regions. To better transfer and utilize the learned task-related representation, we designed a novel multi-task framework to simultaneously achieve ischemic lesion segmentation and TSS classification with limited labeled data. In addition, a multi-modal region-related feature fusion module is proposed to enable the feature correlation and synergy between multi-modal deep image features for more accurate TSS decision-making. Extensive experiments on the large-scale multi-center MRI dataset demonstrate the superiority of the proposed framework. Therefore, it is promising that it helps better stroke evaluation and treatment decision-making.

CMR-net: A cross modality reconstruction network for multi-modality remote sensing classification.

In recent years, the classification and identification of surface materials on earth have emerged as fundamental yet challenging research topics in the fields of geoscience and remote sensing (RS). The classification of multi-modality RS data still poses certain challenges, despite the notable advancements achieved by deep learning technology in RS image classification. In this work, a deep learning architecture based on convolutional neural network (CNN) is proposed for the classification of multimodal RS image data. The network structure introduces a cross modality reconstruction (CMR) module in the multi-modality feature fusion stage, called CMR-Net. In other words, CMR-Net is based on CNN network structure. In the feature fusion stage, a plug-and-play module for cross-modal fusion reconstruction is designed to compactly integrate features extracted from multiple modalities of remote sensing data, enabling effective information exchange and feature integration. In addition, to validate the proposed scheme, extensive experiments were conducted on two multi-modality RS datasets, namely the Houston2013 dataset consisting of hyperspectral (HS) and light detection and ranging (LiDAR) data, as well as the Berlin dataset comprising HS and synthetic aperture radar (SAR) data. The results demonstrate the effectiveness and superiority of our proposed CMR-Net compared to several state-of-the-art methods for multi-modality RS data classification.

Exploring target-related information with reliable global pixel relationships for robust RGB-T tracking

RGB-T Siamese trackers have drawn continuous interest in recent years due to their proper trade-off between accuracy and speed. However, they are sensitive to the background distractors in some challenging cases, thereby inducing unreliable response positions. To overcome such drawbacks, we advance a new RGB-T Siamese tracker, named SiamTIH, which will advance the RGB-T Siamese trackers’ discriminability against distractors by exploiting target-related information and reliable global pixel relationships within multi-modal data. Specifically, we propose a target-related feature enhancement module (TFE) to highlight such areas in the detection branch that are similar to the templates and suppress those background distractor regions that are significantly different from the templates but are greatly informative. Then, we propose an intra- and mutual-modal attention based multi-modal feature fusion module (IMA-MF) to capture the reliable global pixel relationships within multi-modal data. Especially, the intra-modal attention is used to capture the global pixel relationships within each single modality data, and the mutual-modal attention is utilized to enhance the feature representation of the current modality by overall pixel relationships as well as modality-specific relationships. Finally, we propose a hard-focused online classifier (HFOC) that combines an offline classifier and an online classifier to further improve the robustness of our tracker. Besides, the proposed framework is further extended to a Transformer based tracker to verify its generality. Extensive experiments on three RGB-T benchmarks demonstrate that our new RGB-T tracker outperforms the existing ones and maintains real-time performance, exceeding on average 30 frames per second (FPS). The code will be available at https://github.com/Tianlu-Zhang/SiamTIH.

E-Learning system application in art entrepreneurship teaching based on multimodal feature fusion and neural network

College student entrepreneurship teams can leverage the advantages of mobile internet resource integration and online collaborative production platforms to easily integrate various resources, thereby achieving rapid start-up and development of entrepreneurial projects. This also makes the threshold for college students to start businesses lower and easier to achieve. This article designs an interactive aesthetic education and entrepreneurship teaching system based on multimodal feature fusion and interactive systems. The core concept of the interactive aesthetic education and entrepreneurship teaching system is multimodal feature fusion and interactive system. Multimodal feature fusion can integrate different teaching resources, so that students can better understand the factors that need to be considered in the entrepreneurial process and practice. Interactive systems can enable closer interaction and communication between students and teachers, thereby better understanding and mastering entrepreneurial knowledge and skills. The design and implementation of a systematic system will help enhance the entrepreneurial ability and quality of college students, help them better understand the entrepreneurial process and factors that need to be considered in practice, and provide entrepreneurial practice reference for students in Chinese universities who have the intention of designing entrepreneurship.