Multimodal Fusion Research Articles

Multimodal Co-Attention Fusion Network With Online Data Augmentation for Cancer Subtype Classification.

It is an essential task to accurately diagnose cancer subtypes in computational pathology for personalized cancer treatment. Recent studies have indicated that the combination of multimodal data, such as whole slide images (WSIs) and multi-omics data, could achieve more accurate diagnosis. However, robust cancer diagnosis remains challenging due to the heterogeneity among multimodal data, as well as the performance degradation caused by insufficient multimodal patient data. In this work, we propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification. Specifically, a multimodal mutual-guided co-attention (MMC) module is proposed to effectively perform dense multimodal interactions. It enables multimodal data to mutually guide and calibrate each other during the integration process to alleviate inter- and intra-modal heterogeneities. Subsequently, a self-normalizing network (SNN)-Mixer is developed to allow information communication among different omics data and alleviate the high-dimensional small-sample size problem in multi-omics data. Most importantly, to compensate for insufficient multimodal samples for model training, we propose an ODA module in MCFN. The ODA module leverages the multimodal knowledge to guide the data augmentations of WSIs and maximize the data diversity during model training. Extensive experiments are conducted on the public TCGA dataset. The experimental results demonstrate that the proposed MCFN outperforms all the compared algorithms, suggesting its effectiveness.

CLIPMulti: Explore the performance of multimodal enhanced CLIP for zero-shot text classification

Zero-shot text classification does not require large amounts of labeled data and is designed to handle text classification tasks that lack annotated training data. Existing zero-shot text classification uses either a text-text matching paradigm or a text-image matching paradigm, which shows good performance on different benchmark datasets. However, the existing classification paradigms only consider a single modality for text matching, and little attention is paid to the help of multimodality for text classification. In order to incorporate multimodality into zero-shot text classification, we propose a multimodal enhanced CLIP framework (CLIPMulti), which employs a text-image&text matching paradigm to enhance the effectiveness of zero-shot text classification. Three different image and text combinations are tested for their effects on zero-shot text classification, and a matching method (Match-CLIPMulti) is further proposed to find the corresponding text based on the classified images automatically. We conducted experiments on seven publicly available zero-shot text classification datasets and achieved competitive performance. In addition, we analyzed the effect of different parameters on the Match-CLIPMulti experiments. We hope this work will bring more thoughts and explorations on multimodal fusion in language tasks.

A novel multimodal image feature fusion mechanism:Application to rabbit liveweight estimation in commercial farms

In the realm of commercial rabbit farming, assessment of liveweight for each rabbit is crucial for production management. Traditional manual weighing methods are labor-intensive, time-consuming, and can induce stress and disease transmission. To solve this problem, this paper introduced a liveweight estimation method for rabbit utilizing near-infrared and depth image, including a segmentation stage and a weight estimation stage. Specifically, a dual-stream feature fusion mechanism was proposed to effectively integrate information from both near-infrared and depth imaging. The network was trained and validated on a dataset of 1,957 overhead images from 300 rabbits collected from a commercial farm, achieving superior performance with an R-Square (R2) of 0.95, Root Mean Square Error (RMSE) of 194.95 g, Mean Absolute Deviation (MAD) of 155.10 g, Mean Absolute Percentage Error (MAPE) of 4.08%, and Mean Coefficient of Variation (MCV) of 3.03%. Ablation experiments revealed that employing the proposed dual-stream feature fusion mechanism led to reductions of 52.87g, 38.19g, 1.01%, and 0.75% in RMSE, MAD, MAPE, and MCV, respectively, compared to not employing the mechanism. Compared to machine learning method, our proposed approach demonstrates superior performance in RMSE, MAD, MAPE, and MCV, achieving more accurate liveweight estimation for large-scale commercial farms.

Integrated Sensing and Communications Toward Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

Integrated Sensing and Communications Toward Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma

Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

Enhanced Preoperative Planning for Intracranial Aneurysms Through Multimodal Image Fusion of Silent/Time-of-Magnetic Resonance Angiography and Computed Tomography Using 3DSlicer: A Comparative Efficacy Analysis With Computed Tomography Angiography.

This study evaluates the effectiveness of multimodal image fusion (MIF) using silent and time-of-flight (TOF) magnetic resonance angiography (MRA) and computed tomography (CT) for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast media. A retrospective study included 40 patients with intracranial aneurysms, diagnosed using three-dimensional computed tomography angiography (CTA). These patients underwent both Silent and TOF MRA scans, followed by a CTA scan. The multi-image fusion (MIF) technique, applied using 3DSlicer software, integrated the silent/TOF-MRA with CT images for preoperative assessment. This study compared the image quality, aneurysm detection sensitivity, and anatomic accuracy of the MIF images with those of three-dimensional CTA. Silent-MRA-CT fusion images demonstrated higher sensitivity (95.5%) and lower false negative rates (4.5%) compared with TOF-MRA-CT. Furthermore, silent-MRA-CT fusion images outperformed TOF-MRA-CT in terms of signal homogeneity, venous signal interference suppression, and aneurysm visibility (all P < 0.05). The interclass correlation coefficient and kappa values for aneurysm morphology and shape indicated superior measurement consistency and shape concordance of silent-MRA-CT with CTA compared with TOF-MRA-CT (all P < 0.01). This study supports the use of silent/TOF-MRA-CT fusion imaging as a reliable alternative to CTA, noting that silent-MRA-CT closely mirrors CTA. Contrast-free MRA-CT fusion images have the potential to be used for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast.

An intelligence enhancement method for USV navigation visual measurement based on variable gradient soft-threshold correction

Maritime image enhancement is crucial in marine engineering for applications like unmanned surface vessels (USV) visual measurement and path planning. This paper proposes an image enhancement algorithm based on variable gradient soft-threshold correction (VGSC), tailored for USV intelligent navigation. By calculating the gradient of the two-dimensional difference quotient function of the pixels, we determine the pixel distribution and threshold. An innovative dynamic equalization function is applied to compress maximum pixel values and stretch minimum ones, allowing for adaptive adjustment of the threshold. The results indicate that the VGSC algorithm effectively suppresses background noise, thereby facilitating the extraction of target contour pixel information. When evaluated using four image assessment indicators, the VGSC algorithm enhances the image quality by 20.09 % and 28.92 % compared to bilateral filtering and Gamma correction, respectively. This method excels in enhancing image details and suppressing noise, promising robust application in multi-modal data fusion and precise maritime target detection.

AFDFusion: An adaptive frequency decoupling fusion network for multi-modality image

The multi-modality image fusion goal is to create a single image that provides a comprehensive scene description and conforms to visual perception by integrating complementary information about the merits of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images. Although some works explore decoupled representations of multi-modality images, they struggle with complex nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an adaptive frequency decoupling module to perceive the associative invariant and inherent specific among cross-modality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize a contrastive learning loss for restricting the solution space of feature decoupling to learn representations of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling, low frequency features, which are similar in the representation space, should be pulled closer to each other, signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II, two feature fusion modules are added to integrate the invariant and specific features and output the fused image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.

Multimodal fish maw type recognition based on Wasserstein generative adversarial network combined with gradient penalty and spectral fusion

There are many types of fish maw with significantly varying prices. The specific type directly affects its market value and medicinal efficacy. This paper proposes a fish maw type recognition method based on Wasserstein generative adversarial network combined with gradient penalty (WGAN-GP) and spectral fusion. By collecting Raman and near-infrared (NIR) spectral data of four types of fish maw (Beihai Male Fish Maw, Beihai Female Fish Maw, Yellow Croaker Fish Maw, and Red Mouth Croaker Fish Maw), we used WGAN-GP for data augmentation. The performance of three spectral fusion strategies (data layer, feature layer, and decision layer) was explored based on two one-dimensional convolutional neural network (1D-CNN) models. The results indicate that, after applying data augmentation and expanding the training set to 3,600 samples, the performances of the 1D-VGG (NIR), 1D-VGG (Raman), 1D-ResNet (NIR), and 1D-ResNet (Raman) models all reach optimal levels. The accuracies on the test set are improved by 15.48%, 13.10%, 1.19%, and 5.95%, respectively. Under different fusion strategies, the 1D-VGG (Raman)-1D-VGG (NIR) model at the feature layer and 1D-ResNet (Raman)(1.0)-1D-ResNet (NIR)(1.0) model at the decision layer achieved the same classification results. They exceeded other models in accuracy (98.21%), precision (98.27%), recall (98.21%), and F1-score (98.21%) on the test set. In summary, this study demonstrated the great potential of data enhancement and multimodal spectral data fusion in fish maw type identification, providing analytical tools for the development of fish maw detection equipment based on multimodal techniques.

An objective quantitative diagnosis of depression using a local-to-global multimodal fusion graph neural network

An objective quantitative diagnosis of depression using a local-to-global multimodal fusion graph neural network

Design and research of a monitoring and environmental testing analysis system for substation status

Abstract By testing and analyzing the real substation environment, and analyzing and fitting on-site data, a system that simulates the operating environment of the real substation is designed, and the goal of replicating a typical on-site electricity environment is achieved. Testing and analysis are conducted on the selected substation area, and light and heavy load noise is collected for spectrum analysis and measurement point attenuation analysis. The LMF (Late Multimodal Fusion) algorithm is used and test data are fitted based on the trust region theory. Conducting noise testing and attenuation analysis in the substation area has laid the foundation for research on substation status monitoring and environmental monitoring.

Adaptive Transformer-Based Multi-Modal Image Fusion for Real-Time Medical Diagnosis and Object Detection

In recent years, medical diagnosis and object detection have been significantly enhanced by the integration of multi-modal image fusion techniques. This study proposes an Adaptive Transformer-Based Multi-Modal Image Fusion (AT-MMIF) framework designed for real-time medical diagnosis and object detection. The framework employs a Transformer architecture to capture both global and local feature correlations across multiple imaging modalities, including MRI, CT, PET, and X-ray, for more accurate diagnostic results and faster object detection in medical imagery. The fusion process incorporates spatial and frequency-domain information to improve the clarity and detail of the output images, enhancing diagnostic accuracy. The adaptive attention mechanism within the Transformer dynamically adjusts to the relevant features of different image types, optimizing fusion in real time. This leads to an improved sensitivity (98.5%) and specificity (96.7%) in medical diagnosis. Additionally, the model significantly reduces false positives and negatives, with an F1 score of 97.2% in object detection tasks. The AT-MMIF framework is further optimized for real-time processing with an average inference time of 120 ms per image and a model size reduction of 35% compared to existing multi-modal fusion models. By leveraging the strengths of Transformer architectures and adaptive learning, the proposed framework offers a highly efficient and scalable solution for real-time medical diagnosis and object detection in various clinical settings, including radiology, oncology, and pathology.

Parallel-way: Multi-modality-based brain tumor segmentation using parallel capsule network

ABSTRACT Brain tumors present a formidable diagnostic challenge due to their aberrant cell growth. Accurate determination of tumor location and size is paramount for effective diagnosis. Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) are pivotal tools in clinical diagnosis, yet tumor segmentation within their images remains challenging, particularly at boundary pixels, owing to limited sensitivity. Recent endeavors have introduced fusion-based strategies to refine segmentation accuracy, yet these methods often prove inadequate. In response, we introduce the Parallel-Way framework to surmount these obstacles. Our approach integrates MRI and PET data for a holistic analysis. Initially, we enhance image quality by employing noise reduction, bias field correction, and adaptive thresholding, leveraging Improved Kalman Filter (IKF), Expectation Maximization (EM), and Improved Vibe Algorithm (IVib), respectively. Subsequently, we conduct multi-modality image fusion through the Dual-Tree Complex Wavelet Transform (DTWCT) to amalgamate data from both modalities. Following fusion, we extract pertinent features using the Advanced Capsule Network (ACN) and reduce feature dimensionality via Multi-objective Diverse Evolution-based selection. Tumor segmentation is then executed utilizing the Twin Vision Transformer with dual attention mechanism. Implemented our Parallel-Way framework which exhibits heightened model performance. Evaluation across multiple metrics, including accuracy, sensitivity, specificity, F1-Score, and AUC, underscores its superiority over existing methodologies.

A cross-attention-based deep learning approach for predicting functional stroke outcomes using 4D CTP imaging and clinical metadata

Acute ischemic stroke (AIS) remains a global health challenge, leading to long-term functional disabilities without timely intervention. Spatio-temporal (4D) Computed Tomography Perfusion (CTP) imaging is crucial for diagnosing and treating AIS due to its ability to rapidly assess the ischemic core and penumbra. Although traditionally used to assess acute tissue status in clinical settings, 4D CTP has also been explored in research for predicting stroke tissue outcomes. However, its potential for predicting functional outcomes, especially in combination with clinical metadata, remains unexplored. Thus, this work aims to develop and evaluate a novel multimodal deep learning model for predicting functional outcomes (specifically, 90-day modified Rankin Scale) in AIS patients by combining 4D CTP and clinical metadata. To achieve this, an intermediate fusion strategy with a cross-attention mechanism is introduced to enable a selective focus on the most relevant features and patterns from both modalities. Evaluated on a dataset comprising 70 AIS patients who underwent endovascular mechanical thrombectomy, the proposed model achieves an accuracy (ACC) of 0.77, outperforming conventional late fusion strategies (ACC = 0.73) and unimodal models based on either 4D CTP (ACC = 0.61) or clinical metadata (ACC = 0.71). The results demonstrate the superior capability of the proposed model to leverage complex inter-modal relationships, emphasizing the value of advanced multimodal fusion techniques for predicting functional stroke outcomes.

Drug–drug interaction extraction based on multimodal feature fusion by Transformer and BiGRU

Understanding drug–drug interactions (DDIs) plays a vital role in the fields of drug disease treatment, drug development, preventing medical error, and controlling health care-costs. Extracting potential from biomedical corpora is a major complement of existing DDIs. Most existing DDI extraction (DDIE) methods do not consider the graph and structure of drug molecules, which can improve the performance of DDIE. Considering the different advantages of bi-directional gated recurrent units (BiGRU), Transformer, and attention mechanisms in DDIE tasks, a multimodal feature fusion model combining BiGRU and Transformer (BiGGT) is here constructed for DDIE. In BiGGT, the vector embeddings of medical corpora, drug molecule topology graphs, and structure are conducted by Word2vec, Mol2vec, and GCN, respectively. BiGRU and multi-head self-attention (MHSA) are integrated into Transformer to extract the local–global contextual DDIE features, which is important for DDIE. The extensive experiment results on the DDIExtraction 2013 shared task dataset show that the BiGGT-based DDIE method outperforms state-of-the-art DDIE approaches with a precision of 78.22%. BiGGT expands the application of multimodal deep learning in the field of multimodal DDIE.

Feature fusion and context interaction for RGB-D indoor semantic segmentation

Depth information in indoor scenes is crucial to understanding the spatial and occluding relationships of objects. Depth information contains rich geometric details and is unaffected by lighting, texture, and color, significantly enhancing traditional color images. RGB images and depth images have been widely applied in various image analysis. However, effectively leveraging the complementary information from both modalities remains a challenge. To address this issue, we propose a unified and effective feature fusion and context interaction network (FCINet). It includes three modules. Firstly, we construct a multi-modal feature fusion module (MFFM), which can achieve the aggregation of two modalities along spatial and channel dimensions. Secondly, we construct a global and local information interaction context module (GLCM) to encode rich semantic information. Finally, we construct a feature alignment and fusion module (FAFM), which integrates upsampled low-level features with high-level features to mitigate spatial information loss. Experimental results indicate that the model achieved favorable outcomes on both the NYU Depth v2 dataset and the more complex SUN RGB-D dataset.

Design of a hazard prediction system with intelligent multimodal fusion based on artificial intelligence &amp; internet of things technology: taking a crib as an example

Problem How to design an intelligent multimodal fusion hazard prediction system using AIoT (Artificial Intelligence &amp; Internet of Things) technology to predict some potential dangers. This article will provide some ideas and methods. Introduction This article designs an intelligent multimodal fusion hazard prediction system based on AIoT technology. The system mainly consists of an IoT hardware device and an AI multimodal and multi-dimensional Hazard Prediction Algorithm. Method This article will take baby cribs as an example, using this system to empower traditional baby cribs and transform them into intelligent baby cribs. Results In this example, the system can detect the real-time status of the baby and predict upcoming dangers, including kicking the quilt, wetting the bed, fever, crying, climbing over the crib, and turning over. Hypothesis Assuming that intelligent systems can detect the status of infants in real-time and make predictions before potential dangers occur, to promptly alert parents. Perhaps it can prevent some irreversible dangers from occurring. Background This system solves the pain point problem of parents having to take care of their babies after working hard. Purpose In this scenario, the system provides a design scheme for an intelligent multimodal fusion hazard prediction system with a temporary care function. The application of this system to baby cribs not only reduces the burden on parents but also ensures the safety and comfort of the baby. Transferability At the same time, the system has transferability, and its design concept can be transferred to other application scenarios, such as hospital care for infants, young children, or elderly patients, as well as childcare stations for abandoned children in society. Significance This innovative system design scheme has a positive significance for family harmony and social development.

Multi-Modal LiDAR Point Cloud Semantic Segmentation with Salience Refinement and Boundary Perception

Point cloud segmentation is essential for scene understanding, which provides advanced information for many applications, such as autonomous driving, robots, and virtual reality. To improve the accuracy and robustness of point cloud segmentation, many researchers have attempted to fuze camera images to complement the color and texture information. The common fusion strategy is the combination of convolutional operations with concatenation, element-wise addition or element-wise multiplication. However, conventional convolutional operators tend to confine the fusion of modal features within their receptive fields, which can be incomplete and limited. In addition, the inability of encoder–decoder segmentation networks to explicitly perceive segmentation boundary information results in semantic ambiguity and classification errors at object edges. These errors are further amplified in point cloud segmentation tasks, significantly affecting the accuracy of point cloud segmentation. To address the above issues, we propose a novel self-attention multi-modal fusion semantic segmentation network for point cloud semantic segmentation. Firstly, to effectively fuze different modal features, we propose a self-cross fusion module (SCF), which models long-range modality dependencies and transfers complementary image information to the point cloud to fully leverage the modality-specific advantages. Secondly, we design the salience refinement module (SR), which calculates the importance of channels in the feature maps and global descriptors to enhance the representation capability of salient modal features. Finally, we propose the local-aware anisotropy loss measure the element-level importance in the data and explicitly provide boundary information for the model, which alleviates the inherent semantic ambiguity problem in segmentation networks. Extensive experiments on two benchmark datasets demonstrate that our proposed method surpasses current state-of-the-art methods.

DecoupleCLIP: A Novel Cross-Modality Decouple Model for Painting Captioning

Image captioning aims to describe the content in an image, which plays a critical role in image understanding. Existing methods tend to generate the text for more distinct natural images. These models can not be well for paintings containing more abstract meaning due to the limitation of objective parsing without related knowledge. To alleviate, we propose a novel cross-modality decouple model to generate the objective and subjective parsing separately. Concretely, we propose to encode both subjective semantic and implied knowledge contained in the paintings. The key point of our framework is decoupled CLIP-based branches (DecoupleCLIP). For the objective caption branch, we utilize the CLIP model as the global feature extractor and construct a feature fusion module for global clues. Based on the objective caption branch structure, we add a multimodal fusion module called the artistic conception branch. In this way, the objective captions can constrain artistic conception content. We conduct extensive experiments to demonstrate our DecoupleCLIP’s superior ability over our new dataset. Our model achieves nearly 2% improvement over other comparison models on CIDEr.