Fusion Network Research Articles

A multimodal fusion network based on a cross-attention mechanism for the classification of Parkinsonian tremor and essential tremor

A multimodal fusion network based on a cross-attention mechanism for the classification of Parkinsonian tremor and essential tremor

Fault diagnosis of rolling bearings driven by multi-channel data fusion and feature fusion under time-varying speed conditions

Abstract Bearings, as the core component for power transmission, are crucial in ensuring the safe and reliable operation of equipment. However, the fault information contained in a single-channel vibration signal is inherently limited. Additionally, under time-varying speed conditions, features are prone to drift, and the cross-domain diagnostic performance of most traditional domain adaptation (DA) models may drop dramatically. To solve the above problems and enhance the ability of DA models in extracting domain invariant features, this paper introduces a Multi-channel data fusion and Attention-guided Multi-feature Fusion-driven Center-aligned Network (MAMC). Initially, a multi-channel time-frequency information fusion strategy based on wavelet transform is constructed to achieve a comprehensive fusion of multi-channel data, thereby obtaining richer fault feature representations. Subsequently, a multi-branch feature fusion network, integrated with an attention mechanism, is devised to capture significant features across various dimensions and scales, resulting in more comprehensive and representative fault features. Finally, a novel Center-Aligned Domain Adaptation method (CADA) is proposed based on domain adversarial methods and center loss. By minimizing the distance between deep domain invariant features and trainable common class centers, the issue of domain shift between data is effectively alleviated, and the cross-domain diagnostic performance of DA models under the time-varying speed conditions is improved. The experimental results indicate that the MAMC method exhibits superior performance on both bearing datasets and is a promising approach for cross-domain intelligent fault diagnosis.

Physics-aware cross-domain fusion aids learning-driven computer-generated holography

The rapid advancement of computer-generated holography has bridged deep learning with traditional optical principles in recent years. However, a critical challenge in this evolution is the efficient and accurate conversion from the amplitude to phase domain for high-quality phase-only hologram (POH) generation. Existing computational models often struggle to address the inherent complexities of optical phenomena, compromising the conversion process. In this study, we present the cross-domain fusion network (CDFN), an architecture designed to tackle the complexities involved in POH generation. The CDFN employs a multi-stage (MS) mechanism to progressively learn the translation from amplitude to phase domain, complemented by the deep supervision (DS) strategy of middle features to enhance task-relevant feature learning from the initial stages. Additionally, we propose an infinite phase mapper (IPM), a phase-mapping function that circumvents the limitations of conventional activation functions and encapsulates the physical essence of holography. Through simulations, our proposed method successfully reconstructs high-quality 2K color images from the DIV2K dataset, achieving an average PSNR of 31.68 dB and SSIM of 0.944. Furthermore, we realize high-quality color image reconstruction in optical experiments. The experimental results highlight the computational intelligence and optical fidelity achieved by our proposed physics-aware cross-domain fusion.

MFMamba: A Mamba-Based Multi-Modal Fusion Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation of remote sensing images is a fundamental task in computer vision, holding substantial relevance in applications such as land cover surveys, environmental protection, and urban building planning. In recent years, multi-modal fusion-based models have garnered considerable attention, exhibiting superior segmentation performance when compared with traditional single-modal techniques. Nonetheless, the majority of these multi-modal models, which rely on Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs) for feature fusion, face limitations in terms of remote modeling capabilities or computational complexity. This paper presents a novel Mamba-based multi-modal fusion network called MFMamba for semantic segmentation of remote sensing images. Specifically, the network employs a dual-branch encoding structure, consisting of a CNN-based main encoder for extracting local features from high-resolution remote sensing images (HRRSIs) and of a Mamba-based auxiliary encoder for capturing global features on its corresponding digital surface model (DSM). To capitalize on the distinct attributes of the multi-modal remote sensing data from both branches, a feature fusion block (FFB) is designed to synergistically enhance and integrate the features extracted from the dual-branch structure at each stage. Extensive experiments on the Vaihingen and the Potsdam datasets have verified the effectiveness and superiority of MFMamba in semantic segmentation of remote sensing images. Compared with state-of-the-art methods, MFMamba achieves higher overall accuracy (OA) and a higher mean F1 score (mF1) and mean intersection over union (mIoU), while maintaining low computational complexity.

Meta-learning-based fault diagnosis method for rolling bearings under cross working-conditions

Abstract Problems such as domain shift and insufficient sample quantity may occur when conducting fault diagnosis under cross-working conditions, decreasing the accuracy and generalization of deep learning algorithms. In this paper, a fault diagnosis framework based on meta-learning and a dual-channel classification feature fusion network combines the advantages of meta-learning and dual-channel classification feature fusion network to enhance the framework's performance in cross-working condition diagnosis and few-shot learning. Firstly, dual-channel network is used to extract the classification features of different domains, and the features are fused. Then, training is conducted under the meta-learning strategy to acquire prior knowledge for fast model learning under cross-working conditions and solve the problem of a few shots. Finally, two public rolling bearing data sets are used to demonstrate the efficacy of the proposed method across different operational conditions. Before that, the appropriate sample length and fusion domain were selected through experiments. The proposed method also has good fault diagnosis accuracy in cross-device tasks. The experimental results verify the effective classification capability and robustness of the proposed method. Furthermore, the superiority of this method over other meta-learning approaches is evident when compared.

RSG-YOLOV8: Detection of rice seed germination rate based on enhanced YOLOv8 and multi-scale attention feature fusion

The lack of obvious difference between germinated seeds and non-germinated seeds will cause the low accuracy of detecting rice seed germination rate, remains a challenging issue in the field. In view of this, a new model named Rice Seed Germination-YOLOV8 (RSG-YOLOV8) is proposed in this paper. This model initially incorporates CSPDenseNet to streamline computational processes while preserving accuracy. Furthermore, the BRA, a dynamic and sparse attention mechanism is integrated to highlight critical features while minimizing redundancy. The third advancement is the employment of a structured feature fusion network, based on GFPN, aiming to reconfigure the original Neck component of YOLOv8, thus enabling efficient feature fusion across varying levels. An additional detection head is introduced, improving detection performance through the integration of variable anchor box scales and the optimization of regression losses. This paper also explores the influence of various attention mechanisms, feature fusion techniques, and detection head architectures on the precision of rice seed germination rate detection. Experimental results indicate that RSG-YOLOV8 achieves a mAP50 of 0.981, marking a 4% enhancement over the mAP50 of YOLOv8 and setting a new benchmark on the RiceSeedGermination dataset for the detection of rice seed germination rate.

Multi-Head Attention Affinity Diversity Sharing Network for Facial Expression Recognition

Facial expressions exhibit inherent similarities, variability, and complexity. In real-world scenarios, challenges such as partial occlusions, illumination changes, and individual differences further complicate the task of facial expression recognition (FER). To further improve the accuracy of FER, a Multi-head Attention Affinity and Diversity Sharing Network (MAADS) is proposed in this paper. MAADS comprises a Feature Discrimination Network (FDN), an Attention Distraction Network (ADN), and a Shared Fusion Network (SFN). To be specific, FDN first integrates attention weights into the objective function to capture the most discriminative features by using the proposed sparse affinity loss. Then, ADN employs multiple parallel attention networks to maximize diversity within spatial attention units and channel attention units, which guides the network to focus on distinct, non-overlapping facial regions. Finally, SFN deconstructs facial features into generic parts and unique parts, which allows the network to learn the distinctions between these features without having to relearn complete features from scratch. To validate the effectiveness of the proposed method, extensive experiments were conducted on several widely used in-the-wild datasets including RAF-DB, AffectNet-7, AffectNet-8, FERPlus, and SFEW. MAADS achieves the accuracy of 92.93%, 67.14%, 64.55%, 91.58%, and 62.41% on these datasets, respectively. The experimental results indicate that MAADS not only outperforms current state-of-the-art methods in recognition accuracy but also has a relatively low computational complexity.

SRPNet: stroke risk prediction based on two-level feature selection and deep fusion network

BackgroundStroke is one of the major chronic non-communicable diseases (NCDs) with high morbidity, disability and mortality. The key to preventing stroke lies in controlling risk factors. However, screening risk factors and quantifying stroke risk levels remain challenging.MethodsA novel prediction model for stroke risk based on two-level feature selection and deep fusion network (SRPNet) is proposed to solve the problem mentioned above. First, the two-level feature selection method is used to screen comprehensive features related to stroke risk, enabling accurate identification of significant risk factors while eliminating redundant information. Next, the deep fusion network integrating Transformer and fully connected neural network (FCN) is utilized to establish the risk prediction model SRPNet for stroke patients.ResultsWe evaluate the performance of the SRPNet using screening data from the China Stroke Data Center (CSDC), and further validate its effectiveness with census data on stroke collected in affiliated hospital of Jining Medical University. The experimental results demonstrate that the SRPNet model selects features closely related to stroke and achieves superior risk prediction performance over benchmark methods.ConclusionsSRPNet can rapidly identify high-quality stroke risk factors, improve the accuracy of stroke prediction, and provide a powerful tool for clinical diagnosis.

Mf-net: multi-feature fusion network based on two-stream extraction and multi-scale enhancement for face forgery detection

Due to the increasing sophistication of face forgery techniques, the images generated are becoming more and more realistic and difficult for human eyes to distinguish. These face forgery techniques can cause problems such as fraud and social engineering attacks in facial recognition and identity verification areas. Therefore, researchers have worked on face forgery detection studies and have made significant progress. Current face forgery detection algorithms achieve high detection accuracy within-dataset. However, it is difficult to achieve satisfactory generalization performance in cross-dataset scenarios. In order to improve the cross-dataset detection performance of the model, this paper proposes a multi-feature fusion network based on two-stream extraction and multi-scale enhancement. First, we design a two-stream feature extraction module to obtain richer feature information. Secondly, the multi-scale feature enhancement module is proposed to focus the model more on information related to the current sub-region from different scales. Finally, the forgery detection module calculates the overlap between the features of the input image and real images during the training phase to determine the forgery regions. The method encourages the model to mine forgery features and learns generic and robust features not limited to a particular feature. Thus, the model achieves high detection accuracy and performance. We achieve the AUC of 99.70% and 90.71% on FaceForensics++ and WildDeepfake datasets. The generalization experiments on Celeb-DF-v2 and WildDeepfake datasets achieve the AUC of 80.16% and 65.15%. Comparison experiments with multiple methods on other benchmark datasets confirm the superior generalization performance of our proposed method while ensuring model detection accuracy. Our code can be found at: https://github.com/1241128239/MFNet.

SFFNet: Shallow Feature Fusion Network Based on Detection Framework for Infrared Small Target Detection

Infrared small target detection (IRSTD) is the process of recognizing and distinguishing small targets from infrared images that are obstructed by crowded backgrounds. This technique is used in various areas, including ground monitoring, flight navigation, and so on. However, due to complex backgrounds and the loss of information in deep networks, infrared small target detection remains a difficult undertaking. To solve the above problems, we present a shallow feature fusion network (SFFNet) based on detection framework. Specifically, we design the shallow-layer-guided feature enhancement (SLGFE) module, which guides multi-scale feature fusion with shallow layer information, effectively mitigating the loss of information in deep networks. Then, we design the visual-Mamba-based global information extension (VMamba-GIE) module, which leverages a multi-branch structure combining the capability of convolutional layers to extract features in local space with the advantages of state space models in the exploration of long-distance information. The design significantly extends the network’s capacity to acquire global contextual information, enhancing its capability to handle complex backgrounds. And through the effective fusion of the SLGFE and VMamba-GIE modules, the exorbitant computation brought by the SLGFE module is substantially reduced. The experimental results on two publicly available infrared small target datasets demonstrate that the SFFNet surpasses other state-of-the-art algorithms.

Attention-enhanced multimodal feature fusion network for clothes-changing person re-identification

Clothes-Changing Person Re-Identification is a challenging problem in computer vision, primarily due to the appearance variations caused by clothing changes across different camera views. This poses significant challenges to traditional person re-identification techniques that rely on clothing features. These challenges include the inconsistency of clothing and the difficulty in learning reliable clothing-irrelevant local features. To address this issue, we propose a novel network architecture called the Attention-Enhanced Multimodal Feature Fusion Network (AE-Net). AE-Net effectively mitigates the impact of clothing changes on recognition accuracy by integrating RGB global features, grayscale image features, and clothing-irrelevant features obtained through semantic segmentation. Specifically, global features capture the overall appearance of the person; grayscale image features help eliminate the interference of color in recognition; and clothing-irrelevant features derived from semantic segmentation enforce the model to learn features independent of the person’s clothing. Additionally, we introduce a multi-scale fusion attention mechanism that further enhances the model’s ability to capture both detailed and global structures, thereby improving recognition accuracy and robustness. Extensive experimental results demonstrate that AE-Net outperforms several state-of-the-art methods on the PRCC and LTCC datasets, particularly in scenarios with significant clothing changes. On the PRCC and LTCC datasets, AE-Net achieves Top-1 accuracy rates of 60.4% and 42.9%, respectively.

Attention-based hand pose estimation with voting and dual modalities

Hand pose estimation has recently emerged as a compelling topic in the robotic research community, because of its usefulness in learning from human demonstration or safe human–robot interaction. Although deep learning-based methods have been introduced for this task and have shown promise, it remains a challenging problem. To address this, we propose a novel end-to-end architecture for hand pose estimation using red-green-blue (RGB) and depth (D) data (RGB-D). Our approach processes the two data sources separately and utilizes a dense fusion network with an attention module to extract discriminative features. The features extracted include both spatial information and geometric constraints, which are fused to vote for the hand pose. We demonstrate that our voting mechanism in conjunction with the attention mechanism is particularly useful for solving the problem, especially when hands are heavily occluded by objects or are self-occluded. Our experimental results on benchmark datasets demonstrate that our approach outperforms state-of-the-art methods by a significant margin.

AIRHF-Net: an adaptive interaction representation hierarchical fusion network for occluded person re-identification

To tackle the high resource consumption in occluded person re-identification, sparse attention mechanisms based on Vision Transformers (ViTs) have become popular. However, they often suffer from performance degradation with long sequences, omission of crucial information, and token representation convergence. To address these issues, we introduce AIRHF-Net: an Adaptive Interaction Representation Hierarchical Fusion Network, named AIRHF-Net, designed to enhance pedestrian identity recognition in occluded scenarios. Our approach begins with the development of an Adaptive Local-Window Interaction Encoder (AL-WIE), which aims to overcome the inherent subjective limitations of traditional sparse attention mechanisms. This innovative encoder merges window attention, adaptive local attention, and interaction attention, facilitating automatic localization and focusing on visible pedestrian regions within images. It effectively extracts contextual information from window-level features while minimizing the impact of occlusion noise. Additionally, recognizing that ViTs may lose spatial information in deeper structural layers, we implement a Local Hierarchical Encoder (LHE). This component segments the input sequence in the spatial dimension, integrating features from various spatial positions to construct hierarchical local representations that substantially enhance feature discriminability. To further augment the quality and breadth of datasets, we adopt an Occlusion Data Augmentation Strategy (ODAS), which bolsters the model’s capacity to extract critical information under occluded conditions. Extensive experiments demonstrate that our method achieves improved performance on the Occluded-DukeMTMC dataset, with a rank-1 accuracy of 69.6% and an mAP of 61.6%.

Multi-gate Weighted Fusion Network for neuronal morphology classification

Multi-gate Weighted Fusion Network for neuronal morphology classification

Reconstruction of Fine-Spatial-Resolution FY-3D-Based Vegetation Indices to Achieve Farmland-Scale Winter Wheat Yield Estimation via Fusion with Sentinel-2 Data

The spatial resolution (250–1000 m) of the FY-3D MERSI is too coarse for agricultural monitoring at the farmland scale (20–30 m). To achieve the winter wheat yield (WWY) at the farmland scale, based on FY-3D, a method framework is developed in this work. The enhanced deep convolutional spatiotemporal fusion network (EDCSTFN) was used to perform a spatiotemporal fusion on the 10 day interval FY-3D and Sentinel-2 vegetation indices (VIs), which were compared with the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM). In addition, a BP neural network was built to calculate the farmland-scale WWY based on the fused VIs, and the Aqua MODIS gross primary productivity product was used as ancillary data for WWY estimation. The results reveal that both the EDCSTFN and ESTARFM achieve satisfactory precision in the fusion of the Sentinel-2 and FY-3D VIs; however, when the period of spatiotemporal data fusion is relatively long, the EDCSTFN can achieve greater precision than ESTARFM. Finally, the WWY estimation results based on the fused VIs show remarkable correlations with the WWY data at the county scale and provide abundant spatial distribution details about the WWY, displaying great potential for accurate farmland-scale WWY estimations based on reconstructed fine-spatial-temporal-resolution FY-3D data.

SIFusion: Lightweight infrared and visible image fusion based on semantic injection.

The objective of image fusion is to integrate complementary features from source images to better cater to the needs of human and machine vision. However, existing image fusion algorithms predominantly focus on enhancing the visual appeal of the fused image for human perception, often neglecting their impact on subsequent high-level visual tasks, particularly the processing of semantic information. Moreover, these fusion methods that incorporate downstream tasks tend to be overly complex and computationally intensive, which is not conducive to practical applications. To address these issues, a lightweight infrared and visible light image fusion method known as SIFusion, which is based on semantic injection, is proposed in this paper. This method employs a semantic-aware branch to extract semantic feature information, and then integrates these features into the fused features through a Semantic Injection Module (SIM) to meet the semantic requirements of high-level visual tasks. Furthermore, to simplify the complexity of the fusion network, this method introduces an Edge Convolution Module (ECB) based on structural reparameterization technology to enhance the representational capacity of the encoder and decoder. Extensive experimental comparisons demonstrate that the proposed method performs excellently in terms of visual appeal and advanced semantics, providing satisfactory fusion results for subsequent high-level visual tasks even in challenging scenarios.

Deep learning hybrid model ECG classification using AlexNet and parallel dual branch fusion network model.

Cardiovascular diseases are a cause of death making it crucial to accurately diagnose them. Electrocardiography plays a role in detecting heart issues such as heart attacks, bundle branch blocks and irregular heart rhythms. Manual analysis of ECGs is prone to mistakes and time consuming, underscoring the importance of automated methods. This study uses AI models like AlexNet and a dual branch model for categorizing ECG signals from the PTB Diagnostic ECG Database. AlexNet achieved a validation accuracy of 98.64% and a test set accuracy of 99% while the dual branch fusion network model achieved a test set accuracy of 99%. Data preprocessing involved standardizing, balancing and reshaping ECG signals. These models exhibited precision, sensitivity and specificity. In comparison to state of the arts' models such as Hybrid AlexNet SVM and DCNN LSTM our proposed models displayed performance. The high accuracy rates of 99% underscore their potential for ECG classification. These results validate the advantages of incorporating learning models into setups for automated ECG analysis providing adaptable solutions for various healthcare settings including rural areas.

A Dual-Stream Fusion Network for Human Energy Expenditure Estimation with Wearable Sensor

With the increasing awareness of health, using wearable sensors to monitor individual activities and accurately estimate energy expenditure has become a current research focus. However, existing research encounters challenges including low estimation accuracy, a deficiency of frequency domain features, and difficulty in integrating time domain and frequency domain features. To address these issues, we propose an innovative framework called the Dual-Stream Fusion Network (DSFN). This framework combines the Time Domain Encoding (TDE) module, the Frequency Domain Hierarchical-Split Encoding (FDHSE) module, and a Two-Stage Feature Fusion (TSF) module. Specifically, the temporal stream of the framework employs the TDE module to capture deep temporal features that reflect the complex dynamic variations in time-series data. The frequency domain stream introduces the FDHSE module, which extracts frequency domain features using a multi-level, multi-scale approach, ensuring a comprehensive and diverse representation of frequency information. Through this dual-stream architecture, our model effectively learns both time and frequency domain features, addressing the limitations of frequency domain features observed in prior studies. Additionally, we propose the TSF module to fully integrate time and frequency domain features, effectively overcoming the challenge of fusing these two types of features. We conducted experiments on two public datasets, namely the GOTOV dataset (elderly people) and the JSI dataset (young people). Experimental results demonstrate that our method achieves excellent performance across different age groups. Compared to the baseline models, the proposed DSFN significantly improves the accuracy of human energy expenditure estimation.

A Lightweight and Efficient Multimodal Feature Fusion Network for Bearing Fault Diagnosis in Industrial Applications

To address the issues of single-structured feature input channels, insufficient feature learning capabilities in noisy environments, and large model parameter sizes in intelligent diagnostic models for mechanical equipment, a lightweight and efficient multimodal feature fusion convolutional neural network (LEMFN) method is proposed. Compared with existing models, LEMFN captures rich fault features at multiple scales by combining time-domain and frequency-domain signals, thereby enhancing the model’s robustness to noise and improving data adaptability under varying operating conditions. Additionally, the convolutional block attention module (CBAM) and random overlapping sampling technology (ROST) are introduced, and through a feature fusion strategy, the accurate diagnosis of mechanical equipment faults is achieved. Experimental results demonstrate that the proposed method not only possesses high diagnostic accuracy and rapid convergence but also exhibits strong robustness in noisy environments. Finally, a graphical user interface (GUI)-based mechanical equipment fault detection system was developed to promote the practical application of intelligent fault diagnosis in mechanical equipment.

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation