Feature Fusion Network Research Articles

Multi-level feature fusion network for neuronal morphology classification.

Neuronal morphology can be represented using various feature representations, such as hand-crafted morphometrics and deep features. These features are complementary to each other, contributing to improving performance. However, existing classification methods only utilize a single feature representation or simply concatenate different features without fully considering their complementarity. Therefore, their performance is limited and can be further improved. In this paper, we propose a multi-level feature fusion network that fully utilizes diverse feature representations and their complementarity to effectively describe neuronal morphology and improve performance. Specifically, we devise a Multi-Level Fusion Module (MLFM) and incorporate it into each feature extraction block. It can facilitate the interaction between different features and achieve effective feature fusion at multiple levels. The MLFM comprises a channel attention-based Feature Enhancement Module (FEM) and a cross-attention-based Feature Interaction Module (FIM). The FEM is used to enhance robust morphological feature presentations, while the FIM mines and propagates complementary information across different feature presentations. In this way, our feature fusion network ultimately yields a more distinctive neuronal morphology descriptor that can effectively characterize neurons than any singular morphological representation. Experimental results show that our method effectively depicts neuronal morphology and correctly classifies 10-type neurons on the NeuronMorpho-10 dataset with an accuracy of 95.18%, outperforming other approaches. Moreover, our method performs well on the NeuronMorpho-12 and NeuronMorpho-17 datasets and possesses good generalization.

A Depth Awareness and Learnable Feature Fusion Network for Enhanced Geometric Perception in Semantic Correspondence.

Deep learning is becoming the most widely used technology for multi-sensor data fusion. Semantic correspondence has recently emerged as a foundational task, enabling a range of downstream applications, such as style or appearance transfer, robot manipulation, and pose estimation, through its ability to provide robust correspondence in RGB images with semantic information. However, current representations generated by self-supervised learning and generative models are often limited in their ability to capture and understand the geometric structure of objects, which is significant for matching the correct details in applications of semantic correspondence. Furthermore, efficiently fusing these two types of features presents an interesting challenge. Achieving harmonious integration of these features is crucial for improving the expressive power of models in various tasks. To tackle these issues, our key idea is to integrate depth information from depth estimation or depth sensors into feature maps and leverage learnable weights for feature fusion. First, depth information is used to model pixel-wise depth distributions, assigning relative depth weights to feature maps for perceiving an object's structural information. Then, based on a contrastive learning optimization objective, a series of weights are optimized to leverage feature maps from self-supervised learning and generative models. Depth features are naturally embedded into feature maps, guiding the network to learn geometric structure information about objects and alleviating depth ambiguity issues. Experiments on the SPair-71K and AP-10K datasets show that the proposed method achieves scores of 81.8 and 83.3 on the percentage of correct keypoints (PCK) at the 0.1 level, respectively. Our approach not only demonstrates significant advantages in experimental results but also introduces the depth awareness module and a learnable feature fusion module, which enhances the understanding of object structures through depth information and fully utilizes features from various pre-trained models, offering new possibilities for the application of deep learning in RGB and depth data fusion technologies. We will also continue to focus on accelerating model inference and optimizing model lightweighting, enabling our model to operate at a faster speed.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

Estimation of Lower Limb Joint Angles Using sEMG Signals and RGB-D Camera.

Estimating human joint angles is a crucial task in motion analysis, gesture recognition, and motion intention prediction. This paper presents a novel model-based approach for generating reliable and accurate human joint angle estimation using a dual-branch network. The proposed network leverages combined features derived from encoded sEMG signals and RGB-D image data. To ensure the accuracy and reliability of the estimation algorithm, the proposed network employs a convolutional autoencoder to generate a high-level compression of sEMG features aimed at motion prediction. Considering the variability in the distribution of sEMG signals, the proposed network introduces a vision-based joint regression network to maintain the stability of combined features. Taking into account latency, occlusion, and shading issues with vision data acquisition, the feature fusion network utilizes high-frequency sEMG features as weights for specific features extracted from image data. The proposed method achieves effective human body joint angle estimation for motion analysis and motion intention prediction by mitigating the effects of non-stationary sEMG signals.

Aircraft skin defect detection algorithm based on improved YOLOv8

To solve the problem that the traditional aircraft skin defect detection relies on human eye observation, which leads to reduced efficiency due to human eye fatigue and limited individual cognition, an aircraft skin defect detection algorithm based on improved YOLOv8 is proposed. The data enhancement method is improved, and a slice reasoning + mosaic data enhancement method is proposed; the residual block is integrated into the feature extraction network to enhance the network expression ability and improve the accuracy of the model in the aircraft skin defect detection task; the three-branch (Triplet) attention module is applied to improve the feature fusion network to reduce the false detection rate and missed detection rate of small target samples; the structure of the detection head is optimized so that the network can better combine shallow information with deep information. Experimental results show that compared with the latest YOLOv8 algorithm, the improved algorithm improves the mean average precision (mAP) and recall rate (Recall) on the aircraft skin defect dataset by 3.6%and respectively 3.7%; and improves the mean average precision and recall rate on the public dataset VOC2007 by 2.9%and 2.2%.

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition.

EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

DPF-YOLOv8: Dual Path Feature Fusion Network for Traffic Sign Detection in Hazy Weather

Traffic sign detection plays an integral role in intelligent driving systems. It was found that in real driving scenarios, traffic signs were easily obscured by haze leading to traffic sign detection inaccuracy in assisted driving systems. Therefore, we designed a traffic sign detection model for hazy weather that can effectively help drivers to recognize road signs and reduce the incidence of traffic accidents. A high-precision traffic sign detection network has been designed to address the problem of decreased model recognition performance caused by external factors such as small size of traffic signs and haze obstruction in real-world scenarios. First, the default YOLOv8 was found to have low model detection accuracy in hazy weather occlusion conditions through experimental studies. Therefore, a deeper lightweight and efficient multi-branch CSP (Cross Stage Partial) module was introduced. Second, a dual path feature fusion network was designed to address the problem of insufficient feature fusion due to the small size of traffic signs. Finally, in order to be able to better simulate the real haze weather scene, we added fog to the raw data to enrich the data samples. This was verified through experiments on a public Chinese traffic sign detection dataset after fogging treatment, compared to the default YOLOv8 model. The improved DPF-YOLOv8 algorithm achieved 2.1% and 2.2% improvement in mAP@0.5 and mAP@0.5:0.95 performance metrics to 65.0% and 47.4%, respectively.

MDFF-Net: Multi-level Dynamic Feature Fusion Network Combined with Deep Supervision Mechanism for Low-Light Image Enhancement

MDFF-Net: Multi-level Dynamic Feature Fusion Network Combined with Deep Supervision Mechanism for Low-Light Image Enhancement

Sparse-View Photoacoustic Reconstruction Method for Diabetic Retinopathy Using Feature Fusion Network.

Diabetic retinopathy is one of the most prevalent microvascular complications of diabetes mellitus, and photoacoustic imaging is an effective method for imaging diabetic retinal vessels. Photoacoustic imaging is an emerging noninvasive imaging method based on the photoacoustic effect, which offers advantages of contrast, resolution, and depth imaging. Appropriate photoacoustic reconstruction methods are essential for obtaining high-quality photoacoustic images. In this study, a multi-input self-attention multiscale feature fusion network (SAMF-Net) is proposed for photoacoustic reconstruction. The algorithm accepts two inputs, namely the original photoacoustic signal and the traditional reconstructed image. Furthermore, a global feature extraction module based on the self-attention mechanism is employed to focus on the global information. The results demonstrate that the proposed method exhibits superior reconstruction capability under different sparse detection views. The method has instructive value for photoacoustic image reconstruction and has the potential for further application in the diagnosis of diabetic retinopathy.

Joint segmentation of tumors in 3D PET-CT images with a network fusing multi-view and multi-modal information

Objective. Joint segmentation of tumors in positron emission tomography-computed tomography (PET-CT) images is crucial for precise treatment planning. However, current segmentation methods often use addition or concatenation to fuse PET and CT images, which potentially overlooks the nuanced interplay between these modalities. Additionally, these methods often neglect multi-view information that is helpful for more accurately locating and segmenting the target structure. This study aims to address these disadvantages and develop a deep learning-based algorithm for joint segmentation of tumors in PET-CT images.Approach. To address these limitations, we propose the Multi-view Information Enhancement and Multi-modal Feature Fusion Network (MIEMFF-Net) for joint tumor segmentation in three-dimensional PET-CT images. Our model incorporates a dynamic multi-modal fusion strategy to effectively exploit the metabolic and anatomical information from PET and CT images and a multi-view information enhancement strategy to effectively recover the lost information during upsamping. A Multi-scale Spatial Perception Block is proposed to effectively extract information from different views and reduce redundancy interference in the multi-view feature extraction process.Main results. The proposed MIEMFF-Net achieved a Dice score of 83.93%, a Precision of 81.49%, a Sensitivity of 87.89% and an IOU of 69.27% on the Soft Tissue Sarcomas dataset and a Dice score of 76.83%, a Precision of 86.21%, a Sensitivity of 80.73% and an IOU of 65.15% on the AutoPET dataset.Significance. Experimental results demonstrate that MIEMFF-Net outperforms existing state-of-the-art models which implies potential applications of the proposed method in clinical practice.

Lightweight Crack Automatic Detection Algorithm Based on TF-MobileNet

With the progress of social life, the aging of building facilities has become an inevitable phenomenon. The efficiency of manual crack detection is limited, so it is necessary to explore intelligent detection technology. This article proposes a novel crack detection method TF-MobileNet. We took into account the effect of lightweight and crack feature extraction, so we developed a novel crack feature extraction backbone network, which combined Transformer and MobileNetV3. Then we improved the feature fusion network by using the multi-headed attention mechanism of the Bottleneck Transformer, which enables the feature fusion effect to be improved. Then, we integrated SENet and SimAM attention mechanisms into the networks used for feature extraction and feature fusion, thereby further improving the crack detection performance. Finally, we deployed our model in edge devices (NVIDIA Jeston Nano). The findings indicate that our proposed model has achieved 90.8% mAP on the dataset and worked well on the edge device side, which meet the requirements of automatic crack detection. Our model enables real-time monitoring of pavement using edge devices. This approach allows for timely maintenance and repair of the pavement. In the future, we can train the model to recognize more pavement distress features, addressing road safety issues effectively.

Local-global feature fusion network for hyperspectral image classification

ABSTRACT Hyperspectral images (HSIs) are rich in spatial and spectral information, making them crucial for accurate classification. While existing convolutional neural network (CNN) methods have shown promise in HSI classification, they primarily focus on local features and struggle with establishing long-range dependencies. To address these limitations, we propose a Local-Global Feature Fusion Network (LGFFN). LGFFN effectively learns joint local-global information from HSIs. Firstly, we employ a group-general convolution parallel module (GGCPM) to extract both local and global spectral features. Next, we utilize parallel branches: a CNN branch for capturing fine-grained spatial information and an improved transformer encoder (ITE) branch for capturing long-distance spatial-spectral features and exploiting semantic information. Finally, we elegantly integrate the diverse features obtained from these branches to obtain classification results. We validate our method on four datasets and compare it with six state-of-the-art hyperspectral classification methods. Our experimental results demonstrate that LGFFN achieves high classification accuracy, outperforming existing methods.

SFFN: Serial Feature Fusion Network for Visual Tracking

Abstract Recently, The Transformer tracker has excellent visual target tracking performance. The parallel arrangement of structural feature fusion networks proposed by the TransT tracker makes the Transformer learn the erroneous information. To address the issue, this paper proposes the SFFN tracker. It constructed a serial feature fusion network with the Transformer module in a serial interleaved arrangement to avoid learning erroneous information during the Transformer feature fusion process. The experiments show that SFFN performs well on the GOT-10k dataset, and it improves over TransT by 2.6%, 2.6%, and 2.3% in AO, SR0.5, and SR0.75, respectively.

Turbofan Engine's RUL Prediction Based on the CSI-EMD and Double-Channel Multilayer Feature Fusion Network

Turbofan Engine's RUL Prediction Based on the CSI-EMD and Double-Channel Multilayer Feature Fusion Network

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

WITHDRAWN: Non-Contact Blood Pressure Monitoring from Radar Signals by a Temporal-Spatial Feature Fusion Network

Non-contact blood pressure (BP) monitoring offers a comfortable and uninterrupted means of BP assessment, free from the constraints of physical contact. This capability is particularly better for the routine surveillance of BP for individuals with hypertension, burn injuries, skin sensitivities, and other pertinent conditions. In this study, we proposed a Temporal-Spatial Feature Fusion Network (TSFN), a radar-based, non-contact approach for BP monitoring. The TSFN architecture combines Residual Networks (ResNet) for the meticulous extraction of spatial features from radar signals, Gated Recurrent Unit (GRU) for the discernment of temporal dynamics, and Multiple Head Attention (MHA) to selectively emphasize crucial information. These components collectively ensure precise BP detection from radar signals. To enhance the model's robustness, a Pseudo-Huber loss function was employed to refine the optimization process, providing a smoother gradient transition and improved stability. Evaluations demonstrated impressive accuracies, with mean errors of 0.24±6.78 mmHg for systolic blood pressure (SBP) and 0.25±5.13 mmHg for diastolic blood pressure (DBP). These outcomes meet the standards set by the British Hypertension Society (BHS) for grade ‘A’ benchmarks for SBP and DBP measurements. Notably, the TSFN model avoids the need for complex feature engineering, demonstrating its effectiveness in monitoring BP fluctuations across diverse physiological states at 2s intervals. This feature highlights its potential applicability in real-time monitoring systems, offering a promising solution for continuous, non-contact BP monitoring.

YOLO-Faster: An efficient remote sensing object detection method based on AMFFN.

As a pivotal task within computer vision, object detection finds application across a diverse spectrum of industrial scenarios. The advent of deep learning technologies has significantly elevated the accuracy of object detectors designed for general-purpose applications. Nevertheless, in contrast to conventional terrestrial environments, remote sensing object detection scenarios pose formidable challenges, including intricate and diverse backgrounds, fluctuating object scales, and pronounced interference from background noise, rendering remote sensing object detection an enduringly demanding task. In addition, despite the superior detection performance of deep learning-based object detection networks compared to traditional counterparts, their substantial parameter and computational demands curtail their feasibility for deployment on mobile devices equipped with low-power processors. In response to the aforementioned challenges, this paper introduces an enhanced lightweight remote sensing object detection network, denoted as YOLO-Faster, built upon the foundation of YOLOv5. Firstly, the lightweight design and inference speed of the object detection network is augmented by incorporating the lightweight network as the foundational network within YOLOv5, satisfying the demand for real-time detection on mobile devices. Moreover, to tackle the issue of detecting objects of different scales in large and complex backgrounds, an adaptive multiscale feature fusion network is introduced, which dynamically adjusts the large receptive field to capture dependencies among objects of different scales, enabling better modeling of object detection scenarios in remote sensing scenes. At last, the robustness of the object detection network under background noise is enhanced through incorporating a decoupled detection head that separates the classification and regression processes of the detection network. The results obtained from the public remote sensing object detection dataset DOTA show that the proposed method has a mean average precision of 71.4% and a detection speed of 38 frames per second.

Domain expansion fusion single-domain generalization framework for mechanical fault diagnosis under unknown working conditions

In real industrial scenarios, mechanical systems often adjust working conditions based on specific tasks, leading to challenges in collecting data for all possible machine states in advance. Consequently, applying deep learning models trained on data from a single working condition directly to other unknown working condition poses a significant challenge. To tackle this issue, a novel domain expansion fusion single-domain generalization framework is proposed for machinery fault diagnosis under unknown working conditions. Firstly, a domain expansion module that can be controlled via a constraint function is developed to create expanded domains that generate samples with controlled differences from the source domain. Subsequently, a dual-branch feature fusion network is proposed in the feature extraction module. It combines two distinct feature extractors and employs a weighted average fusion strategy to extract discriminative features. Additionally, a state recognition module is implemented through a multi-classifier ensemble strategy to enhance the robustness and accuracy of health state identification. Lastly, an adversarial contrastive training strategy is employed to optimize the network and enhance its generalization capabilities and fault diagnosis performance. Through case studies conducted on two mechanical fault datasets, the proposed method demonstrates good diagnosis performance on single-domain generalized diagnosis tasks with an average accuracy of 87.53%. Its generalization effect is validated. Furthermore, the comparison and ablation experiment results confirm the effectiveness and superior performance of the proposed intelligent fault diagnosis method in scenarios with unknown working conditions with an average accuracy improvement of at least 3.94%.

A Ghost and Attention Mechanism-Based Deep Learning Approach for SAR Small Target Image Detection

Sy nthetic aperture radar (SAR) images play an important role in the task of small target imaging and detection. However, due to its blurred target imaging, complex image background, and challenging feature representation, SAR detection is difficult to get high precision in a complex imaging environment. Therefore, by integrating the ghost model and the Attention-Mechanism (AM), this paper presents a new feature network model based on the improved YOLOv5s for detecting small targets in SAR images efficiently. The key is to design a Ghost_AM module by integrating the ghost bottleneck with the simple AM. This Ghost_AM module is introduced into to the YOLOv5s’ network for improving the YOLOv5s’ extraction performance for small target’s feature from a complex SAR image. Furthermore, to improve the YOLOv5s’ recognition ability for the features extracted from the Ghost_AM, a bidirectional feature pyramid network is use to replace the YOLOv5s’ feature fusion network and get a precise detection. The experiments using some SAR ship image datasets have demonstrated the presented model is better than some traditional SAR target detection algorithms and the original YOLOv5s. Moreover, the presented model’s parameters are reduced by 25.1% compared with the original YOLOv5s, which contributes to its simpler implementation.

Grape clusters detection based on multi-scale feature fusion and augmentation

This paper addresses the challenge of low detection accuracy of grape clusters caused by scale differences, illumination changes, and occlusion in realistic and complex scenes. We propose a multi-scale feature fusion and augmentation YOLOv7 network to enhance the detection accuracy of grape clusters across variable environments. First, we design a Multi-Scale Feature Extraction Module (MSFEM) to enhance feature extraction for small-scale targets. Second, we propose the Receptive Field Augmentation Module (RFAM), which uses dilated convolution to expand the receptive field and enhance the detection accuracy for objects of various scales. Third, we present the Spatial Pyramid Pooling Cross Stage Partial Concatenation Faster (SPPCSPCF) module to fuse multi-scale features, improving accuracy and speeding up model training. Finally, we integrate the Residual Global Attention Mechanism (ResGAM) into the network to better focus on crucial regions and features. Experimental results show that our proposed method achieves a mAP0.5\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$_{0.5}$$\\end{document} of 93.29% on the GrappoliV2 dataset, an improvement of 5.39% over YOLOv7. Additionally, our method increases Precision, Recall, and F1 score by 2.83%, 3.49%, and 0.07, respectively. Compared to state-of-the-art detection methods, our approach demonstrates superior detection performance and adaptability to various environments for detecting grape clusters.