Feature Fusion Research Articles

Disentangled feature fusion network for lightweight image super-resolution

Recently, the quality of generated images in image super-resolution (SR) has significantly improved due to the widespread application of convolutional neural networks. Existing super-resolution methods often overlook the importance of network width and instead prioritize designing deeper network structures to improve performance. Deeper networks, however, demand greater computational resources and are more challenging to train, making them less suitable for devices with limited performance. To address this issue, we propose a novel method called the disentangled feature fusion network (DFFN). Specifically, we construct dual feature extraction streams (DFES) using the strategies of feature disentanglement and parameter sharing, which increase the network's width without increasing the number of parameters. To facilitate the interaction of information between the dual feature extraction streams, we design a feature interaction module (FIM) that separately enhances high and low-frequency features. Furthermore, a feature fusion module (FFM) is presented to efficiently fuse different levels of high and low-frequency feature information. The proposed DFFN integrates DFES, FIM, and FFM, which not only widens the network without increasing parameters but also minimizes the loss of crucial feature information. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms other state-of-the-art lightweight super-resolution methods, achieving a superior balance between parameter quantity, reconstruction performance, and model complexity. The code is available at https://github.com/zhoujy-aust/DFFN.

Hypersonic glide vehicle trajectory prediction based on frequency enhanced channel attention and light sampling-oriented MLP network

Hypersonic Glide Vehicles (HGVs) are advanced aircraft that can achieve extremely high speeds (generally over 5 Mach) and maneuverability within the Earth's atmosphere. HGV trajectory prediction is crucial for effective defense planning and interception strategies. In recent years, HGV trajectory prediction methods based on deep learning have the great potential to significantly enhance prediction accuracy and efficiency. However, it's still challenging to strike a balance between improving prediction performance and reducing computation costs of the deep learning trajectory prediction models. To solve this problem, we propose a new deep learning framework (FECA-LSMN) for efficient HGV trajectory prediction. The model first uses a Frequency Enhanced Channel Attention (FECA) module to facilitate the fusion of different HGV trajectory features, and then subsequently employs a Light Sampling-oriented Multi-Layer Perceptron Network (LSMN) based on simple MLP-based structures to extract long/short-term HGV trajectory features for accurate trajectory prediction. Also, we employ a new data normalization method called reversible instance normalization (RevIN) to enhance the prediction accuracy and training stability of the network. Compared to other popular trajectory prediction models based on LSTM, GRU and Transformer, our FECA-LSMN model achieves leading or comparable performance in terms of RMSE, MAE and MAPE metrics while demonstrating notably faster computation time. The ablation experiments show that the incorporation of the FECA module significantly improves the prediction performance of the network. The RevIN data normalization technique outperforms traditional min-max normalization as well.

An effective multi-scale interactive fusion network with hybrid Transformer and CNN for smoke image segmentation

Smoke has visually elusive appearances, especially in low-light conditions, so it is quite difficult to quickly and accurately detect smoke from images. To address these challenges, we design a dual-encoder structure of Transformer and Convolutional Neural Network (CNN) to propose an effective Multi-scale Interactive Fusion Network (MIFNet) for smoke image segmentation. To enhance the presentation of features, we propose a Local Feature Enhancement Propagation (LFEP) module to enhance spatial details. To optimize global and local features for efficient fusion, we integrate LFEP into the original Transformer to replace the traditional multi-head self-attention mechanism. Then, we propose a Multi-level Attention Coupled Module (MACM) to fuse Transformer and CNN features of the dual-encoder. MACM can flexibly focus on information interaction between different levels of two encoding paths. Finally, we design a Prior-guided Multi-scale Fusion Decoder (PMFD), which combines prior knowledge with a multi-scale feature fusion strategy to improve the performance of segmentation. Experimental results demonstrate that MIFNet substantially outperforms the state-of-the-art methods. MIFNet achieves a mean Intersection over Union (mIoU) of 81.6% on the synthetic smoke (SYN70K) dataset, and a remarkable accuracy of 98.3% on the forest smoke dataset.

DSIFNet: Implicit Feature Network for Nasal Cavity and Vestibule Segmentation from 3D Head CT

This study is dedicated to accurately segment the nasal cavity and its intricate internal anatomy from head CT images, which is critical for understanding nasal physiology, diagnosing diseases, and planning surgeries. Nasal cavity and it’s anatomical structures such as the sinuses, and vestibule exhibit significant scale differences, with complex shapes and variable microstructures. These features require the segmentation method to have strong cross-scale feature extraction capabilities. To effectively address this challenge, we propose an image segmentation network named the Deeply Supervised Implicit Feature Network (DSIFNet). This network uniquely incorporates an Implicit Feature Function Module Guided by Local and Global Positional Information (LGPI-IFF), enabling effective fusion of features across scales and enhancing the network's ability to recognize details and overall structures. Additionally, we introduce a deep supervision mechanism based on implicit feature functions in the network's decoding phase, optimizing the utilization of multi-scale feature information, thus improving segmentation precision and detail representation. Furthermore, we constructed a dataset comprising 7116 CT volumes (including 1,292,508 slices) and implemented PixPro-based self-supervised pretraining to utilize unlabeled data for enhanced feature extraction. Our tests on nasal cavity and vestibule segmentation, conducted on a dataset comprising 128 head CT volumes (including 34006 slices), demonstrate the robustness and superior performance of proposed method, achieving leading results across multiple segmentation metrics.

Multiscale Spatio-Temporal Feature Fusion Based Non-Intrusive Appliance Load Monitoring for Multiple Industrial Industries

The appliance types and power consumption patterns vary greatly across different industries. This can lead to unstable identification results of traditional appliance load monitoring methods in different industries. A non-intrusive appliance load monitoring (NIALM) method for multiple industries based on multiscale spatio-temporal feature fusion has been proposed. Firstly, the ConvNeXt Block with efficient channel attention has strong feature extraction capability. Spatial features of appliance state changes and micro-variations generated during operation can be extracted from mixed industrial load information by it. Meanwhile, the bidirectional gated recurrent neural network is used to learn the bidirectional dependencies of the load data, obtaining temporal features. Then, the multi-scale feature extraction module is used to extract temporal and spatial features from different depths of the network layers. And the extracted multi-scale temporal and spatial features are fully integrated. Finally, the proposed model is optimized using the Stochastic Weight Averaging method. During the training process, a certain number of model weights are randomly averaged, which can improve the model's generalization ability and identification accuracy. The experiment was conducted on six different industries. The evaluation indexes such as accuracy, F1 score, and Wasserstein distance are also used to verify the effectiveness and superiority of the method.

Multistream Adaptive Attention-Enhanced Graph Convolutional Networks for Youth Fencing Footwork Training.

The popularity of fencing and intense sports competition has burdened adolescents with excessive training, harming their immature bodies. Traditional training methods fail to provide timely movement corrections and personalized plans, leading to ineffective exercises. This paper aims to use artificial intelligence technology to reduce ineffective exercises and alleviate the training burden. We propose an action recognition algorithm based on the characteristics of adolescent athletes. This algorithm uses multimodal input data to comprehensively extract action information. Each modality is processed by the same network structure, utilizing attention mechanisms and adaptive graph structures. A multibranch feature fusion method is used to determine the final action category. We gathered the fencing footwork data set 2.0. Our model achieved 93.3% accuracy, with the highest precision at 95.8% and the highest F1-Score at 94.5% across all categories. It effectively recognized actions of adolescents with different heights and speeds, outperforming traditional methods. Our artificial intelligence-based training solution improves training efficiency and reduces the training burden on adolescents.

A multi-scale, multi-task fusion UNet model for accurate breast tumor segmentation

Background and Objective:Breast cancer is the most common cancer type among women worldwide and a leading cause of female death. Accurately interpreting these complex tumors, involving small size and morphology, requires a significant amount of expertise and time. Developing a breast tumor segmentation model to assist clinicians in treatment, therefore, holds great practical significance. Methods:We propose a multi-scale, multi-task model framework named MTF-UNet. Firstly, we differ from the common approach of using different convolution kernel sizes to extract multi-scale features, and instead use the same convolution kernel size with different numbers of convolutions to obtain multi-scale, multi-level features. Additionally, to better integrate features from different levels and sizes, we extract a new multi-branch feature fusion block (ADF). This block differs from using channel and spatial attention to fuse features, but considers fusion weights between various branches. Secondly, we propose to use the number of pixels predicted to be related to tumors and background to assist segmentation, which is different from the conventional approach of using classification tasks to assist segmentation. Results:We conducted extensive experiments on our proprietary DCE-MRI dataset, as well as two public datasets (BUSI and Kvasir-SEG). In the aforementioned datasets, our model achieved excellent MIoU scores of 90.4516%, 89.8408%, and 92.8431% on the respective test sets. Furthermore, our ablation study has demonstrated the efficacy of each component and the effective integration of our auxiliary prediction branch into other models. Conclusion:Through comprehensive experiments and comparisons with other algorithms, the effectiveness, adaptability, and robustness of our proposed method have been demonstrated. We believe that MTF-UNet has great potential for further development in the field of medical image segmentation. The relevant code and data can be found at https://github.com/LCUDai/MTF-UNet.git.

Attention-Based Two-Stage Multi-Sensor Feature Fusion Method for Bearing Fault Diagnosis

Attention-Based Two-Stage Multi-Sensor Feature Fusion Method for Bearing Fault Diagnosis

A novel multimodal image feature fusion mechanism:Application to rabbit liveweight estimation in commercial farms

In the realm of commercial rabbit farming, assessment of liveweight for each rabbit is crucial for production management. Traditional manual weighing methods are labor-intensive, time-consuming, and can induce stress and disease transmission. To solve this problem, this paper introduced a liveweight estimation method for rabbit utilizing near-infrared and depth image, including a segmentation stage and a weight estimation stage. Specifically, a dual-stream feature fusion mechanism was proposed to effectively integrate information from both near-infrared and depth imaging. The network was trained and validated on a dataset of 1,957 overhead images from 300 rabbits collected from a commercial farm, achieving superior performance with an R-Square (R2) of 0.95, Root Mean Square Error (RMSE) of 194.95 g, Mean Absolute Deviation (MAD) of 155.10 g, Mean Absolute Percentage Error (MAPE) of 4.08%, and Mean Coefficient of Variation (MCV) of 3.03%. Ablation experiments revealed that employing the proposed dual-stream feature fusion mechanism led to reductions of 52.87g, 38.19g, 1.01%, and 0.75% in RMSE, MAD, MAPE, and MCV, respectively, compared to not employing the mechanism. Compared to machine learning method, our proposed approach demonstrates superior performance in RMSE, MAD, MAPE, and MCV, achieving more accurate liveweight estimation for large-scale commercial farms.

Integrated Sensing and Communications Toward Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

Integrated Sensing and Communications Toward Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

YOLOv7-based insulator defect detection with progressive feature fusion and DFC attention

Abstract With the development of smart grids, drones have been widely used for inspecting power transmission lines, generating a large amount of insulator image data. Relying on manual inspection for this task results in a huge workload and the risk of missed detections due to human fatigue. Here, we propose an improved model based on YOLOv7-Tiny. First, we replace the activation function with the funnel activation function to optimize the activation domain dynamically. Second, we introduce a lightweight DFC attention mechanism. Experimental results show that the improved model achieves a detection accuracy and recall rate of 96.4% and 92.5%, respectively.

ACGRHA-Net: Accelerated Multi-Contrast MR Imaging with Adjacency Complementary Graph assisted Residual Hybrid Attention Network

Multi-contrast magnetic resonance (MR) imaging is an advanced technology used in medical diagnosis, but the long acquisition process can lead to patient discomfort and limit its broader application. Shortening acquisition time by undersampling k-space data introduces noticeable aliasing artifacts. To address this, we propose a method that reconstructs multi-contrast MR images from zero-filled data by utilizing a fully-sampled auxiliary contrast MR image as a prior to learn an adjacency complementary graph. This graph is then combined with a residual hybrid attention network, forming the adjacency complementary graph assisted residual hybrid attention network (ACGRHA-Net) for multi-contrast MR image reconstruction. Specifically, the optimal structural similarity is represented by a graph learned from the fully sampled auxiliary image, where the node features and adjacency matrices are designed to precisely capture structural information among different contrast images. This structural similarity enables effective fusion with the target image, improving the detail reconstruction. Additionally, a residual hybrid attention module is designed in parallel with the graph convolution network, allowing it to effectively capture key features and adaptively emphasize these important features in target contrast MR images. This strategy prioritizes crucial information while preserving shallow features, thereby achieving comprehensive feature fusion at deeper levels to enhance multi-contrast MR image reconstruction. Extensive experiments on the different datasets, using various sampling patterns and accelerated factors demonstrate that the proposed method outperforms the current state-of-the-art reconstruction methods.

Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma

Self-supervised multi-modal feature fusion for predicting early recurrence of hepatocellular carcinoma

PV-YOLO: A lightweight pedestrian and vehicle detection model based on improved YOLOv8

With the frequent occurrence of urban traffic accidents, fast and accurate detection of pedestrian and vehicle targets has become one of the key technologies for intelligent assisted driving systems. To meet the efficiency and lightweight requirements of smart devices, this paper proposes a lightweight pedestrian and vehicle detection model based on the YOLOv8n model, named PV-YOLO. In the proposed model, receptive-field attention convolution (RFAConv) serves as the backbone network because of its target feature extraction ability, and the neck utilizes the bidirectional feature pyramid network (BiFPN) instead of the original path aggregation network (PANet) to simplify the feature fusion process. Moreover, a lightweight detection head is introduced to reduce the computational burden and improve the overall detection accuracy. In addition, a small target detection layer is designed to improve the accuracy for small distant targets. Finally, to reduce the computational burden further, the lightweight C2f module is utilized to compress the model. The experimental results on the BDD100K and KITTI datasets demonstrate that the proposed PV-YOLO can achieve higher detection accuracy than YOLOv8n and other baseline methods with less model complexity.

Attention-based CNN model for motor imagery classification from nonlinear EEG signals

Motor imagery (MI)-based brain-computer interface (BCI) provides a promising solution for the limb rehabilitation of stroke patients. For better rehabilitation performance, high-precision classification of MI-related EEG signals plays a critical role. However, this is still a challenging problem for multi-category MI signals. In this paper, we focus on four commonly used stroke rehabilitation actions, and propose a modular temporal-spatial attention-based CNN (MTSACNN) model for MI classification. In detail, we carry out the MI experiments and acquire the EEG signals related to imagining left/right fist clenching and left/right wrist dorsiflexion. MTSACNN model firstly extracts the low-order MI features through the temporal-spatial feature extraction module (TSFE module). Especially, a group attention mechanism is proposed for intra-group information interaction. Secondly, considering the short- and long-term working characteristics of brain, high-order temporal features are further extracted and fused by the multi-level feature fusion module (MLFF module). Finally, four auxiliary losses are arranged in the classification module (C module) to speed up the model optimization process. The experimental results show that MTSACNN model can achieve good performance in decoding rehabilitation-related MI brain intentions, achieving an average classification accuracy of 72.05% for fourteen subjects. This work is beneficial to promote the construction of high-performance stroke rehabilitation BCI system.

CI-YOLO: A lightweight foreign object detection model for inspecting transmission line

Timely detection and elimination of foreign objects on transmission lines are of significance for the secure operation of power grids. With the widespread utilization of unmanned aerial vehicles (UAVs) for inspecting transmission lines, how to effectively balance the resource consumption and accuracy of detection model presents a challenge. To tackle this problem, this study proposes a foreign object detection model based on an improved YOLOv8n, realizing both lightweight and enhanced detection performance. Firstly, a SC Block structure is devised to substitute Bottleneck in C2f module of Backbone, minimizing the spatial and channel redundancy among features, enhancing the network performance, and reducing the consumption of computational resources. Subsequently, a novel dual attention mechanism, IEMA, is added after C2f module of Backbone, strengthening the crucial spatial and channel feature expression capability and significantly increasing the detection accuracy. Lastly, a new feature fusion network, CCFF_IEMA, is designed to replace Neck network, effectively integrating detailed features and reducing the parameter count and FLOPs. Experimental results indicate that in comparison to standard YOLOv8n, the parameters of our proposed CI-YOLO model are decreased by 40 %, FLOPs are reduced by 19.8 %, while achieving improvements of 0.7 % in Precision, 3.2 % in Recall, and 3.1 % in mAP0.5, respectively. The CI-YOLO model is beneficial to deployment on UAVs and provides a novel approach for detecting the foreign objects.

BEV-TinySpotter: A novel BEV perception method considering multi-dimensional feature fusion of small target

BEV-TinySpotter: A novel BEV perception method considering multi-dimensional feature fusion of small target

Long Short Observation-driven Prediction Network for pedestrian crossing intention prediction with momentary observation

Pedestrian crossing intention prediction aims to predict whether the pedestrian will cross the road, which is crucial for the decision-making of intelligent vehicles and ensuring traffic safety. Existing methods just rely on long-term observation and rarely consider it challenging to obtain sufficiently long and precise observation in real-world scenarios. Focus on momentary observation, which only contains two frames of the preceding and current time, we propose a novel Long Short Observation-driven Prediction Network (LSOP-Net). LSOP-Net comprises two critical components, the Momentary Observation feature Extraction Module (MOE-Module) and the Multimodal Long-Short-term feature Fusion Module (MLSFusion). Utilizing a hybrid training strategy and an external long-term feature pool, the MOE-Module is proposed to extract features with long-term patterns from momentary observations, which effectively mitigates feature deficiency arising from momentary observations. Based on a feature selection fusion mechanism, the MLSFusion is proposed to explicitly model the importance relationship between various modalities’ long-short-term features and the output, which adaptively fuses the long-short-term features from various modalities. Experimental results on the JAAD and PIE datasets demonstrate that our approach achieves superior performance in pedestrian crossing intention prediction with momentary observation.

Multi-type feature fusion visual navigation for asteroid landing

In order to address the challenges encountered in visual navigation for asteroid landing using traditional point features, such as significant recognition and extraction errors, low computational efficiency, and limited navigation accuracy, a novel approach for multi-type fusion visual navigation is proposed. This method aims to overcome the limitations of single-type features and enhance navigation accuracy. Analytical criteria for selecting multi-type features are introduced, which simultaneously improve computational efficiency and system navigation accuracy. Concerning pose estimation, both absolute and relative pose estimation methods based on multi-type feature fusion are proposed, and multi-type feature normalization is established, which significantly improves system navigation accuracy and lays the groundwork for flexible application of joint absolute-relative estimation. The feasibility and effectiveness of the proposed method are validated through simulation experiments through 4769 Castalia.

Echocardiographic mitral valve segmentation model

Segmentation of mitral valve is not only important for clinical diagnosis, but also has far-reaching impact on prevention and prognosis of the disease by experts and doctors.In this paper, the multi-channel cross fusion transformer based U-Net network model (MCCT-UNet) is proposed according to the classical U-Net architecture. First, the jump connection part of MCCT-UNet is designed by using a multi-channel cross-fusion based attention mechanism module (MCCT) instead of the original jump connection, and this module fuses the feature maps from different scales in different stages of the encoder. Second, the optimization of the feature fusion method is proposed in the decoding stage by designing the cross-compression excitation sub-module (C-SENet) to replace the simple feature splicing, and the C-SENet is used to bridge the inconsistency of the semantic hierarchy by effectively combining the deeper information in the encoding stage with the shallower information. This two modules can establish a close connection between the encoder and decoder by exploring multi-scale global contextual information to solve the semantic divide problem, thus it significantly improves the segmentation performance of the network. The experimental results show that the improvement is effective, and the MCCT-UNet model outperforms the other 9 network models. Specifically, the MCCT-UNet achieved a Dice coefficient of 0.8734, an IoU of 0.7854, and an accuracy of 0.9977, demonstrating significant improvements over the compared models.