Feature Fusion Network Research Articles

Lightweight Water Surface Object Detection Network for Unmanned Surface Vehicles

The detection algorithms for water surface objects considerably assist unmanned surface vehicles in rapidly perceiving their surrounding environment, providing essential environmental information and evaluating object attributes. This study proposes a lightweight water surface target detection algorithm called YOLO-WSD (water surface detection), based on YOLOv8n, to address the need for real-time, high-precision, and lightweight target detection algorithms that can adapt to the rapid changes in the surrounding environment during specific tasks. Initially, we designed the C2F-E module, enriched in gradient flow compared to the conventional C2F module, enabling the backbone network to extract richer multi-level features while maintaining lightness. Additionally, this study redesigns the feature fusion network structure by introducing low-level features and achieving multi-level fusion to enhance the network’s capability of integrating multiple levels. Meanwhile, it investigates the impact of channel number differences in the Concat module fusion on model performance, thereby optimizing the neural network structure. Lastly, it introduces the WIOU localization loss function to bolster model robustness. Experiments demonstrated that YOLO-WSD achieves a 4.6% and 3.4% improvement in mAP0.5 on the water surface object detection dataset and Seaship public dataset, respectively, with recall rates improving by 5.4% and 8.5% relative to the baseline YOLOv8n model. The model’s parameter size is 3.3 M. YOLO-WSD exhibits superior performance compared to other mainstream lightweight algorithms.

Fine-grained sentiment analysis using multidimensional feature fusion and GCN

ABSTRACT The demand for analysing sentiment information in social media data is increasing. However, current fine-grained sentiment analysis methods fail to consider both global and local semantic features simultaneously, leading to the oversight of grammatical information in sentences and an inability to address the issue of polysemy. To address these challenges, we propose a microblog fine-grained sentiment analysis model based on multidimensional feature fusion and graph convolutional neural networks (GCN). Built upon the ALBERT model, we utilize BiLSTM and capsule network models to extract global and local semantic features, thereby capturing bidirectional semantic dependencies and textual positional semantic information. Finally, we employ multi-head self-attention and GCN to select key features and sentence information, ensuring the integrity of fine-grained features. The experimental results indicate that the model outperforms several other models on fine-grained sentiment analysis datasets SMP2020-EWECT, NLPCC2013, NLPCC2014, and the binary classification dataset weibo_senti_100k, achieving accuracies of 80.64%, 67.19%, 71.37%, and 98.43%, respectively.

Improved keypoint localization network for tea bud based on YOLO framework

The precise localization of picking points is the primary challenge that should be addressed by the robotic tea bud harvester. The efficient identification and accurate positioning of picking points directly affects the efficiency of tea buds harvesting. Compared with the target detection and semantic segmentation algorithms, the keypoint detection algorithms do not require to incorporate target shape features for accurately localizing the picking point, which allows to significantly decrease the computational load, and thus facilitates the subsequent point cloud processing. This study introduces a tea bud keypoint detection network referred to as TBKNet. This model uses the concept of structural reparameterization to reconstruct the CSPDarknet53 backbone network, and constructs an asymptotic parallel feature fusion network by incorporating the CA attention mechanism into the asymptotic feature pyramid network. In addition, in order to accelerate the convergence of the model, a keypoint regression loss function, which considers the angle and Euclidean distance, is proposed. The experimental results show that TBNKet reaches the highest accuracy with an average precision of 87.1 % for keypoint detection. It has 5.7 %, 16.9 %, and 3.3 % higher mAP values compared with the DEKR, YOLOPose, and YOLOPv8-Pose models, respectively. This is efficient in identifying and localizing keypoints of tea buds, and it can provide guidance for similar intensive harvesting tasks.

Multisource information fusion method for vegetable disease detection

Automated detection and identification of vegetable diseases can enhance vegetable quality and increase profits. Images of greenhouse-grown vegetable diseases often feature complex backgrounds, a diverse array of diseases, and subtle symptomatic differences. Previous studies have grappled with accurately pinpointing lesion positions and quantifying infection degrees, resulting in overall low recognition rates. To tackle the challenges posed by insufficient validation datasets and low detection and recognition rates, this study capitalizes on the geographical advantage of Shouguang, renowned as the “Vegetable Town,” to establish a self-built vegetable base for data collection and validation experiments. Concentrating on a broad spectrum of fruit and vegetable crops afflicted with various diseases, we conducted on-site collection of greenhouse disease images, compiled a large-scale dataset, and introduced the Space-Time Fusion Attention Network (STFAN). STFAN integrates multi-source information on vegetable disease occurrences, bolstering the model’s resilience. Additionally, we proposed the Multilayer Encoder-Decoder Feature Fusion Network (MEDFFN) to counteract feature disappearance in deep convolutional blocks, complemented by the Boundary Structure Loss function to guide the model in acquiring more detailed and accurate boundary information. By devising a detection and recognition model that extracts high-resolution feature representations from multiple sources, precise disease detection and identification were achieved. This study offers technical backing for the holistic prevention and control of vegetable diseases, thereby advancing smart agriculture. Results indicate that, on our self-built VDGE dataset, compared to YOLOv7-tiny, YOLOv8n, and YOLOv9, the proposed model (Multisource Information Fusion Method for Vegetable Disease Detection, MIFV) has improved mAP by 3.43%, 3.02%, and 2.15%, respectively, showcasing significant performance advantages. The MIFV model parameters stand at 39.07 M, with a computational complexity of 108.92 GFLOPS, highlighting outstanding real-time performance and detection accuracy compared to mainstream algorithms. This research suggests that the proposed MIFV model can swiftly and accurately detect and identify vegetable diseases in greenhouse environments at a reduced cost.

A temperature measurement compensation method for industrial rotary kilns based on infrared multi-feature fusion under dynamic water mist interference

Accurate temperature measurement is vital for assessing the reaction atmosphere of industrial rotary kilns. However, dynamic water mist interference imposes significant challenges to accurate infrared temperature measurement. To this end, this study proposes a temperature measurement compensation method based on infrared multi-feature fusion. First, we developed an industrial system to collect infrared data from the kiln. To address the issue of unclear water mist in infrared images caused by its low resolution and weak texture features, we designed an artificial feature based on infrared images and temperature differences named water mist level (WML). This feature effectively represents the size of the water mist in the image. Additionally, a classification model is established to identify the WML automatically. Subsequently, a novel multi-scale feature fusion network called efficient regression feature pyramid network (ERFPN) is proposed to acquire multi-scale image features. Finally, we propose a grouped feature fusion network (GFFN) to fuse multi-scale image features, WML, and interfered temperature. Experimental results show that the proposed compensation method achieves satisfactory infrared temperature measurement results and significantly reduces the temperature measurement error caused by water mist.

Global and local complementary multi-path feature fusion network for the classification of crop remote sensing images

Abstract The accuracy and efficiency of crop distribution information extraction are pivotal in ensuring global food security. In long-time-series optical satellite data, most existing methods focus on extracting spatial features using Convolutional Neural Networks (CNNs), which do not adequately mine and model the spatial-temporal information. The development of the attention mechanism allows for the extraction of global features in remote-sensing images of long temporal sequences. To extract global attentional features with complementary features in crop remote sensing images, we propose a Global and Local Complementary Multi-path Feature Fusion Network (GLMP), which is capable of extracting global features from remote sensing images of long temporal sequence, that enhances the local characteristics of crop images derived from CNNs, thus obtaining more effective multi-scale complementary features. This extraction of features enhances the comprehension of crop images, thereby boosting the performance of associated tasks. Within GLMP, we introduce two pivotal modules: the Hybrid Attention and Convolutional Paths Module (HACM) and the Multi-path Feature Fusion Module (MPFM). These modules synergistically converge multi-path features, yielding more discriminative feature information. Experimental results on the ZueriCrop dataset show that the proposed GLMP technique is effective; it performs promisingly having a total accuracy of 90.2% and an F1 value of 62.5%. Furthermore, the ablation study verifies the substantial improvement in classification accuracy for remote sensing crop images of long-time series in nature, specifically attributed to the HACM and MPFM modules.

Haze generation and feature fusion network aiming at real-world single image dehazing

Abstract Because most dehazing methods use synthesized haze images for training, they may perform well on synthesized datasets. However, when these methods are applied to real-world scenes, their performance may significantly decrease due to domain shift. Therefore, we propose a dehazing network for real-world hazy scenes. This network includes a haze generation network that can utilize the hazy information of real haze images to generate images that are closer to real hazy scenes, generating training pairs to address the domain shift problem. The network also includes a dehazing network that integrates feature fusion attention mechanisms, which can achieve better dehazing performance.

Accurate UAV Small Object Detection Based on HRFPN and EfficentVMamba.

(1) Background: Small objects in Unmanned Aerial Vehicle (UAV) images are often scattered throughout various regions of the image, such as the corners, and may be blocked by larger objects, as well as susceptible to image noise. Moreover, due to their small size, these objects occupy a limited area in the image, resulting in a scarcity of effective features for detection. (2) Methods: To address the detection of small objects in UAV imagery, we introduce a novel algorithm called High-Resolution Feature Pyramid Network Mamba-Based YOLO (HRMamba-YOLO). This algorithm leverages the strengths of a High-Resolution Network (HRNet), EfficientVMamba, and YOLOv8, integrating a Double Spatial Pyramid Pooling (Double SPP) module, an Efficient Mamba Module (EMM), and a Fusion Mamba Module (FMM) to enhance feature extraction and capture contextual information. Additionally, a new Multi-Scale Feature Fusion Network, High-Resolution Feature Pyramid Network (HRFPN), and FMM improved feature interactions and enhanced the performance of small object detection. (3) Results: For the VisDroneDET dataset, the proposed algorithm achieved a 4.4% higher Mean Average Precision (mAP) compared to YOLOv8-m. The experimental results showed that HRMamba achieved a mAP of 37.1%, surpassing YOLOv8-m by 3.8% (Dota1.5 dataset). For the UCAS_AOD dataset and the DIOR dataset, our model had a mAP 1.5% and 0.3% higher than the YOLOv8-m model, respectively. To be fair, all the models were trained without a pre-trained model. (4) Conclusions: This study not only highlights the exceptional performance and efficiency of HRMamba-YOLO in small object detection tasks but also provides innovative solutions and valuable insights for future research.

A duplex transform heterogeneous feature fusion network for road segmentation

Detecting roads in automatic driving environments poses a challenge due to issues such as boundary fuzziness, occlusion, and glare from light. We believe that two factors are instrumental in addressing these challenges and enhancing detection performance: global context dependency and effective feature representation that prioritizes important feature channels. To tackle these issues, we introduce DTRoadseg, a novel duplex Transformer-based heterogeneous feature fusion network designed for road segmentation. DTRoadseg leverages a duplex encoder architecture to extract heterogeneous features from both RGB images and point-cloud depth images. Subsequently, we introduce a multi-source Heterogeneous Feature Reinforcement Block (HFRB) for fusion of the encoded features, comprising a Heterogeneous Feature Fusion Module (HFFM) and a Reinforcement Fusion Module (RFM). The HFFM leverages the self-attention mechanisms of Transformers to achieve effective fusion through token interactions, while the RFM focuses on emphasizing informative features while downplaying less important ones, thereby reinforcing feature fusion. Finally, a Transformer decoder is utilized to produce the final semantic prediction. Furthermore, we employ a boundary loss function to optimize the segmentation structure area, reduce false detection areas, and improve model accuracy. Extensive experiments are carried out on the KITTI road dataset. The results demonstrate that, compared with state-of-the-art methods, DTRoadseg exhibits superior performance, achieving an average accuracy of 97.01%, a Recall of 96.35%, and runs at a speed of 0.09 s per picture.

FuseNet: a multi-modal feature fusion network for 3D shape classification

FuseNet: a multi-modal feature fusion network for 3D shape classification

Multi-layer adaptive spatial-temporal feature fusion network for efficient food image recognition

Numerous deep learning methods have been developed to tackle the challenges of recognizing food images, including convolutional neural networks, deep feature extraction, and deep feature fusion methods. This research proposes a new architecture called ASTFF-Net that uses deep feature fusion to tackle various challenges in food recognition, including similarity patterns between two categories, multi-object problems, light conditions, camera position, noise objects, and blurred images. ASTFF-Net is a robust and adaptive spatial–temporal fusion network designed to address these challenges effectively. The ASTFF-Net architecture consisted of three networks. In the spatial feature extraction network, the ResNet50 architecture was used to extract robust spatial features, and the reduction operation was utilized to minimize parameter size. Subsequently, the spatial features were passed through a 1D convolution (Conv1D) to fit the features into the recurrent neural networks. In the temporal feature extraction network, the spatial features were given to the long short-term memory, allowing the network to learn from various long sequence patterns. In the adaptive feature fusion network, the robust spatial and temporal features were fused and assigned to the Conv1D, followed by the softmax function. The ASTFF-Net architecture is also intended to decrease the number of network parameters and prevent overfitting problems. Experimental results on four benchmark food image datasets: Food11, UEC Food-100, UEC Food-256, and ETH Food-101, demonstrate that the proposed ASTFF-Nets, particularly ASTFF-NetB3, were more competitive compared with other existing methods.

A novel simulation-assisted transfer method for bearing unknown fault diagnosis

Abstract Supervised data-driven bearing fault diagnosis methods rely on completed datasets of faults, which can be challenging for signals collected in real engineering. Recognizing unknown faults using a data-driven approach is particularly difficult, as purposefully modeling these faults is complex. To address this challenge, this study proposes a new simulation-assisted transfer bearing unknown fault diagnosis method for realizing unknown compound fault diagnosis of rotating machinery. Firstly, finite element method is used to obtain the compound fault data that does not exist in the historical data, and wavelet packet transform is performed on the simulated and measured signals to enhance the detailed features of the signals. Then, a deep convolutional feature fusion network based on hybrid multi-wavelet spatial attention is constructed to fuse the time-frequency information processed by different wavelet bases. Finally, by integrating the concepts of intra-class splitting and transfer learning, the model is fine-tuned using simulation data to recognize unknown compound faults of rolling bearings. The method validates the simulated signals’ feasibility and the unknown faults’ diagnostic validity under the publicly available rolling bearings dataset. Compared to the comparison methods, the method’s accuracy increased by 2.86%, 2.61%, 5.41%, 4.77%, and 7.07%, respectively.

Incorporating long-tail data in complex backgrounds for visual surface defect detection in PCBs

AbstractHigh-quality printed circuit boards (PCBs) are essential components in modern electronic circuits. Nevertheless, most of the existing methods for PCB surface defect detection neglect the fact that PCB surface defects in complex backgrounds are prone to long-tailed data distributions, which in turn affects the effectiveness of defect detection. Additionally, most of the existing methods ignore the intra-scale features of defects and do not utilize auxiliary supervision strategies to improve the detection performance of the network. To tackle these issues, we propose a lightweight long-tailed data mining network (LLM-Net) for identifying PCB surface defects. Firstly, the proposed Efficient Feature Fusion Network (EFFNet) is applied to embed intra-scale feature associations and multi-scale features of defects into LLM-Net. Next, an auxiliary supervision method with a soft label assignment strategy is designed to help LLM-Net learn more accurate defect features. Finally, the issue of inadequate tail data detection is addressed by employing the devised Binary Cross-Entropy Loss Rank Mining method (BCE-LRM) to identify challenging samples. The performance of LLM-Net was evaluated on a homemade dataset of PCB surface soldering defects, and the results show that LLM-Net achieves the best accuracy of mAP@0.5 for the evaluation metric of the COCO dataset, and it has a real-time inference speed of 188 frames per second (FPS).

TTAFPred: Prediction of time to aging failure for software systems based on a two-stream multi-scale features fusion network

TTAFPred: Prediction of time to aging failure for software systems based on a two-stream multi-scale features fusion network

One-Year-Old Precocious Chinese Mitten Crab Identification Algorithm Based on Task Alignment.

The cultivation of the Chinese mitten crab (Eriocheir sinensis) is an important component of China's aquaculture industry and also a field of concern worldwide. It focuses on the selection of high-quality, disease-free juvenile crabs. However, the early maturity rate of more than 18.2% and the mortality rate of more than 60% make it difficult to select suitable juveniles for adult culture. The juveniles exhibit subtle distinguishing features, and the methods for differentiating between sexes vary significantly; without training from professional breeders, it is challenging for laypersons to identify and select the appropriate juveniles. Therefore, we propose a task-aligned detection algorithm for identifying one-year-old precocious Chinese mitten crabs, named R-TNET. Initially, the required images were obtained by capturing key frames, and then they were annotated and preprocessed by professionals to build a training dataset. Subsequently, the ResNeXt network was selected as the backbone feature extraction network, with Convolutional Block Attention Modules (CBAMs) and a Deformable Convolution Network (DCN) embedded in its residual blocks to enhance its capability to extract complex features. Adaptive spatial feature fusion (ASFF) was then integrated into the feature fusion network to preserve the detailed features of small targets such as one-year-old precocious Chinese mitten crab juveniles. Finally, based on the detection head proposed by task-aligned one-stage object detection, the parameters of its anchor alignment metric were adjusted to detect, locate, and classify the crab juveniles. The experimental results showed that this method achieves a mean average precision (mAP) of 88.78% and an F1-score of 97.89%. This exceeded the best-performing mainstream object detection algorithm, YOLOv7, by 4.17% in mAP and 1.77% in the F1-score. Ultimately, in practical application scenarios, the algorithm effectively identified one-year-old precocious Chinese mitten crabs, providing technical support for the automated selection of high-quality crab juveniles in the cultivation process, thereby promoting the rapid development of aquaculture and agricultural intelligence in China.

Surface Defect Detection for Aerospace Aluminum Profiles with Attention Mechanism and Multi-Scale Features

A YOLOv5 aluminum profile defect detection algorithm that integrates attention and multi-scale features is proposed in this paper to address the issues of the low detection accuracy, high false detection rates, and high missed detection rates that are caused by the large-scale variation of surface defects, inconspicuous small defect characteristics, and a lack of concentrated feature information in defect areas. Firstly, an improved CBAM (Channel-Wise Attention Module) convolutional attention module is employed, which effectively focuses on the feature information of defect areas in the aluminum defect dataset with only a small amount of spatial dimension. Secondly, a bidirectional weighted feature fusion network is utilized, incorporating a multi-scale feature fusion network with skip connections to aggregate various high-resolution features, thus enriching the semantic expression of features. Then, new size feature maps that have not been fused are introduced into the detection layer network to improve the detection effect of small target defects. Experimental results indicate that an average detection accuracy (mAP) of 82.6% was achieved by the improved YOLOv5 algorithm on the aluminum surface defect dataset. An improvement of 6.2% over the previous version was observed. The current defect detection requirements of aluminum profile production sites are met by this enhanced algorithm.

TDF-Net: Trusted Dynamic Feature Fusion Network for breast cancer diagnosis using incomplete multimodal ultrasound

Ultrasound is a critical imaging technique for diagnosing breast cancer. However, the multimodal breast ultrasound diagnostic process is time-consuming and labor-intensive, heavily dependent on the physician’s extensive expertise. Therefore, developing a computer-aided diagnosis system for breast cancer is essential. Existing diagnostic systems fail to consider the varying impacts of different ultrasound modalities on diagnostic results and struggle to address the issue of missing modalities in clinical practice. Consequently, this paper proposes the Trusted Dynamic Feature Fusion Network (TDF-Net) for diagnosing breast cancer using incomplete multimodal ultrasound data. Initially, this method introduces a dual-branch feature extraction module to capture modality-specific information. Meanwhile, a contrastive clustering loss is designed to enforce the consistency constraint, ensuring the coherence of different modal features within the semantic space for each sample. Additionally, an invertible neural network-based recovery method is suggested to establish mappings between different modalities, enabling the recovery of missing modalities. Finally, a trusted dynamic feature fusion module based on the Dirichlet distribution is proposed to quantify each modality’s contribution to the diagnostic result by considering uncertainty, thereby achieving the dynamic fusion of each modality’s features across different samples. The proposed method is validated on an established multimodal breast ultrasound dataset, demonstrating superior diagnostic performance compared to existing methods, with an average AUC of 98.31%, a 95% confidence interval of [96.98%, 99.64%], and a p-value < 0.05. A pilot study is planned to assess the effectiveness and usability of TDF-Net in clinical settings.

SPT-UNet: A Superpixel-Level Feature Fusion Network for Water Extraction from SAR Imagery

SPT-UNet: A Superpixel-Level Feature Fusion Network for Water Extraction from SAR Imagery

Hyperspectral image classification based on adaptive spectral feature decoupling with global local feature fusion network

Hyperspectral image classification based on adaptive spectral feature decoupling with global local feature fusion network

Instance Segmentation of Tea Garden Roads Based on an Improved YOLOv8n-seg Model

In order to improve the efficiency of fine segmentation and obstacle removal in the road of tea plantation in hilly areas, a lightweight and high-precision DR-YOLO instance segmentation algorithm is proposed to realize environment awareness. Firstly, the road data of tea gardens in hilly areas were collected under different road conditions and light conditions, and data sets were generated. YOLOv8n-seg, which has the highest operating efficiency, was selected as the basic model. The MSDA-CBAM and DR-Neck feature fusion network were added to the YOLOv8-seg model to improve the feature extraction capability of the network and the feature fusion capability and efficiency of the model. Experimental results show that, compared with the YOLOv8-seg model, the DR-YOLO model proposed in this study has 2.0% improvement in AP@0.5 and 1.1% improvement in Precision. In this study, the DR-YOLO model is pruned and quantitatively compressed, which greatly improves the model inference speed with little reduction in AP. After deploying on Jetson, compared with the YOLOv8n-seg model, the Precision of DR-YOLO is increased by 0.6%, the AP@0.5 is increased by 1.6%, and the inference time is reduced by 17.1%, which can effectively improve the level of agricultural intelligent automation and realize the efficient operation of the instance segmentation model at the edge.