Feature Fusion Research Articles

A hybrid local-global neural network for visual classification using raw EEG signals

EEG-based brain-computer interfaces (BCIs) have the potential to decode visual information. Recently, artificial neural networks (ANNs) have been used to classify EEG signals evoked by visual stimuli. However, methods using ANNs to extract features from raw signals still perform lower than traditional frequency-domain features, and the methods are typically evaluated on small-scale datasets at a low sample rate, which can hinder the capabilities of deep-learning models. To overcome these limitations, we propose a hybrid local-global neural network, which can be trained end-to-end from raw signals without handcrafted features. Specifically, we first propose a reweight module to learn channel weights adaptively. Then, a local feature extraction module is designed to capture basic EEG features. Next, a spatial integration module fuses information from each electrode, and a global feature extraction module integrates overall time-domain characteristics. Additionally, a feature fusion module is proposed to extract efficient features in high sampling rate settings. The proposed model achieves state-of-the-art results on two commonly used small-scale datasets and outperforms baseline methods on three under-studied large-scale datasets. Ablation experimental results demonstrate that the proposed modules have a stable performance improvement ability on multiple datasets across different sample rates, providing a robust end-to-end learning framework.

Identification of sweetpotato virus disease-infected leaves from field images using deep learning

IntroductionSweetpotato virus disease (SPVD) is widespread and causes significant economic losses. Current diagnostic methods are either costly or labor-intensive, limiting both efficiency and scalability.MethodsThe segmentation algorithm proposed in this study can rapidly and accurately identify SPVD lesions from field-captured photos of sweetpotato leaves. Two custom datasets, DS-1 and DS-2, are utilized, containing meticulously annotated images of sweetpotato leaves affected by SPVD. DS-1 is used for training, validation, and testing the model, while DS-2 is exclusively employed to validate the model’s reliability. This study employs a deep learning-based semantic segmentation network, DeepLabV3+, integrated with an Attention Pyramid Fusion (APF) module. The APF module combines a channel attention mechanism with multi-scale feature fusion to enhance the model’s performance in disease pixel segmentation. Additionally, a novel data augmentation technique is utilized to improve recognition accuracy in the edge background areas of real large images, addressing issues of poor segmentation precision in these regions. Transfer learning is applied to enhance the model’s generalization capabilities.ResultsThe experimental results indicate that the model, with 62.57M parameters and 253.92 Giga Floating Point Operations Per Second (GFLOPs), achieves a mean Intersection over Union (mIoU) of 94.63% and a mean accuracy (mAcc) of 96.99% on the DS-1 test set, and an mIoU of 78.59% and an mAcc of 79.47% on the DS-2 dataset.DiscussionAblation studies confirm the effectiveness of the proposed data augmentation and APF methods, while comparative experiments demonstrate the model’s superiority across various metrics. The proposed method also exhibits excellent detection results in simulated scenarios. In summary, this study successfully deploys a deep learning framework to segment SPVD lesions from field images of sweetpotato foliage, which will contribute to the rapid and intelligent detection of sweetpotato diseases.

Research on Target Hybrid Recognition and Localization Methods Based on an Industrial Camera and a Depth Camera in Complex Scenes

In order to improve the target visual recognition and localization accuracy of robotic arms in complex scenes with similar targets, hybrid recognition and localization methods based on an industrial camera and depth camera are proposed. First, according to the speed and accuracy requirements of target recognition and localization, YOLOv5s is introduced as the basic algorithm model for target hybrid recognition and localization. Then, in order to improve the accuracy of target recognition and coarse localization based on an industrial camera (eye-to-hand), the AFPN feature fusion module, simple and parameter-free attention module (SimAM), and soft non-maximum suppression (Soft NMS) are introduced. In order to improve the accuracy of target recognition and fine localization based on a depth camera (eye-in-hand), the SENetV2 backbone network structure, dynamic head module, deformable attention mechanism, and chain-of-thought prompted adaptive enhancer network are introduced. After that, on the basis of constructing a dual camera platform for target hybrid recognition and localization, the hand–eye calibration, collection and production of image datasets required for model training are completed. Finally, for the docking of the oil filling port, the hybrid recognition and localization experimental tests are completed in sequence. The test results show that in target recognition and coarse localization based on the industrial camera, the recognition accuracy of the designed model reaches 99%, and the average localization errors in the horizontal and vertical directions are 2.22 mm and 3.66 mm, respectively. In target recognition and fine localization based on the depth camera, the recognition accuracy of the designed model reaches 98%, and the average errors in depth, horizontal, and vertical directions are 0.12 mm, 0.28 mm, and 0.16 mm, respectively. These not only verify the effectiveness of the target hybrid recognition and localization methods based on dual cameras, but also demonstrate that they meet the high-precision recognition and localization requirements in complex scenes.

Lightweight defect detection algorithm of tunnel lining based on knowledge distillation

Due to the influence of construction quality, engineering geology and hydrological environment, defects such as dehollowing and insufficient compaction can occur in tunnels. Aiming at the problems of complex detection model, poor real-time performance and low accuracy of the current tunnel lining defect detection methods, the study proposes a lightweight defect detection algorithm of tunnel lining based on knowledge distillation. Firstly, a high-precision teacher model based on yolov5s was constructed by constructing a C3CSFM module that combines residual structure and attention mechanism, a MDFPN network structure with multi-scale feature fusion and a reweighted RWNMS re-screening mechanism. Secondly, in the distillation process, the feature and output dimension results are fused to improve the detection accuracy, and the mask feature relationship is learned in the space and channel dimension to improve the real-time detection. Tests on the tunnel lining radar defect image dataset showed that the number of parameters of the improved model was reduced from 16.03 MB to 3.20 MB, a reduction of 80%, and the average accuracy was improved from 83.4 to 86.5%, an increase of 3.1%. On the basis of maintaining the structure and detection performance of the model, the lightweight degree of the model is greatly improved, and the high-precision and real-time detection of tunnel lining defects is realized.

A Multimodal Image Registration Method for UAV Visual Navigation Based on Feature Fusion and Transformers

A Multimodal Image Registration Method for UAV Visual Navigation Based on Feature Fusion and Transformers

Research of anomaly detection based on dynamic anomaly detection enhancement framework

Abstract Anomaly detection plays a crucial role in various fields, from industrial defect inspection to geological detection. However, traditional approaches often struggle with insufficient discriminability and an inability to generalize to unseen anomalies. These limitations stem from the practical difficulty in gathering a comprehensive set of anomalies and the tendency to overlook anomalous instances in favor of normal samples. To address these challenges, we propose a novel Dynamic Anomaly Detection Enhancement Framework, integrating three key innovations: (1) SaliencyAug: An adaptive saliency-guided augmentation method that generates realistic pseudo-samples to enhance learning of rare anomalies, improving model generalization. (2) DynAB: A dynamic attention block that achieves effective multi-level feature fusion while minimizing redundant information, enhancing detection accuracy. (3) DualOM: A dual-head optimization module which employs separate heads for normal and anomalous sample learning, creating more explicit and discriminative decision boundaries. Extensive experiments across multiple real-world datasets demonstrate our framework's superior performance in detecting a wide range of anomalies, demonstrating 2.4% improvement over state-of-the-art methods.

An unpaired SAR-to-optical image translation method based on Schrödinger bridge network and multi-scale feature fusion

SAR-to-optical (S2O) translation is able to covert SAR into optical images, which help the interpreter to extract information efficiently. In the absence of strictly matched datasets, it is difficult for existing methods to complete training on unpaired data with a minimum amount of data. By employing the recent Schrödinger bridge-based transformation framework, a multiscale axial residual module (MARM) based on the concept of multi-scale feature fusion has been proposed in this paper. To enable efficient translation of SAR to optical images, the generator and discriminator of the model have been designed. Extensive experiments on the SEN1-2 dataset conducted, and the results show the superiority of the proposed method in terms of the generation quality. Compared with the classical CycleGAN, the proposed method can improve the FID metrics by 42.05%.

A Lightweight and Efficient Multimodal Feature Fusion Network for Bearing Fault Diagnosis in Industrial Applications

To address the issues of single-structured feature input channels, insufficient feature learning capabilities in noisy environments, and large model parameter sizes in intelligent diagnostic models for mechanical equipment, a lightweight and efficient multimodal feature fusion convolutional neural network (LEMFN) method is proposed. Compared with existing models, LEMFN captures rich fault features at multiple scales by combining time-domain and frequency-domain signals, thereby enhancing the model’s robustness to noise and improving data adaptability under varying operating conditions. Additionally, the convolutional block attention module (CBAM) and random overlapping sampling technology (ROST) are introduced, and through a feature fusion strategy, the accurate diagnosis of mechanical equipment faults is achieved. Experimental results demonstrate that the proposed method not only possesses high diagnostic accuracy and rapid convergence but also exhibits strong robustness in noisy environments. Finally, a graphical user interface (GUI)-based mechanical equipment fault detection system was developed to promote the practical application of intelligent fault diagnosis in mechanical equipment.

A Dual-Stream Fusion Network for Human Energy Expenditure Estimation with Wearable Sensor

With the increasing awareness of health, using wearable sensors to monitor individual activities and accurately estimate energy expenditure has become a current research focus. However, existing research encounters challenges including low estimation accuracy, a deficiency of frequency domain features, and difficulty in integrating time domain and frequency domain features. To address these issues, we propose an innovative framework called the Dual-Stream Fusion Network (DSFN). This framework combines the Time Domain Encoding (TDE) module, the Frequency Domain Hierarchical-Split Encoding (FDHSE) module, and a Two-Stage Feature Fusion (TSF) module. Specifically, the temporal stream of the framework employs the TDE module to capture deep temporal features that reflect the complex dynamic variations in time-series data. The frequency domain stream introduces the FDHSE module, which extracts frequency domain features using a multi-level, multi-scale approach, ensuring a comprehensive and diverse representation of frequency information. Through this dual-stream architecture, our model effectively learns both time and frequency domain features, addressing the limitations of frequency domain features observed in prior studies. Additionally, we propose the TSF module to fully integrate time and frequency domain features, effectively overcoming the challenge of fusing these two types of features. We conducted experiments on two public datasets, namely the GOTOV dataset (elderly people) and the JSI dataset (young people). Experimental results demonstrate that our method achieves excellent performance across different age groups. Compared to the baseline models, the proposed DSFN significantly improves the accuracy of human energy expenditure estimation.

A lightweight deep-learning model for parasite egg detection in microscopy images.

Intestinal parasitic infections are still a serious public health problem in developing countries, and the diagnosis of parasitic infections requires the first step of parasite/egg detection of samples. Automated detection can eliminate the dependence on professionals, but the current detection algorithms require large computational resources, which increases the lower limit of automated detection. Therefore, we have designed a lightweight deep-learning model, YAC-Net, to achieve rapid and accurate detection of parasitic eggs and reduce the cost of automation. This paper uses the ICIP 2022 Challenge dataset for experiments, and the experiments are conducted using fivefold cross-validation. The YOLOv5n model is used as the baseline model, and then two improvements are made to the baseline model based on the specificity of the egg data. First, the neck of the YOLOv5n is modified to from a feature pyramid network (FPN) to an asymptotic feature pyramid network (AFPN) structure. Different from the FPN structure, which mainly integrates semantic feature information at adjacent levels, the hierarchical and asymptotic aggregation structure of AFPN can fully fuse the spatial contextual information of egg images, and its adaptive spatial feature fusion mode can help the model select beneficial feature and ignore redundant information, thereby reducing computational complexity and improving detection performance. Second, the C3 module of the backbone of the YOLOv5n is modified to a C2f module, which can enrich gradient information, improving the feature extraction capability of the backbone. Moreover, ablation studies are designed by us to verify the effectiveness of the AFPN and C2f modules in the process of model lightweighting. The experimental results show that compared with YOLOv5n, YAC-Net improves precision by 1.1%, recall by 2.8%, the F1 score by 0.0195, and mAP_0.5 by 0.0271 and reduces the parameters by one-fifth. Compared with some state-of-the-art detection methods, YAC-Net achieves the best performance in precision, F1 score, mAP_0.5, and parameters. The precision, recall, F1 score, mAP_0.5, and parameters of our method on the test set are 97.8%, 97.7%, 0.9773, 0.9913, and 1,924,302, respectively. Compared with the baseline model, YAC-Net optimizes the model structure and simplifies the parameters while ensuring the detection performance. It helps to reduce the equipment requirements for performing automated detection and can be used to realize the automatic detection of parasite eggs under microscope images.

Study of conveyor belt deviation detection based on improved YOLOv8 algorithm

Conveyor belt deviation is a commmon and severe type of fault in belt conveyor systems, often resulting in significant economic losses and potential environment pollution. Traditional detection methods have obvious limitations in fault localization precision and analysis accuracy, unable to meet the demands of efficient and real-time fault detection in complex industrial scenarios. To address these issues, this paper proposes an improved detection algorithm based on YOLOv8, aiming to achieve efficient and accurate detection during the operation of the belt. Firstly, an Enhanced Squeeze-and-Excitation (ESE) module is incorporated into C2f to boost feature extraction for rollers and belts. Secondly, the construction of the BiFPN_DoubleAttention module in the neck network enhances bidirectional feature fusion and attention mechanism, thereby improving multi-scale object localization accuracy under complex environments. Then, a Multi-Head Self-Attention (MHSA) mechanism is introduced in the head network, better capturing positional features of small roller targets and belt areas in various environments, thus enhancing detection performance. Finally, extensive experiments are conducted on a self-built dataset, achieving an accuracy of 98.1%, mAP0.5 of 99.0%, and a detection speed of 46 frames per second (FPS). This method effectively handles variations and disturbances in the conveyor belt transportation environment, meeting real-time diagnostic needs for belt misalignment in the industry.

Drainage Pipeline Defect Detection System Based on Semantic Segmentation

Since drainage pipes are widely used in daily life and are necessary for cities to function normally, intelligent fault detection in these pipes has emerged as both an urgent necessity and a major area of development. Compared to traditional target detection methods, the semantic segmentation-based pipeline defect detection method has some special qualities. It can effectively detect a variety of pipeline defect types and segment the defects using precise geometrical attributes to support subsequent defect assessment. This study replaces the backbone network and fuses bar pooling and cascade feature fusion modules in the encoding and decoding phases, respectively, to obtain richer semantic information. This approach strikes a balance between segmentation accuracy and model complexity, aiming to address the problems of the large computational volume of semantic segmentation and the loss of detailed semantic information of pipeline defects. In the pipe defect dataset, the proposed approach is compared with the baseline algorithm and the experimental findings indicate that it has a greater segmentation accuracy than the conventional algorithm. Additionally, the lightweight design of the algorithm lowers the model’s complexity.

M-IPISincNet: An explainable multi-source physics-informed neural network based on improved SincNet for rolling bearings fault diagnosis

Timely and accurate diagnosis of bearing faults can effectively reduce the chance of accidents in equipment. However, deep learning methods are mostly completely dependent on data and lack interpretability. It is difficult to deal with the differences between real-time data and training data under changing working conditions and noisy environments. In this study, we proposed M-IPISincNet, an explainability multi-source physics-informed convolutional network based on improved SincNet. Rolling bearing fault diagnosis is realized by extracting fault features from vibration and current signals. Firstly, a physics-informed convolutional layer is designed based on inverse Fourier transform and bandpass filters. Fault features are extracted by multi-scale convolution and multi-layer nonlinear mapping. A DBN network is applied extract unsupervised hidden fusion features in the vibration and current signals. The proposed method is validated under the datasets of Paderborn University (PU) and Case Western Reserve University (CWRU), which proves that the proposed method has explainability, robustness and great accuracy under multiple working conditions and noises.

Synergistic use of handcrafted and deep learning features for tomato leaf disease classification.

This research introduces a Computer-Aided Diagnosis-system designed aimed at automated detections & classification of tomato leaf diseases, combining traditional handcrafted features with advanced deep learning techniques. The system's process encompasses preprocessing, feature extraction, feature fusion, and classification. It utilizes enhancement filters and segmentation algorithms to isolate with Regions-of-Interests (ROI) in images tomato leaves. These features based arranged in ABCD rule (Asymmetry, Borders, Colors, and Diameter) are integrated with outputs from a Convolutional Neural Network (CNN) pretrained on ImageNet. To address data imbalance, we introduced a novel evaluation method that has shown to improve classification accuracy by 15% compared to traditional methods, achieving an overall accuracy rate of 92% in field tests. By merging classical feature engineering with modern machine learning techniques under mutual information-based feature fusion, our system sets a new standard for precision in agricultural diagnostics. Specific performance metrics showcasing the effectiveness of our approach in automated detection and classifying of tomato leaf disease.

MMS-EF: A Multi-Scale Modular Extraction Framework for Enhancing Deep Learning Models in Remote Sensing

The analysis of land cover using deep learning techniques plays a pivotal role in understanding land use dynamics, which is crucial for land management, urban planning, and cartography. However, due to the complexity of remote sensing images, deep learning models face practical challenges in the preprocessing stage, such as incomplete extraction of large-scale geographic features, loss of fine details, and misalignment issues in image stitching. To address these issues, this paper introduces the Multi-Scale Modular Extraction Framework (MMS-EF) specifically designed to enhance deep learning models in remote sensing applications. The framework incorporates three key components: (1) a multiscale overlapping segmentation module that captures comprehensive geographical information through multi-channel and multiscale processing, ensuring the integrity of large-scale features; (2) a multiscale feature fusion module that integrates local and global features, facilitating seamless image stitching and improving classification accuracy; and (3) a detail enhancement module that refines the extraction of small-scale features, enriching the semantic information of the imagery. Extensive experiments were conducted across various deep learning models, and the framework was validated on two public datasets. The results demonstrate that the proposed approach effectively mitigates the limitations of traditional preprocessing methods, significantly improving feature extraction accuracy and exhibiting strong adaptability across different datasets.

YOLOX-S-TKECB: A Holstein Cow Identification Detection Algorithm

Accurate identification of individual cow identity is a prerequisite for the construction of digital farms and serves as the basis for optimized feeding, disease prevention and control, breed improvement, and product quality traceability. Currently, cow identification faces challenges such as poor recognition accuracy, large data volumes, weak model generalization ability, and low recognition speed. Therefore, this paper proposes a cow identification method based on YOLOX-S-TKECB. (1) Based on the characteristics of Holstein cows and their breeding practices, we constructed a real-time acquisition and preprocessing platform for two-dimensional Holstein cow images and built a cow identification model based on YOLOX-S-TKECB. (2) Transfer learning was introduced to improve the convergence speed and generalization ability of the cow identification model. (3) The CBAM attention mechanism module was added to enhance the model’s ability to extract features from cow torso patterns. (4) The alignment between the apriori frame and the target size was improved by optimizing the clustering algorithm and the multi-scale feature fusion method, thereby enhancing the performance of object detection at different scales. The experimental results demonstrate that, compared to the traditional YOLOX-S model, the improved model exhibits a 15.31% increase in mean average precision (mAP) and a 32-frame boost in frames per second (FPS). This validates the feasibility and effectiveness of the proposed YOLOX-S-TKECB-based cow identification algorithm, providing valuable technical support for the application of dairy cow identification in farms.

YOLOv5DA: An Improved YOLOv5 Model for Posture Detection of Grouped Pigs

Accurate posture detection is the foundation for analyzing animal behavior, which can promote animal welfare. With the development of computer vision, such technology has been widely used in analyzing animal behavior without physical contact. However, computer vision technology for pig posture detection often suffers from problems of missed or false detection due to complex scenarios. To solve the problem, this study proposed a novel object detection model YOLOv5DA, which was based on YOLOv5s and designed for pig posture detection from 2D camera video. Firstly, we established the annotated dataset (7220 images) including the training set (5776 images), validation set (722 images), and test set (722 images). Secondly, an object detection model YOLOv5DA based on YOLOv5s was proposed to recognize pig postures (standing, prone lying, and side lying), which incorporated Mosaic9 data augmentation, deformable convolution, and adaptive spatial feature fusion. The comparative and ablation experiments were conducted to verify the model’s effectiveness and reliability. Finally, we used YOLOv5DA to detect the posture distribution of pigs. The results revealed that the standing posture was more frequent in the morning and afternoon and the side-lying posture was most common at noon. This observation demonstrated that the posture of pigs is influenced by temperature variations. The study demonstrated that YOLOv5DA could accurately identify three postures of standing, prone lying, and side lying with an average precision (AP) of 99.4%, 99.1%, and 99.1%, respectively. Compared with YOLOv5s, YOLOv5DA could effectively handle occlusion while increasing the mean precision (mAP) by 1.7%. Overall, our work provided a highly accurate, effective, low-cost, and non-contact strategy of posture detection in grouped pigs, which can be used to monitor pig behavior and assist in the early prevention of disease.

Spatio-temporal graph neural network based on time series periodic feature fusion for traffic flow prediction

Spatio-temporal graph neural network based on time series periodic feature fusion for traffic flow prediction

Autonomous Extraction Technology for Aquaculture Ponds in Complex Geological Environments Based on Multispectral Feature Fusion of Medium-Resolution Remote Sensing Imagery

Coastal aquaculture plays a crucial role in global food security and the economic development of coastal regions, but it also causes environmental degradation in coastal ecosystems. Therefore, the automation, accurate extraction, and monitoring of coastal aquaculture areas are crucial for the scientific management of coastal ecological zones. This study proposes a novel deep learning- and attention-based median adaptive fusion U-Net (MAFU-Net) procedure aimed at precisely extracting individually separable aquaculture ponds (ISAPs) from medium-resolution remote sensing imagery. Initially, this study analyzes the spectral differences between aquaculture ponds and interfering objects such as saltwater fields in four typical aquaculture areas along the coast of Liaoning Province, China. It innovatively introduces a difference index for saltwater field aquaculture zones (DIAS) and integrates this index as a new band into remote sensing imagery to increase the expressiveness of features. A median augmented adaptive fusion module (MEA-FM), which adaptively selects channel receptive fields at various scales, integrates the information between channels, and captures multiscale spatial information to achieve improved extraction accuracy, is subsequently designed. Experimental and comparative results reveal that the proposed MAFU-Net method achieves an F1 score of 90.67% and an intersection over union (IoU) of 83.93% on the CHN-LN4-ISAPS-9 dataset, outperforming advanced methods such as U-Net, DeepLabV3+, SegNet, PSPNet, SKNet, UPS-Net, and SegFormer. This study’s results provide accurate data support for the scientific management of aquaculture areas, and the proposed MAFU-Net method provides an effective method for semantic segmentation tasks based on medium-resolution remote sensing images.

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation

CFFANet: category feature fusion and attention mechanism network for retinal vessel segmentation