Feature Fusion Module Research Articles

Cross-Granularity Infrared Image Segmentation Network for Nighttime Marine Observations

Infrared image segmentation in marine environments is crucial for enhancing nighttime observations and ensuring maritime safety. While recent advancements in deep learning have significantly improved segmentation accuracy, challenges remain due to nighttime marine scenes including low contrast and noise backgrounds. This paper introduces a cross-granularity infrared image segmentation network CGSegNet designed to address these challenges specifically for infrared images. The proposed method designs a hybrid feature framework with cross-granularity to enhance segmentation performance in complex water surface scenarios. To suppress feature semantic disparity against different feature granularity, we propose an adaptive multi-scale fusion module (AMF) that combines local granularity extraction with global context granularity. Additionally, incorporating a handcrafted histogram of oriented gradients (HOG) features, we designed a novel HOG feature fusion module to improve edge detection accuracy under low-contrast conditions. Comprehensive experiments conducted on the public infrared segmentation dataset demonstrate that our method outperforms state-of-the-art techniques, achieving superior segmentation results compared to professional infrared image segmentation methods. The results highlight the potential of our approach in facilitating accurate infrared image segmentation for nighttime marine observation, with implications for maritime safety and environmental monitoring.

Multimodal Data Fusion for Precise Lettuce Phenotype Estimation Using Deep Learning Algorithms

Effective lettuce cultivation requires precise monitoring of growth characteristics, quality assessment, and optimal harvest timing. In a recent study, a deep learning model based on multimodal data fusion was developed to estimate lettuce phenotypic traits accurately. A dual-modal network combining RGB and depth images was designed using an open lettuce dataset. The network incorporated both a feature correction module and a feature fusion module, significantly enhancing the performance in object detection, segmentation, and trait estimation. The model demonstrated high accuracy in estimating key traits, including fresh weight (fw), dry weight (dw), plant height (h), canopy diameter (d), and leaf area (la), achieving an R2 of 0.9732 for fresh weight. Robustness and accuracy were further validated through 5-fold cross-validation, offering a promising approach for future crop phenotyping.

Learning spatial‐frequency interaction for generalizable deepfake detection

AbstractIn recent years, face forgery detection has gained significant attention, resulting in considerable advancements. However, most existing methods rely on CNNs to extract artefacts from the spatial domain, overlooking the pervasive frequency‐domain artefacts present in deepfake content, which poses challenges in achieving robust and generalized detection. To address these issues, we propose the dual‐stream frequency—spatial fusion network is proposed for deepfake detection. The dual‐stream frequency‐spatial fusion network consists of three components: the spatial forgery feature extraction module, the frequency forgery feature extraction module, and the spatial–frequency feature fusion module. The spatial forgery feature extraction module employs spatial‐channel attention to extract spatial domain features, targeting artefacts in the spatial domain. The frequency forgery feature extraction module leverages the focused linear attention to detect frequency domain anomalies in internal regions, enabling the identification of generated content. The spatial–frequency feature fusion module then fuses forgery features extracted from both the spatial and frequency domains, facilitating accurate detection of splicing artefacts and internally generated forgeries. This approach enhances the model's ability to more accurately capture forgery characteristics. Extensive experiments on several widely‐used benchmarks demonstrate that our carefully designed network exhibits superior generalization and robustness, significantly improving deepfake detection performance.

No-Reference Quality Assessment Based on Dual-Channel Convolutional Neural Network for Underwater Image Enhancement

Underwater images are important for underwater vision tasks, yet their quality often degrades during imaging, promoting the generation of Underwater Image Enhancement (UIE) algorithms. This paper proposes a Dual-Channel Convolutional Neural Network (DC-CNN)-based quality assessment method to evaluate the performance of different UIE algorithms. Specifically, inspired by the intrinsic image decomposition, the enhanced underwater image is decomposed into reflectance with color information and illumination with texture information based on the Retinex theory. Afterward, we design a DC-CNN with two branches to learn color and texture features from reflectance and illumination, respectively, reflecting the distortion characteristics of enhanced underwater images. To integrate the learned features, a feature fusion module and attention mechanism are conducted to align efficiently and reasonably with human visual perception characteristics. Finally, a quality regression module is used to establish the mapping relationship between the extracted features and quality scores. Experimental results on two public enhanced underwater image datasets (i.e., UIQE and SAUD) show that the proposed DC-CNN method outperforms a variety of the existing quality assessment methods.

I-DINO: High-Quality Object Detection for Indoor Scenes

Object Detection in Complex Indoor Scenes is designed to identify and categorize objects in indoor settings, with applications in areas such as smart homes, security surveillance, and home service robots. It forms the basis for advanced visual tasks including visual question answering, video description generation, and instance segmentation. Nonetheless, the task faces substantial hurdles due to background clutter, overlapping objects, and significant size differences. To tackle these challenges, this study introduces an indoor object detection approach utilizing an enhanced DINO framework. To cater to the needs of indoor object detection, an Indoor-COCO dataset was developed from the COCO object detection dataset. The model incorporates an advanced Res2net as the backbone feature extraction network, complemented by a deformable attention mechanism to better capture detailed object features. An upgraded Bi-FPN module is employed to replace the conventional feature fusion module, and SIoU loss is utilized to expedite convergence. The experimental outcomes indicate that the refined model attains an mAP of 62.3%, marking a 5.2% improvement over the baseline model. These findings illustrate that the DINO-based indoor object detection model exhibits robust generalization abilities and practical utility for multi-scale object detection in complex environments.

A Deep Learning Model for Accurate Maize Disease Detection Based on State-Space Attention and Feature Fusion

In improving agricultural yields and ensuring food security, precise detection of maize leaf diseases is of great importance. Traditional disease detection methods show limited performance in complex environments, making it challenging to meet the demands for precise detection in modern agriculture. This paper proposes a maize leaf disease detection model based on a state-space attention mechanism, aiming to effectively utilize the spatiotemporal characteristics of maize leaf diseases to achieve efficient and accurate detection. The model introduces a state-space attention mechanism combined with a multi-scale feature fusion module to capture the spatial distribution and dynamic development of maize diseases. In experimental comparisons, the proposed model demonstrates superior performance in the task of maize disease detection, achieving a precision, recall, accuracy, and F1 score of 0.94. Compared with baseline models such as AlexNet, GoogLeNet, ResNet, EfficientNet, and ViT, the proposed method achieves a precision of 0.95, with the other metrics also reaching 0.94, showing significant improvement. Additionally, ablation experiments verify the impact of different attention mechanisms and loss functions on model performance. The standard self-attention model achieved a precision, recall, accuracy, and F1 score of 0.74, 0.70, 0.72, and 0.72, respectively. The Convolutional Block Attention Module (CBAM) showed a precision of 0.87, recall of 0.83, accuracy of 0.85, and F1 score of 0.85, while the state-space attention module achieved a precision of 0.95, with the other metrics also at 0.94. In terms of loss functions, cross-entropy loss showed a precision, recall, accuracy, and F1 score of 0.69, 0.65, 0.67, and 0.67, respectively. Focal loss showed a precision of 0.83, recall of 0.80, accuracy of 0.81, and F1 score of 0.81. State-space loss demonstrated the best performance in these experiments, achieving a precision of 0.95, with recall, accuracy, and F1 score all at 0.94. These results indicate that the model based on the state-space attention mechanism achieves higher detection accuracy and better generalization ability in the task of maize leaf disease detection, effectively improving the accuracy and efficiency of disease recognition and providing strong technical support for the early diagnosis and management of maize diseases. Future work will focus on further optimizing the model’s spatiotemporal feature modeling capabilities and exploring multi-modal data fusion to enhance the model’s application in real agricultural scenarios.

A hybrid local-global neural network for visual classification using raw EEG signals

EEG-based brain-computer interfaces (BCIs) have the potential to decode visual information. Recently, artificial neural networks (ANNs) have been used to classify EEG signals evoked by visual stimuli. However, methods using ANNs to extract features from raw signals still perform lower than traditional frequency-domain features, and the methods are typically evaluated on small-scale datasets at a low sample rate, which can hinder the capabilities of deep-learning models. To overcome these limitations, we propose a hybrid local-global neural network, which can be trained end-to-end from raw signals without handcrafted features. Specifically, we first propose a reweight module to learn channel weights adaptively. Then, a local feature extraction module is designed to capture basic EEG features. Next, a spatial integration module fuses information from each electrode, and a global feature extraction module integrates overall time-domain characteristics. Additionally, a feature fusion module is proposed to extract efficient features in high sampling rate settings. The proposed model achieves state-of-the-art results on two commonly used small-scale datasets and outperforms baseline methods on three under-studied large-scale datasets. Ablation experimental results demonstrate that the proposed modules have a stable performance improvement ability on multiple datasets across different sample rates, providing a robust end-to-end learning framework.

Research on Target Hybrid Recognition and Localization Methods Based on an Industrial Camera and a Depth Camera in Complex Scenes

In order to improve the target visual recognition and localization accuracy of robotic arms in complex scenes with similar targets, hybrid recognition and localization methods based on an industrial camera and depth camera are proposed. First, according to the speed and accuracy requirements of target recognition and localization, YOLOv5s is introduced as the basic algorithm model for target hybrid recognition and localization. Then, in order to improve the accuracy of target recognition and coarse localization based on an industrial camera (eye-to-hand), the AFPN feature fusion module, simple and parameter-free attention module (SimAM), and soft non-maximum suppression (Soft NMS) are introduced. In order to improve the accuracy of target recognition and fine localization based on a depth camera (eye-in-hand), the SENetV2 backbone network structure, dynamic head module, deformable attention mechanism, and chain-of-thought prompted adaptive enhancer network are introduced. After that, on the basis of constructing a dual camera platform for target hybrid recognition and localization, the hand–eye calibration, collection and production of image datasets required for model training are completed. Finally, for the docking of the oil filling port, the hybrid recognition and localization experimental tests are completed in sequence. The test results show that in target recognition and coarse localization based on the industrial camera, the recognition accuracy of the designed model reaches 99%, and the average localization errors in the horizontal and vertical directions are 2.22 mm and 3.66 mm, respectively. In target recognition and fine localization based on the depth camera, the recognition accuracy of the designed model reaches 98%, and the average errors in depth, horizontal, and vertical directions are 0.12 mm, 0.28 mm, and 0.16 mm, respectively. These not only verify the effectiveness of the target hybrid recognition and localization methods based on dual cameras, but also demonstrate that they meet the high-precision recognition and localization requirements in complex scenes.

A Dual-Stream Fusion Network for Human Energy Expenditure Estimation with Wearable Sensor

With the increasing awareness of health, using wearable sensors to monitor individual activities and accurately estimate energy expenditure has become a current research focus. However, existing research encounters challenges including low estimation accuracy, a deficiency of frequency domain features, and difficulty in integrating time domain and frequency domain features. To address these issues, we propose an innovative framework called the Dual-Stream Fusion Network (DSFN). This framework combines the Time Domain Encoding (TDE) module, the Frequency Domain Hierarchical-Split Encoding (FDHSE) module, and a Two-Stage Feature Fusion (TSF) module. Specifically, the temporal stream of the framework employs the TDE module to capture deep temporal features that reflect the complex dynamic variations in time-series data. The frequency domain stream introduces the FDHSE module, which extracts frequency domain features using a multi-level, multi-scale approach, ensuring a comprehensive and diverse representation of frequency information. Through this dual-stream architecture, our model effectively learns both time and frequency domain features, addressing the limitations of frequency domain features observed in prior studies. Additionally, we propose the TSF module to fully integrate time and frequency domain features, effectively overcoming the challenge of fusing these two types of features. We conducted experiments on two public datasets, namely the GOTOV dataset (elderly people) and the JSI dataset (young people). Experimental results demonstrate that our method achieves excellent performance across different age groups. Compared to the baseline models, the proposed DSFN significantly improves the accuracy of human energy expenditure estimation.

Drainage Pipeline Defect Detection System Based on Semantic Segmentation

Since drainage pipes are widely used in daily life and are necessary for cities to function normally, intelligent fault detection in these pipes has emerged as both an urgent necessity and a major area of development. Compared to traditional target detection methods, the semantic segmentation-based pipeline defect detection method has some special qualities. It can effectively detect a variety of pipeline defect types and segment the defects using precise geometrical attributes to support subsequent defect assessment. This study replaces the backbone network and fuses bar pooling and cascade feature fusion modules in the encoding and decoding phases, respectively, to obtain richer semantic information. This approach strikes a balance between segmentation accuracy and model complexity, aiming to address the problems of the large computational volume of semantic segmentation and the loss of detailed semantic information of pipeline defects. In the pipe defect dataset, the proposed approach is compared with the baseline algorithm and the experimental findings indicate that it has a greater segmentation accuracy than the conventional algorithm. Additionally, the lightweight design of the algorithm lowers the model’s complexity.

MMS-EF: A Multi-Scale Modular Extraction Framework for Enhancing Deep Learning Models in Remote Sensing

The analysis of land cover using deep learning techniques plays a pivotal role in understanding land use dynamics, which is crucial for land management, urban planning, and cartography. However, due to the complexity of remote sensing images, deep learning models face practical challenges in the preprocessing stage, such as incomplete extraction of large-scale geographic features, loss of fine details, and misalignment issues in image stitching. To address these issues, this paper introduces the Multi-Scale Modular Extraction Framework (MMS-EF) specifically designed to enhance deep learning models in remote sensing applications. The framework incorporates three key components: (1) a multiscale overlapping segmentation module that captures comprehensive geographical information through multi-channel and multiscale processing, ensuring the integrity of large-scale features; (2) a multiscale feature fusion module that integrates local and global features, facilitating seamless image stitching and improving classification accuracy; and (3) a detail enhancement module that refines the extraction of small-scale features, enriching the semantic information of the imagery. Extensive experiments were conducted across various deep learning models, and the framework was validated on two public datasets. The results demonstrate that the proposed approach effectively mitigates the limitations of traditional preprocessing methods, significantly improving feature extraction accuracy and exhibiting strong adaptability across different datasets.

Research on Fault Diagnosis of Rotating Parts Based on Transformer Deep Learning Model

The rotating parts of large and complex equipment are key components that ensure the normal operation of the equipment. Accurate fault diagnosis is crucial for the safe operation of these systems. To simultaneously extract both local and global valuable fault feature information from key components of complex equipment, this study proposes a fault diagnosis network model, named MultiDilatedFormer, which is based on the fusion of transformer and multi-head dilated convolution. The newly designed multi-head dilated convolution module is sequentially integrated into the transformer-encoder architecture, constructing a feature extraction module where the complementary advantages of both components enhance overall performance. Firstly, the sample is expanded into a two-dimensional feature map and then input into the newly designed feature extraction module. Finally, the diagnostic output is performed by the designed patch feature fusion module and classifier module. Additionally, interpretability research is conducted on the proposed model, aiming to understand the decision-making mechanism of the model through visual analysis of the entire decision process. The experimental results on three different datasets indicate that the proposed model achieved high accuracy in fault diagnosis with relatively short data windows. The highest accuracy reached 97.95%, which was up to 10.97% higher than other models. Furthermore, the feasibility of the model is also verified in the actual dataset of the rotating parts of the injection molding machine. The excellent performance of the model on different datasets demonstrates its effectiveness in extracting comprehensive fault feature information and also proves its great potential in practical industrial applications.

Enhanced light field depth estimation through occlusion refinement and feature fusion

Light field depth estimation is crucial for various applications, but current algorithms often falter when dealing with complex textures and edges. To address this, we propose a light field depth estimation network based on multi-scale fusion and channel attention (LFMCNet). It incorporates a convolutional multi-scale fusion module to enhance feature extraction and utilizes a channel attention mechanism to refine depth map accuracy. Additionally, LFMCNet integrates the Transformer Feature Fusion Module (TFFM) and Channel Attention-Based Perspective Fusion (CAPF) module for improved occlusion refinement, effectively handling challenges in occluded regions. Testing on the 4D HCI and real-world datasets demonstrates that LFMCNet significantly reduces the Bad Pixel (BP) rate and Mean Square Error (MSE).

NFSA-DTI: A Novel Drug-Target Interaction Prediction Model Using Neural Fingerprint and Self-Attention Mechanism.

Existing deep learning methods have shown outstanding performance in predicting drug-target interactions. However, they still have limitations: (1) the over-reliance on locally extracted features by some single encoders, with insufficient consideration of global features, and (2) the inadequate modeling and learning of local crucial interaction sites in drug-target interaction pairs. In this study, we propose a novel drug-target interaction prediction model called the Neural Fingerprint and Self-Attention Mechanism (NFSA-DTI), which effectively integrates the local information of drug molecules and target sequences with their respective global features. The neural fingerprint method is used in this model to extract global features of drug molecules, while the self-attention mechanism is utilized to enhance CNN's capability in capturing the long-distance dependencies between the subsequences in the target amino acid sequence. In the feature fusion module, we improve the bilinear attention network by incorporating attention pooling, which enhances the model's ability to learn local crucial interaction sites in the drug-target pair. The experimental results on three benchmark datasets demonstrated that NFSA-DTI outperformed all baseline models in predictive performance. Furthermore, case studies illustrated that our model could provide valuable insights for drug discovery. Moreover, our model offers molecular-level interpretations.

Disentangled feature fusion network for lightweight image super-resolution

Recently, the quality of generated images in image super-resolution (SR) has significantly improved due to the widespread application of convolutional neural networks. Existing super-resolution methods often overlook the importance of network width and instead prioritize designing deeper network structures to improve performance. Deeper networks, however, demand greater computational resources and are more challenging to train, making them less suitable for devices with limited performance. To address this issue, we propose a novel method called the disentangled feature fusion network (DFFN). Specifically, we construct dual feature extraction streams (DFES) using the strategies of feature disentanglement and parameter sharing, which increase the network's width without increasing the number of parameters. To facilitate the interaction of information between the dual feature extraction streams, we design a feature interaction module (FIM) that separately enhances high and low-frequency features. Furthermore, a feature fusion module (FFM) is presented to efficiently fuse different levels of high and low-frequency feature information. The proposed DFFN integrates DFES, FIM, and FFM, which not only widens the network without increasing parameters but also minimizes the loss of crucial feature information. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms other state-of-the-art lightweight super-resolution methods, achieving a superior balance between parameter quantity, reconstruction performance, and model complexity. The code is available at https://github.com/zhoujy-aust/DFFN.

Long Short Observation-driven Prediction Network for pedestrian crossing intention prediction with momentary observation

Pedestrian crossing intention prediction aims to predict whether the pedestrian will cross the road, which is crucial for the decision-making of intelligent vehicles and ensuring traffic safety. Existing methods just rely on long-term observation and rarely consider it challenging to obtain sufficiently long and precise observation in real-world scenarios. Focus on momentary observation, which only contains two frames of the preceding and current time, we propose a novel Long Short Observation-driven Prediction Network (LSOP-Net). LSOP-Net comprises two critical components, the Momentary Observation feature Extraction Module (MOE-Module) and the Multimodal Long-Short-term feature Fusion Module (MLSFusion). Utilizing a hybrid training strategy and an external long-term feature pool, the MOE-Module is proposed to extract features with long-term patterns from momentary observations, which effectively mitigates feature deficiency arising from momentary observations. Based on a feature selection fusion mechanism, the MLSFusion is proposed to explicitly model the importance relationship between various modalities’ long-short-term features and the output, which adaptively fuses the long-short-term features from various modalities. Experimental results on the JAAD and PIE datasets demonstrate that our approach achieves superior performance in pedestrian crossing intention prediction with momentary observation.

AFDFusion: An adaptive frequency decoupling fusion network for multi-modality image

The multi-modality image fusion goal is to create a single image that provides a comprehensive scene description and conforms to visual perception by integrating complementary information about the merits of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images. Although some works explore decoupled representations of multi-modality images, they struggle with complex nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an adaptive frequency decoupling module to perceive the associative invariant and inherent specific among cross-modality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize a contrastive learning loss for restricting the solution space of feature decoupling to learn representations of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling, low frequency features, which are similar in the representation space, should be pulled closer to each other, signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II, two feature fusion modules are added to integrate the invariant and specific features and output the fused image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.

Attention-based CNN model for motor imagery classification from nonlinear EEG signals

Motor imagery (MI)-based brain-computer interface (BCI) provides a promising solution for the limb rehabilitation of stroke patients. For better rehabilitation performance, high-precision classification of MI-related EEG signals plays a critical role. However, this is still a challenging problem for multi-category MI signals. In this paper, we focus on four commonly used stroke rehabilitation actions, and propose a modular temporal-spatial attention-based CNN (MTSACNN) model for MI classification. In detail, we carry out the MI experiments and acquire the EEG signals related to imagining left/right fist clenching and left/right wrist dorsiflexion. MTSACNN model firstly extracts the low-order MI features through the temporal-spatial feature extraction module (TSFE module). Especially, a group attention mechanism is proposed for intra-group information interaction. Secondly, considering the short- and long-term working characteristics of brain, high-order temporal features are further extracted and fused by the multi-level feature fusion module (MLFF module). Finally, four auxiliary losses are arranged in the classification module (C module) to speed up the model optimization process. The experimental results show that MTSACNN model can achieve good performance in decoding rehabilitation-related MI brain intentions, achieving an average classification accuracy of 72.05% for fourteen subjects. This work is beneficial to promote the construction of high-performance stroke rehabilitation BCI system.

CATNet: A Cross Attention and Texture‐Aware Network for Polyp Segmentation

ABSTRACTPolyp segmentation is a challenging task, as some polyps exhibit similar textures to surrounding tissues, making them difficult to distinguish. Therefore, we present a parallel cross‐attention and texture‐aware network to address this challenging task. CATNet incorporates the parallel cross‐attention mechanism, Residual Feature Fusion Module, and texture‐aware module. Initially, polyp images undergo processing in our backbone network to extract multi‐level polyp features. Subsequently, the parallel cross‐attention mechanism sequentially captures channel and spatial dependencies across multi‐scale polyp features, thereby yielding enhanced representations. These enhanced representations are then input into multiple texture‐aware modules, which facilitate polyp segmentation by accentuating subtle textural disparities between polyps and the background. Finally, the Residual Feature Fusion module integrates the segmentation results with the previous layer of enhanced representations. This process serves to eliminate background noise and enhance intricate details. We assess the efficacy of our proposed method across five distinct polyp datasets. On three unseen datasets, CVC‐300, CVC‐ColonDB, and ETIS. We achieve mDice scores of 0.916, 0.817, and 0.777, respectively. Experimental results unequivocally demonstrate the superior performance of our approach over current models. The proposed CATNet addresses the challenges posed by textural similarities, setting a benchmark for future advancements in automated polyp detection and segmentation.

Frequency line detection in spectrograms using a deep neural network with attention.

In this paper, a frequency line detection network (FLDNet) is proposed to effectively detect multiple weak frequency lines and time-varying frequency lines in underwater acoustic signals under low signal-to-noise ratios (SNRs). FLDNet adopts an encoder-decoder architecture as the basic framework, where the encoder is designed to obtain multilevel features of the frequency lines, and the decoder is responsible for reconstructing the frequency lines. FLDNet includes attention-based feature fusion modules that combine deep semantic features with shallow features learned by the encoder to reduce noise in the decoder's deep feature representation and improve reconstruction accuracy. In addition, a composite loss function was constructed by using the continuity of frequency lines, which improved the detection performance of frequency lines. After training through simulated signal sets, FLDNet can effectively detect frequency lines in spectrograms of simulated and measured signals. The experimental results indicate that FLDNet is superior to other state-of-the-art methods, even at SNRs as low as -28 dB.