Multi-scale Fusion Module Research Articles

PM-YOLO: A Powdery Mildew Automatic Grading Detection Model for Rubber Tree

Powdery mildew has become a significant disease affecting the yield and quality of rubber trees in recent years. It typically manifests on the leaf surface at an early stage, rapidly infecting and spreading throughout the leaves. Therefore, early detection and intervention are essential to reduce the resulting losses due to this disease. However, the conventional methods of disease detection are both time-consuming and labor-intensive. In this study, we proposed a novel deep-learning-based approach for detecting powdery mildew in rubber trees, even in complex backgrounds. First, to address the lack of existing datasets on rubber tree powdery mildew, we constructed a dataset comprising 6200 images and 38,000 annotations. Second, based on the YOLO framework, we integrated a multi-scale fusion module that combines a Feature Focus and Diffusion Mechanism (FFDM) into the neck of the detection architecture. We designed an overall focus diffusion architecture and introduced a Dimension-Aware Selective Integration (DASI) module to enhance the detection of small powdery mildew targets, naming the model PM-YOLO. Furthermore, we proposed an automatic grading detection algorithm to evaluate the severity of powdery mildew on rubber tree leaves. The experimental results demonstrated that the proposed method achieved 86.9% mean average precision (mAP) and 85.6% recall, which outperformed the standard YOLOv10 by 7.6% mAP and 8.2% recall. This approach offered accurate and real-time detection of powdery mildew rubber trees, providing an effective solution for early diagnosis through automated grading.

Cross-Granularity Infrared Image Segmentation Network for Nighttime Marine Observations

Infrared image segmentation in marine environments is crucial for enhancing nighttime observations and ensuring maritime safety. While recent advancements in deep learning have significantly improved segmentation accuracy, challenges remain due to nighttime marine scenes including low contrast and noise backgrounds. This paper introduces a cross-granularity infrared image segmentation network CGSegNet designed to address these challenges specifically for infrared images. The proposed method designs a hybrid feature framework with cross-granularity to enhance segmentation performance in complex water surface scenarios. To suppress feature semantic disparity against different feature granularity, we propose an adaptive multi-scale fusion module (AMF) that combines local granularity extraction with global context granularity. Additionally, incorporating a handcrafted histogram of oriented gradients (HOG) features, we designed a novel HOG feature fusion module to improve edge detection accuracy under low-contrast conditions. Comprehensive experiments conducted on the public infrared segmentation dataset demonstrate that our method outperforms state-of-the-art techniques, achieving superior segmentation results compared to professional infrared image segmentation methods. The results highlight the potential of our approach in facilitating accurate infrared image segmentation for nighttime marine observation, with implications for maritime safety and environmental monitoring.

Laparoscopic stereo matching using 3-Dimensional Fourier transform with full multi-scale features

3-Dimensional (3D) reconstruction of laparoscopic surgical scenes is a key task for future surgical navigation and automated robotic minimally invasive surgery. Binocular laparoscopy with stereo matching enables 3D reconstruction. Stereo matching models used for natural images such as autopilot tend to be less suitable for laparoscopic environments due to the constraints of small samples of laparoscopic images, complex textures, and uneven illumination. In addition, current stereo matching modules use 3D convolutions and transformers in the spatial domain as the base module, which is limited by the ability to learn in the spatial domain. In this paper, we propose a model for laparoscopic stereo matching using 3D Fourier Transform combined with Full Multi-scale Features (FT-FMF Net). Specifically, the proposed Full Multi-scale Fusion Module (FMFM) is able to fuse the full multi-scale feature information from the feature extractor into the stereo matching block, which densely learns the feature information with parallax and FMFM fusion information in the frequency domain using the proposed Dense Fourier Transform Module (DFTM). We validated the proposed method in both the laparoscopic dataset (SCARED) and the endoscopic dataset (SERV-CT). In comparison with other popular and advanced deep learning models available at present, FT-FMF Net achieves the most advanced stereo matching performance available. In the SCARED and SERV-CT public datasets, the End-Point-Error (EPE) was 0.7265 and 2.3119, and the Root Mean Square Error Depth (RMSE Depth) was 4.00 mm and 3.69 mm, respectively. In addition, the inference time is only 0.17s. Our project code is available on https://github.com/wurenkai/FT-FMF.

Enhancing wildfire detection: a novel algorithm for controllable generation of wildfire smoke images

Background The lack of wildfire smoke image data is one of the most important factors hindering the development of image-based wildfire detection. Smoke image generation based on image inpainting techniques is a solution worthy of study. However, it is difficult to generate smoke texture with context consistency in complex backgrounds with current image inpainting methods. Aims This work aims to provide a wildfire smoke image database for specific scenarios. Methods We designed an algorithm based on generative adversarial networks (GANs) to generate smoke images. The algorithm includes a multi-scale fusion module to ensure consistency between the generated smoke and backgrounds. Additionally, a local feature-matching mechanism in the discriminator guides the generator to capture real smoke’s feature distribution. Key results We generated 13,400 wildfire smoke images based on forest background images and early fire simulation from the Fire Dynamics Simulator (FDS). Conclusions A variety of advanced object detection algorithms were trained based on the generated data. The experimental results confirmed that the addition of the generated data to the real datasets can effectively improve model performance. Implications This study paves a way for generating object datasets to enhance the reliability of watchtower or satellite wildfire monitoring.

Enhanced light field depth estimation through occlusion refinement and feature fusion

Light field depth estimation is crucial for various applications, but current algorithms often falter when dealing with complex textures and edges. To address this, we propose a light field depth estimation network based on multi-scale fusion and channel attention (LFMCNet). It incorporates a convolutional multi-scale fusion module to enhance feature extraction and utilizes a channel attention mechanism to refine depth map accuracy. Additionally, LFMCNet integrates the Transformer Feature Fusion Module (TFFM) and Channel Attention-Based Perspective Fusion (CAPF) module for improved occlusion refinement, effectively handling challenges in occluded regions. Testing on the 4D HCI and real-world datasets demonstrates that LFMCNet significantly reduces the Bad Pixel (BP) rate and Mean Square Error (MSE).

Background-Aware Cross-Attention Multiscale Fusion for Multispectral Object Detection

Limited by the imaging capabilities of sensors, research based on single modality is difficult to cope with faults and dynamic perturbations in detection. Effective multispectral object detection, which can achieve better detection accuracy by fusing visual information from different modalities, has attracted widespread attention. However, most of the existing methods adopt simple fusion mechanisms, which fail to utilize the complementary information between modalities while lacking the guidance of a priori knowledge. To address the above issues, we propose a novel background-aware cross-attention multiscale fusion network (BA-CAMF Net) to achieve adaptive fusion in visible and infrared images. First, a background-aware module is designed to calculate the light and contrast to guide the fusion. Then, a cross-attention multiscale fusion module is put forward to enhance inter-modality complement features and intra-modality intrinsic features. Finally, multiscale feature maps from different modalities are fused according to background-aware weights. Experimental results on LLVIP, FLIR, and VEDAI indicate that the proposed BA-CAMF Net achieves higher detection accuracy than the current State-of-the-Art multispectral detectors.

PZS‐Net: Incorporating of Frame Sequence and Multi‐Scale Priors for Prostate Zonal Segmentation in Transrectal Ultrasound

Transrectal ultrasound (TRUS) videos offer valuable histopathologic information about the prostate. Accurate prostate zonal segmentation in TRUS videos is vital for diagnosing prostate cancer and guiding surgery. However, TRUS videos are manually recorded by urologists, resulting in no standardized coordinate system, which limits direct prostate zonal segmentation in these videos. To overcome the limitation, a novel Prostate Zonal Segmentation Network (PZS‐Net), based on U‐Net, which learns critical cross‐frame information and multi‐scale features from sequential frames, is proposed. First, a sequential frame cross‐attention (SFCA) module is designed to capture remote information from sequential frames to enhance the feature representation of the current frame. The SFCA module is embedded at each skip connection layer to extract crucial cross‐frame information. Then, a multi‐scale fusion (MSF) module that utilizes three parallel branches with different atrous convolutions is designed. The MSF module is placed at the bottleneck layer to dynamically fuse multi‐scale context information from high‐level features. Extensive experiments on TRUS image datasets show that the PZS‐Net achieves higher accuracy in both the transitional zone (dice coefficient [Dice]: 68.90% ± 1.73%, mean intersection over union [mIoU]: 59.19% ± 2.09%, 95% Hausdorff distance [HD95]: 5.02 ± 0.83 mm) and the peripheral zone (Dice: 63.99% ± 3.16%, mIoU: 54.60% ± 3.35%, HD95: 5.28 ± 1.12 mm) and demonstrates the effectiveness and competitiveness of its key components via comprehensive ablation studies.

Channel Interaction and Transformer Depth Estimation Network: Robust Self-Supervised Depth Estimation Under Varied Weather Conditions

Monocular depth estimation provides low-cost environmental information for intelligent systems such as autonomous vehicles and robots, supporting sustainable development by reducing reliance on expensive, energy-intensive sensors and making technology more accessible and efficient. However, in practical applications, monocular vision is highly susceptible to adverse weather conditions, significantly reducing depth perception accuracy and limiting its ability to deliver reliable environmental information. To improve the robustness of monocular depth estimation in challenging weather, this paper first utilizes generative models to adjust image exposure and generate synthetic images of rainy, foggy, and nighttime scenes, enriching the diversity of the training data. Next, a channel interaction module and Multi-Scale Fusion Module are introduced. The former enhances information exchange between channels, while the latter effectively integrates multi-level feature information. Finally, an enhanced consistency loss is added to the loss function to prevent the depth estimation bias caused by data augmentation. Experiments on datasets such as DrivingStereo, Foggy CityScapes, and NuScenes-Night demonstrate that our method, CIT-Depth, exhibits superior generalization across various complex conditions.

Verdiff-Net: A Conditional Diffusion Framework for Spinal Medical Image Segmentation.

Spinal medical image segmentation is critical for diagnosing and treating spinal disorders. However, ambiguity in anatomical boundaries and interfering factors in medical images often cause segmentation errors. Current deep learning models cannot fully capture the intrinsic data properties, leading to unstable feature spaces. To tackle the above problems, we propose Verdiff-Net, a novel diffusion-based segmentation framework designed to improve segmentation accuracy and stability by learning the underlying data distribution. Verdiff-Net integrates a multi-scale fusion module (MSFM) for fine feature extraction and a noise semantic adapter (NSA) to refine segmentation masks. Validated across four multi-modality spinal datasets, Verdiff-Net achieves a high Dice coefficient of 93%, demonstrating its potential for clinical applications in precision spinal surgery.

Multi-Scale Frequency-Adaptive-Network-Based Underwater Target Recognition

Due to the complexity of underwater environments, underwater target recognition based on radiated noise has always been challenging. This paper proposes a multi-scale frequency-adaptive network for underwater target recognition. Based on the different distribution densities of Mel filters in the low-frequency band, a three-channel improved Mel energy spectrum feature is designed first. Second, by combining a frequency-adaptive module, an attention mechanism, and a multi-scale fusion module, a multi-scale frequency-adaptive network is proposed to enhance the model’s learning ability. Then, the model training is optimized by introducing a time–frequency mask, a data augmentation strategy involving data confounding, and a focal loss function. Finally, systematic experiments were conducted based on the ShipsEar dataset. The results showed that the recognition accuracy for five categories reached 98.4%, and the accuracy for nine categories in fine-grained recognition was 88.6%. Compared with existing methods, the proposed multi-scale frequency-adaptive network for underwater target recognition has achieved significant performance improvement.

Unsupervised underwater image enhancement with improved CycleGAN

Due to the complexity of underwater environments, acquiring high-quality paired underwater images poses a significant challenge. Water’s absorption and scattering of light often result in images with low contrast, color deviations, and blurred details. To address these challenges, this paper proposes an improved unsupervised learning model based on CycleGAN. This model uses a two-part generator to separate content and style features from underwater images. The model integrates content and style features through a multi-scale fusion module, then uses a decoder to reconstruct them into clear images, enhancing image quality with style transfer techniques. Our experiments show that our algorithm performs better than other advanced models in terms of PSNR and SSIM indices, respectively. It can also produce good-quality enhanced images. Furthermore, feature point matching experiments were conducted to demonstrate the practicality of our model.

DeepFake detection method based on multi-scale interactive dual-stream network

DeepFake face forgery has a serious negative impact on both society and individuals. Therefore, research on DeepFake detection technologies is necessary. At present, DeepFake detection technology based on deep learning has achieved acceptable results on high-quality datasets; however, its detection performance on low-quality datasets and cross-datasets remains poor. To address this problem, this paper presents a multi-scale interactive dual-stream network (MSIDSnet). The network is divided into spatial- and frequency-domain streams and uses a multi-scale fusion module to capture both the facial features of images that have been manipulated in the spatial domain under different circumstances and the fine-grained high-frequency noise information of forged images. The network fully integrates the features of the spatial- and frequency-domain streams through an interactive dual-stream module and uses vision transformer (ViT) to further learn the global information of the forged facial features for classification. Experimental results confirm that the accuracy of this method reached 99.30 % on the high-quality dataset Celeb-DF-v2, and 95.51 % on the low-quality dataset FaceForensics++. Moreover, the results of the cross-dataset experiments were superior to those of the other comparison methods.

Early Smoke Recognition Algorithm for Forest Fires

Forest fires require rapid and precise early smoke detection to minimize damage. This study focuses on employing smoke recognition methods for early warning systems in forest fire detection, identifying smoke as the primary indicator. A significant hurdle lies in the absence of a large-scale dataset for real-world early forest fire smoke detection. Early smoke videos present characteristics such as smoke plumes being small, slow-moving, and/or semi-transparent in color, and include images where there is background interference, posing critical challenges for practical recognition algorithms. To address these issues, this paper introduces a real-world early smoke monitoring video dataset as a foundational resource. The proposed 4D attention-based motion target enhancement network includes an important frame sorting module which adaptively selects essential frame sequences to improve the detection of slow-moving smoke targets. Additionally, a 4D attention-based motion target enhancement module is introduced to mitigate interference from smoke-like objects and enhance recognition of light smoke during the initial stages. Moreover, a high-resolution multi-scale fusion module is presented, incorporating a small target recognition layer to enhance the network’s ability to detect small smoke targets. This research represents a significant advancement in early smoke detection for forest fire surveillance, with practical implications for enhancing fire management.

MSFNet‐2SE: A multi‐scale fusion convolutional network for Alzheimer's disease classification on magnetic resonance images

AbstractAlzheimer's disease (AD) is an irreversible neurodegenerative disease, and the early diagnosis and effective intervention of AD is essential for patients and doctors. Intelligence diagnosis based on magnetic resonance imaging and deep learning has become one of the useful methods for AD identification. To improve the diagnosis effect of AD in early stages, a novel multi‐scale classification model of AD, called MSFNet‐2SE, is proposed. First, a new multi‐scale fusion (MSF) feature extraction module is designed based on the idea of the feature maps split into feature subsets of Res2Net. Second, a channel attention module is embedded into the MSF module through integrating two Squeeze‐and‐Excitation (SE) blocks. Finally, a gradient centralization Adam optimizer is used to improve the model classification performance. Experimental results illustrate that, compared with other available state‐of‐the‐art classification models of AD, the proposed model has excellent classification performance. It is helpful to improve the clinical diagnosis efficiency.

GEMF: a novel geometry-enhanced mid-fusion network for PLA prediction.

Accurate prediction of protein-ligand binding affinity (PLA) is important for drug discovery. Recent advances in applying graph neural networks have shown great potential for PLA prediction. However, existing methods usually neglect the geometric information (i.e. bond angles), leading to difficulties in accurately distinguishing different molecular structures. In addition, these methods also pose limitations in representing the binding process of protein-ligand complexes. To address these issues, we propose a novel geometry-enhanced mid-fusion network, named GEMF, to learn comprehensive molecular geometry and interaction patterns. Specifically, the GEMF consists of a graph embedding layer, a message passing phase, and a multi-scale fusion module. GEMF can effectively represent protein-ligand complexes as graphs, with graph embeddings based on physicochemical and geometric properties. Moreover, our dual-stream message passing framework models both covalent and non-covalent interactions. In particular, the edge-update mechanism, which is based on line graphs, can fuse both distance and angle information in the covalent branch. In addition, the communication branch consisting of multiple heterogeneous interaction modules is developed to learn intricate interaction patterns. Finally, we fuse the multi-scale features from the covalent, non-covalent, and heterogeneous interaction branches. The extensive experimental results on several benchmarks demonstrate the superiority of GEMF compared with other state-of-the-art methods.

A robust multi-scale feature extraction framework with dual memory module for multivariate time series anomaly detection

Although existing reconstruction-based multivariate time series anomaly detection (MTSAD) methods have shown advanced performance, most assume the training data is clean. When faced with noise or contamination in training data, they can also reconstruct the anomaly well, weakening the distinction between normal and anomaly. Some probabilistic generation-based methods have been used to address this issue because of their implicit robust structure to noise, but the training process and suppression of anomalous generalization are not stable. The recently proposed explicit method based on the memory module would also sacrifice the reconstruction effect of normal patterns, resulting in limited performance improvement. Moreover, most existing MTSAD methods use a single fixed-length window for input, which weakens their ability to extract long-term dependency. This paper proposes a robust multi-scale feature extraction framework with the dual memory module to comprehensively extract features fusing different levels of semantic information and lengths of temporal dependency. First, this paper designs consecutive neighboring windows as inputs to allow the model to extract local and long-term dependency information. Secondly, a dual memory-augmented encoder is proposed to extract global typical patterns and local common features. It ensures the reconstruction ability of normal data while suppressing the generalization of the anomaly. Finally, this paper proposes a multi-scale fusion module to fuse latent variables representing different levels of semantic information and uses the reconstructed latent variables to reconstruct samples for anomaly detection. Experimental results on five datasets from diverse domains show that the proposed method outperforms 16 typical baseline methods.

Decoupled and boosted learning for skeleton-based dynamic hand gesture recognition

With the development of cost-effective depth sensors, skeleton-based dynamic hand gesture recognition has made significant progress. Existing methods mostly utilize a single model to learn all spatial–temporal features. Meanwhile, they cannot effectively boost key features and make use of multi-scale features. In this paper, we propose a lightweight dual-stream framework, which consists of a temporal mutual boosted stream (TMB-Stream) and a spatial self-boosted stream (SSB-Stream). In the TMB-Stream, we design a hybrid attention module (HAM) to boost important motion features from temporal sequences, which is composed of a multi-scale multi-head attention module (MMAM) and a spatial–temporal attention module (STAM). In the SSB-Stream, we present a self-boosted learning manner to promote the performance of the spatial stream. Specifically, we design a multi-scale auto-encoder (MAE), which can use limited skeleton data to extract and boost spatial latent features by minimizing the gap between original and reconstructed skeleton images. In addition, we propose a multi-scale fusion module (MFM) to effectively fuse multi-scale features in stages. Experimental results show that our lightweight framework yields satisfactory performance on SHREC’17 Track and DHG-14/28 datasets, as well as very competitive performance on FPHA dataset.

Multi-view orientational attention network combining point-based affinity for polyp segmentation

Most existing deep learning-based polyp segmentation methods neglect two important aspects of polyps: the geometric orientation information of polyps and the point information of the entire colonoscopy area. In this paper, we introduce a multi-view orientational attention network (MVOA-Net), which incorporates orientation and point awareness to effectively address the issue of intra-class inconsistency resulting from variations in polyp shape, size, and position, as well as the inter-class indistinction caused by the high similarity between polyp lesions and surrounding tissues. To achieve robust orientation awareness, we propose a novel geometric orientation transformer encoder (GOTE) based on horizontal and vertical views. Moreover, To simultaneously capture the global context information of GOTE and emphasize the important local information of the convolution-based attention encoder (CBAE), a global and local cross attention fusion module (CAFM) is also proposed to simultaneously model the long-range dependencies of polyps and pay sufficient attention to the local boundaries of polyps. Additionally, a efficient atrous spatial pyramid pooling (E-ASPP) module is proposed to enhance the semantic representation of high-level features. Finally, a point-based affinity module (PBAM) and a multi-scale fusion module (MSFM) are proposed to distinguish the disguise of polyps, further alleviating inter-class indistinction. The ablation study results demonstrate the effectiveness of each component. Quantitative and qualitative experimental results show that MVOA-Net achieves the best segmentation accuracy across domain polyp datasets and has obvious advantages in segmenting multiple polyp objects.

Multi-Scale Feature Fusion Network with Symmetric Attention for Land Cover Classification Using SAR and Optical Images

The complementary characteristics of SAR and optical images are beneficial in improving the accuracy of land cover classification. Deep learning-based models have achieved some notable results. However, how to effectively extract and fuse the unique features of multi-modal images for pixel-level classification remains challenging. In this article, a two-branch supervised semantic segmentation framework without any pretrained backbone is proposed. Specifically, a novel symmetric attention module is designed with improved strip pooling. The multiple long receptive fields can better perceive irregular objects and obtain more anisotropic contextual information. Meanwhile, to solve the semantic absence and inconsistency of different modalities, we construct a multi-scale fusion module, which is composed of atrous spatial pyramid pooling, varisized convolutions and skip connections. A joint loss function is introduced to constrain the backpropagation and reduce the impact of class imbalance. Validation experiments were implemented on the DFC2020 and WHU-OPT-SAR datasets. The proposed model achieved the best quantitative values on the metrics of OA, Kappa and mIoU, and its class accuracy was also excellent. It is worth mentioning that the number of parameters and the computational complexity of the method are relatively low. The adaptability of the model was verified on RGB–thermal segmentation task.

MFIL-FCOS: A Multi-Scale Fusion and Interactive Learning Method for 2D Object Detection and Remote Sensing Image Detection

Object detection is dedicated to finding objects in an image and estimate their categories and locations. Recently, object detection algorithms suffer from a loss of semantic information in the deeper feature maps due to the deepening of the backbone network. For example, when using complex backbone networks, existing feature fusion methods cannot fuse information from different layers effectively. In addition, anchor-free object detection methods fail to accurately predict the same object due to the different learning mechanisms of the regression and centrality of the prediction branches. To address the above problem, we propose a multi-scale fusion and interactive learning method for fully convolutional one-stage anchor-free object detection, called MFIL-FCOS. Specifically, we designed a multi-scale fusion module to address the problem of local semantic information loss in high-level feature maps which strengthen the ability of feature extraction by enhancing the local information of low-level features and fusing the rich semantic information of high-level features. Furthermore, we propose an interactive learning module to increase the interactivity and more accurate predictions by generating a centrality-position weight adjustment regression task and a centrality prediction task. Following these strategic improvements, we conduct extensive experiments on the COCO and DIOR datasets, demonstrating its superior capabilities in 2D object detection tasks and remote sensing image detection, even under challenging conditions.