Multi-scale Feature Aggregation Research Articles

LightVSR: A Lightweight Video Super-Resolution Model with Multi-Scale Feature Aggregation

Video super-resolution aims to generate high-resolution video sequences with realistic details from existing low-resolution video sequences. However, most existing video super-resolution models require substantial computational power and are not suitable for resource-constrained devices such as smartphones and tablets. In this paper, we propose a lightweight video super-resolution (LightVSR) model that employs a novel feature aggregation module to enhance video quality by efficiently reconstructing high-resolution frames from compressed low-resolution inputs. LightVSR integrates several novel mechanisms, including head-tail convolution, cross-layer shortcut connections, and multi-input attention, to enhance computational efficiency while guaranteeing video super-resolution performance. Extensive experiments show that LightVSR achieves a frame rate of 28.57 FPS and a PSNR of 39.25 dB on the UDM10 dataset and 36.91 dB on the Vimeo-90k dataset, validating its efficiency and effectiveness.

Automatic crack defect detection via multiscale feature aggregation and adaptive fusion

Automatic crack defect detection via multiscale feature aggregation and adaptive fusion

Computational spectral imaging reconstruction via a spatial–spectral cross-attention-driven network

Compared with traditional hyperspectral imaging, computational spectral imaging (CSI) has the advantage of snapshot imaging with high spatial and temporal resolution, which has attracted considerable attention. The core challenge of CSI is to achieve computational imaging reconstruction from a single 2D measurement image to the corresponding 3D spatial–hyperspectral image (HSI). Existing reconstruction methods still face problems in exploring spatial–spectral cross correlation, leading to significant spatial–spectral distortion. Furthermore, due to neglect of multi-scale feature reconstruction, their reconstruction quality still needs to be improved. In this paper, to solve the above problems, we propose a spatial–spectral cross-attention-driven network (SSCA-DN). In SSCA, a proposed multi-scale feature aggregation (MFA) module and a spectral-wise transformer (SpeT) are used for multi-scale spatial feature reconstruction and long-range spectral feature reconstruction, respectively. Using spatial attention and spectral attention to interactively guide the reconstruction of the target HSI in spectral and spatial dimensions, the proposed SSCA models spatial–spectral cross correlation with considering multi-scale features. Using the SSCA as a basic module, a novel SSCA-DN network is constructed, in which a proposed supervised preliminary reconstruction subnetwork (SPRNet) learns the generalized prior, and a proposed unsupervised multi-scale feature fusion and refinement subnetwork (UMFFRNet) learns the specific prior. The SSCA module ensures that the learned generalized and specific priors can capture the spatial–spectral cross correlation while considering multi-scale features. In addition, in UMFFRNet, driven by MFA and SSCA, a novel multi-scale fusion and refinement mechanism for multi-level adjacent features is proposed to effectively model the correlation between adjacent level features and the multi-scale spatial–spectral cross correlation, which further improves the reconstruction accuracy. Extensive experiments show that our method achieves state-of-the-art performance on both simulated and real datasets.

MDA-DETR: Enhancing Offending Animal Detection with Multi-Channel Attention and Multi-Scale Feature Aggregation.

Conflicts between humans and animals in agricultural and settlement areas have recently increased, resulting in significant resource loss and risks to human and animal lives. This growing issue presents a global challenge. This paper addresses the detection and identification of offending animals, particularly in obscured or blurry nighttime images. This article introduces Multi-Channel Coordinated Attention and Multi-Dimension Feature Aggregation (MDA-DETR). It integrates multi-scale features for enhanced detection accuracy, employing a Multi-Channel Coordinated Attention (MCCA) mechanism to incorporate location, semantic, and long-range dependency information and a Multi-Dimension Feature Aggregation Module (DFAM) for cross-scale feature aggregation. Additionally, the VariFocal Loss function is utilized to assign pixel weights, enhancing detail focus and maintaining accuracy. In the dataset section, this article uses a dataset from the Northeast China Tiger and Leopard National Park, which includes images of six common offending animal species. In the comprehensive experiments on the dataset, the mAP50 index of MDA-DETR was 1.3%, 0.6%, 0.3%, 3%, 1.1%, and 0.5% higher than RT-DETR-r18, yolov8n, yolov9-C, DETR, Deformable-detr, and DCA-yolov8, respectively, indicating that MDA-DETR is superior to other advanced methods.

FA-MSVNet: multi-scale and multi-view feature aggregation methods for stereo 3D reconstruction

FA-MSVNet: multi-scale and multi-view feature aggregation methods for stereo 3D reconstruction

Multi-scale Unet-based feature aggregation network for lightweight image deblurring

Multi-scale Unet-based feature aggregation network for lightweight image deblurring

Building Change Detection Network Based on Multilevel Geometric Representation Optimization Using Frame Fields

To address the challenges of accurately segmenting irregular building boundaries in complex urban environments faced by existing remote sensing change detection methods, this paper proposes a building change detection network based on multilevel geometric representation optimization using frame fields called BuildingCDNet. The proposed method employs a multi-scale feature aggregation encoder–decoder architecture, leveraging contextual information to capture the characteristics of buildings of varying sizes in the imagery. Cross-attention mechanisms are incorporated to enhance the feature correlations between the change pairs. Additionally, the frame field is introduced into the network to model the complex geometric structure of the building target. By learning the local orientation information of the building structure, the frame field can effectively capture the geometric features of complex building features. During the training process, a multi-task learning strategy is used to align the predicted frame field with the real building outline, while learning the overall segmentation, edge outline, and corner point features of the building. This improves the accuracy of the building polygon representation. Furthermore, a discriminative loss function is constructed through multi-task learning to optimize the polygonal structured information of the building targets. The proposed method achieves state-of-the-art results on two commonly used datasets.

CMPNet: A cross-modal multi-scale perception network for RGB-T crowd counting

The cross-modal crowd counting method demonstrates better scene adaptability under complex conditions by introducing independent supplementary information. However, existing methods still face problems such as insufficient fusion of modal features, underutilization of crowd structure, and the neglect of scale information. In response to the above issues, this paper proposes a cross-modal multi-scale perception network (CMPNet). Specifically, CMPNet mainly consists of a cross-modal perception fusion module and a multi-scale feature aggregation module. The cross-modal perception fusion module effectively suppresses noise features while sharing features between different modalities, thereby significantly improving the robustness of the crowd counting process. The multi-scale feature aggregation module obtains rich crowd structure information through a spatial context aware graph convolution unit, and then integrates feature information from different scales to enhance the network’s perception ability of crowd density. To the best of our knowledge, CMPNet is the first attempt to model the crowd structure and mine its semantics in the field of cross-modal crowd counting. The experimental results show that CMPNet achieves state-of-the-art performance on all RGB-T datasets, providing an effective solution for cross-modal crowd counting. We will release the code at https://github.com/KunChenKKK/CMPNet.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

SPFDNet: Water Extraction Method Based on Spatial Partition and Feature Decoupling

Extracting water information from remote-sensing images is of great research significance for applications such as water resource protection and flood monitoring. Current water extraction methods aggregated richer multi-level features to enhance the output results. In fact, there is a difference in the requirements for the water body and the water boundary. Indiscriminate multi-feature fusion can lead to perturbation and competition of information between these two types of features during the optimization. Consequently, models cannot accurately locate the internal vacancies within the water body with the external boundary. Therefore, this paper proposes a water feature extraction network with spatial partitioning and feature decoupling. To ensure that the water features are extracted with deep semantic features and stable spatial information before decoupling, we first design a chunked multi-scale feature aggregation module (CMFAM) to construct a context path for obtaining deep semantic information. Then, an information interaction module (IIM) is designed to exchange information between two spatial paths with two fixed resolution intervals and the two paths through. During decoding, a feature decoupling module (FDM) is developed to utilize internal flow prediction to acquire the main body features, and erasing techniques are employed to obtain boundary features. Therefore, the deep features of the water body and the detailed boundary information are supplemented, strengthening the decoupled body and boundary features. Furthermore, the integrated expansion recoupling module (IERM) module is designed for the recoupling stage. The IERM expands the water body and boundary features using expansion and adaptively compensates the transition region between the water body and boundary through information guidance. Finally, multi-level constraints are combined to realize the supervision of the decoupled features. Thus, the water body and boundaries can be extracted more accurately. A comparative validation analysis is conducted on the public datasets, including the gaofen image dataset (GID) and the gaofen2020 challenge dataset (GF2020). By comparing with seven SOTAs, the results show that the proposed method achieves the best results, with IOUs of 91.22 and 78.93, especially in the localization of water bodies and boundaries. By applying the proposed method in different scenarios, the results show the stable capability of the proposed method for extracting water with various shapes and areas.

Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral image classification

Hyperspectral image (HSI) classification plays a crucial role in remote sensing (RS) applications, enabling the precise identification of materials and land cover based on spectral information. This supports tasks such as agricultural management and urban planning. While sequential neural models like Recurrent Neural Networks (RNNs) and Transformers have been adapted for this task, they present limitations: RNNs struggle with feature aggregation and are sensitive to noise from interfering pixels, whereas Transformers require extensive computational resources and tend to underperform when HSI datasets contain limited or unbalanced training samples. To address these challenges, Mamba architectures have emerged, offering a balance between RNNs and Transformers by leveraging lightweight, parallel scanning capabilities. Although models like Vision Mamba (ViM) and Visual Mamba (VMamba) have demonstrated improvements in visual tasks, their application to HSI classification remains underexplored, particularly in handling land-cover semantic tokens and multi-scale feature aggregation for patch-wise classifiers. In response, this study introduces the Mamba-in-Mamba (MiM) architecture for HSI classification, marking a pioneering effort in this domain. The MiM model features: (1) a novel centralized Mamba-Cross-Scan (MCS) mechanism for efficient image-to-sequence data transformation; (2) a Tokenized Mamba (T-Mamba) encoder that incorporates a Gaussian Decay Mask (GDM), Semantic Token Learner (STL), and Semantic Token Fuser (STF) for enhanced feature generation; and (3) a Weighted MCS Fusion (WMF) module with a Multi-Scale Loss Design for improved training efficiency. Experimental results on four public HSI datasets—Indian Pines, Pavia University, Houston2013, and WHU-Hi-Honghu—demonstrate that our method achieves an overall accuracy improvement of up to 3.3%, 2.7%, 1.5%, and 2.3% over state-of-the-art approaches (i.e., SSFTT, MAEST, etc.) under both fixed and disjoint training-testing settings.

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement.

In smart classroom environments, accurately recognizing students' facial expressions is crucial for teachers to efficiently assess students' learning states, timely adjust teaching strategies, and enhance teaching quality and effectiveness. In this paper, we propose a student facial expression recognition model based on multi-scale and deep fine-grained feature attention enhancement (SFER-MDFAE) to address the issues of inaccurate facial feature extraction and poor robustness of facial expression recognition in smart classroom scenarios. Firstly, we construct a novel multi-scale dual-pooling feature aggregation module to capture and fuse facial information at different scales, thereby obtaining a comprehensive representation of key facial features; secondly, we design a key region-oriented attention mechanism to focus more on the nuances of facial expressions, further enhancing the representation of multi-scale deep fine-grained feature; finally, the fusion of multi-scale and deep fine-grained attention-enhanced features is used to obtain richer and more accurate facial key information and realize accurate facial expression recognition. The experimental results demonstrate that the proposed SFER-MDFAE outperforms the existing state-of-the-art methods, achieving an accuracy of 76.18% on FER2013, 92.75% on FERPlus, 92.93% on RAF-DB, 67.86% on AffectNet, and 93.74% on the real smart classroom facial expression dataset (SCFED). These results validate the effectiveness of the proposed method.

Crack Detection, Classification, and Segmentation on Road Pavement Material Using Multi-Scale Feature Aggregation and Transformer-Based Attention Mechanisms

This paper introduces a novel approach to pavement material crack detection, classification, and segmentation using advanced deep learning techniques, including multi-scale feature aggregation and transformer-based attention mechanisms. The proposed methodology significantly enhances the model’s ability to handle varying crack sizes, shapes, and complex pavement textures. Trained on a dataset of 10,000 images, the model achieved substantial performance improvements across all tasks after integrating transformer-based attention. Detection precision increased from 88.7% to 94.3%, and IoU improved from 78.8% to 93.2%. In classification, precision rose from 88.3% to 94.8%, and recall improved from 86.8% to 94.2%. For segmentation, the Dice Coefficient increased from 80.3% to 94.7%, and IoU for segmentation advanced from 74.2% to 92.3%. These results underscore the model’s robustness and accuracy in identifying pavement cracks in challenging real-world scenarios. This framework not only advances automated pavement maintenance but also provides a foundation for future research focused on optimizing real-time processing and extending the model’s applicability to more diverse pavement conditions.

DBAII-Net with multiscale feature aggregation and cross-modal attention for enhancing infant brain injury classification in MRI.

MRI is pivotal in diagnosing brain injuries in infants. However, the dynamic development of the brain introduces variability in infant MRI characteristics, posing challenges for MRI-based classification in this population. Furthermore, manual data selection in large-scale studies is labor-intensive, and existing algorithms often underperform with thick-slice MRI data. To enhance research efficiency and classification accuracy in large datasets, we propose an advanced classification model. We introduce the Dual-Branch Attention Information Interactive Neural Network (DBAII-Net), a cutting-edge model inspired by radiologists' use of multiple MRI sequences. DBAII-Net features two innovative modules: (1) the Convolutional Enhancement Module (CEM), which leverages advanced convolutional techniques to aggregate multi-scale features, significantly enhancing information representation; and (2) the Cross-Modal Attention Module (CMAM), which employs state-of-the-art attention mechanisms to fuse data across branches, dramatically improving positional and channel feature extraction. Performances (accuracy, sensitivity, specificity, area under the curve, etc.) of DBAII-Net were compared with eight benchmark models for brain MRI classification in infants aged 6 months to 2 years. Utilizing a self-constructed dataset of 240 thick-slice brain MRI scans (122 with brain injuries, 118 without), DBAII-Net demonstrated superior performance. On a test set of approximately 50 cases, DBAII-Net achieved average performance metrics of 92.53% accuracy, 90.20% sensitivity, 94.93% specificity, and an area under the curve (AUC) of 0.9603. Ablation studies confirmed the effectiveness of CEM and CMAM, with CMAM significantly boosting classification metrics. DBAII-Net with CEM and CMAM outperforms existing benchmarks in enhancing the precision of brain MRI classification in infants, significantly reducing manual effort in infant brain research. Our code is available at https://github.com/jiazhen4585/DBAII-Net.

High-Resolution Reconstruction of Temperature Fields Based on Improved ResNet18.

High-precision measurement of temperature value distributions in production scenarios is of great significance for industrial production, but traditional temperature field reconstruction algorithms rely on the design of manual feature extraction methods with high computational complexity and poor generalization ability. In this paper, we propose a high-precision temperature field reconstruction algorithm based on deep learning, using an efficient adaptive feature extraction method for temperature field reconstruction. We design an improved temperature field reconstruction algorithm based on the ResNet18 neural network; introduce the CBAM attention mechanism in the model; and design a feature pyramid, using M-FPN, a multi-scale feature aggregation network fusing PAN and FPN, to make the extracted feature information propagate multi-dimensionally among different layers to improve the feature characterization ability. Finally, the mean square error is used to guide the model to optimize the training so that the model pays more attention to the data and reduces the large error to ensure that the gap between the predicted value and the real value is small. The experimental results show that the reconstruction accuracy of the improved algorithm presented in this paper is significantly better than that of the original algorithm in the case of typical peaked temperature field distributions.

GS–CDNet: a remote sensing image cloud detection method with geographic spatial data integration

ABSTRACT In optical remote sensing images, clouds exhibit irregular scales and boundaries that vary with elevation across diverse geographical locations. To accurately capture the diverse visual patterns of clouds, we propose a cloud image segmentation approach named GS-CDNet (Geographic Spatial Data-Cloud Detection Network), which is based on the integration of geospatial data with multifaceted self-attention feature extraction, multi-scale feature aggregation, and boundary clarification techniques.Firstly, we utilize geographical coordinates from optical remote sensing images to extract a raster DEM (Digital Elevation Model) from SRTM3. This process creates a dataset consisting of elevation images, longitude, and latitude maps as geospatial data, enhancing the model’s capability in spatial positioning for cloud detection. Secondly, the proposed method consists of three interconnected modules within the cloud detection network: the Interleaved Self-Attention module(ISAM) utilizes a variety of self-attention mechanisms in an interleaved manner to extract multi-scale feature information.The Bidirectional Multi-Scale Feature Fusion Module(BIMFM) is responsible for integrating features, enabling a more comprehensive contextual understanding. The Boundary Extraction Module(BEM) utilizes a residual structure to generate a boundary cloud mask, effectively addressing the common issue of boundary blurring in multi-scale cloud masks. Finally, we compared and evaluated GS-CDNet with other cloud detection methods and conducted an ablation study on the key components of the method. The validation of generalization performance demonstrates the exceptional performance of the proposed model in cloud mask generation. Geospatial data and the different modules of the method play a significant role in the model.

Real-Time ConvNext-Based U-Net with Feature Infusion for Egg Microcrack Detection

Real-time automatic detection of microcracks in eggs is crucial for ensuring egg quality and safety, yet rapid detection of micron-scale cracks remains challenging. This study introduces a real-time ConvNext-Based U-Net model with Feature Infusion (CBU-FI Net) for egg microcrack detection. Leveraging edge features and spatial continuity of cracks, we incorporate an edge feature infusion module in the encoder and design a multi-scale feature aggregation strategy in the decoder to enhance the extraction of both local details and global semantic information. By introducing large convolution kernels and depth-wise separable convolution from ConvNext, the model significantly reduces network parameters compared to the original U-Net. Additionally, a composite loss function is devised to address class imbalance issues. Experimental results on a dataset comprising over 3400 graded egg microcrack image patches demonstrate that CBU-FI Net achieves a reduction in parameters to one-third the amount in the original U-Net, with an inference speed of 21 ms per image (1 million pixels). The model achieves a Crack-IoU of 65.51% for microcracks smaller than 20 μm and a Crack-IoU and MIoU of 60.76% and 80.22%, respectively, for even smaller cracks (less than 5 μm), achieving high-precision, real-time detection of egg microcracks. Furthermore, on the publicly benchmarked CrackSeg9k dataset, CBU-FI Net achieves an inference speed of 4 ms for 400 × 400 resolution images, with an MIoU of 81.38%, proving the proposed method’s robustness and generalization capability across various cracks and complex backgrounds.

MFRENet: efficient detection of drone image based on multiscale feature aggregation and receptive field expanded

MFRENet: efficient detection of drone image based on multiscale feature aggregation and receptive field expanded

CCM-Net: Color compensation and coordinate attention guided underwater image enhancement with multi-scale feature aggregation

Due to the light scattering and wavelength absorption in water, underwater images exhibit blurred details, low contrast, and color deviation. Existing underwater image enhancement methods are divided into traditional methods and deep learning-based methods. Traditional methods either rely on scene prior and lack robustness, or are not flexible enough resulting in poor enhancement effects. Deep learning methods have achieved good results in the field of underwater image enhancement due to their powerful feature representation ability. However, these methods cannot enhance underwater images with various degradations because they do not consider the inconsistent attenuation of different color channels and spatial regions. In this paper, we propose a novel asymmetric encoder-decoder network for underwater image enhancement, called CCM-Net. Concretely, we first introduce the prior knowledge-based encoder, which includes color compensation (CC) modules and feature extraction modules that consist of depth-wise separable convolution and global-local coordinate attention (GLCA). Then, we design a multi-scale feature aggregation (MFA) module to integrate shallow, middle, and deep features. Finally, we deploy a decoder to reconstruct the underwater images with the extracted features. Extensive experiments on publicly available datasets demonstrate that our CCM-Net effectively improves the visual quality of underwater images and achieves impressive performance.

Semantic Segmentation of Point Cloud Scene via Multi-Scale Feature Aggregation and Adaptive Fusion

Point cloud semantic segmentation is a key step in 3D scene understanding and analysis. In recent years, deep learning–based point cloud semantic segmentation methods have received extensive attention from researchers. Multi-scale neighborhood feature learning methods are suitable for inhomogeneous density point clouds, but different scale branching feature learning increases the computational complexity and makes it difficult to accurately fuse different scale features to express local information. In this study, a point cloud semantic segmentation network based on RandLA-Net with multi-scale local feature aggregation and adaptive fusion is proposed. The designed structure can reduce computational complexity and accurately express local features. The mean intersection-over-union is improved by 1.1% on the SemanticKITTI data set with an inference speed of nine frames per second, while the mean intersection-over-union is improved by 0.9% on the S3DIS data set, compared with RandLA-Net. We also conduct ablation studies to validate the effectiveness of the proposed key structure.