Multi-scale Feature Fusion Research Articles

UAV Geo-Localization Dataset and Method Based on Cross-View Matching

The stable flight of drones relies on Global Navigation Satellite Systems (GNSS). However, in complex environments, GNSS signals are prone to interference, leading to flight instability. Inspired by cross-view machine learning, this paper introduces the VDUAV dataset and designs the VRLM network architecture, opening new avenues for cross-view geolocation. First, to address the limitations of traditional datasets with limited scenarios, we propose the VDUAV dataset. By leveraging the virtual–real mapping of latitude and longitude coordinates, we establish a digital twin platform that incorporates 3D models of real-world environments. This platform facilitates the creation of the VDUAV dataset for cross-view drone localization, significantly reducing the cost of dataset production. Second, we introduce a new baseline model for cross-view matching, the Virtual Reality Localization Method (VRLM). The model uses FocalNet as its backbone and extracts multi-scale features from both drone and satellite images through two separate branches. These features are then fused using a Similarity Computation and Feature Fusion (SCFF) module. By applying a weighted fusion of multi-scale features, the model preserves critical distinguishing features in the images, leading to substantial improvements in both processing speed and localization accuracy. Experimental results demonstrate that the VRLM model outperforms FPI on the VDUAV dataset, achieving an accuracy increase to 83.35% on the MA@20 metric and a precision of 74.13% on the RDS metric.

Fabric Defect Detection Method Based on Multi-scale Fusion Attention Mechanisms

Abstract Fabric defect detection is extremely important for the development of the textile industry, but the existing traditional image processing algorithms are not good enough to detect fabric defects, and the detection efficiency and accuracy of the classical deep learning model is not satisfactory, so this paper proposes an improved fabric defect detection method based on multi-scale fusion of attention mechanism YOLOv7-PCBS. Based on the YOLOv7 network structure, some of the standard convolutions of the backbone network are replaced with Partial Convolution (PConv) modules, which reduces the amount of network computation and improves the network detection speed; add Coordinate Attention (CA) to enhance the ability of extracting the positional features of tiny defects in fabrics; reconfiguration of the SPPCSPC module to improve small target detection; optimization of Bidirectional Feature Pyramid Network (BiFPN) and design of Tiny- BiFPN for simple and fast multi-scale feature fusion; finally, a novel loss function SIoU with angular loss is introduced to facilitate the fitting of the true and predicted frames and enhance the accuracy of defect prediction. The results show that the algorithm achieves a mAP value of 94.4% on the detection of defects in solid-colored fabrics of six denim materials, which is an improvement of 15.1% compared to the original YOLOv7 algorithm, while the model achieves a frame rate of 59.5 per second. Compared with other traditional deep learning algorithms SSD and Faster-RCNN, the detection accuracies are improved by 21.6% and 15.2%, and the FPS values are improved by 78.1% and 101.0%, respectively. Therefore, the YOLOv7-PCBS fabric defect detection algorithm proposed in this paper makes the fabric defect detection results more accurate while realizing lightweight, which provides an important technical reference for the subsequent improvement of textile quality.

Colon polyp detection based on multi-scale and multi-level feature fusion and lightweight convolutional neural network

Early diagnosis and treatment of colorectal polyps are crucial for preventing colorectal cancer. This paper proposes a lightweight convolutional neural network for the automatic detection and auxiliary diagnosis of colorectal polyps. Initially, a 53-layer convolutional backbone network is used, incorporating a spatial pyramid pooling module to achieve feature extraction with different receptive field sizes. Subsequently, a feature pyramid network is employed to perform cross-scale fusion of feature maps from the backbone network. A spatial attention module is utilized to enhance the perception of polyp image boundaries and details. Further, a positional pattern attention module is used to automatically mine and integrate key features across different levels of feature maps, achieving rapid, efficient, and accurate automatic detection of colorectal polyps. The proposed model is evaluated on a clinical dataset, achieving an accuracy of 0.9982, recall of 0.9988, F1 score of 0.9984, and mean average precision (mAP) of 0.9953 at an intersection over union (IOU) threshold of 0.5, with a frame rate of 74 frames per second and a parameter count of 9.08 M. Compared to existing mainstream methods, the proposed method is lightweight, has low operating configuration requirements, high detection speed, and high accuracy, making it a feasible technical method and important tool for the early detection and diagnosis of colorectal cancer.

DMDLK-Net: A dynamic multi-scale feature fusion network with deformable large kernel for medical segmentation

Abstract. Skin disease image segmentation is a crucial component of computer-aided diagnosis, providing precise localization and delineation of lesions that enhance diagnostic accuracy and efficiency. Despite significant advancements in convolutional neural networks (CNNs), there remains substantial room for improvement in segmentation performance due to the diverse and complex nature of skin lesions. In this study, we propose DMDLK-Net, a dynamic multi-scale feature fusion network with deformable large kernels, specifically designed to address the challenges in skin disease segmentation. Our network incorporates a Dynamic Deformable Large Kernel (DDLK) module and a Dynamic Multi-Scale Feature Fusion (DMFF) module, enhancing the model's ability to capture intricate lesion features. We present the performance of DMDLK-Net on the ISIC-2018 dataset, highlighting its promising results. Key contributions of this work include the innovative use of deformable large kernels for adaptive feature extraction and the introduction of dynamic multi-scale fusion to balance local and global information. Our experimental results confirm the effectiveness of DMDLK-Net in delivering high-precision segmentation, thus providing a reliable tool for clinical applications.

GLE-net: global-local information enhancement for semantic segmentation of remote sensing images

Remote sensing (RS) images contain a wealth of information with expansive potential for applications in image segmentation. However, Convolutional Neural Networks (CNN) face challenges in fully harnessing the global contextual information. Leveraging the formidable capabilities of global information modeling with Swin-Transformer, a novel RS images segmentation model with CNN (GLE-Net) was introduced. This integration gives rise to a revamped encoder structure. The subbranch initiates the process by extracting features at varying scales within the RS images using the Multiscale Feature Fusion Module (MFM), acquiring rich semantic information, discerning localized finer features, and adeptly handling occlusions. Subsequently, Feature Compression Module (FCM) is introduced in main branch to downsize the feature map, effectively reducing information loss while preserving finer details, enhancing segmentation accuracy for smaller targets. Finally, we integrate local features and global features through Spatial Information Enhancement Module (SIEM) for comprehensive feature modeling, augmenting the segmentation capabilities of model. We performed experiments on public datasets provided by ISPRS, yielding notably remarkable experimental outcomes. This underscores the substantial potential of our model in the realm of RS image segmentation within the context of scientific research.

Multi-Scale and Multi-Network Deep Feature Fusion for Discriminative Scene Classification of High-Resolution Remote Sensing Images

The advancement in satellite image sensors has enabled the acquisition of high-resolution remote sensing (HRRS) images. However, interpreting these images accurately and obtaining the computational power needed to do so is challenging due to the complexity involved. This manuscript proposed a multi-stream convolutional neural network (CNN) fusion framework that involves multi-scale and multi-CNN integration for HRRS image recognition. The pre-trained CNNs were used to learn and extract semantic features from multi-scale HRRS images. Feature extraction using pre-trained CNNs is more efficient than training a CNN from scratch or fine-tuning a CNN. Discriminative canonical correlation analysis (DCCA) was used to fuse deep features extracted across CNNs and image scales. DCCA reduced the dimension of the features extracted from CNNs while providing a discriminative representation by maximizing the within-class correlation and minimizing the between-class correlation. The proposed model has been evaluated on NWPU-RESISC45 and UC Merced datasets. The accuracy associated with DCCA was 10% and 6% higher than discriminant correlation analysis (DCA) in the NWPU-RESISC45 and UC Merced datasets. The advantage of DCCA was better demonstrated in the NWPU-RESISC45 dataset due to the incorporation of richer within-class variability in this dataset. While both DCA and DCCA minimize between-class correlation, only DCCA maximizes the within-class correlation and, therefore, attains better accuracy. The proposed framework achieved higher accuracy than all state-of-the-art frameworks involving unsupervised learning and pre-trained CNNs and 2–3% higher than the majority of fine-tuned CNNs. The proposed framework offers computational time advantages, requiring only 13 s for training in NWPU-RESISC45, compared to a day for fine-tuning the existing CNNs. Thus, the proposed framework achieves a favourable balance between efficiency and accuracy in HRRS image recognition.

Electrical load forecasting based on the fusion of multi-scale features extracted by using neural ordinary differential equation

Electrical load forecasting based on the fusion of multi-scale features extracted by using neural ordinary differential equation

MDAR: A Multiscale Features-Based Network for Remotely Measuring Human Heart Rate Utilizing Dual-Branch Architecture and Alternating Frame Shifts in Facial Videos

Remote photoplethysmography (rPPG) refers to a non-contact technique that measures heart rate through analyzing the subtle signal changes of facial blood flow captured by video sensors. It is widely used in contactless medical monitoring, remote health management, and activity monitoring, providing a more convenient and non-invasive way to monitor heart health. However, factors such as ambient light variations, facial movements, and differences in light absorption and reflection pose challenges to deep learning-based methods. To solve these difficulties, we put forward a measurement network of heart rate based on multiscale features. In this study, we designed and implemented a dual-branch signal processing framework that combines static and dynamic features, proposing a novel and efficient method for feature fusion, enhancing the robustness and reliability of the signal. Furthermore, we proposed an alternate time-shift module to enhance the model’s temporal depth. To integrate the features extracted at different scales, we utilized a multiscale feature fusion method, enabling the model to accurately capture subtle changes in blood flow. We conducted cross-validation on three public datasets: UBFC-rPPG, PURE, and MMPD. The results demonstrate that MDAR not only ensures fast inference speed but also significantly improves performance. The two main indicators, MAE and MAPE, achieved improvements of at least 30.6% and 30.2%, respectively, surpassing state-of-the-art methods. These conclusions highlight the potential advantages of MDAR for practical applications.

A multi-scale feature extraction and fusion-based model for retinal vessel segmentation in fundus images.

In response to the challenge of low accuracy in retinal vessel segmentation attributed to the minute nature of the vessels, this paper proposes a retinal vessel segmentation model based on an improved U-Net, which combines multi-scale feature extraction and fusion techniques. An improved dilated residual module was first used to replace the original convolutional layer of U-Net, and this module, coupled with a dual attention mechanism and diverse expansion rates, facilitates the extraction of multi-scale vascular features. Moreover, an adaptive feature fusion module was added at the skip connections of the model to improve vessel connectivity. To further optimize network training, a hybrid loss function is employed to mitigate the class imbalance between vessels and the background. Experimental results on the DRIVE dataset and CHASE_DB1 dataset show that the proposed model has an accuracy of 96.27% and 96.96%, sensitivity of 81.32% and 82.59%, and AUC of 98.34% and 98.70%, respectively, demonstrating superior segmentation performance.

Remote Sensing LiDAR and Hyperspectral Classification with Multi-Scale Graph Encoder–Decoder Network

The rapid development of sensor technology has made multi-modal remote sensing data valuable for land cover classification due to its diverse and complementary information. Many feature extraction methods for multi-modal data, combining light detection and ranging (LiDAR) and hyperspectral imaging (HSI), have recognized the importance of incorporating multiple spatial scales. However, effectively capturing both long-range global correlations and short-range local features simultaneously on different scales remains a challenge, particularly in large-scale, complex ground scenes. To address this limitation, we propose a multi-scale graph encoder–decoder network (MGEN) for multi-modal data classification. The MGEN adopts a graph model that maintains global sample correlations to fuse multi-scale features, enabling simultaneous extraction of local and global information. The graph encoder maps multi-modal data from different scales to the graph space and completes feature extraction in the graph space. The graph decoder maps the features of multiple scales back to the original data space and completes multi-scale feature fusion and classification. Experimental results on three HSI-LiDAR datasets demonstrate that the proposed MGEN achieves considerable classification accuracies and outperforms state-of-the-art methods.

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement.

In smart classroom environments, accurately recognizing students' facial expressions is crucial for teachers to efficiently assess students' learning states, timely adjust teaching strategies, and enhance teaching quality and effectiveness. In this paper, we propose a student facial expression recognition model based on multi-scale and deep fine-grained feature attention enhancement (SFER-MDFAE) to address the issues of inaccurate facial feature extraction and poor robustness of facial expression recognition in smart classroom scenarios. Firstly, we construct a novel multi-scale dual-pooling feature aggregation module to capture and fuse facial information at different scales, thereby obtaining a comprehensive representation of key facial features; secondly, we design a key region-oriented attention mechanism to focus more on the nuances of facial expressions, further enhancing the representation of multi-scale deep fine-grained feature; finally, the fusion of multi-scale and deep fine-grained attention-enhanced features is used to obtain richer and more accurate facial key information and realize accurate facial expression recognition. The experimental results demonstrate that the proposed SFER-MDFAE outperforms the existing state-of-the-art methods, achieving an accuracy of 76.18% on FER2013, 92.75% on FERPlus, 92.93% on RAF-DB, 67.86% on AffectNet, and 93.74% on the real smart classroom facial expression dataset (SCFED). These results validate the effectiveness of the proposed method.

Extraction of Winter Wheat Planting Plots with Complex Structures from Multispectral Remote Sensing Images Based on the Modified Segformer Model

As one of the major global food crops, the monitoring and management of the winter wheat planting area is of great significance for agricultural production and food security worldwide. Today, the development of high-resolution remote sensing imaging technology has provided rich sources of data for extracting the visual planting information of winter wheat. However, the existing research mostly focuses on extracting the planting plots that have a simple terrain structure. In the face of diverse terrain features combining mountainous areas, plains, and saline alkali land, as well as small-scale but complex planting structures, the extraction of planting plots through remote sensing imaging is subjected to great challenges in terms of recognition accuracy and model complexity. In this paper, we propose a modified Segformer model for extracting winter wheat planting plots with complex structures in rural areas based on the 0.8 m high-resolution multispectral data obtained from the Gaofen-2 satellite, which significantly improves the extraction accuracy and efficiency under complex conditions. In the encoder and decoder of this method, new modules were developed for the purpose of optimizing the feature extraction and fusion process. Specifically, the improvement measures of the proposed method include: (1) The MixFFN module in the original Segformer model is replaced with the Multi-Scale Feature Fusion Fully-connected Network (MSF-FFN) module, which enhances the model’s representation ability in handling complex terrain features through multi-scale feature extraction and position embedding convolution; furthermore, the DropPath mechanism is introduced to reduce the possibility of overfitting while improving the model’s generalization ability. (2) In the decoder part, after fusing features at four different scales, a CoordAttention module is added, which can precisely locate important regions with enhanced features in the images by utilizing the coordinate attention mechanism, therefore further improving the model’s extraction accuracy. (3) The model’s input data are strengthened by incorporating multispectral indices, which are also conducive to the improvement of the overall extraction accuracy. The experimental results show that the accuracy rate of the modified Segformer model in extracting winter wheat planting plots is significantly increased compared to traditional segmentation models, with the mean Intersection over Union (mIOU) and mean Pixel Accuracy (mPA) reaching 89.88% and 94.67%, respectively (an increase of 1.93 and 1.23 percentage points, respectively, compared to the baseline model). Meanwhile, the parameter count and computational complexity are significantly reduced compared to other similar models. Furthermore, when multispectral indices are input into the model, the mIOU and mPA reach 90.97% and 95.16%, respectively (an increase of 3.02 and 1.72 percentage points, respectively, compared to the baseline model).

Multi-scale Adaptive Feature Fusion Hashing for Image Retrieval

AbstractThe hash algorithm has the characteristics of high computational speed and low memory consumption, making it well-suited for massive image search tasks. Currently, most methods rely on deep learning for end-to-end feature extraction and hash encoding. These methods use the last layer feature of the model as the semantic feature of the hash encoding image. However, mainstream research has not realized that the features of different stages of the network contain rich image semantic information, which all affect the performance of retrieval. Based on this, we propose a multi-scale adaptive feature fusion hash image retrieval method, which mines more detailed information about the image by introducing adaptive feature fusion modules at different stages of the network, and incorporates shallow features in the final extracted features to help the model understand the image content. In addition, to maintain the similarity of the generated image hash codes and reduce errors in the process of transitioning from original features to hash codes, similarity loss and quantization loss are applied, while fully utilizing the supervised information of dataset labels to get high-quality hash codes. Experimental proof conducted on the CIFAR-10 and NUS-WIDE datasets that this method outperforms other deep learning-based hash image retrieval methods.

RU-Net, Multi-Scale Feature Fusion and Transfer Learning: Unlocking the Potential of Cuffless Blood Pressure Monitoring with PPG and ECG.

This study introduces an innovative deep-learning model for cuffless blood pressure estimation using PPG and ECG signals, demonstrating state-of-the-art performance on the largest clean dataset, PulseDB. The rU-Net architecture, a fusion of U-Net and ResNet, enhances both generalization and feature extraction accuracy. Accurate multi-scale feature capture is facilitated by short-time Fourier transform (STFT) time-frequency distributions and multi-head attention mechanisms, allowing data-driven feature selection. The inclusion of demographic parameters as supervisory information further elevates performance. On the calibration-based dataset, our model excels, achieving outstanding accuracy (SBP MAE ± std: 4.49 ± 4.86 mmHg, DBP MAE ± std: 2.69 ± 3.10 mmHg), surpassing AAMI standards and earning a BHS Grade A rating. Addressing the challenge of calibration-free data, we propose a fine-tuning-based transfer learning approach. Remarkably, with only 10% data transfer, our model attains exceptional accuracy (SBP MAE ± std: 4.14 ± 5.01 mmHg, DBP MAE ± std: 2.48 ± 2.93 mmHg). This study sets the stage for the development of highly accurate and reliable wearable cuffless blood pressure monitoring devices.

A dual encoder LDCT image denoising model based on cross-scale skip connections

LDCT image denoising is crucial in medical imaging as it aims to minimize patient radiation exposure while maintaining diagnostic image quality. However, current convolutional neural network-based denoising methods struggle to incorporate global contexts, often focusing solely on local features. This limitation poses a significant challenge. To address this, a dual encoder denoising model is introduced that utilizes the Transformer model’s proficiency in capturing long-range dependencies and global context. This model integrates the Transformer branch and the convolutional branch in the encoder. By concatenating the features of these two different branches, the model can capture both global and local image features, substantially enhancing denoising efficacy. A cross-scale skip connection mechanism is introduced to integrate the encoder’ s low-level features with the decoder’ s high-level features, enriching contextual information and preserving image details. In addition, to meet the requirements of multi-scale feature fusion, the decoder is equipped with different multi-scale convolution modules to optimize feature processing. The number of layers in these modules gradually decreases as the depth of the decoder increases. In order to enhance the discriminative ability of the model, a multi-scale discriminator is also introduced, which effectively improves the recognition ability of the image by extracting features from four different scales. Consequently, our approach demonstrates remarkable performance in reducing noise and improving LDCT image quality, as evidenced by the substantial improvements in PSNR (17.75%) and SSIM (7.31%) values.

CBS-YOLOv5: fault detection algorithm of electrolyzer plate with low-resolution infrared images based on improved YOLOv5

Abstract In the process of copper electrorefining, accurate detection of electrode plate faults is extremely challenging due to the low resolution of captured infrared images, significant noise interference, and dense electrode plate arrangements. To address these challenges, this paper proposes an improved YOLOv5-based electrode plate fault detection algorithm called CBS-YOLOv5. This algorithm introduces several innovations over the original YOLOv5, including: the incorporation of coordinate attention to enhance the ability of the feature extraction network to separate target information from noise; the construction of a small object detection module to improve the detection of dense small objects by increasing the resolution of the feature map; the replacement of the traditional path aggregation network with a Bi-directional Feature Pyramid Network (BiFPN) for more flexible multi-scale feature fusion; and the integration of the swin transformer to optimize the cross-stage partial bottleneck structure, significantly enhancing the model’s ability to detect densely packed small objects. Experimental results show that the proposed CBS-YOLOv5 model achieves an accuracy of 88.1%, which is an improvement of 5.7% over the base model. Furthermore, this algorithm demonstrates exceptional detection capabilities for dense small objects in low-resolution infrared images while maintaining real-time detection speed, making it suitable for various complex industrial scenarios, including fault detection in non-ferrous metal electrolysis processes. CBS-YOLOv5 not only improves detection accuracy and robustness but also has broad application prospects, offering a new solution for intelligent manufacturing and industrial inspection.

Waste classification strategy based on multi-scale feature fusion for intelligent waste recycling in office buildings

Waste classification is an important measure to protect the environment. Existing waste classification methods mainly focus on scientific research, but lack attention to the challenges of waste classification in actual scenarios. For example, wastes with similar contours, similar textures, or contaminated appearance are difficult to be classified in actual scenarios. To address these issues, this paper proposes an innovative multi-scale feature fusion strategy (MFFS) to improve the classification accuracy of these wastes. MFFS combines local fine-grained features with global coarse-grained features to improve the feature expression ability of waste. However, how to effectively fuse these two features is a key challenge. This paper proposes a dual-scale feature fusion strategy, first fusing fine-grained features in the first dimension, then fusing coarse-grained features in the second dimension, and introducing spatial features to further enhance feature expression capabilities. In order to reduce the interference of background information, the model in this paper models global relationships based on convolutional features. The MFFS strategy achieved a classification accuracy of 95.5% on the self-built dataset and 94.1% on the public dataset TrashNet. The number of parameters of our model is reduced by 57.2% compared with the classic VGG16 and by 34.2% compared with the Vision Transformer. In addition, we designed an intelligent waste sorting device and deployed the MFFS model on the device to implement the application. Experiments show that our model has ideal accuracy and stability and can be promoted and applied.

ETLSH-YOLO: An Edge–Real-Time Transmission Line Safety Hazard Detection Method

Using deep learning methods to detect potential safety hazards in transmission lines is the mainstream method for power grid security monitoring. However, the existing model is too complex to adapt to edge device deployment and real-time detection. Therefore, an edge–real-time transmission line safety hazard detection method (ETLSH-YOLO) was proposed to reduce the model’s complexity and improve the model’s robustness. Firstly, a re-parameterized Ghost efficient layer aggregation network (RepGhostCSPELAN) was designed to effectively fuse the feature information of different layers while enhancing the model’s expression ability and reducing the number of model parameters and floating-point operations. Then, a spatial channel decoupled downsampling block (CSDovn) was designed to reduce computational redundancy and improve the computational efficiency of the model. Then, coordinate attention (CA) was added in the process of multi-scale feature fusion to suppress the interference of complex background and improve the global perception ability of the model object. Finally, the Mish activation function was used to improve the network’s training speed, convergence, and generalization ability. The experimental results show that the mAP50 of this model improved by 1.73% compared with the baseline model, and the number of parameters and floating-point operations were reduced by 33.96% and 22.22%, respectively. This model lays the foundation for solving the dilemma of edge device deployment.

MSMP-Net: A Multi-Scale Neural Network for End-to-End Monkeypox Virus Skin Lesion Classification

Monkeypox is a zoonotic disease caused by monkeypox virus infection. It is easily transmitted among people and poses a major threat to human health, making it of great significance in public health. Therefore, this paper proposes MSMP-Net, a multi-scale neural network for end-to-end monkeypox virus skin lesion classification ConvNeXt is used as the backbone network, and designs such as inverse bottleneck layers and large convolution kernels are used to enhance the network’s feature extraction capabilities. In order to effectively utilize the multi-level feature maps generated by the backbone network, a multi-scale feature fusion structure was designed. By fusing the deepest feature maps of multi-scale features, the model’s ability to represent monkeypox image features is enhanced. Experimental results show that the accuracy, precision, recall, and F1-score of this method on the MSLD v2.0 dataset are 87.03 ± 3.43%, 87.59 ± 3.37%, 87.03 ± 3.43%, and 86.58 ± 3.66%, respectively.

Enhanced Vehicle Logo Detection Method Based on Self-Attention Mechanism for Electric Vehicle Application

Vehicle logo detection plays a crucial role in various computer vision applications, such as vehicle classification and detection. In this research, we propose an improved vehicle logo detection method leveraging the self-attention mechanism. Our feature-sampling structure integrates multiple attention mechanisms and bidirectional feature aggregation to enhance the discriminative power of the detection model. Specifically, we introduce the multi-head attention for multi-scale feature fusion module to capture multi-scale contextual information effectively. Moreover, we incorporate the bidirectional aggregation mechanism to facilitate information exchange between different layers of the detection network. Experimental results on a benchmark dataset (VLD-45 dataset) demonstrate that our proposed method outperforms baseline models in terms of both detection accuracy and efficiency. Our experimental evaluation using the VLD-45 dataset achieves a state-of-the-art result of 90.3% mAP. Our method has also improved AP by 10% for difficult samples, such as HAVAL and LAND ROVER. Our method provides a new detection framework for small-size objects, with potential applications in various fields.