Multi-scale Feature Fusion Network Research Articles

Research on Rapid Recognition of Moving Small Targets by Robotic Arms Based on Attention Mechanisms

For small target objects on fast-moving conveyor belts, traditional vision detection algorithms equipped with conventional robotic arms struggle to capture the long and short-range pixel dependencies crucial for accurate detection. This leads to high miss rates and low precision. In this study, we integrate the traditional EMA (efficient multi-scale attention) algorithm with the c2f (channel-to-pixel) module from the original YOLOv8, alongside a Faster-Net module designed based on partial convolution concepts. This fusion results in the Faster-EMA-Net module, which greatly enhances the ability of the algorithm and robotic technologies to extract pixel dependencies for small targets, and improves perception of dynamic small target objects. Furthermore, by incorporating a small target semantic information enhancement layer into the multiscale feature fusion network, we aim to extract more expressive features for small targets, thereby boosting detection accuracy. We also address issues with training time and subpar performance on small targets in the original YOLOv8 algorithm by improving the loss function. Through experiments, we demonstrate that our attention-based visual detection algorithm effectively enhances accuracy and recall rates for fast-moving small targets, meeting the demands of real industrial scenarios. Our approach to target detection using industrial robotic arms is both practical and cutting-edge.

XAI-FusionNet: Diabetic foot ulcer detection based on multi-scale feature fusion with explainable artificial intelligence

Diabetic foot ulcer (DFU) poses a significant threat to individuals affected by diabetes, often leading to limb amputation. Early detection of DFU can greatly improve the chances of survival for diabetic patients. This work introduces FusionNet, a novel multi-scale feature fusion network designed to accurately differentiate DFU skin from healthy skin using multiple pre-trained convolutional neural network (CNN) algorithms. A dataset comprising 6963 skin images (3574 healthy and 3389 ulcer) from various patients was divided into training (6080 images), validation (672 images), and testing (211 images) sets. Initially, three image preprocessing techniques - Gaussian filter, median filter, and motion blur estimation - were applied to eliminate irrelevant, noisy, and blurry data. Subsequently, three pre-trained CNN algorithms -DenseNet201, VGG19, and NASNetMobile - were utilized to extract high-frequency features from the input images. These features were then inputted into a meta-tuner module to predict DFU by selecting the most discriminative features. Statistical tests, including Friedman and analysis of variance (ANOVA), were employed to identify significant differences between FusionNet and other sub-networks. Finally, three eXplainable Artificial Intelligence (XAI) algorithms - SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and Grad-CAM (Gradient-weighted Class Activation Mapping) - were integrated into FusionNet to enhance transparency and explainability. The FusionNet classifier achieved exceptional classification results with 99.05 % accuracy, 98.18 % recall, 100.00 % precision, 99.09 % AUC, and 99.08 % F1 score. We believe that our proposed FusionNet will be a valuable tool in the medical field to distinguish DFU from healthy skin.

NAS-MFF: NAS-Guided Multiscale Feature Fusion Network With Pareto Optimization for Sonar Images Classification

NAS-MFF: NAS-Guided Multiscale Feature Fusion Network With Pareto Optimization for Sonar Images Classification

Fish detection based on Gather-and-Distribute mechanism Multi-scale feature fusion network and Structural Re-parameterization method

ABSTRACT To solve the problems of localization and identification of fish in the complex fishway environment, improving the accuracy of fish detection, this paper proposes an object detection algorithm YOLORG, and a fishway fish detection dataset (FFDD). The FFDD contains 4,591 images from the web and lab shots and labeled with the LabelIMG tool, covering fish in a wide range of complex scenarios. The YOLORG algorithm, based on YOLOv8, improves the traditional FPN–PAN network into a C2f Multi-scale feature fusion network with a Gather-and-Distribute mechanism, which solves the problem of information loss accompanied by the network in the fusion of feature maps of different sizes. Also, we propose a C2D Structural Re-parameterization module with a rich gradient flow and good performance to further improve the detection accuracy of the algorithm. The experimental results show that the YOLORG algorithm improves the mAP50 and mAP50-95 by 1.2 and 1.8% compared to the original network under the joint VOC dataset, and also performs very well in terms of accuracy compared to other state-of-the-art object detection algorithms, and is able to detect fish in very turbid environments after training on the FFDD.

A novel diagnostic framework based on vibration image encoding and multi-scale neural network

Intelligent fault diagnosis employing CNN-based techniques has demonstrated promising results in rotating machinery maintenance and management. However, most approaches that rely on time-series vibration signals fail to incorporate the physical knowledge of faults into feature learning, and are subject to obtaining the correlation between input and output characteristics. Furthermore, it is challenging for the neural network to improve feature extraction due to the rigid convolution kernel and the limited receptive field. To address these issues, an intelligent fault diagnosis framework is developed based on vibration image encoding and multi-scale neural networks. Firstly, a novel image encoding rule is designed to convert time-series signals into vibration images, which incorporate the impact characteristics of faults into the model inputs to enhance fault information. Secondly, a multi-scale feature fusion network with dilatated and irregular sampling convolution block is constructed for feature extraction and classification. In addition, a new regularization term is designed to avoid training overfitting. Finally, Grad-CAM is introduced to visualize the region of interest of the model. The proposed method is verified on the datasets with different fault types from our laboratory and DIRG bearing dataset with different fault severities from Politecnico di Torino. The experimental results demonstrate that our diagnostic framework exhibits good generalization and noise robustness, and can achieve high-precision fault diagnosis.

MTC-Net: Multi-scale feature fusion network for medical image segmentation

Image segmentation is critical in medical image processing for lesion detection, localisation, and subsequent diagnosis. Currently, computer-aided diagnosis (CAD) has played a significant role in improving diagnostic efficiency and accuracy. The segmentation task is made more difficult by the hazy lesion boundaries and uneven forms. Because standard convolutional neural networks (CNNs) are incapable of capturing global contextual information, adequate segmentation results are impossible to achieve. We propose a multiscale feature fusion network (MTC-Net) in this paper that integrates deep separable convolution and self-attentive modules in the encoder to achieve better local continuity of images and feature maps. In the decoder, a multi-branch multi-scale feature fusion module (MSFB) is utilized to improve the network’s feature extraction capability, and it is integrated with a global cooperative aggregation module (GCAM) to learn more contextual information and adaptively fuse multi-scale features. To develop rich hierarchical representations of irregular forms, the suggested detail enhancement module (DEM) adaptively integrates local characteristics with their global dependencies. To validate the effectiveness of the proposed network, we conducted extensive experiments, evaluated on the public datasets of skin, breast, thyroid and gastrointestinal tract with ISIC2018, BUSI, TN3K and Kvasir-SEG. The comparison with the latest methods also verifies the superiority of our proposed MTC-Net in terms of accuracy. Our code on https://github.com/gih23/MTC-Net.

An Automated Multi-scale Feature Fusion Network for Spine Fracture Segmentation Using Computed Tomography Images.

Spine fractures represent a critical health concern with far-reaching implications for patient care and clinical decision-making. Accurate segmentation of spine fractures from medical images is a crucial task due to its location, shape, type, and severity. Addressing these challenges often requires the use of advanced machine learning and deep learning techniques. In this research, a novel multi-scale feature fusion deep learning model is proposed for the automated spine fracture segmentation using Computed Tomography (CT) to these challenges. The proposed model consists of six modules; Feature Fusion Module (FFM), Squeeze and Excitation (SEM), Atrous Spatial Pyramid Pooling (ASPP), Residual Convolution Block Attention Module (RCBAM), Residual Border Refinement Attention Block (RBRAB), and Local Position Residual Attention Block (LPRAB). These modules are used to apply multi-scale feature fusion, spatial feature extraction, channel-wise feature improvement, segmentation border results border refinement, and positional focus on the region of interest. After that, a decoder network is used to predict the fractured spine. The experimental results show that the proposed approach achieves better accuracy results in solving the above challenges and also performs well compared to the existing segmentation methods.

FNeXter: A Multi-Scale Feature Fusion Network Based on ConvNeXt and Transformer for Retinal OCT Fluid Segmentation.

The accurate segmentation and quantification of retinal fluid in Optical Coherence Tomography (OCT) images are crucial for the diagnosis and treatment of ophthalmic diseases such as age-related macular degeneration. However, the accurate segmentation of retinal fluid is challenging due to significant variations in the size, position, and shape of fluid, as well as their complex, curved boundaries. To address these challenges, we propose a novel multi-scale feature fusion attention network (FNeXter), based on ConvNeXt and Transformer, for OCT fluid segmentation. In FNeXter, we introduce a novel global multi-scale hybrid encoder module that integrates ConvNeXt, Transformer, and region-aware spatial attention. This module can capture long-range dependencies and non-local similarities while also focusing on local features. Moreover, this module possesses the spatial region-aware capabilities, enabling it to adaptively focus on the lesions regions. Additionally, we propose a novel self-adaptive multi-scale feature fusion attention module to enhance the skip connections between the encoder and the decoder. The inclusion of this module elevates the model's capacity to learn global features and multi-scale contextual information effectively. Finally, we conduct comprehensive experiments to evaluate the performance of the proposed FNeXter. Experimental results demonstrate that our proposed approach outperforms other state-of-the-art methods in the task of fluid segmentation.

Multi-Scale Feature Fusion Network with Symmetric Attention for Land Cover Classification Using SAR and Optical Images

The complementary characteristics of SAR and optical images are beneficial in improving the accuracy of land cover classification. Deep learning-based models have achieved some notable results. However, how to effectively extract and fuse the unique features of multi-modal images for pixel-level classification remains challenging. In this article, a two-branch supervised semantic segmentation framework without any pretrained backbone is proposed. Specifically, a novel symmetric attention module is designed with improved strip pooling. The multiple long receptive fields can better perceive irregular objects and obtain more anisotropic contextual information. Meanwhile, to solve the semantic absence and inconsistency of different modalities, we construct a multi-scale fusion module, which is composed of atrous spatial pyramid pooling, varisized convolutions and skip connections. A joint loss function is introduced to constrain the backpropagation and reduce the impact of class imbalance. Validation experiments were implemented on the DFC2020 and WHU-OPT-SAR datasets. The proposed model achieved the best quantitative values on the metrics of OA, Kappa and mIoU, and its class accuracy was also excellent. It is worth mentioning that the number of parameters and the computational complexity of the method are relatively low. The adaptability of the model was verified on RGB–thermal segmentation task.

Double multi-scale feature fusion network for crowd counting

Double multi-scale feature fusion network for crowd counting

DS-MSFF-Net: Dual-path self-attention multi-scale feature fusion network for CT image segmentation

DS-MSFF-Net: Dual-path self-attention multi-scale feature fusion network for CT image segmentation

Collaborative Attention Guided Multi-Scale Feature Fusion Network for Medical Image Segmentation

Collaborative Attention Guided Multi-Scale Feature Fusion Network for Medical Image Segmentation

PASTFNet: a paralleled attention spatio-temporal fusion network for micro-expression recognition.

Micro-expressions (MEs) play such an important role in predicting a person's genuine emotions, as to make micro-expression recognition such an important resea rch focus in recent years. Most recent researchers have made efforts to recognize MEs with spatial and temporal information of video clips. However, because of their short duration and subtle intensity, capturing spatio-temporal features of micro-expressions remains challenging. To effectively promote the recognition performance, this paper presents a novel paralleled dual-branch attention-based spatio-temporal fusion network (PASTFNet). We jointly extract short- and long-range spatial relationships in spatial branch. Inspired by the composite architecture of the convolutional neural network (CNN) and long short-term memory (LSTM) for temporal modeling, we propose a novel attention-based multi-scale feature fusion network (AMFNet) to encode features of sequential frames, which can learn more expressive facial-detailed features for it implements the integrated use of attention and multi-scale feature fusion, then design an aggregation block to aggregate and acquire temporal features. At last, the features learned by the above two branches are fused to accomplish expression recognition with outstanding effect. Experiments on two MER datasets (CASMEII and SAMM) show that the PASTFNet model achieves promising ME recognition performance compared with other methods.

Exploiting multi-scale hierarchical feature representation for visual tracking

Convolutional neural networks (CNNs) have been the dominant architectures for feature extraction tasks, but CNNs do not look for and focus on some specific image features. Correlation operations play an important role in visual tracking. However, the correlation operation reserves a large amount of unfavorable background information. In this paper, we propose an effective feature recognizer including channel and spatial attention modules to focus on important object feature information. Thus, the representation power of the feature extraction network is improved. Further, we design a multi-scale feature fusion network. The fusion network performs feature fusion on template feature and encoded feature branches to establish connections between features at different scales. Experiments on six benchmarks demonstrate that the proposed tracker outperforms the state-of-the-art trackers. In particular, the proposed tracker achieves an 80.4% AUC on TrackingNet and a 68.4% AUC on GOT-10k while running at a real-time speed.

IMFF-Net: An integrated multi-scale feature fusion network for accurate retinal vessel segmentation from fundus images

Extracting vascular structures from retinal fundus images plays a critical role in the early diagnosis and long-term treatment of ophthalmic diseases. Traditional manual segmentation of retinal vessels is a time-consuming process that demands a high level of expertise. In recent years, deep learning has made significant strides in retinal vessel segmentation, but it still faces certain challenges in fine vessel segmentation, such as the loss of spatial information resulting from multi-level feature extraction and the blurring of fine structural segmentation. To address these issues, we propose a multi-scale feature fusion segmentation network known as IMFF-Net. Specifically, we propose two fusion blocks in the IMFF-Net. Firstly, an Attention Pooling Feature Fusion (APF) block is proposed, which consists of Max Pooling, and Average Pooling and incorporates the SE block. This design effectively mitigates the problem of spatial information loss stemming from multiple pooling operations. Secondly, the Upsampling and Downsampling Feature Fusion block (UDFF) is proposed to weightedly merge the feature maps of each downsampling with the upsampling feature maps, thereby facilitating the precise segmentation of fine structures. To validate the performance of the proposed IMFF-Net, we conducted experiments on three retinal blood vessel segmentation datasets: DRIVE, STARE, and CHASE_DB1. IMFF-Net achieved outstanding results on the test set of these three public datasets with accuracies of 0.9621, 0.9707, and 0.9730, and sensitivities of 0.8575, 0.8634, and 0.8048, respectively. These results demonstrate the superior performance of IMFF-Net compared to the backbone network and other state-of-the-art methods. Our code is available at: https://github.com/wangyunyuwyy/IMFF-Net.

Semantic Correlation Attention-Based Multiorder Multiscale Feature Fusion Network for Human Motion Prediction.

Human motion prediction is to predict future human states based on the observed human states. However, current research ignores the semantic correlations between body parts (joints and bones) in the observed human states and motion time; thus, the prediction accuracy is limited. To address this issue, we propose a novel semantic correlation attention-based multiorder multiscale feature fusion network (SCAFF), which includes an encoder and a decoder. In the encoder, a multiorder difference calculation module (MODC) is designed to calculate the multiorder difference information of joint and bone attributes in the observed human states. Then, multiple semantic correlation attention-based graph calculation operators (SCA-GCOs) are stacked to extract the multiscale features of the multiorder difference information. Each SCA-GCO captures joint and bone dependencies of the multiorder difference information, refines them with a semantic correlation attention module (SCAM), and captures temporal dynamics of the refined joint and bone dependencies as the output features. Note that SCAM learns a semantic attention mask describing the semantic correlations between body parts and motion time for feature refinement. Afterward, multiple multiorder feature fusion modules (MOFFs) and multiscale feature fusion modules (MSFFs) are designed to fuse the multiscale features of the multiorder difference information extracted by multiple SCA-GCOs, thus obtaining the motion features of the observed human states. Based on the obtained motion features, the decoder recurrently recruits a composite gated recurrent module (CGRM) and multilayer perceptrons (MLPs) to predict future human states. As far as we know, this is the first attempt to consider the semantic correlations between body parts and motion time in human motion prediction. The results on public datasets demonstrate that SCAFF outperforms existing models.

Accurate and fast detection of tomatoes based on improved YOLOv5s in natural environments.

Uneven illumination, obstruction of leaves or branches, and the overlapping of fruit significantly affect the accuracy of tomato detection by automated harvesting robots in natural environments. In this study, a proficient and accurate algorithm for tomato detection, called SBCS-YOLOv5s, is proposed to address this practical challenge. SBCS-YOLOv5s integrates the SE, BiFPN, CARAFE and Soft-NMS modules into YOLOv5s to enhance the feature expression ability of the model. First, the SE attention module and the C3 module were combined to form the C3SE module, replacing the original C3 module within the YOLOv5s backbone architecture. The SE attention module relies on modeling channel-wise relationships and adaptive re-calibration of feature maps to capture important information, which helps improve feature extraction of the model. Moreover, the SE module's ability to adaptively re-calibrate features can improve the model's robustness to variations in environmental conditions. Next, the conventional PANet multi-scale feature fusion network was replaced with an efficient, weighted Bi-directional Feature Pyramid Network (BiFPN). This adaptation aids the model in determining useful weights for the comprehensive fusion of high-level and bottom-level features. Third, the regular up-sampling operator is replaced by the Content Aware Reassembly of Features (CARAFE) within the neck network. This implementation produces a better feature map that encompasses greater semantic information. In addition, CARAFE's ability to enhance spatial detail helps the model discriminate between closely spaced fruits, especially for tomatoes that overlap heavily, potentially reducing the number of merging detections. Finally, for heightened identification of occluded and overlapped fruits, the conventional Non-Maximum-Suppression (NMS) algorithm was substituted with the Soft-NMS algorithm. Since Soft-NMS adopts a continuous weighting scheme, it is more adaptable to varying object sizes, improving the handling of small or large fruits in the image. Remarkably, this is carried out without introducing changes to the computational complexity. The outcome of the experiments showed that SBCS-YOLOv5s achieved a mean average precision (mAP (0.5:0.95)) of 87.7%, which is 3.5% superior to the original YOLOv5s model. Moreover, SBCS-YOLOv5s has a detection speed of 2.6 ms per image. Compared to other state-of-the-art detection algorithms, SBCS-YOLOv5s performed the best, showing tremendous promise for tomato detection in natural environments.

Direction-Guided Multiscale Feature Fusion Network for Geo-Localization

Direction-Guided Multiscale Feature Fusion Network for Geo-Localization

A Lightweight Multiscale Feature Fusion Network for Solar Cell Defect Detection

A Lightweight Multiscale Feature Fusion Network for Solar Cell Defect Detection

MFFNet: a building change detection method based on fusion of spectral and geometric information

Accurate detection and extraction of changes in buildings heights is important in monitoring construction (both legal and illegal) and assessing disasters. It is also important information for updating real 3D scenes. However, when using remote sensing images, shadows, vegetation and objects with similar spectral and morphological characteristics as buildings can cause false detections, omissions and incomplete patch edges. To address this issue, we develop the multiscale feature fusion network for dual-modal data (MFFNet), which has two main aspects: (1) The multi-dual-modal feature fusion module detects changes in features with similar spectral and morphological characteristics as buildings. This mitigates false detections by making the model more aware of areas where the elevation has changed over time. (2) Because building extraction is affected by shadows and vegetation, we designed a multiscale feature shuffle module. It takes multiscale features and establishes relationships between neighbouring pixels using the pixel-shuffle algorithm, then fuses and reorganizes the multiscale features to highlight the relationships between global contexts, thereby mitigating the problem of building occlusion by shadows. Comparative experiments show that MFFNet achieves better results on GF7-CD and 3DCD datasets than other similar methods. The proposed method can more accurately monitor building changes over large areas.