Multi-scale Fusion Network Research Articles

Composite Perception and Multi-scale Fusion Network for Arbitrary-Oriented Object Detection in Remote Sensing Imagery

Composite Perception and Multi-scale Fusion Network for Arbitrary-Oriented Object Detection in Remote Sensing Imagery

Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

Densely Multiscale Fusion Network for Lightweight and Accurate Semantic Segmentation of Railway Scenes

Densely Multiscale Fusion Network for Lightweight and Accurate Semantic Segmentation of Railway Scenes

PSMFNet: Lightweight Partial Separation and Multiscale Fusion Network for Image Super-Resolution

PSMFNet: Lightweight Partial Separation and Multiscale Fusion Network for Image Super-Resolution

BFG&MSF-Net: Boundary Feature Guidance and Multi-Scale Fusion Network for Thyroid Nodule Segmentation

BFG&MSF-Net: Boundary Feature Guidance and Multi-Scale Fusion Network for Thyroid Nodule Segmentation

Local–Global Multiscale Fusion Network for Semantic Segmentation of Buildings in SAR Imagery

Local–Global Multiscale Fusion Network for Semantic Segmentation of Buildings in SAR Imagery

Highway Visibility Estimation in Foggy Weather via Multi-Scale Fusion Network.

Poor visibility has a significant impact on road safety and can even lead to traffic accidents. The traditional means of visibility monitoring no longer meet the current needs in terms of temporal and spatial accuracy. In this work, we propose a novel deep network architecture for estimating the visibility directly from highway surveillance images. Specifically, we employ several image feature extraction methods to extract detailed structural, spectral, and scene depth features from the images. Next, we design a multi-scale fusion network to adaptively extract and fuse vital features for the purpose of estimating visibility. Furthermore, we create a real-scene dataset for model learning and performance evaluation. Our experiments demonstrate the superiority of our proposed method to the existing methods.

Attention-Guided Multi-Scale Fusion Network for Similar Objects Semantic Segmentation

Attention-Guided Multi-Scale Fusion Network for Similar Objects Semantic Segmentation

SADFusion: A multi-scale infrared and visible image fusion method based on salient-aware and domain-specific

Image fusion intends to generate an informative image with maximally possible features and details of the source images. However, the existing deep-learning fusion methods are rarely considering the discrepancy of visible and infrared (IVIF) modalities, besides losing control of balance in preserving the texture details and thermal target, owing to the identical structure and parameters of multi-modality feature extractors’ designing and weak domain guidance of network training. In this paper, a novel multi-scale fusion network based on the salient-aware and domain-specific methods is proposed, termed as SADFusion. To boost the multi-modality feature extracting performance of our method, the proposed network adopts dual encoding with short connection and multi-scale structure. Moreover, the domain-specific framework and corresponding training strategy are designed to achieve the identical encoding for different image modalities. Moreover, multi-scale attention fusion modules (MSFAF modules) are proposed to effectively fuse the extracted complementary features at every scale. Finally, we construct the specific salient-aware loss to guide the model to trade off the preserving necessary information, by utilizing salient modality features as pixel-to-pixel intensity and gradient maps. Experiments based on the public datasets demonstrate the superiority of our method over the state-of-art fusion methods, which particularly highlights the targets and retains the effective information.

Retinal artery/vein classification by multi-channel multi-scale fusion network

Retinal artery/vein classification by multi-channel multi-scale fusion network

NRVC: Neural Representation for Video Compression with Implicit Multiscale Fusion Network.

Recently, end-to-end deep models for video compression have made steady advancements. However, this resulted in a lengthy and complex pipeline containing numerous redundant parameters. The video compression approaches based on implicit neural representation (INR) allow videos to be directly represented as a function approximated by a neural network, resulting in a more lightweight model, whereas the singularity of the feature extraction pipeline limits the network's ability to fit the mapping function for video frames. Hence, we propose a neural representation approach for video compression with an implicit multiscale fusion network (NRVC), utilizing normalized residual networks to improve the effectiveness of INR in fitting the target function. We propose the multiscale representations for video compression (MSRVC) network, which effectively extracts features from the input video sequence to enhance the degree of overfitting in the mapping function. Additionally, we propose the feature extraction channel attention (FECA) block to capture interaction information between different feature extraction channels, further improving the effectiveness of feature extraction. The results show that compared to the NeRV method with similar bits per pixel (BPP), NRVC has a 2.16% increase in the decoded peak signal-to-noise ratio (PSNR). Moreover, NRVC outperforms the conventional HEVC in terms of PSNR.

DSNet: Dual-stream multi-scale fusion network for low-quality 3D face recognition

3D face recognition (FR) has become increasingly widespread due to the illumination invariance and pose robustness of 3D face data. Most existing 3D FR methods can only achieve excellent performance on complete and smooth faces. However, low-quality 3D FR with missing facial features still suffers from insufficient discriminative feature extraction for visible face regions. In this paper, we propose a dual-stream multi-scale fusion network (DSNet) for low-quality 3D FR. First, in the first stream, we design a new multi-scale local and global feature fusion network, which consists of an enhanced shallow feature extraction module, an enhanced deep feature extraction module, and a layered multi-scale feature correlation fusion module, aiming to obtain more discriminative details and category information of the facial visible region, reducing the interference of similar features and the redundancy of the same features. Second, we also introduced a capsule network as the second stream to enhance the expression of 3D facial spatial position information, thereby further improving the performance of low-quality 3D FR with missing facial features. We conduct extensive experiments on low-quality datasets (Lock3DFace, KinectFaceDB, and IIIT-D) and cross-quality datasets synthesized by Bosphorus. These results show that our proposed DSNet can achieve state-of-the-art recognition performance and exhibit excellent performance on low-quality 3D faces with missing facial features.

Counting with Self-Weighted Multi-Scale Fusion Networks

Because of the large-scale variation, counting in scenes of different densities is an extremely difficult task. In this paper, based on the attention mechanism, we propose a new self-weighted multi-scale fusion network structure named SMFNet to solve the problem of multi-scale changes and can significantly improve the effect of crowd counting in monitoring scene. The proposed SMFNet uses VGG as the backbone network to extract multi-scale features, uses a SMFNet as the neck to fuse multiple-scale features, and uses the atrous spatial pyramid pooling (ASPP) network and ordinary convolution as the head to generate both the attention map and the density map. The attention map highlighting crowd regions in the image contributes to a high-quality density map, and the density map records the crowd distribution. The number of crowd in the image can be obtained by summing the pixel values of the density map. We conduct experiments on three crowd counting datasets and one vehicle counting dataset to show that our proposed SMFNet can improve the state-of-the-art counting methods.

An Efficient Multi-Scale Fusion Network for 3D Organs at Risk (OARs) Segmentation.

Accurate segmentation of organs-at-risks (OARs) is a precursor for optimizing radiation therapy planning. Existing deep learning-based multi-scale fusion architectures have demonstrated a tremendous capacity for 2D medical image segmentation. The key to their success is aggregating global context and maintaining high resolution representations. However, when translated into 3D segmentation problems, existing multi-scale fusion architectures might underperform due to their heavy computation overhead and substantial data diet. To address this issue, we propose a new OAR segmentation framework, called OARFocalFuseNet, which fuses multi-scale features and employs focal modulation for capturing global-local context across multiple scales. Each resolution stream is enriched with features from different resolution scales, and multi-scale information is aggregated to model diverse contextual ranges. As a result, feature representations are further boosted. The comprehensive comparisons in our experimental setup with OAR segmentation as well as multi-organ segmentation show that our proposed OARFocalFuseNet outperforms the recent state-of-the-art methods on publicly available OpenKBP datasets and Synapse multi-organ segmentation. Both of the proposed methods (3D-MSF and OARFocalFuseNet) showed promising performance in terms of standard evaluation metrics. Our best performing method (OARFocalFuseNet) obtained a dice coefficient of 0.7995 and hausdorff distance of 5.1435 on OpenKBP datasets and dice coefficient of 0.8137 on Synapse multi-organ segmentation dataset. Our code is available at https://github.com/NoviceMAn-prog/OARFocalFuse.

Dual attention guided multi-scale fusion network for RGB-D salient object detection

While recent research on salient object detection (SOD) has shown remarkable progress in leveraging both RGB and depth data, it is still worth exploring how to use the inherent relationship between the two to extract and fuse features more effectively, and further make more accurate predictions. In this paper, we consider combining the attention mechanism with the characteristics of the SOD, proposing the Dual Attention Guided Multi-scale Fusion Network. We design the multi-scale fusion block by combining multi-scale branches with channel attention to achieve better fusion of RGB and depth information. Using the characteristic of the SOD, the dual attention module is proposed to make the network pay more attention to the currently unpredicted saliency regions and the wrong parts in the already predicted regions. We perform an ablation study to verify the effectiveness of each component. Quantitative and qualitative experimental results demonstrate that our method achieves state-of-the-art (SOTA) performance.

VFSN: a ResNet Multi-scale Fusion Network for Metal Defect Detection

Defect detection is an imperative step in ensuring the quality of steel products. To overcome the problem of low accuracy and poor detection of subtle features in current detection methods, a revised neural network model structure has been proposed. VFSN’s main convolutional layer structure is VGG with residuals, where two convolutional blocks are designed. It’s connected to a spatial pyramid pool (SPP) after reducing the aliasing effects and detecting defect features at various scales by constructing multibranch inputs and multilevel feature overlays and fusions. Research on the open-source data set NEU-DET has shown high-accuracy recognition, with the recognition accuracy rate reaching 75.1% and an average accuracy rate improving by 2.7%. The F1 score improved by 1.3%. There’s a significant improvement in detecting small defects using the proposed network structure.

JAMSNet: A Remote Pulse Extraction Network Based on Joint Attention and Multi-Scale Fusion

Remote photoplethysmography (rPPG) has been an active research topic in recent years. While most existing methods are focusing on eliminating motion artifacts in the raw traces obtained from single-scale region-of-interest (ROI), it is worth noting that there are some noise signals that cannot be effectively separated in single-scale space but can be separated more easily in multi-scale space. In this paper, we analyze the distribution of pulse signal and motion artifacts in different layers of a Gaussian pyramid. We propose a method that combines multi-scale analysis and neural network for pulse extraction in different scales, and a layer-wise attention mechanism to adaptively fuse the features according to signal strength. In addition, we propose spatial-temporal joint attention module and channel-temporal joint attention module to learn and exaggerate pulse features in the joint spaces, respectively. The proposed remote pulse extraction network is called Joint Attention and Multi-Scale fusion Network (JAMSNet). Extensive experiments have been conducted on two publicly available datasets and one self-collected dataset. The results show that the proposed JAMSNet shows better performance than state-of-the-art methods.

FE-Fusion-VPR: Attention-Based Multi-Scale Network Architecture for Visual Place Recognition by Fusing Frames and Events

Traditional visual place recognition (VPR), usually using standard cameras, is easy to fail due to glare or high-speed motion. By contrast, event cameras have the advantages of low latency, high temporal resolution, and high dynamic range, which can deal with the above issues. Nevertheless, event cameras are prone to failure in motionless scenes, while standard cameras can still provide appearance information in this case. Thus, exploiting the complementarity of standard cameras and event cameras can effectively improve the performance of VPR algorithms. In the paper, we propose FE-Fusion-VPR, an attention-based multi-scale network architecture for VPR by fusing frames and events. First, the intensity frame and event volume are fed into the two-stream feature extraction network for shallow feature fusion. Next, the three-scale features are obtained through the multi-scale fusion network and aggregated into three sub-descriptors using the VLAD layer. Finally, the weight of each sub-descriptor is learned through the descriptor re-weighting network to obtain the final refined descriptor. Experimental results show that our FE-Fusion-VPR outperforms existing frame-based, event-based and fusion-based VPR methods in most cases on Brisbane-Event-VPR and DDD20 datasets. In a word, compared to the previous works, our FE-Fusion-VPR achieves new state-of-the-art (SOTA) VPR performance in Brisbane-Event-VPR and DDD20 datasets by fusing frames and events.

Hyperspectral Image Classification Based on Multiscale Hybrid Networks and Attention Mechanisms

Hyperspectral image (HSI) classification is one of the most crucial tasks in remote sensing processing. The attention mechanism is preferable to a convolutional neural network (CNN), due to its superior ability to express information during HSI processing. Recently, numerous methods combining CNNs and attention mechanisms have been applied in HSI classification. However, it remains a challenge to achieve high-accuracy classification by fully extracting effective features from HSIs under the conditions of limited labeled samples. In this paper, we design a novel HSI classification network based on multiscale hybrid networks and attention mechanisms. The network consists of three subnetworks: a spectral-spatial feature extraction network, a spatial inverted pyramid network, and a classification network, which are employed to extract spectral-spatial features, to extract spatial features, and to obtain classification results, respectively. The multiscale fusion network and attention mechanisms complement each other by capturing local and global features separately. In the spatial pyramid network, multiscale spaces are formed through down-sampling, which can reduce redundant information while retaining important information. The structure helps the network better capture spatial features at different scales, and to improve classification accuracy. Experimental results on various public HSI datasets demonstrate that the designed network is extremely competitive compared to current advanced approaches, under the condition of insufficient samples.

Deformable multi-scale fusion network for non-uniform single image deblurring

Deformable multi-scale fusion network for non-uniform single image deblurring