Multi-scale Fusion Network Research Articles

Multiscale Structure and Texture Feature Fusion for Image Inpainting

In order to achieve interaction between structure and texture information in generative adversarial image inpainting networks and improve the semantic veracity of the restored images, unlike the original two-stage inpainting ideas where texture and structure are restored separately, this paper constructs a multi-scale fusion approach to image generation, which embeds images into two collaborative subtasks, that is, structure generation and texture synthesis under structural constraints. We also introduce a self-attention mechanism into the partial convolution of the encoder to enhance the long range contextual information acquisition of the model in image inpainting, and design a multi-scale fusion network to fuse the generated structure and texture feature, so that the structure and texture information can be reused for reconstruction, perception and style loss compensation, thus enabling the fused images to achieve global consistency. In the training phase, feature matching loss are introduced to enhance the image in terms of structural generation plausibility. Finally, through comparison experiments with other inpainting networks on the CelebA, Paris StreetView and Places2 datasets, it is demonstrated that our method constructed in this paper has better objective evaluation metrics, more effective inpainting of structural and texture information of corrupted images and better image inpainting performance.

Remote Sensing Scene Classification Based on Attention-Enabled Progressively Searching

Remote sensing image scene classification plays a significant role in remote sensing image analysis. Aiming at the problems of large transformation and scale variation of background and key objects in remote sensing images, we propose a neural architecture search (NAS) method based on attention search space. The network adaptively searches convolution, pooling, and attention operations in the appropriate layers. To ensure the stability of the searching process, a multistage network progressive fusion search method is proposed, which discards useless operations in stages, reduces the burden of search algorithm, and improves the search efficiency. Finally, paying attention to the association information between objects and scenes, a bottom-up multiscale fusion network connection strategy is proposed to fully reuse the semantics of multiscale feature maps in each stage. The experimental results show that the proposed method performs better than the manual method and the current neural network architecture search method.

Automated vessel segmentation in lung CT and CTA images via deep neural networks

The distribution of pulmonary vessels in computed tomography (CT) and computed tomography angiography (CTA) images of lung is important for diagnosing disease, formulating surgical plans and pulmonary research. Based on the pulmonary vascular segmentation task of International Symposium on Image Computing and Digital Medicine 2020 challenge, this paper reviews 12 different pulmonary vascular segmentation algorithms of lung CT and CTA images and then objectively evaluates and compares their performances. First, we present the annotated reference dataset of lung CT and CTA images. A subset of the dataset consisting 7,307 slices for training and 3,888 slices for testing was made available for participants. Second, by analyzing the performance comparison of different convolutional neural networks from 12 different institutions for pulmonary vascular segmentation, the reasons for some defects and improvements are summarized. The models are mainly based on U-Net, Attention, GAN, and multi-scale fusion network. The performance is measured in terms of Dice coefficient, over segmentation rate and under segmentation rate. Finally, we discuss several proposed methods to improve the pulmonary vessel segmentation results using deep neural networks. By comparing with the annotated ground truth from both lung CT and CTA images, most of 12 deep neural network algorithms do an admirable job in pulmonary vascular extraction and segmentation with the dice coefficients ranging from 0.70 to 0.85. The dice coefficients for the top three algorithms are about 0.80. Study results show that integrating methods that consider spatial information, fuse multi-scale feature map, or have an excellent post-processing to deep neural network training and optimization process are significant for further improving the accuracy of pulmonary vascular segmentation.

SDFNet: Automatic segmentation of kidney ultrasound images using multi-scale low-level structural feature

Due to speckle noise, changes in kidney shape and size between patients, and similar regions, segmenting kidneys in ultrasound images is challenging. To alleviate this challenge, we proposed a novel CNN model, namely multi-scale fusion network of structural features and detailed features (SDFNet), to segment kidneys accurately and robustly. Specifically, the architecture includes structure feature extraction network (S-Net), detail information extraction network (D-Net) and multi-scale fusion block (MCBlock), which are in charge of extracting structural features, capturing texture details and merging features, respectively. In S-Net, we designed a boundary detection (BD) module to obtain more complete kidney structural features. In addition, this paper also designed a step-by-step training mechanism to improve the generalization ability of the SDFNet. We validated the proposed method and compared the same kidney ultrasound dataset with several state-of-the-art methods using six quantitative indicators. The results demonstrate that the proposed approach achieves the best overall performance and outperforms all other approaches on kidney ultrasound image segmentation. It is worth noting that this paper quantitatively analyzes the loss function of segment kidney ultrasound images. This work is a good reference for future research.

Unified Information Fusion Network for Multi-Modal RGB-D and RGB-T Salient Object Detection

The use of complementary information, namely depth or thermal information, has shown its benefits to salient object detection (SOD) during recent years. However, the RGB-D or RGB-T SOD problems are currently only solved independently, and most of them directly extract and fuse raw features from backbones. Such methods can be easily restricted by low-quality modality data and redundant cross-modal features. In this work, a unified end-to-end framework is designed to simultaneously analyze RGB-D and RGB-T SOD tasks. Specifically, to effectively tackle multi-modal features, we propose a novel multi-stage and multi-scale fusion network (MMNet), which consists of a cross-modal multi-stage fusion module (CMFM) and a bi-directional multi-scale decoder (BMD). Similar to the visual color stage doctrine in the human visual system (HVS), the proposed CMFM aims to explore important feature representations in feature response stage, and integrate them into cross-modal features in adversarial combination stage. Moreover, the proposed BMD learns the combination of multi-level cross-modal fused features to capture both local and global information of salient objects, and can further boost the multi-modal SOD performance. The proposed unified cross-modality feature analysis framework based on two-stage and multi-scale information fusion can be used for diverse multi-modal SOD tasks. Comprehensive experiments (<inline-formula> <tex-math notation="LaTeX">$\sim 92\text{K}$ </tex-math></inline-formula> image-pairs) demonstrate that the proposed method consistently outperforms the other 21 state-of-the-art methods on nine benchmark datasets. This validates that our proposed method can work well on diverse multi-modal SOD tasks with good generalization and robustness, and provides a good multi-modal SOD benchmark.

Detecting the Early Flowering Stage of Tea Chrysanthemum Using the F-YOLO Model

Detecting the flowering stage of tea chrysanthemum is a key mechanism of the selective chrysanthemum harvesting robot. However, under complex, unstructured scenarios, such as illumination variation, occlusion, and overlapping, detecting tea chrysanthemum at a specific flowering stage is a real challenge. This paper proposes a highly fused, lightweight detection model named the Fusion-YOLO (F-YOLO) model. First, cutout and mosaic input components are equipped, with which the fusion module can better understand the features of the chrysanthemum through slicing. In the backbone component, the Cross-Stage Partial DenseNet (CSPDenseNet) network is used as the main network, and feature fusion modules are added to maximize the gradient flow difference. Next, in the neck component, the Cross-Stage Partial ResNeXt (CSPResNeXt) network is taken as the main network to truncate the redundant gradient flow. Finally, in the head component, the multi-scale fusion network is adopted to aggregate the parameters of two different detection layers from different backbone layers. The results show that the F-YOLO model is superior to state-of-the-art technologies in terms of object detection, that this method can be deployed on a single mobile GPU, and that it will be one of key technologies to build a selective chrysanthemum harvesting robot system in the future.

Crowd counting based on attention-guided multi-scale fusion networks

In this paper, we propose an attention-guided multi-scale fusion network (named as AMS-Net) for crowd counting in dense scenarios. The overall model is mainly comprised by the density and the attention networks. The density network is able to provide a coarse prediction of the crowd distribution (density map), while the attention network helps to distinguish crowded regions from backgrounds. The output of the attention network serves as a mask of the coarse density map. The number of persons in the scene is finally estimated by applying integration on the refined density map. In order to deal with persons of varied resolutions, we introduce a multi-scale fusion strategy which is built upon dilated convolution. Experiments are carried out on the standard benchmark datasets, covering varied over-crowded scenarios. Experimental results demonstrate the effectiveness of the proposed approach.

Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network

Real-time monitoring and surveillance play an important role in the field of remote sensing, where multi-spectral (MS) images with high spatial resolution are widely desired for better analysis. However, high-resolution MS images cannot be directly obtained due to the limitations of sensors and bandwidth. As an essential way to alleviate this problem, pan-sharpening aims at fusing the complementary information of a low-resolution MS image and a high-resolution panchromatic (PAN) image to reconstruct a high-resolution MS image. Most previous deep-learning based methods can meet the real-time requirements with the help of graphics processing unit (GPU). However, they don’t fully exploit the favorable hierarchical information, sparing huge room for performance improvement. In this paper, to meet the requirement of real-time implementation and achieve more effective performance simultaneously, we propose a multi-scale fusion network (MSFN) to make full use of hierarchical complementary features of PAN and MS images. Specifically, we introduce an encoder–decoder structure and coarse-to-fine strategy to effectively extract multi-scale features of PAN and MS images, separately. Meanwhile, an information pool is adopted to preserve primitive information. Then a multi-scale feature fusion module is applied to fuse multi-scale features from the decoder and information pool. Finally, the fused features are utilized to reconstruct the high-resolution MS image. Extensive experiments demonstrate that our proposed method achieves favorable performance against other methods in terms of quantitative metrics and visual quality. Besides, the results on running time indicate that our method can achieve real-time performance.

Strong ghost removal in multi-exposure image fusion using hole-filling with exposure congruency

It is the most crucial problem to remove ghost in the multi-exposure image fusion of dynamic scene. The traditional fusion methods have good effects to remove weak ghosts. However, they cannot effectively remove strong ghosts. This paper proposes a new strong ghost removal method in multi-exposure image fusion using hole-filling with exposure congruency. First, analyzing the characteristics of strong ghosts, a detection scheme for strong ghost regions is designed by combining histogram matching and exposure difference detection. Subsequently, to effectively extract image local features, a multi-scale fusion network for non-strong ghost regions is designed to obtain a pre-fused image. Further, based on the distribution characteristics of strong ghosts, a hole-filling model with exposure congruency is designed to remove the strong ghosts. Experimental results show that compared with the state-of-the-art methods, the proposed method can obtain better performance in both of subjective and objective evaluation, particularly in terms of effectively removing strong ghosts.

Noise Attenuation of Seismic Data via Deep Multiscale Fusion Network

Convolutional neural network‐ (CNN‐) based deep learning (DL) architectures have achieved great success in many fields such as remote sensing, medical image processing, and computer vision. Recently, CNN‐based models have also been attempted to solve geophysical problems. This paper presents a noise attenuation method of seismic data via a novel deep learning (DL) architecture, namely, deep multiscale fusion network (MSFN). Firstly, we integrate multiscale fusion (MSF) block to adaptively exploit local signal features at different scales from seismic data. And then, a series of stacked MSF blocks are formed into MSFN, which can restore the noisy seismic data effectively and preserve more useful signal information. Furthermore, a comparative study of our method and other leading edge ones is conducted by using synthetic seismic records and the SEG/EAGE salt and overthrust models. The results qualitatively and quantitatively show the capability of our method of achieving higher peak signal‐to‐noise ratios (PSNRs) while preserving much more useful information, comparing with other methods. Finally, our method is utilized in the real seismic data processing, obtaining satisfactory results.

Multi-scale fusion and non-local attention mechanism based salient object detection

With the development of deep learning, researches in the field of computer vision are attracting more attention. As the pre-processing operation of visual tasks, a salient model may focus on pure architectures. The paper proposes a new multi-scale fusion network to enrich high-level redundant information with the enlarged receptive field. With the guidance of attention mechanism, the framework can capture more effective correlation spatial and channels information. Building a short-connection between high-level and each level features to transmit the contextual features. The model can be used in a variety of complex scenes for end-to- end image detection, with simple structure and strong versatility. Experimental results obtained on multiple common datasets have shown that the proposed model achieved better performance both in the visual effect and the accuracy for small object and multi-target detection.

Fully Dense Multiscale Fusion Network for Hyperspectral Image Classification

The convolutional neural network (CNN) can automatically extract hierarchical feature representations from raw data and has recently achieved great success in the classification of hyperspectral images (HSIs). However, most CNN based methods used in HSI classification neglect adequately utilizing the strong complementary yet correlated information from each convolutional layer and only employ the last convolutional layer features for classification. In this paper, we propose a novel fully dense multiscale fusion network (FDMFN) that takes full advantage of the hierarchical features from all the convolutional layers for HSI classification. In the proposed network, shortcut connections are introduced between any two layers in a feed-forward manner, enabling features learned by each layer to be accessed by all subsequent layers. This fully dense connectivity pattern achieves comprehensive feature reuse and enforces discriminative feature learning. In addition, various spectral-spatial features with multiple scales from all convolutional layers are fused to extract more discriminative features for HSI classification. Experimental results on three widely used hyperspectral scenes demonstrate that the proposed FDMFN can achieve better classification performance in comparison with several state-of-the-art approaches.

Contourlet Transform based Multi-scale Fusion Network for Face Hallucination

Recently, Convolutional neural network (CNN) has achieved great success in the field of face hallucination. However, such approaches usually generate blurry and over-smoothed Super Resolution (SR) results, and the performance suffers from degradation when super-resolve a very Low Resolution (LR) face image. To solve these problems, this paper proposes an contourlet transform based accurate CNN architecture, namely Multi-scale Fusion CNN (MSFC), which are able to reconstruct a High Resolution (HR) face image from a very low resolution input. First of all, we present multi-scale fusion CNN (MSFC) to fully detect and exploit features from LR inputs. And then, we formulate the SR problem as the prediction of contourlet transform coefficients, which is able to make MSFC further capture the texture details for super-resolve face images. Extensive qualitative and quantitative experiments show that the proposed method is capable of preserving details and achieves superior SR performance compared to the state-of-the-art methods.

Fast and Accurate Super-Resolution of FY-2 Infrared Cloud Images via Multi-Scale Fusion Network

This paper proposes an effective method to improve the spatial resolution of FengYun-2 (FY-2) infrared cloud images via deep convolutional neural networks. The proposed model consists of four parts: shallow feature representation block, stacked multi-scale fusion blocks, global feature fusion block, and feature reconstruction block. The multi-scale fusion block combines dilated convolution, local feature fusion and local residual learning to extract multi-scale local features from the original low-resolution image directly. Then these local features are all merged by the global feature fusion block to reconstruct the residual representations in high-resolution space. For training and testing, we have specially built a dataset of infrared cloud images. We evaluated the proposed method both on simulated and real data. Experimental results demonstrate that the proposed approach achieves improved reconstruction accuracy than the state-of-the-art methods. Besides, the concise structure of the proposed model enables it to be applicable in practice.