Multi-scale Fusion Module Research Articles

Attention Enhanced U-Net for Building Extraction from Farmland Based on Google and WorldView-2 Remote Sensing Images

High-resolution remote sensing images contain abundant building information and provide an important data source for extracting buildings, which is of great significance to farmland preservation. However, the types of ground features in farmland are complex, the buildings are scattered and may be obscured by clouds or vegetation, leading to problems such as a low extraction accuracy in the existing methods. In response to the above problems, this paper proposes a method of attention-enhanced U-Net for building extraction from farmland, based on Google and WorldView-2 remote sensing images. First, a Resnet unit is adopted as the infrastructure of the U-Net network encoding part, then the spatial and channel attention mechanism module is introduced between the Resnet unit and the maximum pool and the multi-scale fusion module is added to improve the U-Net network. Second, the buildings found on WorldView-2 and Google images are extracted through farmland boundary constraints. Third, boundary optimization and fusion processing are carried out for the building extraction results on the WorldView-2 and Google images. Fourth, a case experiment is performed. The method in this paper is compared with semantic segmentation models, such as FCN8, U-Net, Attention_UNet, and DeepLabv3+. The experimental results indicate that this method attains a higher accuracy and better effect in terms of building extraction within farmland; the accuracy is 97.47%, the F1 score is 85.61%, the recall rate (Recall) is 93.02%, and the intersection of union (IoU) value is 74.85%. Hence, buildings within farming areas can be effectively extracted, which is conducive to the preservation of farmland.

DF-Net: Deep fusion network for multi-source vessel segmentation

Accurate retinal vessel segmentation is very challenging. Recently, the deep learning based method has greatly improved performance. However, the non-vascular structures usually harm the performance and some low contrast small vessels are hard to be detected after several down-sampling operations. To solve these problems, we design a deep fusion network (DF-Net) including multiscale fusion, feature fusion and classifier fusion for multi-source vessel image segmentation. The multiscale fusion module allows the network to detect blood vessels with different scales. The feature fusion module fuses deep features with vessel responses extracted from a Frangi filter to obtain a compact yet domain invariant feature representation. The classifier fusion module provides the network more supervision. DF-Net also predicts the parameter of the Frangi filter to avoid manually picking the best parameters. The learned Frangi filter enhances the feature map of the multiscale network and restores the edge information loss caused by down-sampling operations. The proposed end-to-end network is easy to train and the inference time for one image is 41ms on a GPU. The model outperforms state-of-the-art methods and achieves the accuracy of 96.14%, 97.04%, 98.02% from three publicly available fundus image datasets DRIVE, STARE, CHASEDB1, respectively. The code is available at https://github.com/y406539259/DF-Net.

Infrared Thermal Imaging Super-Resolution via Multiscale Spatio-Temporal Feature Fusion Network

Infrared thermal imaging is widely used in military and civil fields like security and protecting monitoring, military reconnaissance, road traffic monitoring. However, most current infrared sensors have limited resolution for their technological limitation. To solve it, we propose a novel end-to-end infrared thermal imaging super-resolution via multiscale spatio-temporal feature fusion network, that can fuse the temporal aligned features of the neighboring frames and the spatial feature of the reference frame. In our network, a pyramid multiscale fusion module is designed as major kernel of the whole structure, which could release the spatio-temporal misalignment between the neighboring frames and the reference frame. Specifically, we introduce the channel attention mechanism to further improve the infrared thermal imaging video super-resolution performance. Qualitative and quantitative evaluation results prove the effectiveness of our framework on both Vid4 dataset and raw infrared dataset collected by ourselves. We provide the infrared thermal imaging super-resolution results at https://github.com/qq123-jpg/Infrared-Thermal-Imaging-Super-resolution-via-Multiscale-Spatio-temporal-Feature-Fusion-Network.

Multiscale and Multitemporal Road Detection from High Resolution SAR Images Using Attention Mechanism

Road detection from images has emerged as an important way to obtain road information, thereby gaining much attention in recent years. However, most existing methods only focus on extracting road information from single temporal intensity images, which may cause a decrease in image resolution due to the use of spatial filter methods to avoid coherent speckle noises. Some newly developed methods take into account the multi-temporal information in the preprocessing stage to filter the coherent speckle noise in the SAR imagery. They ignore the temporal characteristic of road objects such as the temporal consistency for the road objects in the multitemporal SAR images that cover the same area and are taken at adjacent times, causing the limitation in detection performance. In this paper, we propose a multiscale and multitemporal network (MSMTHRNet) for road detection from SAR imagery, which contains the temporal consistency enhancement module (TCEM) and multiscale fusion module (MSFM) that are based on attention mechanism. In particular, we propose the TCEM to make full use of multitemporal information, which contains temporal attention submodule that applies attention mechanism to capture temporal contextual information. We enforce temporal consistency constraint by the TCEM to obtain the enhanced feature representations of SAR imagery that help to distinguish the real roads. Since the width of roads are various, incorporating multiscale features is a promising way to improve the results of road detection. We propose the MSFM that applies learned weights to combine predictions of different scale features. Since there is no public dataset, we build a multitemporal road detection dataset to evaluate our methods. State-of-the-art semantic segmentation network HRNetV2 is used as a baseline method to compare with MSHRNet that only has MSFM and the MSMTHRNet. The MSHRNet(TAF) whose input is the SAR image after the temporal filter is adopted to compare with our proposed MSMTHRNet. On our test dataset, MSHRNet and MSMTHRNet improve over the HRNetV2 by 2.1% and 14.19%, respectively, in the IoU metric and by 3.25% and 17.08%, respectively, in the APLS metric. MSMTHRNet improves over the MSMTHRNet(TAF) by 8.23% and 8.81% in the IoU metric and APLS metric, respectively.

CovSegNet: A Multi Encoder-Decoder Architecture for Improved Lesion Segmentation of COVID-19 Chest CT Scans.

Automatic lung lesion segmentation of chest computer tomography (CT) scans is considered a pivotal stage toward accurate diagnosis and severity measurement of COVID-19. Traditional U-shaped encoder-decoder architecture and its variants suffer from diminutions of contextual information in pooling/upsampling operations with increased semantic gaps among encoded and decoded feature maps as well as instigate vanishing gradient problems for its sequential gradient propagation that result in suboptimal performance. Moreover, operating with 3-D CT volume poses further limitations due to the exponential increase of computational complexity making the optimization difficult. In this article, an automated COVID-19 lesion segmentation scheme is proposed utilizing a highly efficient neural network architecture, namely CovSegNet, to overcome these limitations. Additionally, a two-phase training scheme is introduced where a deeper 2-D network is employed for generating region-of-interest (ROI)-enhanced CT volume followed by a shallower 3-D network for further enhancement with more contextual information without increasing computational burden. Along with the traditional vertical expansion of Unet, we have introduced horizontal expansion with multistage encoder-decoder modules for achieving optimum performance. Additionally, multiscale feature maps are integrated into the scale transition process to overcome the loss of contextual information. Moreover, a multiscale fusion module is introduced with a pyramid fusion scheme to reduce the semantic gaps between subsequent encoder/decoder modules while facilitating the parallel optimization for efficient gradient propagation. Outstanding performances have been achieved in three publicly available datasets that largely outperform other state-of-the-art approaches. The proposed scheme can be easily extended for achieving optimum segmentation performances in a wide variety of applications. Impact Statement-With lower sensitivity (60-70%), elongated testing time, and a dire shortage of testing kits, traditional RTPCR based COVID-19 diagnostic scheme heavily relies on postCT based manual inspection for further investigation. Hence, automating the process of infected lesions extraction from chestCT volumes will be major progress for faster accurate diagnosis of COVID-19. However, in challenging conditions with diffused, blurred, and varying shaped edges of COVID-19 lesions, conventional approaches fail to provide precise segmentation of lesions that can be deleterious for false estimation and loss of information. The proposed scheme incorporating an efficient neural network architecture (CovSegNet) overcomes the limitations of traditional approaches that provide significant improvement of performance (8.4% in averaged dice measurement scale) over two datasets. Therefore, this scheme can be an effective, economical tool for the physicians for faster infection analysis to greatly reduce the spread and massive death toll of this deadly virus through mass-screening.

Object detection algorithm based on feature enhancement

Recently, many excellent algorithms have made great progress in object detection, but there are also problems in these algorithms’ performance on targets of different sizes, and in particular in small object detection. Aiming at the problem of insufficient feature representation by the feature extractor, in this paper we propose a lightweight algorithm to improve feature extraction. The algorithm includes three modules. First, considering that the shallow features in feature extraction contain much background noise, in this paper we design a multi-level feedback propagation model based on a Gaussian high-pass filter. The shallow layers are enhanced using the filter and then back-propagated to add the upper shallow layer features and obtain new shallow layer features. This process is performed on the newly generated shallow layer for n iterations, which is beneficial for enhancing targets in the foreground area and suppressing background noise. Second, we form a stacked dilated convolution module with different dilation rates to cover the entire deep feature layer densely, which enlarges the receptive field and enriches the contextual information. Finally, we build a multi-scale fusion module to fuse the above-mentioned enhanced shallow and deep features to obtain output features with powerful representational ability for detection tasks. In addition, the model is easily embedded into existing approaches to enhance their performance. We build the model on the VGG-16 and ResNet-50 backbones and successfully applied it on Darknet-19 and Darknet-53 to verify its effectiveness and stability. The experiments on the COCO dataset prove that the proposed algorithm outperforms the state-of-art methods, with a mean average precision improvement reaching 2% on average. The effect is remarkable on small targets and complex backgrounds. Furthermore, it does not affect the detection speed significantly, so real time detection requirements can still be met.

Edge-preserving MRI image synthesis via adversarial network with iterative multi-scale fusion

Magnetic resonance imaging (MRI) is a major imaging technique for studying neuroanatomy. By applying different pulse sequences and parameters, different modalities can be generated regarding the same anatomical structure, which can provide complementary information for diagnosis. However, limited by the scanning time and related cost, multiple different modalities are often not available for the same patient in clinic. Recently, many methods have been proposed for cross-modality MRI synthesis, but most of them only consider pixel-level differences between the synthetic and ground-truth images, ignoring the edge information, which is critical to provide clinical information. In this paper, we propose a novel edge-preserving MRI image synthesis method with iterative multi-scale feature fusion based generative adversarial network (EP_IMF-GAN). Particularly, the generator consists of a shared encoder and two specific decoders to carry out different tasks: 1) a primary task aiming to generate the target modality and 2) an auxiliary task aiming to generate the corresponding edge image of target modality. We assume that infusing the auxiliary edge image generation task can help preserve edge information and learn better latent representation features through the shared encoder. Meanwhile, an iterative multi-scale fusion module is embedded in the primary decoder to fuse supplementary information of feature maps at different scales, thereby further improving quality of the synthesized target modality. Experiments on the BRATS dataset indicate that our proposed method is superior to the state-of-the-art image synthesis approaches in both qualitative and quantitative measures. Ablation study further validates the effectiveness of the proposed components.

Cascaded Cross-Modality Fusion Network for 3D Object Detection.

We focus on exploring the LIDAR-RGB fusion-based 3D object detection in this paper. This task is still challenging in two aspects: (1) the difference of data formats and sensor positions contributes to the misalignment of reasoning between the semantic features of images and the geometric features of point clouds. (2) The optimization of traditional IoU is not equal to the regression loss of bounding boxes, resulting in biased back-propagation for non-overlapping cases. In this work, we propose a cascaded cross-modality fusion network (CCFNet), which includes a cascaded multi-scale fusion module (CMF) and a novel center 3D IoU loss to resolve these two issues. Our CMF module is developed to reinforce the discriminative representation of objects by reasoning the relation of corresponding LIDAR geometric capability and RGB semantic capability of the object from two modalities. Specifically, CMF is added in a cascaded way between the RGB and LIDAR streams, which selects salient points and transmits multi-scale point cloud features to each stage of RGB streams. Moreover, our center 3D IoU loss incorporates the distance between anchor centers to avoid the oversimple optimization for non-overlapping bounding boxes. Extensive experiments on the KITTI benchmark have demonstrated that our proposed approach performs better than the compared methods.

Pulmonary Textures Classification via a Multi-Scale Attention Network.

Precise classification of pulmonary textures is crucial to develop a computer aided diagnosis (CAD) system of diffuse lung diseases (DLDs). Although deep learning techniques have been applied to this task, the classification performance is not satisfied for clinical requirements, since commonly-used deep networks built by stacking convolutional blocks are not able to learn discriminative feature representation to distinguish complex pulmonary textures. For addressing this problem, we design a multi-scale attention network (MSAN) architecture comprised by several stacked residual attention modules followed by a multi-scale fusion module. Our deep network can not only exploit powerful information on different scales but also automatically select optimal features for more discriminative feature representation. Besides, we develop visualization techniques to make the proposed deep model transparent for humans. The proposed method is evaluated by using a large dataset. Experimental results show that our method has achieved the average classification accuracy of 94.78% and the average f-value of 0.9475in the classification of 7 categories of pulmonary textures. Besides, visualization results intuitively explain the working behavior of the deep network. The proposed method has achieved the state-of-the-art performance to classify pulmonary textures on high resolution CT images.