Multi-level Feature Maps Research Articles

Multi-Level Feature Aggregation-Based Joint Keypoint Detection and Description

Image keypoint detection and description is a popular method to find pixel-level connections between images, which is a basic and critical step in many computer vision tasks. The existing methods are far from optimal in terms of keypoint positioning accuracy and generation of robust and discriminative descriptors. This paper proposes a new end-to-end self-supervised training deep learning network. The network uses a backbone feature encoder to extract multi-level feature maps, then performs joint image keypoint detection and description in a forward pass. On the one hand, in order to enhance the localization accuracy of keypoints and restore the local shape structure, the detector detects keypoints on feature maps of the same resolution as the original image. On the other hand, in order to enhance the ability to percept local shape details, the network utilizes multi-level features to generate robust feature descriptors with rich local shape information. A detailed comparison with traditional feature-based methods Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) and deep learning methods on HPatches proves the effectiveness and robustness of the method proposed in this paper.

A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images

The fully convolutional network (FCN) with an encoder-decoder architecture has been the standard paradigm for semantic segmentation. The encoder-decoder architecture utilizes an encoder to capture multilevel feature maps, which are incorporated into the final prediction by a decoder. As the context is crucial for precise segmentation, tremendous effort has been made to extract such information in an intelligent fashion, including employing dilated/atrous convolutions or inserting attention modules. However, these endeavors are all based on the FCN architecture with ResNet or other backbones, which cannot fully exploit the context from the theoretical concept. By contrast, we introduce the Swin Transformer as the backbone to extract the context information and design a novel decoder of densely connected feature aggregation module (DCFAM) to restore the resolution and produce the segmentation map. The experimental results on two remotely sensed semantic segmentation datasets demonstrate the effectiveness of the proposed scheme.

A Densely Attentive Refinement Network for Change Detection Based on Very-High-Resolution Bitemporal Remote Sensing Images

Detecting changes using bitemporal remote sensing imagery is vital to understand the dynamics of the land surface. Existing change detection models based on deep learning suffer from the problem of scale variation and pseudochange due to their insufficient multilevel aggregation and inadequate capability of feature representation, which limits the accuracy. This study proposes a densely attentive refinement network (DARNet) to improve change detection on bitemporal very-high-resolution remote sensing images. DARNet is based on the U-shape encoder–decoder architecture with the Siamese network as a feature extractor. The dense skip connection module (DSCM) is employed between the decoder and the encoder to aggregate multilevel feature maps. The hybrid attention module (HAM) is integrated to exploit contextual information and generate discriminative features. The recurrent refinement module (RRM) is exploited to progressively refine the predicted change maps during the decoding process. Experiments on testing the model performance were conducted on three benchmark datasets: the season-varying change detection (SVCD) dataset, the Sun Yat-sen University change detection (SYSU-CD) dataset, and the Learning Vision and Remote Sensing Laboratory building change detection (LEVIR-CD) dataset. The experimental results demonstrate that DARNet outperforms state-of-the-art models with kappa of 96.58%, 75.35%, and 90.69% for the SVCD, SYSU-CD, and LEVIR-CD datasets, respectively.

Multiscale residual fusion network for image denoising

Deep-learning methods have been developed in recent years and have achieved dramatic improvements for image denoising. The existing deep-learning methods can be conducted using two major models: Encoder–decoder and high-resolution, where the high-resolution model has superior resolution ability for detail description and restoration. In this study, a high-resolution-based network called multiscale residual fusion network (MRF-Net) is proposed, which employed the spatial and contextual information of images. In detail, dilated convolution layers are used to enlarge the network's receptive field and learned sufficient features in a multiscale feature extracting module. The function of dilated convolution is reinterpreted here and it is viewed as a complex downsampling operation. Therefore, multiscale feature analysis could be performed in the proposed network by dilated convolution. Multilevel feature maps are sequentially obtained through a residual projection module, where considerable contextual and spatial information was collected from the multiscale features. In a residual fusion module, all maps were aggregated to generate a residual image effectively for noise removal. Experiments demonstrated that the MRF-Net outperformed several state-of-the-art model-based and deep-learning methods in both blind and non-blind image denoising tests. Meanwhile, ablation studies were executed to verify the denoising performance of each module. Moreover, this method exhibited high computational efficiency, thus demonstrating its practicability.

Self-supervised learning of monocular depth using quantized networks

Learning monocular depth in a self-supervised manner is desirable for numerous applications ranging from autonomous driving, robotics to augmented reality. However, the current challenges lie in the problems of scale ambiguity, dynamic scene and hardware limitations. To this end, a self-supervised approach is proposed in this work for monocular depth learning and estimation. Specifically, we first introduce a self-supervised depth learning framework that learns a part of the camera intrinsics and the stereo extrinsics, which ensures the absoluteness of the predicted depth while achieving enhanced performance (i.e., lower error between the predicted depth and the ground-truth) on depth estimation. Besides, we further improve the accuracy and the efficiency (i.e., shorter inference time and lower weight/activation footprints) via a specially-designed network that exploits multi-scale context across multi-level feature maps. In addition, we propose a quantization scheme for our depth estimation networks. The scheme allows the network inference to be carried out using INT4-INT8 arithmetic while keeping a high performance on depth estimation. Extensive experiments on KITTI and Make3D datasets demonstrate that our approach substantially boosts the performance compared to the existing state-of-the-art methods on monocular depth estimation.

High-resolution representations and multistage region-based network for ship detection and segmentation from optical remote sensing images

Ship detection and segmentation in optical remote sensing (ORS) images is a current concern actively addressed by the academic community owing to its vast applications. Most present methods only detect the location of the ship but do not segment it at the pixel level. Considering the complex background of ORS images, identifying ships from interferences, such as clouds, waves, and some land architectures similar to ships, proves to be difficult. To address this issue, we propose a high-resolution representation and multistage region-based network (HR-MSRN) for ship detection and segmentation from ORS images. HR-MSRN mainly consists of three parts: the high-resolution feature pyramid network (HRFPN), region proposal network (RPN), and multistage detection and segmentation network (MSDSN). First, HRFPN is built as a backbone network to extract and fuse multilevel image feature maps. Second, ship candidate boxes are generated by defining numerous anchors through RPN. Third, using the idea of a cascade mask R-CNN as the reference method, the MSDSN is proposed to obtain the ship localization and mask shape. We utilize the proposed framework to evaluate an Airbus-ship dataset, and the experiments indicate that (1) HRFPN provides better feature representation ability than the ResNet-FPN when maintaining the same detection framework, especially for small ships; (2) the direct flow between mask branches refines the mask information, and the semantic segmentation branch enhances context information, which indicates that MSDSN is effective and promotes further improvements in ship detection and segmentation from ORS images; (3) in comparison to other region-based methods, HR-MSRN obtains superior performance of ship detection and segmentation in the ORS imagery.

Remote Sensing Image Super-Resolution Based on Dense Channel Attention Network

In the recent years, convolutional neural networks (CNN)-based super resolution (SR) methods are widely used in the field of remote sensing. However, complicated remote sensing images contain abundant high-frequency details, which are difficult to capture and reconstruct effectively. To address this problem, we propose a dense channel attention network (DCAN) to reconstruct high-resolution (HR) remote sensing images. The proposed method learns multi-level feature information and pays more attention to the important and useful regions in order to better reconstruct the final image. Specifically, we construct a dense channel attention mechanism (DCAM), which densely uses the feature maps from the channel attention block via skip connection. This mechanism makes better use of multi-level feature maps which contain abundant high-frequency information. Further, we add a spatial attention block, which makes the network have more flexible discriminative ability. Experimental results demonstrate that the proposed DCAN method outperforms several state-of-the-art methods in both quantitative evaluation and visual quality.

End-to-End Detection-Segmentation System for Face Labeling

In this paper, we propose an end-to-end detection-segmentation system to implement detailed face labeling. Fully convolutional networks (FCN) has become the mainstream algorithm in the field of semantic segmentation due to the state-of-the-art performance. However, a general FCN usually produces smooth and homogeneous results. Moreover, when semantic category is extremely unbalanced in samples such as face labeling problem, features for some categories cannot be well explored by FCN. To alleviate these problems, a face image is firstly encoded to multi-level feature maps by a pyramid FCN, then features of different facial components are extracted separately according to the bounding box provided by a one-stage detection head. Three class-specific sub-networks are employed to process the extracted features to obtain the respective segmentation results. The skin-hair region can be decoded directly from the back end of the pyramid FCN. Finally, the overall segmentation result is obtained by combining different branches. Moreover, the proposed method trained on a single-face labeled dataset, can be directly used to implement detailed multi-face labeling tasks without any network modification and additional module or data. The overall structure can be trained in an end-to-end manner while maintaining a small network size (12 MB). Experiments show that the proposed method can generate more accurate (single or multi) face labeling results comparing with previous works and gets the state-of-the-art results in HELEN face dataset.

Image super-resolution via enhanced multi-scale residual network

Recently, a very deep convolutional neural network (CNN) has achieved impressive results in image super-resolution (SR). In particular, residual learning techniques are widely used. However, the previously proposed residual block can only extract one single-level semantic feature maps of one single receptive field. Therefore, it is necessary to stack the residual blocks to extract higher-level semantic feature maps, which will significantly deepen the network. While a very deep network is hard to train and limits the representation for reconstructing the hierarchical information. Based on the residual block, we propose an enhanced multi-scale residual network (EMRN) to take advantage of hierarchical image features via dense connected enhanced multi-scale residual blocks (EMRBs). Specifically, the newly proposed residual block (EMRB) is capable of constructing multi-level semantic feature maps by a two-branch inception. The two-branch inception in our proposed EMRB consists of 2 convolutional layers and 4 convolutional layers in each branch respectively, therefore we have different ranges of receptive fields within one single EMRB. Meanwhile, the local feature fusion (LFF) is used in every EMRB to adaptively fuse the local feature maps extracted by the two-branch inception. Furthermore, global feature fusion (GFF) in EMRN is then used to obtain abundant useful features from previous EMRBs and subsequent ones in a holistic manner. Experiments on benchmark datasets suggest that our EMRN performs favorably over the state-of-the-art methods in reconstructing further superior super-resolution (SR) images.

Salient detection network for lung nodule detection in 3D Thoracic MRI Images

The detection of lung nodule in magnetic resonance imaging (MRI) images is the key to the diagnosis of lung cancer. However, few computer-aided diagnosis (CAD) methods have been carried out for lung nodule detection in MRI images. Moreover, there is a challenge that small nodules are easily missed. In this paper, a salient detection network (SDN) is proposed for nodule detection in thoracic MRI images. To effectively aggregate multi-level feature maps and improve the detection rate of small nodules, an encoding-decoding block is designed, which output the salient detection map (SDM). To highlight the whole nodule region and suppress the distraction regions, a cascade salient map generation scheme is constructed by bifurcated VGG16 network, which employs the initial salient map to refine feature maps. Finally, one false positive (FP) reduction criterion is adopted to reduce FPs and preserve true nodules. The proposed method is tested on 142 T2-weighted MRI scans from the First Affiliated Hospital of Guangzhou Medical University. The true positive rate (TPR) of the proposed method is 94.0 % with 7.19 FPs/scan. Experimental results demonstrate that the proposed SDN can reduce the dependence on nodule size and improve the detection rate for small nodules. It shows better performance compared with existing nodule detection methods.

Towards real-time object tracking with deep Siamese network and layerwise aggregation

Siamese networks have drawn increasing interest in the field of visual object tracking due to their balance of precision and efficiency. However, Siamese trackers use relatively shallow backbone networks, such as AlexNet, and therefore do not take full advantage of the capabilities of modern deep convolutional neural networks (CNNs). Moreover, the feature representations of the target object in a Siamese tracker are extracted through the last layer of CNNs and mainly capture semantic information, which causes the tracker's precision to be relatively low and to drift easily in the presence of similar distractors. In this paper, a new nonpadding residual unit is designed and used to stack a 22-layer deep ResNet, referred as ResNet22. After utilizing ResNet22 as the backbone network, we can build a deep Siamese network, which can greatly enhance the tracking performance. Considering that the different levels of the feature maps of the CNN represent different aspects of the target object, we aggregated different deep convolutional layers to make use of ResNet22’s multilevel feature maps, which can form hyperfeature representations of targets. The final network architecture is named DSiamLA. Experimental results show that DSiamLA has achieved significant improvement on multiple benchmark datasets.

Rotating Objects Detection in Aerial Images via Attention Denoising and Angle Loss Refining

With the rapid development of deep learning technology, the problem of detecting objects has been widely studied. A lot of works have made tremendous success on both detection speed and accuracy. However, when detecting small objects, which have less pixels and are more susceptible to noise, the accuracy declines significantly, and this problem is rather serious in Aerial images because most Aerial images contain a lot of small and clustered targets. In this paper, we propose a novel two-stage detector which can not only improve the detection accuracy on small objects but also predict the oriented bounding boxes of them rather than the horizontal ones. Our detection framework can be described as follow: (1) firstly, a basic feature pyramid network is used to generate multi-level feature maps of input images; (2) secondly, Instead of manually designing anchors, we propose a Multi-scale K-means anchor Generator (MKG) to dynamically generate anchors for feature pyramid, which can solve the problem of imbalance distribution of positive anchors; (3) finally, an Attention Denoise Predictor (ADP) and Refine Angle Loss (RAL) are induced to precisely make the rotating bounding boxes predictions. Experiments on the remote sensing public dataset DOTA show the state-of-art performance of our detection framework.

A Lightweight Adaptive RoI Extraction Network for Precise Aerial Image Instance Segmentation

Bounding boxes have been widely implemented into aerial object detection for their simplicity. They perform instance-level locations with the coordinates and orientation for each target. However, the defects such as coarse edge information impede semantic interpretation in Earth observation. Besides, in terms of the aerial imaging instruments, it is essential to recognize the exterior appearance and contour of the objects. In this article, we propose a novel aerial instance segmentation method termed the adaptive region of interest (RoI) extraction network (ARE-Net), which bridges the gap of accurately delineating instances under the complex background of aerial images. To exert instance segmentation under the proprietary property, e.g., complex background and densely distributed instances, of aerial images, RoIs are pooled from multilevel feature maps and integral region proposals. On this basis, global attention RoI extractor (GA-RoIE) and perceptual RoI extractor (PRoIE) are, respectively, introduced for detection branch and mask branch to perform adaptive RoI extraction for aerial images. Meanwhile, to reconcile the probability distribution regional distribution of pixelwise prediction in aerial images, we present the adaptive compound loss function to improve the integrating degree of the predicted binary mask to the ground-truth mask. In addition, we adopt RegNetx with deformable convolution to optimize ARE-Net and name it as R-ARE-Net. Despite implementing pixelwise prediction, comprehensive experiments on iSAID and NWPU VHR-10 instance segmentation datasets still have verified the effectiveness and efficiency of ARE-Net and R-ARE-Net. Experimental results indicate that our proposed methods receive the highest AP value (38.0% AP on iSAID and 64.2% AP on NWPU VHR-10 instance segmentation dataset) and the lowest floating-point operations (FLOPs) and parameters’ consumption (~46% reduced FLOPs and 61.5% reduced parameters than SCNet) among the mainstream methods. Besides, the false alarms, missing segmentations, poorly predicted masks, and undersegmentations that appeared in the mainstream methods can be avoided to some extend for R-ARE-Net.

HOReID: Deep High-Order Mapping Enhances Pose Alignment for Person Re-Identification.

Despite the remarkable progress in recent years, person Re-Identification (ReID) approaches frequently fail in cases where the semantic body parts are misaligned between the detected human boxes. To mitigate such cases, we propose a novel High-Order ReID (HOReID) framework that enables semantic pose alignment by aggregating the fine-grained part details of multilevel feature maps. The HOReID adopts a high-order mapping of multilevel feature similarities in order to emphasize the differences of the similarities between aligned and misaligned part pairs in two person images. Since the similarities of misaligned part pairs are reduced, the HOReID enhances pose-robustness within the learned features. We show that our method derives from an intuitive and interpretable motivation and elegantly reduces the misalignment problem without using any prior knowledge from human pose annotations or pose estimation networks. This paper theoretically and experimentally demonstrates the effectiveness of the proposed HOReID, achieving superior performance over the state-of-the-art methods on the four large-scale person ReID datasets.

Salient object detection via light-weight multi-path cascaded networks

Recently, deep learning-based saliency detection has achieved fantastic performance over conventional works. However, repeated subsampling operations like pooling or convolution striding in deep CNNs lead to a significant decrease in the initial image resolution. It is difficult for a network to learn saliency at the boundaries of salient regions. In this paper, a Light-Weight Multi-Path Cascaded Network (LMCN) is proposed for image saliency detection – an encoder-decoder architecture that explicitly exploits all the information available along the down-sampling process to enable full-resolution prediction using long-range residual connections. The Res2Net is adopted as a backbone network to better extract multi-level and multi-scale feature maps by constructing hierarchical residual-like connections within one single residual block. Also, the network is lightweighted by the bottleneck structure and depth-wise convolution, which ultimately reduces the number of parameters by more than 50% while maintaining the same performance. Comprehensive experiments are carried out and new state-of-the-art results are set in eight public datasets. Experiment results show that the proposed approach substantially improves previous state-of-the-art performances.

Predicting atypical visual saliency for autism spectrum disorder via scale-adaptive inception module and discriminative region enhancement loss

Autism spectrum disorder (ASD) is a prevalent lifelong neurodevelopmental disorder. Individuals with ASD perform atypical attention toward the visual stimulus. Despite that the number and performance of visual attention models are ongoing, the atypical attention prediction stagnates. To better understand abnormal visual behaviors of individuals with ASD, this paper models the atypical visual saliency via a deep neural network. First, the multi-level side-out feature maps are extracted from images. Second, the representation of features is enhanced by a scale-adaptive coarse-and-fine inception module that is specialized in integrating multi-scale features. Consequently, the enhanced features are transmitted from deep layers to shallow layers, boosting the precision of saliency maps. Furthermore, to facilitate our model transferring from typical saliency prediction to atypical saliency prediction, a loss function, namely discriminative region enhancement loss, is designed to intensify the discrepancy between the atypical saliency and typical saliency, driving the model to fit the atypical pattern of visual attention. Qualitative and quantitative experiments demonstrate that the proposed model achieves state-of-the-art performance in terms of various metrics.

Road Segmentation for Remote Sensing Images Using Adversarial Spatial Pyramid Networks

Road extraction in remote sensing images is of great importance for a wide range of applications. Because of the complex background, and high density, most of the existing methods fail to accurately extract a road network that appears correct and complete. Moreover, they suffer from either insufficient training data or high costs of manual annotation. To address these problems, we introduce a new model to apply structured domain adaption for synthetic image generation and road segmentation. We incorporate a feature pyramid (FP) network into generative adversarial networks to minimize the difference between the source and target domains. A generator is learned to produce quality synthetic images, and the discriminator attempts to distinguish them. We also propose a FP network that improves the performance of the proposed model by extracting effective features from all the layers of the network for describing different scales' objects. Indeed, a novel scale-wise architecture is introduced to learn from the multilevel feature maps and improve the semantics of the features. For optimization, the model is trained by a joint reconstruction loss function, which minimizes the difference between the fake images and the real ones. A wide range of experiments on three data sets prove the superior performance of the proposed approach in terms of accuracy and efficiency. In particular, our model achieves state-of-the-art 78.86 IOU on the Massachusetts data set with 14.89M parameters and 86.78B FLOPs, with 4× fewer FLOPs but higher accuracy (+3.47% IOU) than the top performer among state-of-the-art approaches used in the evaluation.

Combining Prior Knowledge With CNN for Weak Scratch Inspection of Optical Components

Scratches as the major defects in precision optical components are caused inevitably in the manufacturing process, which is harmful to the whole optical system. Most scratches on the surface of optical components are weak scratches with low contrast and uneven distribution of gray scale, which poses a significant problem for inspection. In this article, an end-to-end weak scratch inspection method based on novel scratch-enhancement methods and convolutional neural network (CNN) is proposed for optical components. To enhance weak scratches, a local maximum index (LMI) module and a direction-sensitive convolution (DSC) module are proposed to generate multilevel-feature maps using prior knowledge about scratch. Different from previous works utilizing the raw dark-field image as network input, these multilevel features are used as the inputs of encoder-decoder module for training. After training, the whole inspection model can infer weak scratches from raw dark-field test images in an end-to-end manner. Experimental results show that the proposed model achieves pixel accuracy of 92.48% and IoU at 77.27% on the test data set. It outperforms the networks without adding prior knowledge, which shows that prior knowledge is much helpful for weak scratch inspection. Moreover, compared with other classical methods and CNN-based methods, the proposed method achieves the best performance in the weak scratch inspection.

CDnetV2: CNN-Based Cloud Detection for Remote Sensing Imagery With Cloud-Snow Coexistence

Cloud detection is a crucial preprocessing step for optical satellite remote sensing (RS) images. This article focuses on the cloud detection for RS imagery with cloud-snow coexistence and the utilization of the satellite thumbnails that lose considerable amount of high resolution and spectrum information of original RS images to extract cloud mask efficiently. To tackle this problem, we propose a novel cloud detection neural network with an encoder-decoder structure, named CDnetV2, as a series work on cloud detection. Compared with our previous CDnetV1, CDnetV2 contains two novel modules, that is, adaptive feature fusing model (AFFM) and high-level semantic information guidance flows (HSIGFs). AFFM is used to fuse multilevel feature maps by three submodules: channel attention fusion model (CAFM), spatial attention fusion model (SAFM), and channel attention refinement model (CARM). HSIGFs are designed to make feature layers at decoder of CDnetV2 be aware of the locations of the cloud objects. The high-level semantic information of HSIGFs is extracted by a proposed high-level feature fusing model (HFFM). By being equipped with these two proposed key modules, AFFM and HSIGFs, CDnetV2 is able to fully utilize features extracted from encoder layers and yield accurate cloud detection results. Experimental results on the ZY-3 satellite thumbnail data set demonstrate that the proposed CDnetV2 achieves accurate detection accuracy and outperforms several state-of-the-art methods.

A Deep Hyper Siamese Network for Real-Time Object Tracking

Siamese networks have drawn increasing interest in the field of visual object tracking due to their balance of precision and efficiency. However, Siamese trackers use relatively shallow backbone networks, such as AlexNet, and therefore do not take full advantage of the capabilities of modern deep convolutional neural networks (CNNs). Moreover, the feature representations of the target object in a Siamese tracker are extracted through the last layer of CNNs and mainly capture semantic information, which causes the tracker's precision to be relatively low and to drift easily in the presence of similar distractors. In this paper, a new nonpadding residual unit (NPRU) is designed and used to stack a 22-layer deep ResNet, referred as ResNet22. After utilizing ResNet22 as the backbone network, we can build a deep Siamese network, which can greatly enhance the tracking performance. Considering that the different levels of the feature maps of the CNN represent different aspects of the target object, we aggregated different deep convolutional layers to make use of ResNet22's multilevel feature maps, which can form hyperfeature representations of targets. The designed deep hyper Siamese network is named DHSiam. Experimental results show that DHSiam has achieved significant improvement on multiple benchmark datasets.