Multi-layer Feature Maps Research Articles

SSN: Scale Selection Network for Multi-Scale Object Detection in Remote Sensing Images

The rapid growth of deep learning technology has made object detection in remote sensing images an important aspect of computer vision, finding applications in military surveillance, maritime rescue, and environmental monitoring. Nonetheless, the capture of remote sensing images at high altitudes causes significant scale variations, resulting in a heterogeneous range of object scales. These varying scales pose significant challenges for detection algorithms. To solve the scale variation problem, traditional detection algorithms compute multi-layer feature maps. However, this approach introduces significant computational redundancy. Inspired by the mechanism of cognitive scaling mechanisms handling multi-scale information, we propose a novel Scale Selection Network (SSN) to eliminate computational redundancy through scale attentional allocation. In particular, we have devised a lightweight Landmark Guided Scale Attention Network, which is capable of predicting potential scales in an image. The detector only needs to focus on the selected scale features, which greatly reduces the inference time. Additionally, a fast Reversible Scale Semantic Flow Preserving strategy is proposed to directly generate multi-scale feature maps for detection. Experiments demonstrate that our method facilitates the acceleration of image pyramid-based detectors by approximately 5.3 times on widely utilized remote sensing object detection benchmarks.

Focus-Attention Approach in Optimizing DETR for Object Detection from High-Resolution Images

Transformer-based detectors have recently achieved remarkable success in object detection, revolutionizing the field with their efficiency and accuracy. However, applying these models to high-resolution images presents significant challenges due to the increased computational demands and complexity of processing dense, high-resolution data. In this paper, we introduce a novel model specifically designed for object detection from high-resolution imagery. This model incorporates a multi-layer object-focus network along with a transformer encoder-decoder structure. Specifically, the model employs a dual-head strategy in the object-focus network to balance the detailed analysis of small objects with computational efficiency. This is achieved by leveraging data sparsity to reduce unnecessary computations in massive background regions. Additionally, to improve detection performance for small objects, we propose a method to effectively apply the transformer encoder-decoder structure combined with the object-focus network on the multi-layer feature maps of the feature pyramid. Our extensive evaluations of the VisDrone, MS-COCO, and UAVid datasets demonstrate that our model outperforms other DETR-based detectors in both detection accuracy and computational speed, highlighting its superior performance. These results indicate a significant advancement in the field of high-resolution object detection utilizing transformer architecture.

Robust RGB-T Tracking via Graph Attention-Based Bilinear Pooling.

RGB-T tracker possesses strong capability of fusing two different yet complementary target observations, thus providing a promising solution to fulfill all-weather tracking in intelligent transportation systems. Existing convolutional neural network (CNN)-based RGB-T tracking methods often consider the multisource-oriented deep feature fusion from global viewpoint, but fail to yield satisfactory performance when the target pair only contains partially useful information. To solve this problem, we propose a four-stream oriented Siamese network (FS-Siamese) for RGB-T tracking. The key innovation of our network structure lies in that we formulate multidomain multilayer feature map fusion as a multiple graph learning problem, based on which we develop a graph attention-based bilinear pooling module to explore the partial feature interaction between the RGB and the thermal targets. This can effectively avoid uninformed image blocks disturbing feature embedding fusion. To enhance the efficiency of the proposed Siamese network structure, we propose to adopt meta-learning to incorporate category information in the updating of bilinear pooling results, which can online enforce the exemplar and current target appearance obtaining similar sematic representation. Extensive experiments on grayscale-thermal object tracking (GTOT) and RGBT234 datasets demonstrate that the proposed method outperforms the state-of-the-art methods for the task of RGB-T tracking.

Channel and spatial attention mechanism for fashion image captioning

<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p><span lang="EN-US">Image captioning aims to automatically generate one or more description sentences for a given input image. Most of the existing captioning methods use encoder-decoder model which mainly focus on recognizing and capturing the relationship between objects appearing in the input image. However, when generating captions for fashion images, it is important to not only describe the items and their relationships, but also mention attribute features of clothes (shape, texture, style, fabric, and more). In this study, one novel model is proposed for fashion image captioning task which can capture not only the items and their relationship, but also their attribute features. Two different attention mechanisms (spatial-attention and channel-wise attention) is incorporated to the traditional encoder-decoder model, which dynamically interprets the caption sentence in multi-layer feature map in addition to the depth dimension of the feature map. We evaluate our proposed architecture on Fashion-Gen using three different metrics (CIDEr, ROUGE-L, and BLEU-1), and achieve the scores of 89.7, 50.6 and 45.6, respectively. Based on experiments, our proposed method shows significant performance improvement for the task of fashion-image captioning, and outperforms other state-of-the-art image captioning methods.</span></p></div></div></div>

A balanced random learning strategy for CNN based Landsat image segmentation under imbalanced and noisy labels

Landsat image segmentation is important for obtaining large-scale land cover maps. The accuracy of CNN-based Landsat image segmentation highly depends on the quantity and quality of the training samples. However, enough accurate labels for Landsat images are difficult to access. Fortunately, traditional classifier induced segmentation results can be considered as an alternative, although they are noisy and unbalanced to a certain extent. To resist noisy labels and alleviate the impact of imbalanced samples, this paper proposes a confidence interval based balanced random learning strategy. Firstly, a confidence interval-based mask is employed to control the random learning rate of the network from the entire noisy training set. Then, the multi-layer feature maps of CNN are fully utilized to compensate for the information loss in random learning, in which down-sampled labels are used to decrease the uncertainty brought by up-sampling CNN feature maps. In addition, considering the corruption of noisy labels on different classes, a balanced random learning with different confidence levels is performed on each class to further improve the learning ability of CNN. Experimental results on two widely used backbones, namely VGGNet and ResNet, demonstrate that the proposed balanced random learning strategy can effectively improve the performance of CNN under imbalanced and noisy labels, which can be improved by 3.41%.

Dynamic attention-based detector and descriptor with effective and derivable loss for image matching

Jointly learned detectors and descriptors are becoming increasingly popular because they can simplify the matching process and obtain more correspondences than traditional tools. However, most methods yield low keypoint detection accuracy due to the large receptive field of the detection score map. In addition, existing methods lack efficient detector loss functions because the coordinates of keypoints are discrete and nonderivable. To mitigate these two problems, we propose a method called dynamic attention-based detector and descriptor with effective and derivable loss (DA-Net). For the first problem, a dynamic attention convolution-based feature extraction module is proposed to select the most suitable parameters for different samples. In addition, a multilayer feature self-difference detection (MFSD) module is proposed to detect keypoints with high accuracy. In the MFSD module, multilayer feature maps are used to calculate their feature self-difference maps, and they are fused to obtain a detection score map. For the second problem, an approximate keypoint distance loss function is proposed by approximately regressing the coordinates of the local maximum as keypoint coordinates, allowing the calculations involving keypoint coordinates to backpropagate. Moreover, two descriptor loss functions are proposed to learn reliable descriptors. A series of experiments based on widely used datasets show that DA-Net outperforms other learned detection and description methods.

FFR-SSD: feature fusion and reconstruction single shot detector for multi-scale object detection

Object detection is one of the most fundamental tasks toward image content understanding. Although numerous algorithms have been proposed, implementing effective and efficient object detection is still very challenging for now. To solve the challenges of small and multi-scale objects, we propose a hierarchical feature fusion and reconstruction method called feature fusion and reconstruction single shot detector (FFR-SSD). We first present a multi-scale visual attention model, which incorporates the channel-wise and space-wise information into the multi-branch feature enhancement module to improve the feature representation capacities. Second, a hierarchical feature map weighing mechanism is developed to fuse multi-layer feature maps, which contributes to describing the intact objects for the subsequent module. Third, we present an effective feature map reconstruction module to encourage the model to focus on pivotal information. Therefore, the overall contour information is preserved for the shallow enhanced response map and the semantic feature information is enriched for the deep enhanced feature map. Numerous experiments on two public benchmark datasets show that the proposed method achieves significant improvements over the state of the arts.

Faster RCNN based on Feature Fusion and its Application in Steel Surface Defect Detection

Surface defects are an inevitable problem in steel production and processing. To solve the problems of error detection, leakage detection and low accuracy of traditional manual methods for steel surface defect detection, this paper proposes a steel surface defect detection algorithm based on Faster RCNN, a classical model in the field of computer vision object detection. The proposed model uses ResNet50 as its backbone and introduces FPN to fuse the multi-layer feature maps of the backbone to improve the detection capability for defects of different scales. The experimental results on NEU-DET dataset show that the proposed model in this paper has different detection accuracies for six different types of defects in the dataset, and the overall mAP reaches 0.708. In addition, the model execution speed reaches 43.3 frame/s, which meets the applications in industrial scenarios.

Infrared and visible image fusion based on tight frame learning via VGG19 network

A composite image of infrared and visible images should contain salient features of two source images which come from different sensors in the same scene. In this paper, we propose a fusion method based on a tight frame combined with VGG 19 to integrate the infrared and visible images. After training the tight frame, the pre-registered source images are decomposed into the base and detailed images by classifying all the learned frame filters into two sets with a total-variation-like measure. Then, the max-ℓ1 rule is adopted to fuse the base images, while the VGG19-based method is adopted to fuse the detailed images. For the detailed images, the pre-trained VGG19 is used to extract the multi-layer feature maps, multiple fused detailed images are reconstructed by the ℓ1-norm combined with the weighted evaluation rules, and the final fused detailed image is selected by the max rule. Finally, the final fused image is obtained by directly combining the base image and the detailed image. The experimental results demonstrate that the proposed method achieves the state-of-the-art fusion performance in both quantitative evaluation and subjective quality.

Feature Split–Merge–Enhancement Network for Remote Sensing Object Detection

Recently, multicategory object detection in high-resolution remote sensing images is still a challenge. First, objects with significant scale differences exist in one scene simultaneously, so it is generally difficult for the detectors to balance the detection performance of large and small objects. Second, because of the complex background and the objects&#x2019; densely distributed characteristics in the remote sensing images, the extracted features usually have noise and blurred boundaries, which interfere with the detection performance of the object detectors. With this observation, we propose an end-to-end scale-aware network called feature split&#x2013;merge&#x2013;enhancement network (SME-Net) for remote sensing object detection, composed of the feature split-and-merge (FSM) module, the offset-error rectification (OER) module, and the object saliency enhancement (OSE) strategy. FSM eliminates salient information of large objects to highlight the features of small objects in the shallow feature maps. It also transmits the effective detailed features of large objects to the deep feature maps, alleviating feature confusion between multiscale objects. OER corrects the inconsistency of the features spatial layout among the multilayer feature maps by the proposed offset loss, so as to achieve supervised elimination and transmission in FSM. OSE enhances the features of interests and suppresses the background information by the proposed membership function, thus preventing false detection and missed detection caused by noise and blurred boundaries. The effectiveness of the proposed algorithm has been verified on multiple datasets. Our code is available at: <uri>https://github.com/Momuli/SMENet.git</uri>

Multilayer Feature Fusion Network With Spatial Attention and Gated Mechanism for Remote Sensing Scene Classification

Remote sensing (RS) scene classification has attracted extensive attention due to its large number of applications. Recently, convolutional neural networks (CNNs) methods have shown impressive ability of feature learning in RS scene classification. However, the performance is still limited by large-scale variance and complex background. To address these problems, we present a multilayer feature fusion network with spatial attention and gated mechanism (MLF2Net_SAGM) for RS scene classification. At first, the backbone is employed to extract multilayer convolutional features. Then, a residual spatial attention module (RSAM) is proposed to enhance discriminative regions of the multilayer feature maps, and key areas can be harvested. Finally, the multilayer spatial calibration features are fused to form the final feature map, and a gated fusion module (GFM) is designed to eliminate feature redundancy and mutual exclusion (FRME). To verify the effectiveness of the proposed method, we conduct comparative experiments based on three widely used RS image scene classification benchmarks. The results show that the direct fusion of multilayer features via element-wise addition leads to FRME, whereas our method fuses multilayer features more effectively and improves the performance of scene classification.

FAGNet: Multi-Scale Object Detection Method in Remote Sensing Images by Combining MAFPN and GVR

<p indent=0mm>Remote sensing images of large scenes are complex, and have the characteristics of many categories of objects, different scales and changeable directions, which lead to the problem of multi-class, multi-scale and multi-oriented of objects in remote sensing images. A remote sensing image object detection method based on multi-scale attention feature pyramid network (MAFPN) and gliding vertex regression (GVR) mechanism is proposed. Firstly, multi-layer feature maps are extracted from backbone network as input of MAFPN, which combines feature fusion and attention mechanism. On the basis of fusing feature maps of multi-scale, channel attention and spatial attention mechanisms are used to suppress noise, enhance effective feature reuse, and improve the network’s adaptability to object multi-scale features. The fusion feature map output by MAFPN is input to the region proposal network to generate the regions of interest, and then they will be sent to the classification regression network. The GVR mechanism is used in the object classification regression network and the four vertex offset ratio parameters and rotation factors are added on the basis of predicting the horizontal boxes, which converts the horizontal boxes into a rotating box to reduce the redundant area in the bounding boxes, makes the predicted rotating bounding boxes fit the object more closely. The experimental results on the DOTA public dataset, compared with many classical detection algorithms based on convolutional neural networks, show that the average detection accuracy of the proposed method is significantly improved, which can detect objects of multi-scales and multi-oriented more accurately, and achieve the robust detection of multi-scale objects.

Densely Semantic Enhancement for Domain Adaptive Region-Free Detectors

Unsupervised domain adaptive object detection aims to adapt a well-trained detector from its original source domain with rich labeled data to a new target domain with unlabeled data. Previous works focus on improving the domain adaptability of region-based detectors, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , Faster-RCNN, through matching cross-domain instance-level features that are explicitly extracted from a region proposal network (RPN). However, this is unsuitable for region-free detectors such as single shot detector (SSD), which perform a dense prediction from all possible locations in an image and do not have the RPN to encode such instance-level features. As a result, they fail to align important image regions and crucial instance-level features between the domains of region-free detectors. In this work, we propose an adversarial module, namely, densely semantic enhancement module (DSEM), to strengthen the cross-domain matching of instance-level features for region-free detectors. Firstly, to emphasize the important regions of image, the DSEM learns to predict a transferable foreground enhancement mask that can be utilized to suppress the background disturbance in an image. Secondly, considering that region-free detectors recognize objects of different scales using multi-layer feature maps, the DSEM encodes multi-scale representations across different domains. Finally, the DSEM is pluggable into different region-free detectors, ultimately achieving the densely semantic feature matching via adversarial learning. Extensive experiments have been conducted on PASCAL VOC, Clipart, Comic, W atercolor, and FoggyCityscape benchmarks, and their results well demonstrate that the proposed approach not only improves the domain adaptability of region-free detectors but also outperforms existing domain adaptive region-based detectors under various domain shift settings.

Panchromatic Image Super-Resolution Via Self Attention-Augmented Wasserstein Generative Adversarial Network.

Panchromatic (PAN) images contain abundant spatial information that is useful for earth observation, but always suffer from low-resolution ( LR) due to the sensor limitation and large-scale view field. The current super-resolution (SR) methods based on traditional attention mechanism have shown remarkable advantages but remain imperfect to reconstruct the edge details of SR images. To address this problem, an improved SR model which involves the self-attention augmented Wasserstein generative adversarial network ( SAA-WGAN) is designed to dig out the reference information among multiple features for detail enhancement. We use an encoder-decoder network followed by a fully convolutional network (FCN) as the backbone to extract multi-scale features and reconstruct the High-resolution (HR) results. To exploit the relevance between multi-layer feature maps, we first integrate a convolutional block attention module (CBAM) into each skip-connection of the encoder-decoder subnet, generating weighted maps to enhance both channel-wise and spatial-wise feature representation automatically. Besides, considering that the HR results and LR inputs are highly similar in structure, yet cannot be fully reflected in traditional attention mechanism, we, therefore, designed a self augmented attention (SAA) module, where the attention weights are produced dynamically via a similarity function between hidden features; this design allows the network to flexibly adjust the fraction relevance among multi-layer features and keep the long-range inter information, which is helpful to preserve details. In addition, the pixel-wise loss is combined with perceptual and gradient loss to achieve comprehensive supervision. Experiments on benchmark datasets demonstrate that the proposed method outperforms other SR methods in terms of both objective evaluation and visual effect.

TruingDet: Towards high-quality visual automatic defect inspection for mental surface

Visual surface defect detection, which aims to obtain the locations of defects and classify each defect into the corresponding category in a given image, is a critical task in an actual production process. Nowadays, more and more methods have made excellent progress in visual defect inspection. However, there still exist three tough challenges where these methods cannot handle well: large defect shape change, large-scale variation, and high-quality defect localization. In this paper, a Convolutional Neural Networks (CNN) based visual defect detection framework is proposed, which elegantly mitigated these three problems by introducing three well-designed components including deformable convolution module, balanced feature pyramid module and cascade head module. First, the feature maps contained with defect shape information are adaptively extracted by Resnet/ResneXt network with the deformable convolution operator. Then the balanced feature pyramid module is attached to the feature extraction module to obtain information-fused multilayer feature maps. Finally, the cascade head is applied to refine the predicted bounding box to achieve high-quality defect localization. Under the COCO evaluation metrics, our method significantly obtains 45.2 mAP with a large margin (4.9 AP) compared with Faster RCNN baseline.

Multi-layer attention for person re-identification

Person re-identification has been a significant application in the field of video surveillance analysis, yet it remains a challenging work to recognize the person of interest across disjoint cameras of different viewpoints. The factors affecting the identification results include the variation in background, different illumination conditions and the changes of human body poses. Existing person re-identification methods mainly focus on the feature extraction of the whole frame and metric learning functions. However, most of those algorithms treat different areas without distinction. It is worth emphasizing that different local regions make different contributions to image representaion, which exactly conforms to the attention mechanism. In this paper, we introduce a novel attention network which explores spatial attention in a convolutional neural network. Our algorithm learns the visual attention in multi-layer feature maps. The proposed model not only pays attention to the spatial probabilities of local regions, but also takes the features in different levels into consideration. We evaluate this multi-layer spatial attention model on three benchmark person re-identification datasets: Market-1501, CUHK03, and DukeMTMC-reID. The experiment results validate the advances of our adopted network by comparing with state-of-the-art baselines.

Remote Sensing Scene Classification Using Multilayer Stacked Covariance Pooling

This paper proposes a new method, called multilayer stacked covariance pooling (MSCP), for remote sensing scene classification. The innovative contribution of the proposed method is that it is able to naturally combine multilayer feature maps, obtained by pretrained convolutional neural network (CNN) models. Specifically, the proposed MSCP-based classification framework consists of the following three steps. First, a pretrained CNN model is used to extract multilayer feature maps. Then, the feature maps are stacked together, and a covariance matrix is calculated for the stacked features. Each entry of the resulting covariance matrix stands for the covariance of two different feature maps, which provides a natural and innovative way to exploit the complementary information provided by feature maps coming from different layers. Finally, the extracted covariance matrices are used as features for classification by a support vector machine. The experimental results, conducted on three challenging data sets, demonstrate that the proposed MSCP method can not only consistently outperform the corresponding single-layer model but also achieve better classification performance than other pretrained CNN-based scene classification methods.