DOTA Dataset Research Articles

Data Augmentation Using Bitplane Information Recombination Model.

The performance of deep learning heavily depend on the quantity and quality of training data. But in many fields, well-annotated data are so difficult to collect, which makes the data scale hard to meet the needs of network training. To deal with this issue, a novel data augmentation method using the bitplane information recombination model (termed as BIRD) is proposed in this paper. Considering each bitplane can provide different structural information at different levels of detail, this method divides the internal hierarchical structure of a given image into different bitplanes, and reorganizes them by bitplane extraction, bitplane selection and bitplane recombination, to form an augmented data with different image details. This method can generate up to 62 times of the training data, for a given 8-bits image. In addition, this generalized method is model free, parameter free and easy to combine with various neural networks, without changing the original annotated data. Taking the task of target detection for remotely sensed images and classification for natural images as an example, experimental results on DOTA dataset and CIFAR-100 dataset demonstrated that, our proposed method is not only effective for data augmentation, but also helpful to improve the accuracy of target detection and image classification.

A Light Anchor-Free Detection Network for Remote Sensing Images via Heatmap-Saliency Distillation

Object detection in remote sensing (RS) images is a challenging task because of complex background and multi-scale objects. Recently, much research has been devoted to improving detection accuracy, but they ignore the speed and memory size. In this letter, we introduce a light anchor-free detection model for resource-limited satellite devices trained with the proposed heatmap-saliency distillation (HSD) strategy, enhancing the performance of small models by learning teachers’ heatmaps in output layers and middle layers. On the one hand, the students mimic the teacher’s localization prediction of negative targets indicating the object shapes. On the other hand, saliency coefficient maps are generated by the ground truth heatmaps with rectangular masks to help students obtain features of multi-scale objects and local contexts across the complex background. Significantly, the model with a Res-9-256 backbone achieves 94.60% mAP on the NWPU VHR-10 dataset with only 8.5 MB of memory. And the test time of this tiny network performs 16.2 ms per image. Additional experiments are conducted on the DOTA dataset. Comprehensive evaluations demonstrate the effectiveness of our light anchor-free object detector trained with the HSD method.

Data-Free Low-Bit Quantization for Remote Sensing Object Detection

Convolutional neural networks (CNNs) have been extensively used in remote sensing (RS) detection models. Although the memory size of detection models is massive, quantization can offer access to using these models in satellite embedded devices. However, for privacy reasons, institutions in charge of quantification operations may not obtain the original images. Based on this circumstance, current data-based quantization methods are no longer applicable, and data-free methods have poor performance for low-bit quantization. To address this problem, we propose a data-free quantization method for the CNN-based RS detection model. First, we use a generative adversarial network (GAN) to generate fake scene images. These images represent the global contextual information of each category. Second, we quantify the full-precision pretrained detection network. Finally, we train the quantized model to mimic the performance of the full-precision model by the proposed alternate training strategy with the generated fake scene images. We apply our method on CenterNet with a ResNet-18 backbone and evaluate the quantized model on the NWPU VHR-10 and DOTA datasets. The results show that our 5 bit quantized detection network obtains 94.1% mAP on NWPU VHR-10 and compresses the memory size to 0.158 times that of the full-precision network. Experiments verify that our data-free scene generation quantization algorithm maintains high performance with a large model compression ratio.

Learning Efficient and Accurate Detectors With Dynamic Knowledge Distillation in Remote Sensing Imagery

Deep convolutional neural networks (CNNs) have brought a tremendous increase in detection accuracy, but too cumbersome model makes them hard to deploy on low computation edge devices, such as satellites and unmanned aerial vehicles. A promising method to tackle this problem is knowledge distillation (KD), which makes models lightweight with satisfactory accuracy. For remote sensing images, the objects are usually environment-related and located in a cluttered scene. The features that objects’ semantic information relies on are tangled. However, existing distillation methods only imitate feature distribution derived from regions, including objects resulting in poor performance. Furthermore, masses of instances generated by teachers are blindly inherited, even if some of them are outliers. In this article, we propose a general and effective KD framework called dynamic knowledge distillation (DKD). First, our framework leverages the dynamic global distillation (GD) module to discover valuable regions from the foreground and background for multiscale features imitation, avoiding ignoring the potential geographical spatial relationship. Second, we propose a dynamic instance selection distillation (ISD) module to give students the ability of self-judgment through the magnitude of detection loss. Third, toward more accurate handling of hard samples in regression, a training-status-aware loss is tailored to guide students mine knowledge about objects with large aspect ratio or small size. Extensive experiments are conducted to show the effectiveness of DKD framework. The detection results on DOTA and NWPU VHR-10 dataset illustrate that our method is suitable for single-stage, two-stage and even anchor-free detectors. It shows the state-of-the-art performance. The code will be publicly available.

Enhancing Mid–Low-Resolution Ship Detection With High-Resolution Feature Distillation

To enhance mid-low-resolution ship detection, existing methods generally use image super-resolution (SR) as a preprocessing step and feed the super-resolved images to the detectors. However, these methods only use high-resolution (HR) images as ground-truth labels to supervise the training of their SR module but overlook the rich HR information in the detection stage. Inspired by the recent advances in knowledge distillation, in this letter, we design a feature distillation framework to fully exploit the information in ground-truth HR images to handle mid-low-resolution ship detection. Our framework consists of a student network and a teacher network. The student network first super-resolves input images using an SR module and then feeds the super-resolved images to the detection module. The teacher network whose architecture is the same as the student detection module directly takes HR images as input to generate HR feature representation and then distills these HR features to the student network through a distillation loss. Using our feature distillation framework, HR images are not only used as ground-truth labels to train the SR module but also provide ``ground-truth'' features to train the detection module, which enhances the detection performance of the student network. We apply our framework to several popular detectors, including FCOS, Faster-RCNN, Mask-RCNN, and Cascase-RCNN, and conduct extensive ablation studies to validate its effectiveness and generality. Experimental results on the HRSC2016, DOTA, and NWPU VHR-10 datasets demonstrate that, when applying our framework to Faster-RCNN, our method can outperform several state-of-the-art detection methods in terms of mAP50 and mAP75.

Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

Lightweight object detector is currently gaining more and more popularity in remote sensing. In general, it’s hard for lightweight detectors to achieve competitive performance compared to traditional deep models, while knowledge distillation is a promising training method to tackle the issue. Since the background is more complicated and the object size varies extremely in remote sensing images, it will deliver lots of noise and affect the training performance when directly applying the existing knowledge distillation methods. To tackle the above problems, we propose an Adaptive Reinforcement Supervision Distillation (ARSD) framework to promote the detection capability of the lightweight model. Firstly, we put forward a multiscale core features imitation (MCFI) module for transferring the knowledge of features, which can adaptively select the multiscale core features of objects for distillation and focus more on the features of small objects by an area-weighted strategy. In addition, a strict supervision regression distillation (SSRD) module is designed to select the optimal regression results for distillation, which facilitates the student to effectively imitate the more precise regression output of the teacher network. Massive experiments on the DOTA, DIOR, and NWPU VHR-10 datasets prove that ARSD outperforms the existing distillation SOTA methods. Moreover, the performance of lightweight model trained with our method transcends other classic heavy and lightweight detectors, which beneficiates the development of lightweight models.

Teaching Teachers First and Then Student: Hierarchical Distillation to Improve Long-Tailed Object Recognition in Aerial Images

Remote sensing data distribution generally exposes the long-tail characteristic. This will limit the object recognition performance of existing deep models when they are trained with such unbalanced data. In this paper, we propose a novel hierarchical distillation framework to address the long-tailed object recognition in aerial images. Firstly, we notice that not only student model should learn feature representations from teachers, but also teacher models should learn feature representations from each other. Therefore, we build hierarchical teacher-wise distillation to improve the feature representations of the teacher models trained with middle and tail data, which is achieved by distilling the feature representations of the teacher model trained with head data. Secondly, we notice that the feature representations of the middle and tail classes can not be effectively distilled from the teacher to the student, since too little middle and tail data can be used to learn. Thus, we propose self-calibrated sampling learning that enforces the student to strengthen the learning of the middle and tail data, thereby improving the student’ feature learning ability. Extensive experiments on two widely-used DOTA and FGSC-23 datasets demonstrate superior performance of the proposed method compared with state-of-the-art methods. Model and code are publicly available at: https://github.com/wdzhao123/T2FTS.

MStrans: Multiscale Vision Transformer for Aerial Objects Detection

Detecting objects in aerial images is a challenging task due to the large-scale variations and arbitrary orientations with tiny instances. A new multi-scale transformer-based aerial objects detector called MStrans is proposed in this paper to deal with the challenges in aerial detection. To detect remote instances, MStrans adopts a multi-scale patch embedding transformer (MViT) to extract the global features of the image effectively. Furthermore, to capture the different discriminant features for classification and regression branch tasks, the partial interactive fusion module (PIFM) is designed to enhance the semantic expression of the key features of classification and regression tasks by using the strategy of interactive modeling of adjacent layer features. In addition, considering that the transformer may worsen the local feature details while capturing long-distance feature dependencies, this paper designs a global to local interactive fusion module (GLIFM). It uses the advantage of convolution to extract local features to enrich the detailed information in the transformer. Experiments were carried out on DOTA and DIOR datasets, and the MStrans achieves superior detection performances compared with other approaches.

Learning Orientation Information From Frequency-Domain for Oriented Object Detection in Remote Sensing Images

Object detection in remote sensing images (RSIs) poses great difficulties due to arbitrary orientations, various scales and dense location of the targets over the ground. Recent evidence suggests that encoding the orientation information is of great use for training an accurate object detector for oriented object detection (OOD). In this paper, we propose a new frequency-domain orientation learning (FDOL) module with two main components: the frequency domain feature extraction (FFE) network and an orientation enhanced self-attention layer (OES-Layer). The FFE network models the interactions among spatial locations in the frequency domain to determine the frequency of spatial features. Then, these features are fed into our OES-Layer to learn the orientation information. Moreover, the orientation weights are adopted to guide the feature selection in a self-attention architecture, using them as a control gate to emphasize the spatial responses of target instances. Considering that the original similarity weights (calculated by the self-attention algorithm) do not distinctly model the orientation variation, the considered orientation weights provide an efficient asset to emphasize the orientation of objects. Extensive experiments on the DOTA and HRSC2016 datasets demonstrate that our method achieves state-of-the-art performance among single-scale methods, while achieving competitive performance over multi-scale methods.

Anchor-Free Oriented Proposal Generator for Object Detection

Oriented object detection is a practical and challenging task in remote sensing image interpretation. Nowadays, oriented detectors mostly use horizontal boxes as intermedium to derive oriented boxes from them. However, the horizontal boxes are inclined to get small Intersection-over-Unions (IoUs) with ground truths, which may have some undesirable effects, such as introducing redundant noise, mismatching with ground truths, detracting from the robustness of detectors, etc. In this paper, we propose a novel Anchor-free Oriented Proposal Generator (AOPG) that abandons horizontal box-related operations from the network architecture. AOPG first produces coarse oriented boxes by a Coarse Location Module (CLM) in an anchor-free manner and then refines them into high-quality oriented proposals. After AOPG, we apply a Fast R-CNN head to produce the final detection results. Furthermore, the shortage of large-scale datasets is also a hindrance to the development of oriented object detection. To alleviate the data insufficiency, we release a new dataset on the basis of our DIOR dataset and name it DIOR-R. Massive experiments demonstrate the effectiveness of AOPG. Particularly, without bells and whistles, we achieve the accuracy of 64.41%, 75.24% and 96.22% mAP on the DIOR-R, DOTA and HRSC2016 datasets respectively. Code and models are available at https://github.com/jbwang1997/AOPG.

Target Detection Model Distillation Using Feature Transition and Label Registration for Remote Sensing Imagery

Deep convolution networks have been widely used in remote sensing target detection for various applications in recent years. Target detection models with many parameters provide better results but are not suitable for resource-constrained devices due to their high computational cost and storage requirements. Furthermore, current lightweight target detection models for remote sensing imagery rarely have the advantages of existing models. Knowledge distillation can improve the learning ability of a small student network from a large teacher network due to acceleration and compression. However, current knowledge distillation methods typically use mature backbones as teacher and student networks are unsuitable for target detection in remote sensing imagery. In this paper, we propose a target detection model distillation (TDMD) framework using feature transition and label registration for remote sensing imagery. A lightweight attention network is designed by ranking the importance of the convolutional feature layers in the teacher network. Multi-scale feature transition based on a feature pyramid is utilized to constrain the feature maps of the student network. A label registration procedure is proposed to improve the TDMD model's learning ability of the output distribution of the teacher network. The proposed method is evaluated on the DOTA and NWPU VHR-10 remote sensing image datasets. The results show that the TDMD achieves a mean Average Precision (mAP) of 75.47% and 93.81% on the DOTA and NWPU VHR-10 datasets, respectively. Moreover, the model size is 43% smaller than that of the predecessor model (11.8 MB and 11.6 MB for the two datasets).

R²IPoints: Pursuing Rotation-Insensitive Point Representation for Aerial Object Detection

Anchor-free aerial object detection methods have recently attracted much attention due to their simplicity and efficiency. However, the performance is still unsatisfactory due to the following two main limitations. On the one hand, the anchor-free detector employs ordinary convolution layers with axis-aligned receptive fields to extract object features, resulting in lacking internal mechanisms to handle the rotation variance. On the other hand, the detector sacrifices much semantic information to achieve faster detection, leading to the inability to deal with objects’ high inter-class similarity and intra-class diversity. To address these issues, in this paper, we present a unique anchor-free detector, termed Rotation-Insensitive Point Representation (R2IPoints), of which a set of category-aware points are employed to encode the spatial and semantic information of the arbitrary-oriented objects. Specifically, we first devise a Stacked Rotation convolution Module (SRM) to encourage the learning of rotation-insensitive point representation by adaptively modelling orientation-agnostic interdependencies over stochastically rotated features. Meanwhile, we further introduce a Class-specific Semantic enhancement Module (CSM). It performs category-aware semantic activation to recalibrate features, thus enabling the point representation to be aware of object categories. Through jointly optimizing the two proposed modules in an end-to-end manner, R2IPoints could simultaneously generate rotation-insensitive and category-aware point representation. Extensive experiments on the challenging DIOR and DOTA datasets demonstrate the superiority of the proposed method. We achieve 72.7% mAP on DIOR and 74.34% mAP on DOTA, surpassing the baseline method of +2.4% mAP and +2.49% mAP, respectively. The code is available at https://github.com/shnew/R2IPoints.

Rethinking Classification of Oriented Object Detection in Aerial Images

With the help of the rapid development of technology, especially the prevalence of UAVs (unmanned aerial vehicles), object detection in aerial images gains much more attention in computer vision and deep learning. However, traditional methods use horizontal bounding boxes for object representation leading to inconsistency between objects and features. Therefore, many detectors are being built to tackle this problem, and normally they use the conventional approaches of training and testing to achieve the results. Our pipeline proposed to strengthen not only the classification but also localization via independent training processes using convex-hull transformation in data pre-processing phase. We experimented with the well-designed S2ANet, R3Det, ReDet, RoI Transformer and Oriented R-CNN on the well-established oriented object detection dataset DOTA. Then we adopt the best detectors with the well-known classification network EfficientNet to our proposed pipeline and achieve promising results on the oriented object detection DOTA dataset. Moreover, our pipeline can flexibly be adapted to various oriented object detection baselines improving the results in classification via independent extensive training cycles.

Multiscale and Direction Target Detecting in Remote Sensing Images via Modified YOLO-v4

Traditional target detection algorithms have difficulty to adapt complex environmental changes and have limited applicable scenarios. However, the deep-learning-based target detection model can automatically learn with strong generalization capability. In this article, we choose a single-stage deep-learning-based target detection model for research based on the model’s real-time processing requirements and to improve the accuracy and the robustness of target detection in remote sensing images. In addition, we improve the YOLOv4 network and present a new approach. First, we propose a classification setting of the nonmaximum suppression threshold to increase the accuracy without affecting the speed. Second, we study the anchor frame allocation problem in YOLOv4 and propose two allocation schemes. The proposed anchor frame scheme also improves the detection performance, and experimental results on the DOTA dataset validate their effectiveness.

Cycle-Spinning Convolution for Object Detection

Deep convolutional networks are prominently used in object detection tasks due to their notable performances. These networks typically have pooling layers following the convolution, which effectively subsamples the convolution output, potentially introducing aliasing. An aliased signal emerging in the earlier layers inevitably propagates throughout the network and such distortion potentially prevents getting the best performance out of a network. In this study, we propose integrating the cycle-spinning (CS) into the convolutional layer to have a more robust object detection pipeline. CS makes use of the shift-variant characteristics of pooling, where each shifted version of the image results in a different output. The CS combines these outputs after unshifting them to remove artifacts and alleviate distortion. The proposed method does not introduce any additional trainable parameters and can be straightforwardly integrated into convolution layers. The experimental results show that integrating CS into object detection algorithms in the DOTA dataset results in an up to 5.5% increase in the mAP values.

Multi-Oriented Rotation-Equivariant Network for Object Detection on Remote Sensing Images

Object detection has attracted a lot of attention in the field of image automatic interpretation. Detectors based on convolution neural networks (CNNs) applied in natural scene images encode detection results with horizontal bounding boxes (HBBs), which can not accurately calibrate the position and shape of the arbitrary-orientation objects on remote sensing images (RSIs). To solve these issues, we propose an object detection framework named multi-oriented rotation-equivariant network (MORE-Net) in this letter. The MORE-Net consists of a multi-oriented rotating filter (MORF) and a multi-oriented ground objects detector. The MORF generates rotation-equivariant features by rotating canonical filter according to the predefined discrete directions. Equipped with MORF, the multi-oriented ground objects detector extracts rotation-invariant semantic representations for each decoded rotated region of interest (RRoI). We also propose area smooth (AS) L1 loss to impose tighter shape and position constraints to the RROIs. Extensive experiments and comprehensive evaluations on the large-scale DOTA dataset demonstrate the effectiveness of the proposed framework, which achieves a mean average precision (mAP) value of 81.27 on the DOTA-v1.0 dataset and a mAP value of 78.03 on the DOTA-v1.5 dataset.

Structurally re-parameterized rotation detector for arbitrary-oriented objects in high-resolution remote sensing images

ABSTRACT Horizontal bounding boxes are inflexible for precisely locating geospatial objects with arbitrary orientations in high-resolution remote sensing images. Recently, rotation detectors with oriented bounding boxes have been found to have a positive effect on the detection of arbitrary-oriented objects. However, this method usually suffers from the requirement of a heavy network structure to learn orientation information. Therefore, a structurally re-parameterized rotation detector (SRep-RDet) for arbitrary-oriented objects in high-resolution remote sensing images is proposed. (1) A structurally re-parameterized backbone network, RepVGG-B1g2, is introduced to the detector based on RetinaNet to decouple the training-time multi-branch and inference-time single-path architecture. (2) The multiscale features of RepVGG-B1g2 are refined by using a lightweight channel attention structure. (3) The multiscale features are fused by structurally re-parameterizing a portion of the convolution layers in the feature pyramid network (FPN). (4) Arbitrary-oriented objects are detected by using a rotation detector with oriented bounding boxes. The experimental results on DOTA and HRSC2016 datasets achieve competitive performance.

Toward object alphabet augmentation for object detection in very high‐resolution satellite images

SummaryObject detection in very high‐resolution satellite images has become an important tool in many fields. So there is a vital need to build a more precise and accurate detector. There is a great challenge to detect very small and condensed objects with their different semantics shapes. Recently, convolutional neural networks with different structures have achieved awesome performance. We propose object alphabet augmentation method that extracts object instants during the training process to form an alphabet for these objects in a newly generated dataset. The proposed method uses the DarkNet‐53 framework to get the main features for each object using two new generated datasets for alphabet augmentation up‐down where objects are placed in up down, and alphabet augmentation horizontal where objects are placed in their original setting. Experiments are conducted on datasets randomly generated from the DOTA dataset. The experimental results show that the proposed method has improved the accuracy of detection of the target objects of the DOTA dataset for most object classes, especially for the up‐down dataset.

ECAP-YOLO: Efficient Channel Attention Pyramid YOLO for Small Object Detection in Aerial Image

Detection of small targets in aerial images is still a difficult problem due to the low resolution and background-like targets. With the recent development of object detection technology, efficient and high-performance detector techniques have been developed. Among them, the YOLO series is a representative method of object detection that is light and has good performance. In this paper, we propose a method to improve the performance of small target detection in aerial images by modifying YOLOv5. The backbone is was modified by applying the first efficient channel attention module, and the channel attention pyramid method was proposed. We propose an efficient channel attention pyramid YOLO (ECAP-YOLO). Second, in order to optimize the detection of small objects, we eliminated the module for detecting large objects and added a detect layer to find smaller objects, reducing the computing power used for detecting small targets and improving the detection rate. Finally, we use transposed convolution instead of upsampling. Comparing the method proposed in this paper to the original YOLOv5, the performance improvement for the mAP was 6.9% when using the VEDAI dataset, 5.4% when detecting small cars in the xView dataset, 2.7% when detecting small vehicle and small ship classes from the DOTA dataset, and approximately 2.4% when finding small cars in the Arirang dataset.

Comparison of deep learning models in terms of multiple object detection on satellite images

The images obtained by remote sensing contain important data about ground surface. It is an important issue to detect objects on the ground surface with these images. Deep learning models are known to give better results in studies on object detection. However, the superiority of the deep learning models over each other is unknown. For this reason, it should be clarified which model is superior in terms of object detection and which model should be used in studies. In this study, it was aimed to reveal the superiorities of deep learning models by comparing their performance in detecting multiple objects. By using 11 deep learning models that are frequently encountered in the literature, the application of detecting objects of 14 classes in the DOTA dataset were made. 49,053 objects in 888 images were used for training by using AlexNet, Vgg16, Vgg19, GoogleNet, SequezeeNet, Resnet18, Resnet50, Resnet101, Inceptionresnetv2, inceptionv3, DenseNet201 models. After the training, 13,772 objects consisting of 14 classes in 277 images were used for testing with RCNN, which is one of the object detection methods. The performance of each algorithm in 14 classes has been demonstrated by using Average Precision (AP) and Mean Average Precision (mAP) to measure the performance of the models from their metrics. In a particular class of each deep learning model, difference in performance was observed The model with the highest performance varies in each class. In the application, the most successful average mAP value of 14 classes was Vgg16 with 24.64, while the lowest was InceptionResnetV2 with 11.78. In this article, the success of deep learning models in detecting multiple objects has been demonstrated practically and it is thought to be an important resource for researchers who will study on this subject.