Remote Sensing Object Detection Research Articles

Learning Lightweight and Superior Detectors with Feature Distillation for Onboard Remote Sensing Object Detection

CubeSats provide a low-cost, convenient, and effective way of acquiring remote sensing data, and have great potential for remote sensing object detection. Although deep learning-based models have achieved excellent performance in object detection, they suffer from the problem of numerous parameters, making them difficult to deploy on CubeSats with limited memory and computational power. Existing approaches attempt to prune redundant parameters, but this inevitably causes a degradation in detection accuracy. In this paper, the novel Context-aware Dense Feature Distillation (CDFD) is proposed, guiding a small student network to integrate features extracted from multi-teacher networks to train a lightweight and superior detector for onboard remote sensing object detection. Specifically, a Contextual Feature Generation Module (CFGM) is designed to rebuild the non-local relationships between different pixels and transfer them from teacher to student, thus guiding students to extract rich contextual features to assist in remote sensing object detection. In addition, an Adaptive Dense Multi-teacher Distillation (ADMD) strategy is proposed, which performs adaptive weighted loss fusion of students with multiple well-trained teachers, guiding students to integrate the learning of helpful knowledge from multiple teachers. Extensive experiments were conducted on two large-scale remote sensing object detection datasets with various network structures; the results demonstrate that the trained lightweight network achieves auspicious performance. Our approach also shows good generality for existing state-of-the-art remote sensing object detectors. Furthermore, by experimenting on large general object datasets, we demonstrate that our approach is equally practical for general object detection distillation.

Remote Sensing Object Detection Based on Convolution and Swin Transformer

Remote sensing object detection is an essential task for surveying the earth. It is challenging for the target detection algorithm in natural scenes to obtain satisfactory detection results in remote sensing images. In this paper, the RAST-YOLO (You only look once with Regin Attention and Swin Transformer) algorithm is proposed to address the problems of remote sensing object detection, such as significant differences in target scales, complex backgrounds, and tightly arranged small-size targets. To increase the information interaction range of the feature map, make full use of the background information of the object, and improve the detection accuracy of the object with a complex background, the Regin Attention (RA) mechanism combined with Swin Transformer as the backbone is proposed to extract features. To improve the detection accuracy of small objects, the C3D module is used to fuse deep and shallow semantic information and optimize the multi-scale problem of remote sensing targets. To evaluate the performance of RAST-YOLO, extensive experiments are performed on DIOR and TGRS-HRRSD datasets. The experimental results show that RAST achieves state-of-the-art detection accuracy with high efficiency and robustness. Specifically, compared with the baseline network, the mean average precision (mAP) of detection results is improved by 5% and 2.3% on DIOR and TGRS-HRRSD datasets, respectively, which demonstrates RAST-YOLO is effective and superior. Moreover, the lightweight structure of RAST-YOLO can ensure the real-time detection speed and obtain excellent detection results.

OBBStacking: An Ensemble Method for Remote Sensing Object Detection

Ensemble methods are a reliable way to combine several models to achieve superior performance. However, research on the application of ensemble methods in the remote sensing object detection scenario is mostly overlooked. Two problems arise. First, one unique characteristic of remote sensing object detection is the Oriented Bounding Boxes (OBB) of the objects and the fusion of multiple OBBs requires further research attention. Second, the widely used deep learning object detectors provide a score for each detected object as an indicator of confidence, but how to use these indicators effectively in an ensemble method remains a problem. Trying to address these problems, this paper proposes OBBStacking, an ensemble method that is compatible with OBBs and combines the detection results in a learned fashion. This ensemble method helps take 1st place in the Challenge Track <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fine-grained Object Recognition in High-Resolution Optical Images</i> , which was featured in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2021 Gaofen Challenge on Automated High-Resolution Earth Observation Image Interpretation</i> . The experiments on DOTA dataset and FAIR1M dataset demonstrate the improved performance of OBBStacking and the features of OBBStacking are analyzed. Code will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Haoning724/obbstacking</uri>

RT-YOLO: A Residual Feature Fusion Triple Attention Network for Aerial Image Target Detection

In recent years, target detection of aerial images of unmanned aerial vehicle (UAV) has become one of the hottest topics. However, target detection of UAV aerial images often presents false detection and missed detection. We proposed a modified you only look once (YOLO) model to improve the problems arising in object detection in UAV aerial images: (1) A new residual structure is designed to improve the ability to extract features by enhancing the fusion of the inner features of the single layer. At the same time, triplet attention module is added to strengthen the connection between space and channel and better retain important feature information. (2) The feature information is enriched by improving the multi-scale feature pyramid structure and strengthening the feature fusion at different scales. (3) A new loss function is created and the diagonal penalty term of the anchor frame is introduced to improve the speed of training and the accuracy of reasoning. The proposed model is called residual feature fusion triple attention YOLO (RT-YOLO). Experiments showed that the mean average precision (mAP) of RT-YOLO is increased from 57.2% to 60.8% on the vehicle detection in aerial image (VEDAI) dataset, and the mAP is also increased by 1.7% on the remote sensing object detection (RSOD) dataset. The results show that the RT-YOLO outperforms other mainstream models in UAV aerial image object detection.

Object-Centric Masked Image Modeling-Based Self-Supervised Pretraining for Remote Sensing Object Detection

Masked image modeling (MIM) has been proved to be an optimal pretext task for self-supervised pretraining (SSP), which can facilitate the model to capture an effective task-agnostic representation at the pretraining step and then advance the fine-tuning performance of various downstream tasks. However, under the high randomly masked ratio of MIM, the scene-level MIM-based SSP is hard to capture the small-scale objects or local details from complex remote sensing scenes. Then, when the pretrained models capturing more scene-level information are directly applied for object-level fine-tuning step, there is an obvious representation learning misalignment between model pretraining and fine-tuning steps. Therefore, in this article, a novel object-centric masked image modeling (OCMIM) strategy is proposed to make the model better capture the object-level information at the pretraining step and then further advance the object detection fine-tuning step. First, to better learn the object-level representation involving full scales and multicategories at MIM-based SSP, a novel object-centric data generator is proposed to automatically setup targeted pretraining data according to objects themselves, which can provide the specific data condition for object detection model pretraining. Second, an attention-guided mask generator is designed to generate a guided mask for MIM pretext task, which can lead the model to learn more discriminative representation of highly attended object regions than by using the randomly masking strategy. Finally, several experiments are conducted on six remote sensing object detection benchmarks, and results proved that the proposed OCMIM-based SSP strategy is a better pretraining way for remote sensing object detection than normally used methods.

Remote Sensing Object Detection Based on Strong Feature Extraction and Prescreening Network

Remote sensing object detection has been an important and challenging research hotspot in computer vision that is widely used in military and civilian fields. Recently, the combined detection model of CNN and Transformer has achieved good results, but the problem of poor detection performance of small objects still needs to be solved urgently. This letter proposes a deformable DETR-based framework for object detection in remote sensing images. Firstly, Multi-Scale Split Attention (MSSA) is designed to extract more detailed feature information by grouping. Next, we propose Multi-Scale Deformable Prescreening Attention (MSDPA) mechanism in decoding layer, which achieves the purpose of pre-screening, so that the encoder-decoder structure can obtain attention map more efficiently. Finally, the A-D loss function is applied to the prediction layer, increasing the attention of small objects and optimizing the IOU function. We conduct extensive experiments on the DOTA v1.5 dataset and the HRRSD dataset, which show that the reconstructed detection model is more suitable for remote sensing objects, especially for small objects. The average detection accuracy in DOTA dataset has improved by 4.4% (up to 75.6%), especially the accuracy of small objects has raised by 5%.

Transformer Based Remote Sensing Object Detection With Enhanced Multispectral Feature Extraction

As a convention, satellites and drones are equipped with sensors of both the visible light spectrum and the infrared (IR) spectrum. However, existing remote sensing object detection methods mostly use RGB images captured by the visible light camera while ignoring IR images. Even for algorithms that take RGB-IR image pairs as input, they may fail to extract all potential features in both spectrums. This letter proposes Multispectral DETR, a remote sensing object detector based on the deformable attention mechanism. To enhance multispectral feature extraction and attention, DropSpectrum and SwitchSpectrum methods are further proposed. DropSpectrum facilitates the extraction of multispectral features by requiring the model to detection some of the targets with only one spectrum. SwitchSpectrum eliminates the level bias caused by the fixed order of RGB-IR feature maps and enhances attention on multispectral features. Experiments on the VEDAI dataset show the state-of-the-art performance of Multispectral DETR and the effectiveness of both DropSpectrum and SwitchSpectrum.

TransMIN: Transformer-Guided Multi-Interaction Network for Remote Sensing Object Detection

Remote sensing (RS) object detectors based on convolutional neural networks (CNNs) are hard to model global context dependencies. Transformer-based detectors can overcome this problem via global pairwise interactions, but more comprehensive information interactions are not systemically investigated to boost the detection performance. In view of this issue, we propose a transformer-guided multi-interaction network (TransMIN), which uses ResNet50-feature pyramid network (FPN) as the backbone for remote sensing object detection (RSOD). Specifically, we implement local–global feature interactions (LGFIs) by combining convolution and transformer in the residual blocks of ResNet50 to learn complementary features. We implement cross-view feature interactions (CVFIs) via transformers in the pyramid layers of FPN to capture the correlation between reference features (spatial edge priors and channel statistics) and pyramid features. This enhances edge information and suppresses background interference. In the detection head, we adopt a task-interactive sample assigner (TISA) by considering the interactions of classification and localization losses to obtain high-quality predictions. Experiments on two benchmark datasets demonstrate the superior detection performance of TransMIN over state-of-the-art methods.

Collaborative Learning-Based Network for Weakly Supervised Remote Sensing Object Detection

Existing object detection algorithms rely excessively on instance-level labels, which are both time-consuming and expensive. In particular, for remote sensing images (RSI) with small and dense objects, the labeling cost is much higher than that of general images. Moreover, the propagation process of the labels over the noisy channel results in blurred and noisy information. To address the problem of obtaining RSI instance-level labels, we aim to propose a collaborative learning-based network for weakly supervised remote sensing object detection (CLN-RSOD). Compared with the state-of-the-art, the proposed model combines the advantages of two object detection sub-networks by jointly training. This improves the model's capability and thereby enhances the detection effect for multi-object in RSI. Moreover, we employ a mask-based proposal refinement algorithm for remote sensing images (MPR-RS) to optimize the candidate boxes. In addition, according to the data distribution characteristics of RSI, we introduce a new joint pooling module (JPM) in CLN-RSOD to enhance the backbone network's characterization of RSI. Finally, the experimental results on two public remote sensing datasets illustrate that the proposed weakly supervised learning method is superior to other weakly supervised methods and demonstrates the effectiveness of the proposed CLN-RSOD.

FANet: An Arbitrary Direction Remote Sensing Object Detection Network Based on Feature Fusion and Angle Classification

High-precision remote sensing image object detection has broad application prospects in military defense, disaster emergency, urban planning, and other fields. However, the arbitrary orientation, dense arrangement, and small size of objects in remote sensing images lead to poor detection accuracy of existing methods. To achieve accurate detection, this paper proposes an arbitrary directional remote sensing object detection method, called FANet, based on feature fusion and angle classification. Initially, the angle prediction branch is introduced, and the circular smooth label method is used to transform the angle regression problem into a classification problem, which solves the difficult problem of abrupt changes in the boundaries of the rotating frame while realizing the object frame rotation. Subsequently, to extract robust remote sensing objects, innovative introduce pure convolutional model as a backbone network, while Conv is replaced by GSConv to reduce the number of parameters in the model along with ensuring detection accuracy. Finally, the strengthen connection feature pyramid network (SC-FPN) is proposed to redesign the lateral connection part for deep and shallow layer feature fusion, and add jump connections between the input and output of the same level feature map to enrich the feature semantic information. In addition, add a variable parameter to the original localization loss function to satisfy the bounding box regression accuracy under different IoU thresholds, and thus obtain more accurate object detection. The comprehensive experimental results on two public datasets for rotated object detection DOTA and HRSC2016 demonstrate the effectiveness of our method.

MashFormer: A Novel Multiscale Aware Hybrid Detector for Remote Sensing Object Detection

Object detection is a critical and demanding topic in the subject of processing satellite and airborne images. The targets acquired in remote sensing imagery are at various sizes, and the backgrounds are complicated, which makes object detection extremely challenging. We address these aforementioned issues in this paper by introducing the MashFormer, an innovative multi-scale aware CNN and Transformer integrated hybrid detector. Specifically, MashFormer employs the transformer block to complement the convolutional neural network (CNN) based feature extraction backbone, which could obtain the relationships between long-range features and enhance the representative ability in complex background scenarios. With the intention of improving the detection performance for objects with multi-scale characteristic, since in remote sensing scenarios, the size of object varies greatly. A multi-level feature aggregation component, incoperate with a cross-level feature alignment module is designed to alleviate the semantic discrepancy between features from shallow and deep layers. To verify the effectiveness of the suggested MashFormer, comparative experiments are carried out with other cutting-edge methodologies using the publicly available High Resolution Remote Sensing Detection (HRRSD) and Northwestern Polytechnical University (NWPU) VHR-10 datasets. The experimental findings confirm the effectiveness and superiority of our suggested model by indicating that our approach has greater mean average precision (mAP) than the other methodologies.

Task Alignment Interaction and Cross-Scale Guided Enhancement for Remote Sensing Object Detection

Task Alignment Interaction and Cross-Scale Guided Enhancement for Remote Sensing Object Detection

Instance-Aware Distillation for Efficient Object Detection in Remote Sensing Images

Practical applications ask for object detection models that achieve high performance at low overhead. Knowledge distillation demonstrates favorable potential in this case by transferring knowledge from a cumbersome teacher model to a lightweight student model. However, previous distillation methods are plagued with massive misleading background information in remote sensing images and ignore investigating the relationships between different instances. In this article, we propose an instance-aware distillation (InsDist for short) method to derive efficient remote sensing object detectors. Our InsDist combines feature-based and relation-based knowledge distillation to make the most of instance-related information in the knowledge transfer from the teacher to the student. On one hand, we propose a parameter-free masking module to decouple instance-related foreground from instance-irrelevant background in multiscale features. On the other hand, we construct the relationships between different instances to enhance the learning of intraclass compactness and interclass dispersion. The student comprehensively imitates both features and relationships from the teacher, yielding considerable effectiveness in dealing with complex remote sensing images. In addition, our InsDist can be easily built on mainstream object detectors with negligible extra cost. Extensive experiments on two large-scale remote sensing object detection datasets, namely DIOR and DOTA, show that our InsDist obtains noticeable gains over other distillation methods for both one-stage and two-stage, as well as both anchor-based and anchor-free detectors. The source code will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/swift1988/InsDist</uri> .

Capsule-inferenced Object Detection for Remote Sensing Images

Frequent and accurate object detection based on remote sensing images can effectively monitor dynamic objects on the Earth's surface. While the Detection Transformer (DETR) offers a simple encoder-decoder structure and a direct set prediction approach to object detection, it falls short in complex remote sensing scenes where entity information and relative positions between objects are critical to target reasoning. Notably, the DETR model's feedforward neural network (FFN) relies on weighted summation for target reasoning, disregarding interactive feature information, which is a major factor affecting detection effectiveness. To address these shortcomings, we propose a DETR-based detection model called (CI_DETR), which uses capsule inference to improve remote sensing object detection. Our approach adds a Multi-level Feature Fusion module to the DETR network, allowing the network to learn how to spatially alter features at different levels, preserving only beneficial information for combination. In addition, we introduce a capsule reasoning module to mine entity information during inference and more effectively model the hierarchical correlation of internal knowledge representation in the neural network, consistent with the thinking model of the human brain. Lastly, we employ a sausage model to measure the similarities and differences of capsules, projecting them onto a curved surface for nonlinear function approximation and maximum preservation of the local responsiveness of capsule entities. Our experiments on public datasets confirm the superior detection performance of our proposed algorithm relative to many current detectors.

Hierarchical Information Enhancing Detector for Remotely Sensed Object Detection

For the remote sensing object detection task, two-stage networks are widely used due to their high accuracy. These networks roughly predict the proposal regions containing potential objects. It is assumed in these methods that the sizes of these regions are close to that of the corresponding real object. However, this assumption is not always true. Consequently, the detector is affected by the size-unfitting proposal regions. In this letter, a hierarchical information enhancing detector (HIE-Det) is advocated to deal with this issue. First, the important semantic reinjection (ISR) module is proposed to mitigate the lack of object semantics caused by the size-unfitting problem. Compared with the normal detectors, the ISR module increases the proportion of information on objects and improves the effectiveness of the detection model. Second, the object boundary enhancing (OBE) module is proposed to improve the robustness of the regression. The OBE module introduces the convolutional branch stacking multigranularity grids for the same proposal region. Multiple granularity levels improve the robustness of the model to the different degrees of proposal size unfitting. Finally, to evaluate the effectiveness of the HIE-Det on multiscale datasets in a balanced and effective manner, we propose the scale-modulating scores (S-scores), i.e., scale-modulating average precision (sAP) and scale-modulating average recall (sAR). Compared with the other comprehensive scores, the S-scores are rid of the sample amounts and give priority to weaker indices. Implementing the proposed HIE-Det, S-scores {sAP, sAR} are, respectively, improved from {17.3%, 29.7%} to {34.8%, 43.2%}, reaching the state-of-the-art performance on the HRRSD dataset. These experiments verify the effectiveness of the proposed HIE-Det.

OAB-YOLOv5: One-Anchor-Based YOLOv5 for Rotated Object Detection in Remote Sensing Images

Remote sensing images are widely distributed, small in object size, and complex in background, resulting in low accuracy and slow speed of remote sensing image detection. Existing remote sensing object detection is generally based on the detector with anchors. With the proposal of a feature pyramid network (FPN) and focal loss, an anchorless detector emerges, however, the accuracy of anchorless detection is often low. First, this study analyzes the differences and characteristics of the intersection of union (IoU) and shape matchings based on anchors in mainstream algorithms and indicates that in dense or complex scenes, some labels are not easily assigned to positive samples, which leads to detection failure. Subsequently, we proposean one-anchor-based (OAB) object detection algorithm based on the idea of central point sampling in the anchor-free detector. The positive samples and negative samples are defined according to the central point sampling and distance constraint, and an anchor box is preset for each positive sample to accelerate its convergence. It reduces the complexity of the anchor-based detector, improves the inference speed, and reduces the setting of hyperparameters in the traditional matching strategy, rendering the model more flexible. Finally, in order to suppress background noise in remote sensing images, the vision transformer (ViT) is adopted to connect the neck and head, making it easier for the network to pay attention to key information. Thus, it is not easy to lose in the training process. Experiments on challenging public dataset—DOTA dataset- verified the effectiveness of the proposed algorithm. The experimental results show that the mAP of the optimized OAB-YOLOv5 method is improved by 2.79%, the number of parameters is reduced by 13.2%, and the inference time is reduced by 11% compared with the YOLOv5 baseline.

Absorption Pruning of Deep Neural Network for Object Detection in Remote Sensing Imagery

In recent years, deep convolutional neural networks (DCNNs) have been widely used for object detection tasks in remote sensing images. However, the over-parametrization problem of DCNNs hinders their application in resource-constrained remote sensing devices. In order to solve this problem, we propose a network pruning method (named absorption pruning) to compress the remote sensing object detection network. Unlike the classical iterative three-stage pruning pipeline used in existing methods, absorption pruning is designed as a four-stage pruning pipeline that only needs to be executed once, which differentiates it from existing methods. Furthermore, the absorption pruning no longer identifies unimportant filters, as in existing pruning methods, but instead selects filters that are easy to learn. In addition, we design a method for pruning ratio adjustment based on the object characteristics in remote sensing images, which can help absorption pruning to better compress deep neural networks for remote sensing image processing. The experimental results on two typical remote sensing data sets—SSDD and RSOD—demonstrate that the absorption pruning method not only can remove 60% of the filter parameters from CenterNet101 harmlessly but also eliminate the over-fitting problem of the pre-trained network.

AFSPNet: an adaptive feature selection pyramid network for efficient object detection in remote sensing images

In recent studies, remote sensing object detection methods based on deep learning have emerged as a primary concern in environmental monitoring, military investigation, and hazard response. However, many difficulties, such as complex backgrounds, dense target quantities, large-scale variations, and non-uniform distribution, lead to many parameters and complex network structures, thus limiting the accuracy of the detector and slowing the inference speed. To address these issues, we propose a lightweight and efficient object detector for remote sensing images. First, an asymmetric convolution with the visual attention mechanism is reconstructed to decrease the complexity and strengthen the feature representation ability. Then, an adaptive feature selection structure is designed to extract discriminative feature information, which can adaptively model the shapes of objects by introducing deformable convolution to obtain a stronger geometric feature representation. To reduce information loss across different channels and spatial locations, a hybrid receptive field module is also proposed to increase the receptive field model by mixing dilated convolutional layers with different dilation rates. Finally, experimental results on the DIOR dataset show that our approach significantly improves detection accuracy and running speed.

Spiral Search Grasshopper Features Selection with VGG19-ResNet50 for Remote Sensing Object Detection

Remote sensing object detection plays a major role in satellite imaging and is required in various scenarios such as transportation, forestry, and the ocean. Deep learning techniques provide efficient performance in remote sensing object detection. The existing techniques have the limitations of data imbalance, overfitting, and lower efficiency in detecting small objects. This research proposes the spiral search grasshopper (SSG) optimization technique to increase the exploitation in feature selection. Augmentation is applied in input images to generate more images, and this helps to train the model and reduces data imbalance problems. The VGG-19 and ResNet50 model is applied for feature extraction, and this helps to extract deep features to represent objects. The SSG feature selection technique increases the exploitation and select unique features for object detection that helps to overcome the data imbalance and overfitting problem. The SSG feature selection model helps to balance the exploration and exploitation that escape from the local optima trap. The SSG model has 82.45% mAP, the SSD model has 52.6% mAP, and the MPFP-Net model has 80.43% mAP.

Fast and accurate multi-class geospatial object detection with large-size remote sensing imagery using CNN and Truncated NMS

Multi-class geospatial object detection with remote sensing imagery has broad prospects in urban planning, natural disaster warning, industrial production, military surveillance and other applications. Accuracy and efficiency are two common measures for evaluating object detection models, and it is often difficult to achieve both at the same time. Developing a practical remote sensing object detection algorithm that balances the accuracy and efficiency is thus a big challenge in the Earth observation community. Here, we propose a comprehensive high-speed multi-class remote sensing object detection method. Firstly, we obtain a multi-volume YOLO (You Only Look Once) v4 model for balancing speed and accuracy, based on a pruning strategy of the convolutional neural network (CNN) and the one-stage object detection network YOLO v4. Moreover, we apply the Manhattan-Distance Intersection of Union (MIOU) loss function to the multi-volume YOLO v4 to further improve the accuracy without additional computational burden.Secondly, mainly due to computing limitations, a remote sensing image that is large-size relative to a natural image must first be divided into multiple smaller tiles, which are then detected separately, and finally, the detection results are spliced back to match the original image. In the process of remote sensing image slicing, a large number of truncated objects appear at the edge of tiles, which will produce a large number of false results in the subsequent detection links. To solve this problem, we propose a Truncated Non-Maximum Suppression (NMS) algorithm to filter out repeated and false detection boxes from truncated targets in the spliced detection results. We compare the proposed algorithm with the state-of-the-art methods on the Dataset for Object deTection in Aerial images (DOTA) and DOTA v2. Quantitative evaluations show that mAP and FPS reach 77.3 and 35 on DOTA, and 61.0 and 74 on DOTA v2. Overall, our method reaches the optimal balance between efficiency and accuracy, and realizes the high-speed remote sensing object detection.