Rotated Bounding Boxes Research Articles

A Dual-Polarimetric SAR Ship Detection Dataset and a Memory-Augmented Autoencoder-Based Detection Method.

With the development of imaging and space-borne satellite technology, a growing number of multipolarized SAR imageries have been implemented for object detection. However, most of the existing public SAR ship datasets are grayscale images under single polarization mode. To make full use of the polarization characteristics of multipolarized SAR, a dual-polarimetric SAR dataset specifically used for ship detection is presented in this paper (DSSDD). For construction, 50 dual-polarimetric Sentinel-1 SAR images were cropped into 1236 image slices with the size of 256 × 256 pixels. The variances and covariance of both VV and VH polarization were fused into R,G,B channels of the pseudo-color image. Each ship was labeled with both a rotatable bounding box (RBox) and a horizontal bounding box (BBox). Apart from 8-bit pseudo-color images, DSSDD also provides 16-bit complex data for readers. Two prevalent object detectors R3Det and Yolo-v4 were implemented on DSSDD to establish the baselines of the detectors with the RBox and BBox respectively. Furthermore, we proposed a weakly supervised ship detection method based on anomaly detection via advanced memory-augmented autoencoder (MemAE), which can significantly remove false alarms generated by the two-parameter CFAR algorithm applied upon our dual-polarimetric dataset. The proposed advanced MemAE method has the advantages of a lower annotation workload, high efficiency, good performance even compared with supervised methods, making it a promising direction for ship detection in dual-polarimetric SAR images. The dataset is available on github.

유도무기 영상의 회전 객체 탐지를 위한 분류 및 회귀방법의 통합적 접근

In this paper, a novel missile detection method based on a rotated bounding box is proposed. The angle information included in the rotated bounding box is employed to reduce the effect of clutter noise or to track the missile’s forehead when the separation of the missile proceeds. For angle prediction, YOLOv5 used with additional angle parameters considerably degrades the estimation accuracy and improves the computational complexity. To prevent learning performance degradation owing to additional parameters, an angle prediction approach based on a combination of regression and classification is proposed in this paper. As the number parameters of the last layer of the network head are lower than that when using only regression, the proposed method provides excellent detection performance and leads to a significant reduction of error in terms of angle prediction.

A novel directional object detection method for piled objects using a hybrid region-based convolutional neural network

Digital transformation is an information technology (IT) process that integrates digital information with operating processes. Its introduction to the workplace can promote the development of progressively efficient manufacturing processes, accelerating competition in terms of speed and production capacity. Equipment combined with computer vision has begun to replace manpower in certain industries including manufacturing. However, current object detection methods are unable to identify the actual rotation angle of a specific grasped target while objects are piled. Hence this study proposes a framework based on deep learning that integrates two object detection models. Faster R-CNN (region based convolutional neural network) is utilized to search for the direction reference point of the target, and Mask R-CNN is adopted to obtain the segmentation that not only forms the basis of an area filter but also generates a rotated bounding box by minAreaRect function. After integrating the output from two models, the location and actual rotated angle of target can be obtained. The purpose of this research is to provide the robot arm with the position and angle information of the object located on the top for grasping. An empirical dataset of piled footwear insoles was employed to test the proposed method during the assembly process. Results show that the accuracy of the detection reached 96.26%. The implementation of proposed method in the manufacturing process not only can save man power who responsible for sorting out products but also reduce process time to enlarge production capacity. The proposed method can serve as a part of smart manufacturing system to enhance the enterprise’s competitiveness in the future.

Ship detection in SAR images based on recurrent feature pyramid network and rotatable bounding box

Ship detection in synthetic aperture radar (SAR) images plays an important role in remote sensing, but it is still full of challenges in the deep learning area. The primary problem is that ships in SAR images have different sizes and orientations. The off-the-shelf detectors are not able to adapt to the situation. Recurrent feature pyramid networks are presented to detect ships with different sizes especially the small ones. Rotatable region proposal network is used for locating ships with a tighter rectangle. Rotatable anchors with sizes, aspect ratios, and angles are designed according to the distribution of ships in dataset. Multiratio region-of-interest pooling is used for projecting arbitrary-oriented proposals to fixed length vectors. Angle-related intersection-of-unit (ArIoU) is used for evaluating the intersection of rotatable proposals. ArIoU can be an indicator for nonmaximum suppression (NMS) and also is used for preparing negative and positive proposals. A loss function is proposed to compute loss between bounding boxes. The sinusoidal function is used for solving the problem of unstable angle. We also use a dataset called SSDD+ (SAR ship detection dataset plus) to evaluate different methods. Experiments based on SSDD+ show that our method achieves state-of-the-art performance. The dataset and the code will be public at https://zhuanlan.zhihu.com/p/143794468.

SAR Ship Detection Dataset (SSDD): Official Release and Comprehensive Data Analysis

SAR Ship Detection Dataset (SSDD) is the first open dataset that is widely used to research state-of-the-art technology of ship detection from Synthetic Aperture Radar (SAR) imagery based on deep learning (DL). According to our investigation, up to 46.59% of the total 161 public reports confidently select SSDD to study DL-based SAR ship detection. Undoubtedly, this situation reveals the popularity and great influence of SSDD in the SAR remote sensing community. Nevertheless, the coarse annotations and ambiguous standards of use of its initial version both hinder fair methodological comparisons and effective academic exchanges. Additionally, its single-function horizontal-vertical rectangle bounding box (BBox) labels can no longer satisfy the current research needs of the rotatable bounding box (RBox) task and the pixel-level polygon segmentation task. Therefore, to address the above two dilemmas, in this review, advocated by the publisher of SSDD, we will make an official release of SSDD based on its initial version. SSDD’s official release version will cover three types: (1) a bounding box SSDD (BBox-SSDD), (2) a rotatable bounding box SSDD (RBox-SSDD), and (3) a polygon segmentation SSDD (PSeg-SSDD). We relabel ships in SSDD more carefully and finely, and then explicitly formulate some strict using standards, e.g., (1) the training-test division determination, (2) the inshore-offshore protocol, (3) the ship-size reasonable definition, (4) the determination of the densely distributed small ship samples, and (5) the determination of the densely parallel berthing at ports ship samples. These using standards are all formulated objectively based on the using differences of existing 75 (161 × 46.59%) public reports. They will be beneficial for fair method comparison and effective academic exchanges in the future. Most notably, we conduct a comprehensive data analysis on BBox-SSDD, RBox-SSDD, and PSeg-SSDD. Our analysis results can provide some valuable suggestions for possible future scholars to further elaborately design DL-based SAR ship detectors with higher accuracy and stronger robustness when using SSDD.

Accurate IoU computation for rotated bounding boxes in $${\mathbb {R}}^2$$ and $${\mathbb {R}}^3$$

Accurate IoU computation for rotated bounding boxes in $${\mathbb {R}}^2$$ and $${\mathbb {R}}^3$$

ADT-Det: Adaptive Dynamic Refined Single-Stage Transformer Detector for Arbitrary-Oriented Object Detection in Satellite Optical Imagery

The detection of arbitrary-oriented and multi-scale objects in satellite optical imagery is an important task in remote sensing and computer vision. Despite significant research efforts, such detection remains largely unsolved due to the diversity of patterns in orientation, scale, aspect ratio, and visual appearance; the dense distribution of objects; and extreme imbalances in categories. In this paper, we propose an adaptive dynamic refined single-stage transformer detector to address the aforementioned challenges, aiming to achieve high recall and speed. Our detector realizes rotated object detection with RetinaNet as the baseline. Firstly, we propose a feature pyramid transformer (FPT) to enhance feature extraction of the rotated object detection framework through a feature interaction mechanism. This is beneficial for the detection of objects with diverse patterns in terms of scale, aspect ratio, visual appearance, and dense distributions. Secondly, we design two special post-processing steps for rotated objects with arbitrary orientations, large aspect ratios and dense distributions. The output features of FPT are fed into post-processing steps. In the first step, it performs the preliminary regression of locations and angle anchors for the refinement step. In the refinement step, it performs adaptive feature refinement first and then gives the final object detection result precisely. The main architecture of the refinement step is dynamic feature refinement (DFR), which is proposed to adaptively adjust the feature map and reconstruct a new feature map for arbitrary-oriented object detection to alleviate the mismatches between rotated bounding boxes and axis-aligned receptive fields. Thirdly, the focus loss is adopted to deal with the category imbalance problem. Experiments on two challenging satellite optical imagery public datasets, DOTA and HRSC2016, demonstrate that the proposed ADT-Det detector achieves a state-of-the-art detection accuracy (79.95% mAP for DOTA and 93.47% mAP for HRSC2016) while running very fast (14.6 fps with a 600 × 600 input image size).

CPS-Det: An Anchor-Free Based Rotation Detector for Ship Detection

Ship detection is a significant and challenging task in remote sensing. At present, due to the faster speed and higher accuracy, the deep learning method has been widely applied in the field of ship detection. In ship detection, targets usually have the characteristics of arbitrary-oriented property and large aspect ratio. In order to take full advantage of these features to improve speed and accuracy on the base of deep learning methods, this article proposes an anchor-free method, which is referred as CPS-Det, on ship detection using rotatable bounding box. The main improvements of CPS-Det as well as the contributions of this article are as follows. First, an anchor-free based deep learning network was used to improve speed with fewer parameters. Second, an annotation method of oblique rectangular frame is proposed, which solves the problem that periodic angle and bounded coordinates in conjunction with the regression calculation can lead to the problem of loss anomalies. For the annotation scheme proposed in this paper, a scheme for calculating Angle Loss is proposed, which makes the loss function of angle near the boundary value more accurate and greatly improves the accuracy of angle prediction. Third, the centerness calculation of feature points is optimized in this article so that the center weight distribution of each point is suitable for the rotation detection. Finally, a scheme combining centerness and positive sample screening is proposed and its effectiveness in ship detection is proved. Experiments on remote sensing public dataset HRSC2016 show the effectiveness of our approach.

Learning Modulated Loss for Rotated Object Detection

Popular rotated detection methods usually use five parameters (coordinates of the central point, width, height, and rotation angle) or eight parameters (coordinates of four vertices) to describe the rotated bounding box and l1 loss as the loss function. In this paper, we argue that the aforementioned integration can cause training instability and performance degeneration. The main reason is the discontinuity of loss which is caused by the contradiction between the definition of the rotated bounding box and the loss function. We refer to the above issues as rotation sensitivity error (RSE) and propose a modulated rotation loss to dismiss the discontinuity of loss. The modulated rotation loss can achieve consistent improvement on the five parameter methods and the eight parameter methods. Experimental results using one stage and two stages detectors demonstrate the effectiveness of our loss. The integrated network achieves competitive performances on several benchmarks including DOTA and UCAS AOD. The code is available at https://github.com/yangxue0827/RotationDetection.

Adaptive multi-level feature fusion and attention-based network for arbitrary-oriented object detection in remote sensing imagery

Compared with the classic object detection problem, detecting objects in aerial images has some special challenges including huge orientation variations, complicated and large background, and wide multi-scale distribution. Considering these three challenges together, we propose a novel arbitrary-oriented object detection framework consisting of three main parts. Firstly, the Cascading Attention Network (CA-Net) composed of a patching self-attention module and a supervised spatial attention module is proposed for enhancing the feature representations from objects of interest and suppressing the background noises in Feature Pyramid Network (FPN) from coarse to fine. Then, the Adaptive Feature Concatenate Network (AFC-Net) is proposed to adaptively stack the feature maps pooled from all FPN levels as well as the global semantic features, for dealing with the multi-scale change of objects. Lastly, the OBB Multi-Definition and Selection Strategy (OBB-MDS-Strategy) is proposed to regress rotated bounding boxes more smoothly and detect oriented objects more accurately in the training process. Our experiments are conducted on two common and challenging aerial datasets, i.e., DOTA and HRSC2016. Experiments results show that the proposed method has superior performances in multi-orientated objects detection compared with the representative methods.

Bolt damage identification based on orientation-aware center point estimation network

With the development of deep learning, object detection algorithms based on horizontal box are widely used in the field of damage identification. However, damages can be in any direction and position, and they are not necessarily horizontal or vertical. This article proposes a bolt damage identification network, namely, orientation-aware center point estimation network, which models a damage as a center point of its rotated bounding box. The proposed orientation-aware center point estimation network uses deep layer aggregation network to search center points and regress to all other damage properties, such as size and angle. A loss function is designed to improve the optimization efficiency of network. Orientation-aware center point estimation network is applied to bolt damage detection, and comparison with the well-known Faster Region-Convolutional Neural Network (a benchmark using horizontal bounding box) demonstrates the accuracy of the proposed method. Finally, videos were utilized to verify the capability of the proposed orientation-aware center point estimation network in real-time detection of bolt damages.

Graph Convolutional Networks for the Automated Production of Building Vector Maps From Aerial Images

Building footprint delineation from remote sensing imagery is a basic task in surveying and mapping and geographic information system (GIS), which benefits many engineering applications but requires an enormous amount of human delineation. In this study, we aim to find a way to replace human delineation with automated algorithms. For this purpose, we designed a novel pipeline for the automated production of building vector maps from aerial images, which consists of a bounding-box generation module, a graph convolutional network (GCN)-based polygon prediction module, and an empirical polygon regularization module. First, we introduce the bounding box generation module based on region-based object detection, which along with our overlap cropping strategy is used to generate a bounding box for each building instance. The generated bounding boxes have a horizontal rectangle version and a rotated version, which are used to initialize the next stage of our method. Second, we propose a GCN-based method, which is the core of this study, to conduct the initial accurate building polygon prediction by integrating multiresolution optimization and multilevel loss constraints. Our proposed method, to the authors’ best knowledge, is the first of its kind that introduces GCN to building extraction. The final step is to apply a regularization algorithm to translate the predicted polygons into structured and highly accurate building footprints. We validated the proposed method on two large aerial building data sets, WHU data set and Inria data set, where it was shown to significantly outperform other state-of-the-art methods more than 10%. Specifically, with the ground-truth rotated bounding boxes of buildings, our method is able to automatically delineate 91% of the buildings in the WHU data set and with the predicted rotated bounding the percent reached human-level delineation.

Locality-Aware Rotated Ship Detection in High-Resolution Remote Sensing Imagery Based on Multiscale Convolutional Network

Ship detection has been an active and vital topic in the field of remote sensing for a decade, but it is still a challenging problem due to the large-scale variations, the high aspect ratios, the intensive and rotated arrangement, and the background clutter disturbance. In this letter, we propose a locality-aware rotated ship detection (LARSD) framework based on a multiscale convolutional neural network (CNN) to tackle these issues. The proposed framework applies a UNet-like multiscale CNN to generate multiscale feature maps with high-level semantic information in high resolution. Then, an anchor-based rotated bounding box regression is applied for directly predicting the probability, the edge distances, and the angle of ships. Finally, a locality-aware score alignment (LASA) is proposed to fix the mismatch between classification results and location results caused by the independence of each subnet. Furthermore, to enlarge the data sets of ship detection, we build a new high-resolution ship detection (HRSD) data set, where 2499 images and 9269 instances were collected from Google Earth with different resolutions. Experiments based on public data set high-resolution ship collection 2016 (HRSC2016) and our HRSD data set demonstrate that our detection method achieves the state-of-the-art performance.

A Novel CNN-Based Detector for Ship Detection Based on Rotatable Bounding Box in SAR Images

Thanks to the excellent feature representation capabilities of neural networks, deep learning-based methods perform far better than traditional methods on target detection tasks such as ship detection. Although various network models have been proposed for SAR ship detection such as DRBox-v1, DRBox-v2, and MSR2N, there are still some problems such as mismatch of feature scale, contradictions between different learning tasks, and unbalanced distribution of positive samples, which have not been mentioned in these studies. In this article, an improved one-stage object detection framework based on RetinaNet and rotatable bounding box (RBox), which is referred as R-RetinaNet, is proposed to solve the above problems. The main improvements of R-RetinaNet as well as the contributions of this article are threefold. First, a scale calibration method is proposed to align the scale distribution of the output backbone feature map with the scale distribution of the targets. Second, a feature fusion network based on task-wise attention feature pyramid network is designed to decouple the feature optimization process of different tasks, which alleviates the conflict between different learning goals. Finally, an adaptive intersection over union (IoU) threshold training method is proposed for RBox-based model to correct the unbalanced distribution of positive samples caused by the fixed IoU threshold on RBox. Experimental results show that our method obtains 13.26%, 9.49%, 8.92%, and 4.55% gains in average precision under an IoU threshold of 0.5 on the public SAR ship detection dataset compared with four state-of-the-art RBox-based methods, respectively.

Bounding Box Projection for Regression Uncertainty in Oriented Object Detection

Oriented object detection has recently attracted increasing attention for its importance in aerial image processing. Popular detection methods for oriented and densely packed objects usually utilize the rotation angle to reduce the overlap of bounding boxes over the horizontal line. However, those angle-based methods remain challenging due to the angular periodicity and regression inconsistent, which are uniformly summarized as regression uncertainty. Previous methods usually heal the regression uncertainty problem by adding complex limits on the loss function. In this paper, instead of expressing accurate oriented information by the rotation angle, we propose a novel projection-based method that employs six-parameter to describe rotated bounding box: two points position on the projected line(one center point and one chosen vertex), a projection ratio and a quadrant label, named as ProjBB . ProjBB is angle-free and has three main advantages: i) More accurate angle distance without periodicity; ii) Projection ratio which is aspect ratio sensitive; iii) Theoretical uniqueness guarantee. Extensive experiments on two large-scale public aerial image datasets(i.e. DOTA and HRSC2016) and the scene text dataset ICDAR2015 show the competitive results of our proposed approach.

DIAMONDNET: SHIP DETECTION IN REMOTE SENSING IMAGES BY EXTRACTING AND CLUSTERING KEYPOINTS IN A DIAMOND

Abstract. Ship detection plays an important role in military and civil fields. Despite it has been studied for decays, ship detection in remote sensing images is still a challenging topic. In this work, we come up with a novel ship detection framework based on the keypoint extraction technique. We use a convolutional neural network to detect ship keypoints and then cluster the keypoints into groups, where each group is composed of keypoints belonging to the same ship. The choice of the keypoints is specifically considered to derive an effective ship representation. One keypoint is located at the center of the ship and the rest four keypoints are located at the head, the tail, the midpoint of the left side and the midpoint of the right side, respectively. Since these keypoints are distributed in a diamond, we name our network DiamondNet. In addition, a corresponding clustering algorithm based on the geometric characteristics of the ships is proposed to cluster keypoints into groups. We demonstrate that our method provides a more flexible and effective way to represent ships than the popular anchor-based methods, since either the rectangular bounding box or the rotated bounding box of each ship instance can be easily derived from the ship keypoints. Experiments on two datasets reveal that our DiamondNet reaches the state-of-the-art results.

A Novel False Alarm Suppression Method for CNN-Based SAR Ship Detector

Synthetic aperture radar (SAR) ship detection is an important part of remote sensing applications. With the development of computer vision, SAR ship detection methods based on convolutional neural network (CNN) can directly perform end-to-end detection of near-shore ship targets. However, CNN-based methods are prone to generate false targets on land areas, especially when using a rotatable bounding box (RBox) for detection. Therefore, how to reduce the false alarm rate becomes a key direction in research for SAR ship detection. In this letter, the problem of negative sample intraclass imbalance in the training stage of CNN-based detection methods is pointed out for the first time, which is considered to be an important reason for the excessive false alarm rate in the land area. Then, a method is proposed to reduce the false targets generated in the land area by CNN-based detection methods. First, an RBox-based model is proposed as the basic architecture for detection. Then, a new loss function is adopted to guide the model to balance the loss contribution of different negative samples during the training stage. The experimental results prove that the proposed method can effectively reduce the false alarm rate of the model and boost the performance of CNN-based detection methods.

A Novel CNN-Based Method for Accurate Ship Detection in HR Optical Remote Sensing Images via Rotated Bounding Box

Currently, reliable and accurate ship detection in optical remote sensing images is still challenging. Even the state-of-the-art convolutional neural network (CNN) based methods cannot obtain very satisfactory results. To more accurately locate the ships in diverse orientations, some recent methods conduct the detection via the rotated bounding box. However, it further increases the difficulty of detection, because an additional variable of ship orientation must be accurately predicted in the algorithm. In this paper, a novel CNN-based ship detection method is proposed, by overcoming some common deficiencies of current CNN-based methods in ship detection. Specifically, to generate rotated region proposals, current methods have to predefine multi-oriented anchors, and predict all unknown variables together in one regression process, limiting the quality of overall prediction. By contrast, we are able to predict the orientation and other variables independently, and yet more effectively, with a novel dual-branch regression network, based on the observation that the ship targets are nearly rotation-invariant in remote sensing images. Next, a shape-adaptive pooling method is proposed, to overcome the limitation of typical regular ROI-pooling in extracting the features of the ships with various aspect ratios. Furthermore, we propose to incorporate multilevel features via the spatially-variant adaptive pooling. This novel approach, called multilevel adaptive pooling, leads to a compact feature representation more qualified for the simultaneous ship classification and localization. Finally, detailed ablation study performed on the proposed approaches is provided, along with some useful insights. Experimental results demonstrate the great superiority of the proposed method in ship detection.

MSR2N: Multi-Stage Rotational Region Based Network for Arbitrary-Oriented Ship Detection in SAR Images.

In synthetic aperture radar (SAR) images, ships are often arbitrary-oriented and densely arranged in complex backgrounds, posing enormous challenges for ship detection. However, most existing methods detect ships with horizontal bounding boxes, which leads to the redundancy of detected regions. Furthermore, the high Intersection-over-Union (IoU) between two horizontal bounding boxes of densely arranged ships can cause missing detection. In this paper, a multi-stage rotational region based network (MSR2N) is proposed to solve the above problems. In MSR2N, the rotated bounding boxes, which can reduce background noise and prevent missing detection caused by high IoUs, are utilized to represent ship regions. MSR2N consists of three modules: feature pyramid network (FPN), rotational region proposal network (RRPN), and multi-stage rotational detection network (MSRDN). First of all, the FPN is applied to combine high-resolution features with semantically strong features. Second, in RRPN, a rotation-angle-dependent strategy is employed to generate multi-angle anchors which can represent arbitrary-oriented ship regions more felicitously than horizontal anchors. Finally, the MSRDN with three sub-networks is proposed to regress proposals of ship regions stage by stage. Meanwhile, the incrementally increasing IoU thresholds are selected for resampling positive and negative proposals in sequential stages of MSRDN, which eliminates close false positive proposals successively. With the above characteristics, MSR2N is more suitable and robust for ship detection in SAR images. The experimental results on SAR ship detection dataset (SSDD) show that the MSR2N has achieved state-of-the-art performance.

Deep Learning for Mango (Mangifera indica) Panicle Stage Classification

Automated assessment of the number of panicles by developmental stage can provide information on the time spread of flowering and thus inform farm management. A pixel-based segmentation method for the estimation of flowering level from tree images was confounded by the developmental stage. Therefore, the use of a single and a two-stage deep learning framework (YOLO and R2CNN) was considered, using either upright or rotated bounding boxes. For a validation image set and for a total panicle count, the models MangoYOLO(-upright), MangoYOLO-rotated, YOLOv3-rotated, R2CNN(-rotated) and R2CNN-upright achieved weighted F1 scores of 76.5, 76.1, 74.9, 74.0 and 82.0, respectively. For a test set of the images of another cultivar and using a different camera, the R2 for machine vision to human count of panicles per tree was 0.86, 0.80, 0.83, 0.81 and 0.76 for the same models, respectively. Thus, there was no consistent benefit from the use of rotated over the use of upright bounding boxes. The YOLOv3-rotated model was superior in terms of total panicle count, and the R2CNN-upright model was more accurate for panicle stage classification. To demonstrate practical application, panicle counts were made weekly for an orchard of 994 trees, with a peak detection routine applied to document multiple flowering events.