Optical Remote Sensing Images Research Articles

A comprehensive review of optical remote-sensing image object detection datasets

A comprehensive review of optical remote-sensing image object detection datasets

Boundary-Aware Salient Object Detection in Optical Remote-Sensing Images

Different from the traditional natural scene images, optical remote-sensing images (RSIs) suffer from diverse imaging orientations, cluttered backgrounds, and various scene types. Therefore, the object-detection methods salient to optical RSIs require effective localization and segmentation to deal with complex scenarios, especially small targets, serious occlusion, and multiple targets. However, the existing models’ experimental results are incapable of distinguishing salient objects and backgrounds using clear boundaries. To tackle this problem, we introduce boundary information to perform salient object detection in optical RSIs. Specifically, we first combine the encoder’s low-level and high-level features (i.e., abundant local spatial and semantic information) via a feature-interaction operation, yielding boundary information. Then, the boundary cues are introduced into each decoder block, where the decoder features are directed to focus more on the boundary details and objects simultaneously. In this way, we can generate high-quality saliency maps which can highlight salient objects from optical RSIs completely and accurately. Extensive experiments are performed on a public dataset (i.e., ORSSD dataset), and the experimental results demonstrate the effectiveness of our model when compared with the cutting-edge saliency models.

Cloudformer: A Cloud-Removal Network Combining Self-Attention Mechanism and Convolution

Optical remote-sensing images have a wide range of applications, but they are often obscured by clouds, which affects subsequent analysis. Therefore, cloud removal becomes a necessary preprocessing step. In this paper, a novel and superior transformer-based network is proposed, named Cloudformer. The proposed method novelly combines the advantages of convolution and a self-attention mechanism: it uses convolution layers to extract simple features over a small range in the shallow layer, and exerts the advantage of a self-attention mechanism in extracting correlation in a large range in the deep layer. This method also introduces Locally-enhanced Positional Encoding (LePE) to flexibly generate suitable positional encodings for different inputs and to utilize local information to enhance encoding capabilities. Exhaustive experiments on public datasets demonstrate the superior ability of the method to remove both thin and thick clouds, and the effectiveness of the proposed modules is validated by ablation studies.

Review for Region Localization in Large-Scale Optical Remote Sensing Images

For the massive large-scale visible image data obtained by satellite, unmanned aerial vehicles, and other reconnaissance platforms, if only relying on manual visual interpretation, there will be problems such as heavy workload, low efficiency, high repeatability, strong subjectivity, and high cost, which cannot meet the demand of modern society for efficient information. Therefore, in order to improve work efficiency, it is necessary to study the rapid automatic region localization in large-scale remote sensing images. That will play an important role in change detection, temperature retrieval, and other files. The development and present situation of the region localization algorithms are analyzed. This paper summarizes the development, improvement, and deficiency of the traditional algorithm, as well as the difficulties and challenges. And the authors make a comparison to the deep learning-based methods. Finally, a possible development direction is prospected.

Mapping of Forest Biomass in Shangri-La City Based on LiDAR Technology and Other Remote Sensing Data

Forest ecosystems can be regarded as huge carbon sinks. In order to effectively assess carbon balance in such ecosystems, rapid and accurate estimation of the aboveground biomass of a forest is critically needed. However, the current methods for biomass estimation and mapping are of limited spatial resolution and mostly depend on large numbers of measurements. In order to obtain better biomass estimation outcomes with higher spatial resolution, a rapid method is introduced for region-scale biomass estimation in alpine and canyon areas using space-borne light detection and ranging (LiDAR) data and optical remote-sensing images. Specifically, we explored alpine and canyon areas in Shangri-La City in China using space-borne LiDAR data from ICESAT-2 and optical remote-sensing images from Landsat8 OLI, Sentinel-2, and Microwave remote sensing Sentinel-1. An extrapolation model of the forest canopy heights in these areas was constructed with a 30-m resolution of continuous canopy height outputs. For continuously estimating the diameter at breast height (DBH) in Shangri-La City, a tree height-DBH growth model was constructed based on the LiDAR and remote-sensing measurements. Finally, based on the average DBH of the explored forests, a model was constructed for estimating and mapping the aboveground biomass and carbon storage in Shangri-La with a spatial resolution of 30 m. The results show that the forest canopy height in Shangri-La City is mainly in the range of 2.82–30.96 m, and that the estimation accuracy is verified by the LiDAR-based canopy height model (CHM) with a coefficient of determination of R2 = 0.7143. The inversion results were still largely affected by geospatial location factors (longitude, latitude), terrain factors (slope, elevation), and vegetation indices (NBR, NDGI, NDVI). Based on the relationship between the tree height and the DBH, the DBH of trees in Shangri-La City was estimated to be mainly in the range of 20 cm to 30 cm, and this estimate was verified by actual measurements with R2 greater than 0.7 all. Finally, the established model estimated the aboveground forest biomass and carbon storage of the study area of Shangri-La City in 2020 to be 1.28 × 108 t and 6.41 × 107 t, respectively. These estimates correspond to total accuracies of 92.28%, respectively.

Improving Typical Urban Land-Use Classification with Active-Passive Remote Sensing and Multi-Attention Modules Hybrid Network: A Case Study of Qibin District, Henan, China

The study of high-precision land-use classification is essential for the sustainable development of land resources. This study addresses the problem of classification errors in optical remote-sensing images under high surface humidity, cloud cover, and hazy weather. The synthetic aperture radar (SAR) images are sensitive to soil moisture, and the microwave can penetrate clouds, haze, and smoke. By using both the active and passive remote-sensing data, the Sentinel-1A SAR and Sentinel-2B multispectral (MS) images are combined synergistically. The full-band data combining the SAR + MS + spectral indexes is thus constructed. Based on the high dimensionality and heterogeneity of this data set, a new framework (MAM-HybridNet) based on two-dimensional (2D) and three-dimensional (3D) hybrid convolutional neural networks combined with multi-attention modules (MAMs) is proposed for improving the accuracy of land-use classification in cities with high surface humidity. In addition, the same training samples supported by All bands data (SAR + MS + spectral index) are selected and compared with k-Nearest Neighbors (KNN), support vector machine (SVM), 2D convolutional neural networks, 3D convolutional neural networks, and hybridSN classification models to verify the accuracy of the proposed classification model. The results show that (1) fusion classification based on Sentinel-2B MSI and Sentinel-1A SAR data produce an overall accuracy (OA) of 95.10%, a kappa coefficient (KC) of 0.93, and an average accuracy (AA) of 92.86%, which is better than the classification results using Sentinel-2B MSI and Sentinel-1A SAR images separately. (2) The classification accuracy improves upon adding the spectral index, and the OA, KC, and AA improve by 3.77%, 0.05, and 5.5%, respectively. (3) With the support of full-band data, the algorithm proposed herein produces better results than other classification algorithms, with an OA of 98.87%, a KC of 0.98, and an AA of 98.36%. These results indicate that the synergistic effect of active-passive remote-sensing data improves land-use classification. Additionally, the results verify the effectiveness of the proposed deep-learning classification model for land-use classification.

Deep-Separation Guided Progressive Reconstruction Network for Semantic Segmentation of Remote Sensing Images

The success of deep learning and the segmentation of remote sensing images (RSIs) has improved semantic segmentation in recent years. However, existing RSI segmentation methods have two inherent problems: (1) detecting objects of various scales in RSIs of complex scenes is challenging, and (2) feature reconstruction for accurate segmentation is difficult. To solve these problems, we propose a deep-separation-guided progressive reconstruction network that achieves accurate RSI segmentation. First, we design a decoder comprising progressive reconstruction blocks capturing detailed features at various resolutions through multi-scale features obtained from various receptive fields to preserve accuracy during reconstruction. Subsequently, we propose a deep separation module that distinguishes various classes based on semantic features to use deep features to detect objects of different scales. Moreover, adjacent middle features are complemented during decoding to improve the segmentation performance. Extensive experimental results on two optical RSI datasets show that the proposed network outperforms 11 state-of-the-art methods.

MSSDet: Multi-Scale Ship-Detection Framework in Optical Remote-Sensing Images and New Benchmark

Ships comprise the only and most important ocean transportation mode. Thus, ship detection is one of the most critical technologies in ship monitoring, which plays an essential role in maintaining marine safety. Optical remote-sensing images contain rich color and texture information, which is beneficial to ship detection. However, few optical remote-sensing datasets are open publicly due to the issue of sensitive data and copyrights, and only the HRSC2016 dataset is built for the ship-detection task. Moreover, almost all general object detectors suffer from the failure of multi-scale ship detection because of the diversity of spatial resolution and ship size. In this paper, we re-annotate the HRSC2016 dataset and supplement 610 optical remote-sensing images to build a new open source ship-detection benchmark dataset with rich multi-scale ship objects named the HRSC2016-MS dataset. In addition, we further explore the potential of a recursive mechanism in the field of object detection and propose a novel multi-scale ship-detection framework (MSSDet) in optical remote-sensing images. The success of detecting multi-scale objects depends on the hierarchical pyramid structure in the object-detection framework. However, the inherent semantic and spatial gaps among hierarchical pyramid levels seriously affect detection performance. To alleviate this problem, we propose a joint recursive feature pyramid (JRFP), which can generate semantically strong and spatially refined multi-scale features. Extensive experiments were conducted on the HRSC2016-MS, HRSC2016, and DIOR datasets. Detailed ablation studies directly demonstrated the effectiveness of the proposed JRFP architecture and also showed that the proposed method has excellent generalizability. Comparisons with state-of-the-art methods showed that the proposed method achieves competitive performance, i.e., 77.3%, 95.8%, and 73.3% mean average precision accuracy on the HRSC2016-MS, HRSC2016, and DIOR datasets, respectively.

Super-Resolution of Remote Sensing Images for ×4 Resolution without Reference Images

Sentinel-2 satellites can provide free optical remote-sensing images with a spatial resolution of up to 10 M, but the spatial details provided are not enough for many applications, so it is worth considering improving the spatial resolution of Sentinel-2 satellites images through super-resolution (SR). Currently, the most effective SR models are mainly based on deep learning, especially the generative adversarial network (GAN). Models based on GAN need to be trained on LR–HR image pairs. In this paper, a two-step super-resolution generative adversarial network (TS-SRGAN) model is proposed. The first step is having the GAN train the degraded models. Without supervised HR images, only the 10 m resolution images provided by Sentinel-2 satellites are used to generate the degraded images, which are in the same domain as the real LR images, and then to construct the near-natural LR–HR image pairs. The second step is to design a super-resolution generative adversarial network with strengthened perceptual features, to enhance the perceptual effects of the generated images. Through experiments, the proposed method obtained an average NIQE as low as 2.54, and outperformed state-of-the-art models according to other two NR-IQA metrics, such as BRISQUE and PIQE. At the same time, the comparison of the intuitive visual effects of the generated images also proved the effectiveness of TS-SRGAN.

An MLC and U-Net Integrated Method for Land Use/Land Cover Change Detection Based on Time Series NDVI-Composed Image from PlanetScope Satellite

Land use/land cover change (LUCC) detection based on optical remote-sensing images is an important research direction in the field of remote sensing. The key to it is to select an appropriate data source and detection method. In recent years, the continuous expansion of construction land in urban areas has become the main reason for the increase in LUCC demand. However, due to the complexity and diversity of land-cover types, it is difficult to obtain high-precision classification results. In this article, a 12-month time series NDVI (Normalized Difference Vegetation Index) image of the study area was generated based on the high spatial and temporal resolution PlanetScope satellite images. According to the time series NDVI image, representative land-cover samples were selected, and the changed land samples were selected at the same time. This method could directly obtain the LUCC detection results of the study area through land-cover classification. First, Maximum Likelihood Classification (MLC), a classical machine-learning method, was used for supervised classification, and the samples needed for deep learning were selected according to the classification results. Then, the U-Net model, which can fully identify and explore the deep semantic information of the time series NDVI image, was used for land classification. Finally, this article made a comparative analysis of the two classification results. The results demonstrate that the overall classification accuracy based on time series NDVI is significantly higher than that of single-scene NDVI and mean NDVI. The LUCC detection method proposed in this article can effectively extract changed areas. The overall accuracy of the MLC and U-Net model is 79.38% and 85.26%, respectively. Therefore, the deep-learning method can effectively improve the accuracy of land-cover classification and change detection.

A New Ship Detection Algorithm in Optical Remote Sensing Images Based on Improved R3Det

The task of ship target detection based on remote sensing images has attracted more and more attention because of its important value in civil and military fields. To solve the problem of low accuracy in ship target detection in optical remote sensing ship images due to complex scenes and large-target-scale differences, an improved R3Det algorithm is proposed in this paper. On the basis of R3Det, a feature pyramid network (FPN) structure is replaced by a search architecture-based feature pyramid network (NAS FPN) so that the network can adaptively learn and select the feature combination update and enrich the multiscale feature information. After the feature extraction network, a shallow feature is added to the context information enhancement (COT) module to supplement the small target semantic information. An efficient channel attention (ECA) module is added to make the network gather in the target area. The improved algorithm is applied to the ship data in the remote sensing image data set FAIR1M. The effectiveness of the improved model in a complex environment and for small target detection is verified through comparison experiments with R3Det and other models.

Ship Target Detection in Optical Remote Sensing Images Based on Multiscale Feature Enhancement.

Due to the multiscale characteristics of ship targets in ORSIs (optical remote sensing images), ship target detection in ORSIs based on depth learning is still facing great challenges. Aiming at the low accuracy of multiscale ship target detection in ORSIs, this paper proposes a ship target detection algorithm based on multiscale feature enhancement based on YOLO v4. Firstly, an improved mixed convolution is introduced into the IRes (inverted residual block) to form an MIRes (mixed inverted residual block). The MIRes are used to replace the Res (residual block) in the deep CSP module of the backbone network to enhance the multiscale feature extraction capability of the backbone network. Secondly, for different scale feature maps' perception fields, feature information, and the scale of the detected objects, the multiscale feature enhancement modules—SFEM (small scale feature enhancement module) and MFEM (middle scale feature enhancement module)—are proposed to enhance the feature information of the middle- and low-level feature maps, respectively, and then the enhanced feature maps are sent to the detection head for detection. Finally, experiments were implemented on the LEVIR-ship dataset and the NWPU VHR-10 dataset. The accuracy of the proposed algorithm in ship target detection reached 79.55% and 90.70%, respectively, which is improved by 3.25% and 3.56% compared with YOLO v4.

A Universal Landslide Detection Method in Optical Remote Sensing Images Based on Improved YOLOX

Using deep learning-based object detection algorithms for landslide hazards detection is very popular and effective. However, most existing algorithms are designed for landslides in a specific geographical range. This paper constructs a set of landslide detection models YOLOX-Pro, based on the improved YOLOX (You Only Look Once) target detection model to address the poor detection of complex mixed landslides. Wherein the VariFocal is used to replace the binary cross entropy in the original classification loss function to solve the uneven distribution of landslide samples and improve the detection recall; the coordinate attention (CA) mechanism is added to enhance the detection accuracy. Firstly, 1200 historical landslide optical remote sensing images in thirty-eight areas of China were extracted from Google Earth to create a mixed sample set for landslide detection. Next, the three attention mechanisms were compared to form the YOLOX-Pro model. Then, we tested the performance of YOLOX-Pro by comparing it with four models: YOLOX, YOLOv5, Faster R-CNN, and Single Shot MultiBox Detector (SSD). The results show that the YOLOX-Pro(m) has significantly improved the detection accuracy of complex and small landslides than the other models, with an average precision (AP0.75) of 51.5%, APsmall of 36.50%, and ARsmall of 49.50%. In addition, optical remote sensing images of a 12.32 km2 group-occurring landslides area located in Mibei village, Longchuan County, Guangdong, China, and 750 Unmanned Aerial Vehicle (UAV) images collected from the Internet were also used for landslide detection. The research results proved that the proposed method has strong generalization and good detection performance for many types of landslides, which provide a technical reference for the broad application of landslide detection using UAV.

Systematic Comparison of Objects Classification Methods Based on ALS and Optical Remote Sensing Images in Urban Areas

Geographical object classification and information extraction is an important topic for the construction of 3D virtual reality and digital twin cities in urban areas. However, the majority of current multi-target classification of urban scenes uses only a single source data (optical remote sensing images or airborne laser scanning (ALS) point clouds), which is limited by the restricted information of the data source itself. In order to make full use of the information carried by multiple data sources, we often need to set more parameters as well as algorithmic steps. To address the above issues, we compared and analyzed the object classification methods based on data fusion of airborne LiDAR point clouds and optical remote sensing images, systematically. Firstly, the features were extracted and determined from airborne LiDAR point clouds and high-resolution optical images. Then, some key feature sets were selected and were composed of median absolute deviation of elevation, normalized elevation values, texture features, normal vectors, etc. The feature sets were fed into various classifiers, such as random forest (RF), decision tree (DT), and support vector machines (SVM). Thirdly, the suitable feature sets with appropriate dimensionality were composed, and the point clouds were classified into four categories, such as trees (Tr), houses and buildings (Ho), low-growing vegetation (Gr), and impervious surfaces (Is). Finally, the single data source and multiple data sources, the crucial feature sets and their roles, and the resultant accuracy of different classifier models were compared and analyzed. Under the conditions of different experimental regions, sampling proportion parameters and machine learning models, the results showed that: (1) the overall classification accuracy obtained by the feature-level data fusion method was 76.2% compared with the results of only a single data source, which could improve the overall classification accuracy by more than 2%; (2) the accuracy of the four classes in the urban scenes can reach 88.5% (Is), 76.7% (Gr), 87.2% (Tr), and 88.3% (Ho), respectively, while the overall classification accuracy can reach 87.6% with optimal sampling parameters and random forest classifiers; (3) the RF classifier outperforms DT and SVM for the same sample conditions. In this paper, the method based on ALS point clouds and image data fusion can accurately classify multiple targets in urban scenes, which can provide technical support for 3D scene reconstruction and digital twin cities in complex geospatial environments.

Improved YOLOv5 Based on Hybrid Domain Attention for Small Object Detection in Optical Remote Sensing Images

The object detection technology of optical remote sensing images has been widely applied in military investigation, traffic planning, and environmental monitoring, among others. In this paper, a method is proposed for solving the problem of small object detection in optical remote sensing images. In the proposed method, the hybrid domain attention units (HDAUs) of channel and spatial attention mechanisms are combined and employed to improve the feature extraction capability and suppress background noise. In addition, we designed a multiscale dynamic weighted feature fusion network (MDW-Net) to improve adaptive optimization and deep fusion of shallow and deep feature layers. The model is trained and tested on the DIOR dataset, and some ablation and comparative experiments are carried out. The experimental results show that the mAP of the proposed model surpasses that of YOLOv5 by a large margin of +2.3 and has obvious advantages regarding the detection performance for small object categories, such as airplane, ship, and vehicle, which support its application for small target detection in optical remote sensing images.

Precursors to large rockslides visible on optical remote-sensing images and their implications for landslide early detection

Precursors to large rockslides visible on optical remote-sensing images and their implications for landslide early detection

Sea and Land Segmentation of Optical Remote Sensing Images Based on U-Net Optimization

At present, some related studies on semantic segmentation are becoming complicated, adding a lot of feature layers and various jump splicing to improve the level of refined segmentation, which often requires a large number of parameters to ensure a better segmentation effect. When faced with lightweight tasks, such as sea and land segmentation, the modeling capabilities of these models far exceed the complexity of the task, and reducing the size of the model can easily weaken the original effect of the model. In response to this problem, this paper proposes a U-net optimization structure combining Atrous Spatial Pyramid Pooling (ASPP) and FReLU, namely ACU-Net. ACU-Net replaces the two-layer continuous convolution in the feature extraction part of U-Net with a lightweight ASPP module, retains the symmetric U-shaped structure of the original U-Net structure, and splices the output of the ASPP module with the upsampling part. Use FReLU to improve the modeling ability between pixels, and at the same time cooperate with the attention mechanism to improve the perception ability and receptive field of the network, reduce the training difficulty of the model, and fully tap the hidden information of the samples to capture more effective features. The experimental results show that the ACU-Net in this paper surpasses the reduced U-Net and its optimized improved network U-Net++ in terms of segmentation accuracy and IoU with a smaller volume.

A Novel Method of Ship Detection under Cloud Interference for Optical Remote Sensing Images

In this paper, we propose a novel method developed for detecting incomplete ship targets under cloud interference and low-contrast ship targets in thin fog based on superpixel segmentation, and outline its application to optical remote sensing images. The detection of ship targets often requires the target to be complete, and the overall features of the ship are used for detection and recognition. When the ship target is obscured by clouds, or the contrast between the ship target and the sea-clutter background is low, there may be incomplete targets, which reduce the effectiveness of recognition. Here, we propose a new method combining constant false alarm rate (CFAR) and superpixel segmentation with feature points (SFCFAR) to solve the above problems. Our newly developed SFCFAR utilizes superpixel segmentation to divide large scenes into many small regions which include target regions and background regions. In remote sensing images, the target occupies a small proportion of pixels in the entire image. In our method, we use superpixel segmentation to divide remote sensing images into meaningful blocks. The target regions are identified using the characteristics of clusters of ship texture features and the texture differences between the target and background regions. This step not only detects the ship target quickly, but also detects ships with low contrast and under cloud cover. In optical remote sensing, ships at sea under thin clouds are not common in practice, and the sample size generated is relatively small, so this problem is not applicable to deep learning algorithms for training, while the SFCFAR algorithm does not require data training to complete the detection task. Experiments show that the proposed SFCFAR algorithm enhances the detection of obscured ship targets under clouds and low-contrast targets in thin fog, compared with traditional target detection methods and as deep learning algorithms, further complementing existing ship detection methods.

Outlier Reconstruction of NDVI for Vegetation-Cover Dynamic Analyses

The normalized difference vegetation index (NDVI) contains important data for providing vegetation-cover information and supporting environmental analyses. However, understanding long-term vegetation cover dynamics remains challenging due to data outliers that are found in cloudy regions. In this article, we propose a sliding-window-based tensor stream analysis algorithm (SWTSA) for reconstructing outliers in NDVI from multitemporal optical remote-sensing images. First, we constructed a tensor stream of NDVI that was calculated from clear-sky optical remote-sensing images corresponding to seasons on the basis of the acquired date. Second, we conducted tensor decomposition and reconstruction by SWTSA. Landsat series remote-sensing images were used in experiments to demonstrate the applicability of the SWTSA. Experiments were carried out successfully on the basis of data from the estuary area of Salween River in Southeast Asia. Compared with random forest regression (RFR), SWTSA has higher accuracy and better reconstruction capabilities. Results show that SWTSA is reliable and suitable for reconstructing outliers of NDVI from multitemporal optical remote-sensing images.

Cloudformer: Supplementary Aggregation Feature and Mask-Classification Network for Cloud Detection

Cloud detection is an important step in the processing of optical satellite remote-sensing data. In recent years, deep learning methods have achieved excellent results in cloud detection tasks. However, most of the current models have difficulties to accurately classify similar objects (e.g., clouds and snow) and to accurately detect clouds that occupy a few pixels in an image. To solve these problems, a cloud-detection framework (Cloudformer) combining CNN and Transformer is being proposed to achieve high-precision cloud detection in optical remote-sensing images. The framework achieves accurate detection of thin and small clouds using a pyramidal structure encoder. It also achieves accurate classification of similar objects using a dual-path decoder structure of CNN and Transformer, reducing the rate of missed detections and false alarms. In addition, since the Transformer model lacks the perception of location information, an asynchronous position-encoding method is being proposed to enhance the position information of the data entering the Transformer module and to optimize the detection results. Cloudformer is experimented on two datasets, AIR-CD and 38-Cloud, and the results show that it has state-of-the-art performance.