ISPRS Data Set Research Articles

Remote sensing image semantic segmentation via class-guided structural interaction and boundary perception

Existing remote sensing semantic segmentation methods generally ignore the structural information of objects that is vital in the human visual recognition system. The absence of overall structural information often results in weak perceptions of subtle textures and fragmented predictions, especially for complex and variable ground object scenarios. Besides, they still suffer from the semantic ambiguity caused by the unclear object boundary features in remote sensing images. In this paper, we propose a novel remote sensing semantic segmentation framework, called CSBNet, which aims to enhance the capacity of class-guided structural interaction and boundary perception simultaneously. It consists of a class-guided structure interaction module (CSIM), a Transformer-based context aggregation module (TCAM) and a class-guided boundary supervision module (CBSM). The CSIM has the ability to progressively extract the class-specific structural features, i.e., refining the structural information of each class by iteratively exchanging information between initial coarse class tokens and contexts. Meanwhile, the TCAM is constructed to provide CSIM with more discriminative multi-scale contexts without losing spatial features. In particular, the CBSM plays an auxiliary role, which applies the boundary information obtained from the class tokens to supervise the segmentation of boundary regions. When tested on the ISPRS dataset, LoveDA dataset, UAVid dataset, our method significantly outperforms the state-of-the-art remote sensing semantic segmentation approaches.

A serial semantic segmentation model based on encoder-decoder architecture

The thriving progress of Convolutional Neural Networks (CNNs) and the outstanding efficacy of Visual Transformers (ViTs) have delivered impressive outcomes in the domain of semantic segmentation. However, each model in isolation entails a trade-off between high computational complexity and compromised computational efficiency. To address this challenge, we effectively combine the CNN and encoder-decoder structures in a Transformer-inspired fashion, presenting the Serial Semantic Segmentation Trans via CNN Former (SSS-Former) model. To augment the feature extraction capability, we utilize the meticulously crafted SSS-CSPNet, resulting in a well-designed architecture for the holistic model. We propose a novel SSS-PN attention network that enhances the spatial topological connections of features, leading to improved overall performance. Additionally, the integration of SASPP bridges the semantic gap between multi-scale features and enhances segmentation ability for overlapping objects. To fulfill the requirement of real-time segmentation, we leverage a novel restructuring technique to devise a more lightweight and faster ResSSS-Former model. Abundant experimental results demonstrate that both SSS-Former and ResSSS-Former outperform existing state-of-the-art methods in terms of computational efficiency, result precision, and speed. Remarkably, SSS-Former achieves a mIoU of 58.63 % at 89.1FPS on the ADE20K dataset. On the validation and testing datasets of CityScapes, it obtains mIoU scores of 85.1 % and 85.2 % respectively, with a speed of 94.1FPS. Our optimized ResSSS-Former achieves impressive real-time segmentation results, with an astonishing 100+FPS while maintaining high segmentation accuracy. The compelling results from the ISPRS datasets further validate the effectiveness of our proposed models in segmenting multi-scale and overlapping objects.

AN ANALYSIS OF NEIGHBOURHOOD TYPES FOR POINTNET++ IN SEMANTIC SEGMENTATION OF AIRBORNE LASER SCANNING DATA

Abstract. The objective of the study is to conduct a comprehensive examination of how different neighbourhood types, namely spherical, cylindrical, and k-nearest neighbour (kNN), influence the feature extraction capabilities of the PointNet++ architecture in the semantic segmentation of Airborne Laser Scanning (ALS) point clouds. Two datasets are utilized for semantic segmentation analysis: the Dayton Annotated LiDAR Earth Scan (DALES) and the ISPRS 3D Semantic Labelling Benchmark datasets. In the experiments, the kNN method exhibited approximately 1% higher accuracy in weighted mean F1 and intersection over union (IoU) metrics compared to the spherical and cylindrical neighbourhood types on the DALES dataset. However, in the generalization experiment conducted on the ISPRS dataset, the spherical neighbourhood achieved the best results in these metrics, outperforming the cylindrical neighbourhood by a small margin. Notably, the kNN method was the least accurate, with a decrease in accuracy of approximately 1% in both weighted mean IoU and F1 scores. These findings suggest that the features extracted from spherical and cylindrical neighbourhood types are more generalizable compared to those from the kNN method.

Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory.

As a front-burner problem in incremental learning, class incremental semantic segmentation (CISS) is plagued by catastrophic forgetting and semantic drift. Although recent methods have utilized knowledge distillation to transfer knowledge from the old model, they are still unable to avoid pixel confusion, which results in severe misclassification after incremental steps due to the lack of annotations for past and future classes. Meanwhile data-replay-based approaches suffer from storage burdens and privacy concerns. In this paper, we propose to address CISS without exemplar memory and resolve catastrophic forgetting as well as semantic drift synchronously. We present Inherit with Distillation and Evolve with Contrast (IDEC), which consists of a Dense Knowledge Distillation on all Aspects (DADA) manner and an Asymmetric Region- wise Contrastive Learning (ARCL) module. Driven by the devised dynamic class-specific pseudo-labelling strategy, DADA distils intermediate-layer features and output-logits collaboratively with more emphasis on semantic-invariant knowledge inheritance. ARCL implements region- wise contrastive learning in the latent space to resolve semantic drift among known classes, current classes, and unknown classes. We demonstrate the effectiveness of our method on multiple CISS tasks by state-of-the-art performance, including Pascal VOC 2012, ADE20K and ISPRS datasets. Our method also shows superior anti-forgetting ability, particularly in multi-step CISS tasks.

SERNet: Squeeze and Excitation Residual Network for Semantic Segmentation of High-Resolution Remote Sensing Images

The semantic segmentation of high-resolution remote sensing images (HRRSIs) is a basic task for remote sensing image processing and has a wide range of applications. However, the abundant texture information and wide imaging range of HRRSIs lead to the complex distribution of ground objects and unclear boundaries, which bring huge challenges to the segmentation of HRRSIs. To solve this problem, in this paper we propose an improved squeeze and excitation residual network (SERNet), which integrates several squeeze and excitation residual modules (SERMs) and a refine attention module (RAM). The SERM can recalibrate feature responses adaptively by modeling the long-range dependencies in the channel and spatial dimensions, which enables effective information to be transmitted between the shallow and deep layers. The RAM pays attention to global features that are beneficial to segmentation results. Furthermore, the ISPRS datasets were processed to focus on the segmentation of vegetation categories and introduce Digital Surface Model (DSM) images to learn and integrate features to improve the segmentation accuracy of surface vegetation, which has certain prospects in the field of forestry applications. We conduct a set of comparative experiments on ISPRS Vaihingen and Potsdam datasets. The results verify the superior performance of the proposed SERNet.

AN ADAPTIVE SUPERPIXELS FOR VEGETATION DETECTION ON HIGH RESOLUTION IMAGES BASED ON MLP

Abstract. Vegetation detection aims to find the area which should be attributed with the labels of vegetation on the captured images, such as forest, grass land etc., and nowadays it is a key research topic in the field of remote sensing information processing and application. Over the last few years, the deep learning method based on convolutional neural network (CNN) has become the mainstream method for vegetation detection. However, due to the peculiarities of the underlying encoding and decoding structures, it is common for some CNN methods to loss some boundary details of vegetation when employing high-resolution images with rich details and clear boundaries. In order to improve the boundary localization capability of vegetation, this paper proposes a hybrid solution, i.e., an MLP (MultiLayer Perceptron)-based high-resolution image adaptive superpixels vegetation detection method. Compared with the traditional watershed transform algorithm, this method adopts the two-step boundary marching criterion to generate superpixels with more adherent boundary and compact regularity which contains adaptive neighborhood information by design. Based on the generated superpixels with boundary detail information, this paper applies MLP for binary predictions, i.e., vegetation or non-vegetation. The experimental results show that our method has more precise vegetation boundary localization and higher accuracy compared with several state-of-the-art methods on the UAV image data set and ISPRS data set.

RegGAN: An End-to-End Network for Building Footprint Generation with Boundary Regularization

Accurate and reliable building footprint maps are of great interest in many applications, e.g., urban monitoring, 3D building modeling, and geographical database updating. When compared to traditional methods, the deep-learning-based semantic segmentation networks have largely boosted the performance of building footprint generation. However, they still are not capable of delineating structured building footprints. Most existing studies dealing with this issue are based on two steps, which regularize building boundaries after the semantic segmentation networks are implemented, making the whole pipeline inefficient. To address this, we propose an end-to-end network for the building footprint generation with boundary regularization, which is termed RegGAN. Our method is based on a generative adversarial network (GAN). Specifically, a multiscale discriminator is proposed to distinguish the input between false and true, and a generator is utilized to learn from the discriminator’s response to generate more realistic building footprints. We propose to incorporate regularized loss in the objective function of RegGAN, in order to further enhance sharp building boundaries. The proposed method is evaluated on two datasets with varying spatial resolutions: the INRIA dataset (30 cm/pixel) and the ISPRS dataset (5 cm/pixel). Experimental results show that RegGAN is able to well preserve regular shapes and sharp building boundaries, which outperforms other competitors.

Improved Pseudomasks Generation for Weakly Supervised Building Extraction From High-Resolution Remote Sensing Imagery

Benefiting from free labeling pixel-level samples, weakly supervised semantic segmentation (WSSS) is making progress in automatically extracting building from high-resolution (HR) remote sensing (RS) imagery. For WSSS methods, generating high-quality pseudomasks is crucial for accurate building extraction.To improve the performance of generating pseudomasks by using image-level labels, this article proposes a weakly supervised building extraction method by combining adversarial climbing and gated convolution. The proposed method optimizes class activation maps (CAMs) by using adversarial climbing strategy, generates accurate class boundary maps by introducing a gated convolution module, and further refines building pseudomasks by fusing pairing semantic affinities and CAMs with a random walk strategy. Experimental results on three datasets—two ISPRS datasets and a self-annotated dataset—demonstrate that the proposed approach outperformed SOTA WSSS methods, leading to improvement of building extraction from HR RS imager. This article provides a new approach for optimizing pseudomasks generation, and a methodological reference for the applications of weakly supervised on RS images.

Building Footprint Generation Through Convolutional Neural Networks With Attraction Field Representation

Building footprint generation is a vital task in a wide range of applications, including, to name a few, land use management, urban planning and monitoring, and geographical database updating. Most existing approaches addressing this problem fall back on convolutional neural networks (CNNs) to learn semantic masks of buildings. However, one limitation of their results is blurred building boundaries. To address this, we propose to learn attraction field representation for building boundaries, which is capable of providing an enhanced representation power. Our method comprises two elemental modules: an Img2AFM module and an AFM2Mask module. More specifically, the former aims at learning an attraction field representation conditioned on an input image, which is capable of enhancing building boundaries and suppressing the background. The latter module predicts segmentation masks of buildings using the learned attraction field map. The proposed method is evaluated on three datasets with different spatial resolutions: the ISPRS dataset, the INRIA dataset, and the Planet dataset. From experimental results, we find that the proposed framework can well preserve geometric shapes and sharp boundaries of buildings, which brings significant improvements over other competitors. The trained model and code are available at https://github.com/lqycrystal/AFM_building.

Adaptive neighborhood size and effective geometric features selection for 3D scattered point cloud classification

Classification of 3D scatter and unorganized point cloud (PC) is an ongoing hard problem due to high redundancy, unbalanced sampling density, and large data structure of PC. Geometric and spectral features derived from the PC are generally used for classification. In this paper, an Omnivariance based adaptive neighborhood size selection method and a new feature set composed of 14 features are proposed for extraction of geometric features for each individual point within the local neighborhood. Performance of 8 modern classifiers with different strategies (i.e., boosting, ensemble, and deep learning etc.) were evaluated on the Oakland, Vaihingen, and ISPRS datasets. These 3 datasets are identified by 5, 9, and 2 distinct object classes, respectively. The results were compared with different neighborhood size selection methods (i.e., eigenentropy based, fixed number of the k-nearest neighbors) and feature set (i.e., 21 features). Only 3D local features were employed to classify datasets with varying characteristics and properties. The proposed optimum neighbor selection method and feature set provided the best statistical results with Auto-Encoder classifier (the overall accuracies are over 85%, 60% and, 90% the Oakland, Vaihingen, and ISPRS datasets, respectively). Especially for the ISPRS dataset, the Auto-Encoder obtained over 94%, 90%, and 93% precision, recall, and f-score, respectively.

Pairwise registration for terrestrial laser scanner point clouds based on the covariance matrix

ABSTRACT Automatic pairwise registration for unordered terrestrial laser scanner (TLS) point clouds is the pre-requisite for many applications, including 3D model reconstruction, cultural heritage management and landslide monitoring. However, most of the existing registration methods are still suffering from several limitations: (1) difficult to find the correct corresponding point pairs from the source and target point cloud, (2) low accuracy of pairwise registration and (3) the iterative process of registration is time-consuming. To overcome these challenges, based on the covariance matrix, this paper presents a robust and descriptive feature descriptor vector (FDV) to locally describe a point, by which the corresponding point pairs are obtained and the registration matrix is calculated. First, the FDVs of the original point cloud’s keypoints are calculated. Second, the corresponding point pairs are found via their FDVs. Third, the coarse transformation matrix is calculated by the corresponding point pairs using the singular value decomposition (SVD) algorithm, and which is further refined by the iterative closest point (ICP) algorithm to get a better result. Finally, the experiments are conducted on the Autonomous Systems Lab (ASL) and ISPRS datasets; our results show that the proposed algorithm can obtain good registration performance, and outperforms the compared methods.

Scale invariant line-based co-registration of multimodal aerial data using L1 minimization of spatial and angular deviations

In this work, we investigate the coregistration of multimodal data, such as photogrammetric/LiDAR point clouds, digital surface models, orthoimages, or 3D CAD city models, using corresponding line segments. The lines are analytically derived as intersections of adjacent planar surfaces, which can be determined more robustly and are deemed more accurate compared to single point based features. We propose a two-stage approach, which first focuses on finding optimal line correspondences between the datasets using a scale-invariant graph matching method, and then utilizes the found matching as a basis for calculating the optimal coregistration transform. By decoupling the correspondence search from the transform calculation, our approach can use more line pairs for determining the optimal transform than would be practicable with a combined, sampling-style approach. As opposed to competing methods, our transform computation is based on explicitly minimizing the average L1 distance on the matched line set. The assumed model accounts for an isotropic scaling factor, three translations and three rotation angles. We conducted experiments on two publicly available ISPRS datasets: Vaihingen and Dortmund, and compared the performance of several variations of our approach with three competing methods. The results indicate that the L1 methods decreased the median matched line distance by up to one third in case of pre-aligned Z axes. Moreover, when coregistering two photogrammetric datasets acquired from distinct viewing perspectives, our method was able to triple the number of matched lines (under a strict proximity-based criterion) compared to its competitor. Our results show that it is worthwhile to base the transform calculation on significantly more line pairs than is customary for sample consensus-based approaches. Our established validation dataset for line-based coregistration has been published and made available online (https://doi.org/10.17632/dmp7tkn8kc.2).

Height estimation from single aerial images using a deep convolutional encoder-decoder network

Extracting 3D information from aerial images is an important and still challenging topic in photogrammetry and remote sensing. Height estimation from only a single aerial image is an ambiguous and ill-posed problem. To address this challenging problem, in this paper, an architecture based on a deep convolutional neural network (CNN) is proposed in order to estimate the height values from a single aerial image. Methodologies for data preprocessing, selection of training data as well as data augmentation are presented. Subsequently, a deep CNN architecture is proposed consisting of encoding and decoding steps. In the encoding part, a deep residual learning is employed for extracting the local and global features. An up-sampling approach is proposed in the decoding part for increasing the output resolution and skip connections are employed in each scale to modify the estimated height values at the object boundaries. Finally, a post-processing approach is proposed to merge the predicted height image patches and generate a seamless continuous height map. The quantitative evaluation of the proposed approaches on the ISPRS datasets indicates relative and root mean square errors of approximately 0.9 m and 3.2 m, respectively.

Vehicle Instance Segmentation From Aerial Image and Video Using a Multitask Learning Residual Fully Convolutional Network

Object detection and semantic segmentation are two main themes in object retrieval from high-resolution remote sensing images, which have recently achieved remarkable performance by surfing the wave of deep learning and, more notably, convolutional neural networks (CNNs). In this paper, we are interested in a novel, more challenging problem of vehicle instance segmentation, which entails identifying, at a pixel-level, where the vehicles appear as well as associating each pixel with a physical instance of a vehicle. In contrast, vehicle detection and semantic segmentation each only concern one of the two. We propose to tackle this problem with a semantic boundary-aware multi-task learning network. More specifically, we utilize the philosophy of residual learning (ResNet) to construct a fully convolutional network that is capable of harnessing multi-level contextual feature representations learned from different residual blocks. We theoretically analyze and discuss why residual networks can produce better probability maps for pixel-wise segmentation tasks. Then, based on this network architecture, we propose a unified multi-task learning network that can simultaneously learn two complementary tasks, namely, segmenting vehicle regions and detecting semantic boundaries. The latter subproblem is helpful for differentiating closely spaced vehicles, which are usually not correctly separated into instances. Currently, datasets with pixel-wise annotation for vehicle extraction are ISPRS dataset and IEEE GRSS DFC2015 dataset over Zeebrugge, which specializes in semantic segmentation. Therefore, we built a new, more challenging dataset for vehicle instance segmentation, called the Busy Parking Lot UAV Video dataset, and we make our dataset available at http://www.sipeo.bgu.tum.de/download so that it can be used to benchmark future vehicle instance segmentation algorithms.

Joint Clusters and Iterative Graph Cuts for ALS Point Cloud Filtering

A novel method that combines joint clusters and iterative graph cuts for ALS point cloud filtering is proposed in this paper. The method first extracts clusters of points from the raw point cloud, and then classifies ground points at the cluster level. There are four main steps, i.e., two-step point cloud clustering, critical point extraction, initial terrain determination, and terrain densification based on iterative graph cuts. Smooth clusters, rough clusters, and scattered points are extracted by the two-step clustering to depict the raw point cloud, which reduces the complexity of raw data and the judgment difficulty in the subsequent procedures. Critical points of each cluster are extracted, and the initial terrain is determined among the smooth clusters. Using the initial terrain and critical points, iterative graph cuts is performed to segment ground and nonground points at the cluster level. Experiments on ISPRS dataset with a low point density and Utah dataset with a moderate point density show that our approach provides a satisfactory trade off between Type I and Type II errors. Moreover, our method significantly outperforms progressive TIN densification based filters, and successfully controls Type II errors.

VOLUME BASED DTM GENERATION FROM VERY HIGH RESOLUTION PHOTOGRAMMETRIC DSMS

In this paper we propose a new algorithm for digital terrain (DTM) model reconstruction from very high spatial resolution digital surface models (DSMs). It represents a combination of multi-directional filtering with a new metric which we call <i>normalized volume above ground</i> to create an above-ground mask containing buildings and elevated vegetation. This mask can be used to interpolate a ground-only DTM. The presented algorithm works fully automatically, requiring only the processing parameters <i>minimum height</i> and <i>maximum width</i> in metric units. Since slope and breaklines are not decisive criteria, low and smooth and even very extensive flat objects are recognized and masked. The algorithm was developed with the goal to generate the normalized DSM for automatic 3D building reconstruction and works reliably also in environments with distinct hillsides or terrace-shaped terrain where conventional methods would fail. A quantitative comparison with the ISPRS data sets <i>Potsdam</i> and <i>Vaihingen</i> show that 98-99% of all building data points are identified and can be removed, while enough ground data points (~66%) are kept to be able to reconstruct the ground surface. Additionally, we discuss the concept of <i>size dependent height thresholds</i> and present an efficient scheme for pyramidal processing of data sets reducing time complexity to linear to the number of pixels, <i>O(WH)</i>.

An Automatic Building Extraction and Regularisation Technique Using LiDAR Point Cloud Data and Orthoimage

The development of robust and accurate methods for automatic building detection and regularisation using multisource data continues to be a challenge due to point cloud sparsity, high spectral variability, urban objects differences, surrounding complexity, and data misalignment. To address these challenges, constraints on object’s size, height, area, and orientation are generally benefited which adversely affect the detection performance. Often the buildings either small in size, under shadows or partly occluded are ousted during elimination of superfluous objects. To overcome the limitations, a methodology is developed to extract and regularise the buildings using features from point cloud and orthoimagery. The building delineation process is carried out by identifying the candidate building regions and segmenting them into grids. Vegetation elimination, building detection and extraction of their partially occluded parts are achieved by synthesising the point cloud and image data. Finally, the detected buildings are regularised by exploiting the image lines in the building regularisation process. Detection and regularisation processes have been evaluated using the ISPRS benchmark and four Australian data sets which differ in point density (1 to 29 points/m2), building sizes, shadows, terrain, and vegetation. Results indicate that there is 83% to 93% per-area completeness with the correctness of above 95%, demonstrating the robustness of the approach. The absence of over- and many-to-many segmentation errors in the ISPRS data set indicate that the technique has higher per-object accuracy. While compared with six existing similar methods, the proposed detection and regularisation approach performs significantly better on more complex data sets (Australian) in contrast to the ISPRS benchmark, where it does better or equal to the counterparts.