International Society For Photogrammetry And Remote Sensing Research Articles

An improved semantic segmentation algorithm for high-resolution remote sensing images based on DeepLabv3+

High-precision and high-efficiency Semantic segmentation of high-resolution remote sensing images is a challenge. Existing models typically require a significant amount of training data to achieve good classification results and have numerous training parameters. A novel model called MST-DeepLabv3+ was suggested in this paper for remote sensing image classification. It’s based on the DeepLabv3+ and can produce better results with fewer train parameters. MST-DeepLabv3+ made three improvements: (1) Reducing the number of model parameters by substituting MobileNetV2 for the Xception in the DeepLabv3+’s backbone network. (2) Adding the attention mechanism module SENet to increase the precision of semantic segmentation. (3) Increasing Transfer Learning to enhance the model's capacity to recognize features, and raise the segmentation accuracy. MST-DeepLabv3+ was tested on international society for photogrammetry and remote sensing (ISPRS) dataset, Gaofen image dataset (GID), and practically applied to the Taikang cultivated land dataset. On the ISPRS dataset, the mean intersection over union (MIoU), overall accuracy (OA), Precision, Recall, and F1-score are 82.47%, 92.13%, 90.34%, 90.12%, and 90.23%, respectively. On the GID dataset, these values are 73.44%, 85.58%, 84.10%, 84.86%, and 84.48%, respectively. The results were as high as 90.77%, 95.47%, 95.28%, 95.02%, and 95.15% on the Taikang cultivated land dataset. The experimental results indicate that MST-DeepLabv3+ effectively improves the accuracy of semantic segmentation of remote sensing images, recognizes the edge information with more completeness, and significantly reduces the parameter size.

Preface: ISPRS Geospatial Week 2023

Abstract. The International Society for Photogrammetry and Remote Sensing (ISPRS) Geospatial Week 2023 (GSW’2023) is a combination of 29 workshops organized by the ISPRS Working Groups active in areas of interest of ISPRS. The Geospatial Week 2023 is held from 2–7 September 2023, at the Semiramis Cairo Hotel, on the magnificent river Nile, Egypt, and is convened by the Arab Academy for Science, Technology & Maritime Transport (AASTMT) acting as the local organizer. The GSW’2023 is the first in Africa and the Middle East. The conference was chaired by Prof. Naser EL-Sheimy and Prof. Ismail Abdelgafhar (President of AASTMT) as the Honorary Chair and host of the conference under the Auspices of H.E. Professor Mostafa Madbouly, Prime Minister of the Arab Republic of Egypt. The GSW’2023 is the fifth edition, after, Antalya Turkey in 2013, La Grande Motte France in 2015, Wuhan China in 2017, and Enschede The Netherlands in 2019. The following 29 workshops provide state of the art and future trends in geospatial, sensors, photogrammetry, remote sensing, and spatial information sciences technologies in their applications in many industries and economic sectors: Cultural Heritage Visualization and Virtual Restoration SpACE – Spectral Remote Sensing in the era of AI, Cloud and Edge Computing Youth Presentation Forum Openness in Geospatial and Remote Sensing Precision GNSS: Technology Advances and Applications for Navigation and Mapping Photogrammetric 3D Reconstruction for Geo-Applications (PhotoGA 2023) Geospatial Data Analytics for Physical Geography Impact Assessment on Environment, Health andSociety Intelligent Systems in Sensor Web and Internet of Things Underwater Mapping Workshop: Geospatial techniques for underwater documentation, mapping andmonitoring SO&C: Sensor orientation and calibration for mapping and navigation purposes Smart Forests – Forest ecosystem assessment and monitoring using Remote Sensing, ArtificialIntelligence, and Robotics Satellite Remote Sensing and Its Applications Advanced Data Preparation and Data Management for Geospatial and Remote Sensing Scenarios Laser Scanning 2023 NGC of AV: Navigation, Guidance and Control of Autonomous Vehicles ISSDQ 2023- Artificial Intelligence and Uncertainty Modeling in Spatial Analysis Semantics3D – Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences GeoHB 2023: Geo-Spatial Computing for Understanding Human Behaviours GI4SDGs: The Geospatial Information and SDG Nexus SARcon 2023 – SAR constellations and applications Digital Construction CrowdMapping: Crowdsourcing for Global Mapping Indoor 3D IAMS – Intelligent and autonomous mapping systems AI-PC: AI-based Point Cloud and Image Understanding UAV-based mapping with imaging and LiDAR systems: challenges, data processing, and applications 3DS Smart Cities – 3D Sensing for Smart Cities Robotics for Mapping – SLAM approaches for mobile mapping and robot intelligence MMT and HD Maps – Mobile Mapping Technologies and HD Maps Many of the workshops are part of well-established series of workshops convened in the past. They cover topics like UAV for mapping, laser scanning, Semantics3D, SARcon, ISSDQ hyperspectral imaging, and crowd sourcing and collaborative mapping with applications ranging from indoor mapping, smart forest and smart cities to global mapping studies and planetary mapping. The conference also includes for the first-time few workshops organized by other working groups from sister organizations and the ISPRS youth forum. In total 172 full papers and 423 extended abstracts were submitted by authors from 68 countries. 1400 reviews have been delivered by 283 reviewers. A total of 137 full papers have been accepted for the volume X-1/W1-2023 of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Annals). Another 260 papers are published in volume XLVIII-1/W2-2023 of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Archives). The editors would like to thank all contributing authors, reviewers and all workshop organizers for their role in preparing and organizing the Geospatial Week 2023. Thanks to their contributions, we can offer an excessive and varying collection in the Annals and the Archives. The editors would like to extend their great appreciation to the President of AASTMT Prof. Ismail Abdel Ghafar for his great support in hosting the ISPRS GSW’2023. We also would like to extend our thanks to the members of the local organizing committee for great effort in making GSW’2023 conference a success, namely, Prof. Amr Ali Hanafi, Prof. Ahmed Shaker, Research Eng. Ayman Elzagh, Dr. Mengchi Ai, Dr. Adel Moussa, Dr. Yiran Luo, Mr. Yasser Elrashidi, Eng. Nagy K. Aly, Prof. Amira Zaki, Dr. Radwa Osman, Eng. Mohamed ElQurashi and Mr. Mohamed Abdallah Mahmoud. We hope you enjoy reading the proceedings. Editor: Prof. Naser El-Sheimy, Geospatial Week Director 2023, General Chair (University of Calgary, Canada)Co-Editors, The Arab Academy for Science, Technology & Maritime Transport:Prof. Alaa Abdelwahed Abdelbary, Chair Local Organizing CommitteeProf. Nashwa El-BendaryProf. Yahya Mohasseb

Preface: ISPRS Geospatial Week 2023

Abstract. The International Society for Photogrammetry and Remote Sensing (ISPRS) Geospatial Week 2023 (GSW’2023) is a combination of 29 workshops organized by the ISPRS Working Groups active in areas of interest of ISPRS. The Geospatial Week 2023 is held from 2–7 September 2023, at the Semiramis Cairo Hotel, on the magnificent river Nile, Egypt, and is convened by the Arab Academy for Science, Technology & Maritime Transport (AASTMT) acting as the local organizer. The GSW’2023 is the first in Africa and the Middle East. The conference was chaired by Prof. Naser EL-Sheimy and Prof. Ismail Abdelgafhar (President of AASTMT) as the Honorary Chair and host of the conference under the Auspices of H.E. Professor Mostafa Madbouly, Prime Minister of the Arab Republic of Egypt. The GSW’2023 is the fifth edition, after, Antalya Turkey in 2013, La Grande Motte France in 2015, Wuhan China in 2017, and Enschede The Netherlands in 2019. The following 29 workshops provide state of the art and future trends in geospatial, sensors, photogrammetry, remote sensing, and spatial information sciences technologies in their applications in many industries and economic sectors: Cultural Heritage Visualization and Virtual Restoration SpACE – Spectral Remote Sensing in the era of AI, Cloud and Edge Computing Youth Presentation Forum Openness in Geospatial and Remote Sensing Precision GNSS: Technology Advances and Applications for Navigation and Mapping Photogrammetric 3D Reconstruction for Geo-Applications (PhotoGA 2023) Geospatial Data Analytics for Physical Geography Impact Assessment on Environment, Health andSociety Intelligent Systems in Sensor Web and Internet of Things Underwater Mapping Workshop: Geospatial techniques for underwater documentation, mapping andmonitoring SO&C: Sensor orientation and calibration for mapping and navigation purposes Smart Forests – Forest ecosystem assessment and monitoring using Remote Sensing, ArtificialIntelligence, and Robotics Satellite Remote Sensing and Its Applications Advanced Data Preparation and Data Management for Geospatial and Remote Sensing Scenarios Laser Scanning 2023 NGC of AV: Navigation, Guidance and Control of Autonomous Vehicles ISSDQ 2023- Artificial Intelligence and Uncertainty Modeling in Spatial Analysis Semantics3D – Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences GeoHB 2023: Geo-Spatial Computing for Understanding Human Behaviours GI4SDGs: The Geospatial Information and SDG Nexus SARcon 2023 – SAR constellations and applications Digital Construction CrowdMapping: Crowdsourcing for Global Mapping Indoor 3D IAMS – Intelligent and autonomous mapping systems AI-PC: AI-based Point Cloud and Image Understanding UAV-based mapping with imaging and LiDAR systems: challenges, data processing, and applications 3DS Smart Cities – 3D Sensing for Smart Cities Robotics for Mapping – SLAM approaches for mobile mapping and robot intelligence MMT and HD Maps – Mobile Mapping Technologies and HD Maps Many of the workshops are part of well-established series of workshops convened in the past. They cover topics like UAV for mapping, laser scanning, Semantics3D, SARcon, ISSDQ hyperspectral imaging, and crowd sourcing and collaborative mapping with applications ranging from indoor mapping, smart forest and smart cities to global mapping studies and planetary mapping. The conference also includes for the first-time few workshops organized by other working groups from sister organizations and the ISPRS youth forum. In total 172 full papers and 423 extended abstracts were submitted by authors from 68 countries. 1400 reviews have been delivered by 283 reviewers. A total of 137 full papers have been accepted for the volume X-1/W1-2023 of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Annals). Another 260 papers are published in volume XLVIII-1/W2-2023 of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Archives). The editors would like to thank all contributing authors, reviewers and all workshop organizers for their role in preparing and organizing the Geospatial Week 2023. Thanks to their contributions, we can offer an excessive and varying collection in the Annals and the Archives. The editors would like to extend their great appreciation to the President of AASTMT Prof. Ismail Abdel Ghafar for his great support in hosting the ISPRS GSW’2023. We also would like to extend our thanks to the members of the local organizing committee for great effort in making GSW’2023 conference a success, namely, Prof. Amr Ali Hanafi, Prof. Ahmed Shaker, Research Eng. Ayman Elzagh, Dr. Mengchi Ai, Dr. Adel Moussa, Dr. Yiran Luo, Mr. Yasser Elrashidi, Eng. Nagy K. Aly, Prof. Amira Zaki, Dr. Radwa Osman, Eng. Mohamed ElQurashi and Mr. Mohamed Abdallah Mahmoud. We hope you enjoy reading the proceedings. Editor: Prof. Naser El-Sheimy, Geospatial Week Director 2023, General Chair (University of Calgary, Canada)Co-Editors, The Arab Academy for Science, Technology & Maritime Transport:Prof. Alaa Abdelwahed Abdelbary, Chair Local Organizing CommitteeProf. Nashwa El-BendaryProf. Yahya Mohasseb

DMAU-Net: An Attention-Based Multiscale Max-Pooling Dense Network for the Semantic Segmentation in VHR Remote-Sensing Images

High-resolution remote-sensing images cover more feature information, including texture, structure, shape, and other geometric details, while the relationships among target features are more complex. These factors make it more complicated for classical convolutional neural networks to obtain ideal results when performing a feature classification on remote-sensing images. To address this issue, we proposed an attention-based multiscale max-pooling dense network (DMAU-Net), which is based on U-Net for ground object classification. The network is designed with an integrated max-pooling module that incorporates dense connections in the encoder part to enhance the quality of the feature map, and thus improve the feature-extraction capability of the network. Equally, in the decoding, we introduce the Efficient Channel Attention (ECA) module, which can strengthen the effective features and suppress the irrelevant information. To validate the ground object classification performance of the multi-pooling integration network proposed in this paper, we conducted experiments on the Vaihingen and Potsdam datasets provided by the International Society for Photogrammetry and Remote Sensing (ISPRS). We compared DMAU-Net with other mainstream semantic segmentation models. The experimental results show that the DMAU-Net proposed in this paper effectively improves the accuracy of the feature classification of high-resolution remote-sensing images. The feature boundaries obtained by DMAU-Net are clear and regionally complete, enhancing the ability to optimize the edges of features.

IMPROVED TORNADO METHOD FOR GROUND POINT FILTERING FROM LIDAR POINT CLOUDS

Abstract. In this paper, an improved Tornado method for filtering LiDAR (Light Detection and Ranging) point clouds is presented. The original method uses a vertical cone with a downward vertex and an upward base to remove the points within it as non-ground points. The remaining points are ground points. The cone moves on the ground surface over the entire region of the point cloud. In this work, the regions of the objects are predicted by extracting the vertical features that have points in the vertical plane or vertical column. Therefore, the tornado method is only used in regions that contain objects. In addition, our improved method uses a specific height for a tornado to reduce the Type I error in mountainous areas. Also, a cylinder surrounding the cone is used to reduce the distance calculations between the cone and the point cloud. The results show that this method is very effective and fast compared to the original method. It also has promising results for the Type I error. In addition, this method was tested on the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets and produced outstanding results. The results show that this method achieves high filtering accuracy. Moreover, the proposed method achieves an overall average error of 6.83%, which is lower than most other methods.

An improved progressive triangular irregular network densification filtering algorithm for airborne LiDAR data

Airborne lidar is a technology for mapping surface spatial information and has been widely used in many areas of geospatial information disciplines. The filtering process of removing non-ground points has always been the focus of research. PTD (Progressive Triangular Irregular Network Densification) filtering algorithm is a widely used filtering algorithm for airborne lidar data. However, this algorithm has shortcomings in retaining ground points in steep areas, leading to large type Ⅰ errors. Therefore, this paper proposes an improved PTD algorithm. The improvement is the addition of the seed points filtering. Specifically, after the potential seed points are obtained by the progressive morphological filter, the seed points filtering is performed on it to remove the non-ground points, so that the obtained seed points are more accurate. The benchmark dataset of ISPRS (International Society for Photogrammetry and Remote Sensing) Working Group III is used to assess the proposed method. Results show that the method is effective in decreasing type Ⅰ error in steep areas. Comparing with the classic PTD algorithm, the type Ⅰ error and total error are decreased by 8.46% and 5.06% respectively. In addition, the proposed method shows a great advantage in computational efficiency, that is eight times more efficient than the classic PTD algorithm.

BIBED-Seg: Block-in-Block Edge Detection Network for Guiding Semantic Segmentation Task of High-Resolution Remote Sensing Images

Edge optimization of semantic segmentation results is a challenging issue in remote sensing image processing. This article proposes a semantic segmentation model guided by a block-in-block edge detection network named BIBED-Seg. This is a two-stage semantic segmentation model, where edges are extracted first and then segmented. We do two key works: The first work is edge detection, and we present BIBED, a block-in-block edge detection network, to extract the accurate boundary features. Here, the edge detection of multiscale feature fusion is first realized by creating the block-in-block residual network structure and devising the multilevel loss function. Second, we add the channel and spatial attention module into the residual structure to improve high-resolution remote sensing images' boundary positioning and detection accuracy by focusing on their channel and spatial dimensions. Finally, we evaluate our method on International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen data sets and obtain ODS F-measure of 0.6671 and 0.7432, higher than other excellent edge detection methods. The second work is two-stage segmentation. First, the proposed BIBED is individually pretrained, and subsequently, the pretrained model is introduced into the entire segmentation network to extract boundary features. In the second segmentation stage, the edge detection network is used to constrain semantic segmentation results by loss cycles and feature bootstrapping. Our best model obtains the OA of 90.2%, 87.7%, and 81.5%, the IOU of 76.0%, 69.6%, and 61.3% on the ISPRS and WHDLD datasets, respectively.

Roof Segmentation From Airborne LiDAR Using Octree-Based Hybrid Region Growing and Boundary Neighborhood Verification Voting

Building roof segmentation is a key step in the process of three-dimensional (3D) building reconstruction using airborne Light Detection and Ranging (LiDAR) point cloud data. Voxel-based region growing is one of the most widely used methods to segment planes because of its high efficiency and easy implementation, but it is easy to omit roof planes due to the unreasonable voxel size and the complex roof structures. In addition, boundaries between adjacent roof planes are inaccurate. To solve the issues, a roof segmentation method using octree-based hybrid region growing and boundary neighborhood verification voting is proposed. First, an octree-based voxelization is performed on the raw building points to generate two basic units: planar voxels and individual points (i.e., points that are not in the planar voxels). Then, the hybrid region growing is conducted on these two basic units to segment coarse roof planes. A parameter-free boundary neighborhood verification voting strategy is used to assign the boundary points to the correct roof planes by verifying the neighborhoods of the boundary points and using reliable neighborhood information. Experimental results of four datasets, including two datasets provided by the International Society for Photogrammetry and Remote Sensing (ISPRS), and two high-density datasets provided by OpenTopography, verify that roof planes can be successfully segmented by the proposed method with over 96.8% completeness and a minimum 93.2% correctness. In addition, boundary points are assigned to the correct roof planes by the neighborhood verification voting strategy. Thus, the segmented roof planes can be used in various applications.

Dual-Range Context Aggregation for Efficient Semantic Segmentation in Remote Sensing Images

Although introducing self-attention mechanisms is beneficial to establish long-range dependencies and explore global context information in the task of remote sensing image semantic segmentation, it results in expensive computation and large memory cost. In this letter, we address this dilemma by proposing a lightweight dual-range context aggregation network (LDCANet) for efficient remote sensing image semantic segmentation. First, a dual-range context aggregation module (DCAM) is designed to aggregate the local features and the global semantic context acquired by convolutions and self-attention, respectively, where self-attention is implemented easily by applying two cascaded linear layers to reduce the computational complexity. Furthermore, a simple and lightweight decoder is employed to combine information from different levels, in which a multilayer perceptron (MLP)-based efficient linear block (ELB) is proposed to yield a strong and efficient representation. Experiments conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen dataset and the Gaofen Image dataset (GID) prove that our LDCANet achieves an excellent trade-off between segmentation accuracy and computational efficiency. In particular, our method achieves 74.12% mean intersection over union (mIoU) on the ISPRS Vaihingen dataset and 61.42% mIoU on the GID with only 4.98-M parameter size.

Unsupervised Domain Adaptation Augmented by Mutually Boosted Attention for Semantic Segmentation of VHR Remote Sensing Images

This work investigates unsupervised domain adaptation (UDA)-based semantic segmentation of very high-resolution (VHR) remote sensing (RS) images from different domains. Most existing UDA methods resort to generative adversarial networks (GANs) to cope with the domain shift problem caused by the discrepancies across different domains. However, these GAN-based UDA methods directly align two domains in the appearance, latent, or output space based on convolutional neural networks (CNNs), making them ineffective in exploiting long-range dependencies across the high-level feature maps derived from different domains. Unfortunately, such high-level features play an essential role in characterizing RS images with complex content. To circumvent this obstacle, a mutually boosted attention transformer (MBATrans) is proposed to capture cross-domain dependencies of semantic feature representations in this work. Compared with conventional UDA methods, MBATrans can significantly reduce domain discrepancies by capturing transferable features using global attention. More specifically, MBATrans utilizes a novel mutually boosted attention (MBA) module to align cross-domain feature maps while enhancing domain-general features. Furthermore, a novel GAN-based network with improved discriminative capability is devised by integrating an additional discriminator to learn domain-specific features. Extensive experiments on two large-scale VHR RS datasets, namely, International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen, confirm the superior performance of the proposed MBATrans-augmented GAN (MBATA-GAN) architecture. The source code in this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sstary/SSRS</uri> .

Multilevel Heterogeneous Domain Adaptation Method for Remote Sensing Image Segmentation

Due to more abundant data sources, more various objects of interest, and more time-consuming annotations, there is a large amount of out-of-distribution (OOD) data in the remote sensing field, on which the performance of high-accuracy image segmentation models trained under ideal experimental conditions generally degrades dramatically. Domain adaptation (DA) consequently comes into being, which aims to learn the predictor for the label-scarce target domain of interest with the help of the label-sufficient source domain in the presence of the distribution difference, namely, domain shift, between the two domains. However, the off-the-shelf DA methods for image segmentation not only struggle to cope with the more complex domain shift problems in remote sensing imagery but also almost cannot process heterogeneous data directly without information loss. While the current heterogeneous DA methods mostly still rely on some supervision information from the target domain, which is typically inaccessible in the real world. To overcome these drawbacks, we propose the multilevel heterogeneous unsupervised DA (UDA) method, termed MHDA, which unifies the instance-level DA based on cycle consistency, the feature-level DA based on contrastive learning, and the decision-level DA based on task consistency into a framework to more effectively handle the complex domain shift and heterogeneous data. After that, extensive DA experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) dataset, the BigCity dataset constructed by ourselves, and the Wuhan University (WHU) dataset, to explore the effect of each module in MHDA, the necessity of heterogeneous DA, and the effectiveness of multilevel DA. And the results demonstrate that MHDA can achieve superior performance on the remote sensing image segmentation task, compared with several state-of-the-art DA methods.

Unsupervised domain adaptation using modified cycle generative adversarial network for aerial image classification

The unsupervised domain adaptation (UDA) of aerial image classification is of great significance in the traffic management and public safety applications. Due to the differences in the distribution of source and target domain datasets from different sensors and cities, and the acquisition difficulty and high annotation cost of target domain datasets, existing domain adaptive methods have difficulty obtaining high accuracy and stability. In this work, a modified cycle generative adversarial network (CycleGAN) (MGAN) based on the multiscale residual block (MSRB) and multipooling coordinate attention (MPCA) modules is proposed to generate fake images, which is the most critical stage of the UDA method. In the MGAN, the MSRB module can extract the multiscale information of the image, and the MPCA module can fuse the position and channel information in the image. These two modules better extract the features of edges, regions and semantic classes, which makes the generated fake images better retain the information of the source domain and possess the style of the target domain. Otherwise, the Deeplabv3+ model is used to obtain a pretrained and final segmentation model in the first and third stages. In the experiment, the proposed method is validated on the aerial images of the Potsdam and Vaihingen datasets in the International Society for Photogrammetry and Remote Sensing (ISPRS) two-dimensional semantic labeling benchmark and improves overall accuracy from 52% to 60% on the segmentation of the target domain, compared with the existing optimal method; it also achieves an accuracy of 77% for buildings. The results demonstrate that the proposed method can achieve a higher accuracy and stability.

Multi-information PointNet++ fusion method for DEM construction from airborne LiDAR data

Airborne light detection and ranging (LiDAR) is a popular technology in remote sensing that can significantly improve the efficiency of digital elevation model (DEM) construction. However, it is challenging to identify the real terrain features in complex areas using LiDAR data. To solve this problem, this work proposes a multi-information fusion method based on PointNet++ to improve the accuracy of DEM construction. The RGB data and normalized coordinate information of the point cloud was added to increase the number of channels on the input side of the PointNet++ neural network, which can improve the accuracy of the classification during feature extraction. Low and high density point clouds obtained from the International Society for Photogrammetry and Remote Sensing (ISPRS) and the United States Geological Survey (USGS) were used to test this proposed method. The results suggest that the proposed method improves the Kappa coefficient by 8.81% compared to PointNet++. The type I error was reduced by 2.13%, the type II error was reduced by 8.29%, and the total error was reduced by 2.52% compared to the conventional algorithm. Therefore, it is possible to conclude that the proposed method can obtain DEMs with higher accuracy.

AUTOMATIC TEXTURE MAPPING METHOD FOR 3D MODELS TO CIRCUMVENT OCCLUSION THROUGH 3D SPATIAL THROUGH-VIEW RELATIONSHIPS BETWEEN IMAGES,MODELS, AND POINT CLOUDS

Abstract. Photorealistic 3D models play an important role in various applications related to smart cities. Texture mapping plays a crucial role as the last step of 3D reconstruction. The texture of the model directly reflects the visualization and realism of the model. In complex city scenes, objects often have varying degrees of occlusion from one another. Traditional methods to circumvent occlusion by image or model data are not very effective. In this paper, we propose an automatic texture mapping method for 3D models to circumvent occlusion through 3D spatial through-view relationships between images, models, and point clouds. First of all, calculate the density and voxel resolution of the point clouds, the point clouds here can be either dense matching point clouds or laser point clouds. Voxelization is based on point clouds, voxels generated by voxelization can reflect the spatial location of the point clouds. Secondly, to determine whether there is an occlusion of voxels between the image and the model vertices based on the spatial through-view relationship between the image, the model, and the voxels, and filter out the unobstructed images. Finally, by optimizing the data items of the Markov random field, the shading function is added to the original data items for evaluating the shading condition, based on which the most suitable image is selected for texture mapping of a triangular surface of the model. Loop computation to complete the unobstructed texture mapping of the entire model. In this paper, the public test datasets released by the International Society for Photogrammetry and Remote Sensing (ISPRS) and several urban area datasets collected by ourselves are incorporated to test the performance of the proposed method. Both qualitative and quantitative cross-sectional comparisons with existing texture mapping algorithms are conducted and presented. Experimental results indicate that our method can effectively circumvent occlusion for automated texture mapping, and generate 3D models with fewer obstructed textures, which largely improves the visual quality and realism of the models.

LiDAR Filtering in 3D Object Detection Based on Improved RANSAC

At present, the LiDAR ground filtering technology is very mature. There are fewer applications in 3D-object detection due to the limitations of filtering accuracy and efficiency. If the ground can be removed quickly and accurately, the 3D-object detection algorithm can detect objects more accurately and quickly. In order to meet the application requirements of 3D-object detection, inspired by Universal-RANSAC, we analyze the detailed steps of RANSAC and propose a precise and efficient RANSAC-based ground filtering method. The principle of GroupSAC is analyzed, and the sampled points are grouped by attributes to make it easier to sample the correct point. Based on this principle, we devise a method for limiting sampled points that is applicable to point clouds. We describe preemptive RANSAC in detail. Its breadth-first strategy is adopted to obtain the optimal plane without complex iterations. We use the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets and the KITTI dataset for testing. Experiments show that our method has higher filtering accuracy and efficiency compared with the currently widely used methods. We explore the application of ground filtering methods in 3D-object detection, and the experimental results show that our method can improve the object detection accuracy without affecting the efficiency.

Geometry-Aware Segmentation of Remote Sensing Images via Joint Height Estimation

Recent studies have shown the benefits of using additional elevation data [e.g., digital surface model (DSM) or normalized DSM (nDSM)] for enhancing the performance of the semantic labeling of aerial images. However, previous methods mostly adopt 3-D elevation information as additional inputs, while, in many real-world applications, one does not have the corresponding DSM images at hand, and the spatial resolution of acquired DSM images usually does not match the aerial images. To alleviate this data constraint and also take advantage of 3-D elevation information, in this letter, a geometry-aware segmentation model is introduced to achieve accurate semantic labeling of aerial images via joint height estimation. Instead of using a single-stream encoder-decoder network for semantic labeling, we design a separate decoder branch to predict the height map and use the DSM images as side supervision to train this newly designed decoder branch. With the newly designed decoder branch, our model can distill the 3-D geometric features from 2-D appearance features under the supervision of ground-truth DSM images. Moreover, we develop a new geometry-aware convolution module that fuses the 3-D geometric features from the height decoder branch and the 2-D contextual features from the semantic segmentation branch. The fused feature embeddings can produce geometry-aware segmentation maps with enhanced performance. Our model is trained with DSM images as side supervision, while, in the inference stage, it does not require DSM data and directly predicts the semantic labels. Experiments on International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam data sets demonstrate the effectiveness of the proposed method for the semantic segmentation of aerial images.

CCANet: Class-Constraint Coarse-to-Fine Attentional Deep Network for Subdecimeter Aerial Image Semantic Segmentation

Semantic segmentation is important for the understanding of subdecimeter aerial images. In recent years, deep convolutional neural networks (DCNNs) have been used widely for semantic segmentation in the field of remote sensing. However, because of the highly complex subdecimeter resolution of aerial images, inseparability often occurs among some geographic entities of interest in the spectral domain. In addition, the semantic segmentation methods based on DCNNs mostly obtain context information using extra information within the added receptive field. However, the context information obtained this way is not explicit. We propose a novel class-constraint coarse-to-fine attentional (CCA) deep network, which enables the formation of class information constraints to obtain explicit long-range context information. Further, the performance of subdecimeter aerial image semantic segmentation can be improved, particularly for fine-structured geographic entities. Based on coarse-to-fine technology, we obtained a coarse segmentation result and constructed an image class feature library. We propose the use of the attention mechanism to obtain strong class-constrained features. Consequently, pixels of different geographic entities can adaptively match the corresponding categories in the class feature library. Additionally, we employed a novel loss function, CCA-loss to realize end-to-end training. The experimental results obtained using two popular open benchmarks, International Society for Photogrammetry and Remote Sensing (ISPRS) 2-D semantic labeling Vaihingen data set and Institute of Electrical and Electronics Engineers (IEEE) Geoscience and Remote Sensing Society (GRSS) Data Fusion Contest Zeebrugge data set, validated the effectiveness and superiority of our proposed model. The proposed method achieved state-of-the-art performance on the IEEE GRSS Data Fusion Contest Zeebrugge data set.

A Deep Neural Network Using Double Self-Attention Mechanism for ALS Point Cloud Segmentation

Airborne laser scanning (ALS) point cloud segmentation is an essential procedure for 3D data understanding and applications. This task is challenging due to the unstructured, disordered, and sparse distribution of the point cloud. PointNet++ is a well known end-to-end learning network for point cloud segmentation without fully exploring the local and contextual features, which are less efficient and accurate in capturing the complexity of point clouds. On this basis, we design a novel encoder-decoder network architecture to obtain the semantic features of the ALS point cloud at different levels and achieve a better segmentation effect. The improved local feature aggregation module can merge the deep feature of the point cloud, combining local and global self-attention convolutional networks. It can adaptively explore the inherent semantics feature of points and capture more extensive context information of ALS point cloud, termed DSPNet++. Finally, the conditional random field optimization model can be used to refine the segmentation results. We evaluated the performance of our method on the Vaihingen dataset of the International Society for Photogrammetry and Remote Sensing (ISPRS) and the GML(B) 3D dataset. Experimental results show that our method fully exploits the semantic feature of the ALS point cloud and can achieve higher accuracy. A comparative study with established deep learning models also confirms that our proposed method has outstanding performance in the ALS point cloud segmentation task.

Few-Shot Segmentation of Remote Sensing Images Using Deep Metric Learning

Current convolutional neural network (CNN)-based methods for remote sensing image segmentation require a large number of densely annotated images for model training and have limited generalization abilities for unseen object categories. In this letter, we propose a novel few-shot learning-based method for the semantic segmentation of remote sensing images. Our method can perform semantic labeling for unseen object categories with only a few annotated samples. More specifically, our model starts by using a deep CNN to extract high-level semantic features. The prototype representation of each class is then generated by using a masked average pooling on the feature embeddings of the support images with ground truth masks. Finally, our model performs semantic labeling over the query images by matching the feature embedding of each pixel to its nearest prototypes in the embedding space. Our model is optimized with a nonparametric metric learning-based loss function to maximize the intra-class similarity of learned prototypes while minimizing the inter-class similarity. Experiments on International Society for Photogrammetry and Remote Sensing (ISPRS) 2-D semantic labeling dataset demonstrate satisfying in-domain and cross-domain transferring abilities of our model.

A Stepwise Domain Adaptive Segmentation Network With Covariate Shift Alleviation for Remote Sensing Imagery

Semantic segmentation for remote sensing images (RSI) is critical for the Earth monitoring system. However, the covariate shift between RSI datasets under different capture conditions cannot be alleviated by directly using the unsupervised domain adaptation (UDA) method, which negatively affects the segmentation accuracy in RSI. We propose a stepwise domain adaptive segmentation network with covariate shift alleviation (Cov-DA) for RSI parsing to solve this issue. Specifically, to alleviate domain shift generated by different sensors, both the source and target domains are projected into a colorspace with normalized distribution through an elaborate colorspace mapping unified module (CMUM). The color distributions of these two domains tend to be more uniform. Furthermore, in the target domain, the multistatistics joint evaluation module (MJEM) is proposed to capture different statistical characteristics of subscenarios for selecting plain scenarios regarded as high-confidence segmentation results to assist the further improvement of segmentation performance. In addition, a pyramid perceptual attention module (PPAM) containing omnidirectional features without computational burdens is added to our network for effectively enhancing the multiscale feature capture ability. In the cross-city DA experiments based on the International Society for Photogrammetry and Remote Sensing (ISPRS) and aerial benchmarks, the superiority of our algorithm is significantly demonstrated. Furthermore, we release a large-scale Martian terrain dataset noted as “Mars-Seg” containing 5 K images with pixel-level accurate annotations regarding issues, such as the lack of semantic segmentation datasets for unknown scenes.