International Society For Photogrammetry Research Articles

Category-Based Interactive Attention and Perception Fusion Network for Semantic Segmentation of Remote Sensing Images

With the development of CNNs and the application of transformers, the segmentation performance of high-resolution remote sensing image semantic segmentation models has been significantly improved. However, the issue of category imbalance in remote sensing images often leads to the model’s segmentation ability being biased towards categories with more samples, resulting in suboptimal performance for categories with fewer samples. To make the network’s learning and representation capabilities more balanced across different classes, in this paper we propose a category-based interactive attention and perception fusion network (CIAPNet), where the network divides the feature space by category to ensure the fairness of learning and representation for each category. Specifically, the category grouping attention (CGA) module utilizes self-attention to reconstruct the features of each category in a grouped manner, and optimize the foreground–background relationship and its feature representation for each category through the interactive foreground–background relationship optimization (IFBRO) module therein. Additionally, we introduce a detail-aware fusion (DAF) module, which uses shallow detail features to complete the semantic information of deep features. Finally, a multi-scale representation (MSR) module is deployed for each class in the CGA and DAF modules to enhance the description capability of different scale information for each category. Our proposed CIAPNet achieves mIoUs of 54.44%, 85.71%, and 87.88% on the LoveDA urban–rural dataset, and the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam urban datasets, respectively. Compared with current popular methods, our network not only achieves excellent performance but also demonstrates outstanding class balance.

Multi-scale fusion pixel and instance contrastive self-supervised learning for semantic segmentation of high-resolution Earth surface images

In the field of remote sensing (RS) image semantic segmentation, existing supervised learning methods rely on a large amount of labeled data, which limits their application scope. To solve this problem, we propose a new self-supervised learning method, called multi-scale fusion pixel and instance contrastive learning network (MPINet). This method first uses focal frequency loss to optimize the learning of high-level semantic information, and then strengthens the spatial information in the shallow feature map through multi-scale fusion pixel contrast learning, thereby improving the model’s ability to mine detailed features. Experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam, LoveDA and UAVid datasets. The results show that our method achieves 49.89%, 70.98% and 63.55% in mIoU, OA and mF1 indicators on the ISPRS Potsdam dataset, which is 1.48%, 1.71% and 0.95% higher than the best method. On the LoveDA dataset, our method achieves 38.05%, 52.89% and 54.10% in mIoU, OA and mF1 indicators, which is 0.6%, 0.91% and 1.04% higher than the best method. On the UAVid dataset, our method achieved mIoU, OA, and mF1 scores of 57.92%, 76.97%, and 73.28%, respectively, representing improvements of 1.06%, 1.46%, and 1.15% compared to the best method. Experimental results show that the proposed method outperforms the existing optimal self-supervised learning method and the ImageNet pre-training method.

Filtering airborne LiDAR data based on multi-view window and multi-resolution hierarchical cloth simulation

ABSTRACT Ground filtering is a fundamental step in airborne LiDAR data processing toward a variety of applications. However, existing algorithms remain tremendously challenging in complex environments, e.g. steep hillsides, ridges, valleys, discontinuities, and numerous objects. We presented a new ground filtering algorithm that can handle various landscapes. First, the multi-view window is developed to increase the number of ground seeds on the various terrains. Second, multi-resolution hierarchical cloth simulation is used to rapidly construct the high-resolution reference terrain, and bidirectional internal force operation is proposed to improve the accuracy of reference terrain by smoothing the spikes in cloth. Finally, ground and non-ground points are classified based on the height differences between points and the reference terrain. The proposed algorithm was validated not only in the International Society for Photogrammetry and Remote Sensing (ISPRS) but also karst datasets, where particularly complex environments is contained. Results showed that the proposed algorithm outperformed the existing algorithms, with the lowest average total error of 3.85% and the highest average kappa coefficient of 87.75%. Moreover, the proposed algorithm can completely preserve complex terrain, e.g. extremely steep hillsides, and sharp ridges. This study had great potential to provide a useful tool for LiDAR data processing.

An improved semantic segmentation algorithm for high-resolution remote sensing images based on DeepLabv3+

High-precision and high-efficiency Semantic segmentation of high-resolution remote sensing images is a challenge. Existing models typically require a significant amount of training data to achieve good classification results and have numerous training parameters. A novel model called MST-DeepLabv3+ was suggested in this paper for remote sensing image classification. It’s based on the DeepLabv3+ and can produce better results with fewer train parameters. MST-DeepLabv3+ made three improvements: (1) Reducing the number of model parameters by substituting MobileNetV2 for the Xception in the DeepLabv3+’s backbone network. (2) Adding the attention mechanism module SENet to increase the precision of semantic segmentation. (3) Increasing Transfer Learning to enhance the model's capacity to recognize features, and raise the segmentation accuracy. MST-DeepLabv3+ was tested on international society for photogrammetry and remote sensing (ISPRS) dataset, Gaofen image dataset (GID), and practically applied to the Taikang cultivated land dataset. On the ISPRS dataset, the mean intersection over union (MIoU), overall accuracy (OA), Precision, Recall, and F1-score are 82.47%, 92.13%, 90.34%, 90.12%, and 90.23%, respectively. On the GID dataset, these values are 73.44%, 85.58%, 84.10%, 84.86%, and 84.48%, respectively. The results were as high as 90.77%, 95.47%, 95.28%, 95.02%, and 95.15% on the Taikang cultivated land dataset. The experimental results indicate that MST-DeepLabv3+ effectively improves the accuracy of semantic segmentation of remote sensing images, recognizes the edge information with more completeness, and significantly reduces the parameter size.

I2-FaçadeNet: An Illumination-invariant Façade Recognition Network Leveraging Sparsely Gated Mixture of Multi-color Space Experts for Aerial Oblique Imagery

Façade image recognition under complex illumination conditions is crucial for various applications, including urban three-dimensional modeling and building identification. Existing methods relying solely on Red-Green-Blue (RGB) images are prone to texture ambiguity in complex illumination environments. Furthermore, façades display varying orientations and camera viewing angles, resulting in performance issues within the RGB color space. In this study, we introduce an illumination-invariant façade recognition network (I2-FaçadeNet) that leverages sparsely gated multi-color space experts for enhanced façade image recognition in challenging illumination environments. First, RGB façade images are converted into multi-color spaces to eliminate the ambiguous texture in complex illumination. Second, we train expert networks using separate channels of multi-color spaces. Finally, a sparsely gated mechanism is introduced to manage the expert networks, enabling dynamic activation of expert networks and the merging of results. Experimental evaluations leveraging both the International Society for Photogrammetry and Remote Sensing benchmark data sets and the Shenzhen data sets reveal that our proposed I2 -FaçadeNet surpasses various depths of ResNet in façade recognition under complex illumination conditions. Specifically, the classification accuracy for poorly illuminated façades in Zurich improves by nearly 8%, while the accuracy for over-illuminated areas in Shenzhen increases by approximately 3%. Moreover, ablation studies conducted on façade images with complex illumination indicate that compared to traditional RGB-based ResNet, the proposed network achieves an accuracy improvement of 3% to 4% up to 100% for overexposed images and an accuracy improvement of 3% to 10% for underexposed images.

Preface: ISPRS Geospatial Week 2023

Abstract. The International Society for Photogrammetry and Remote Sensing (ISPRS) Geospatial Week 2023 (GSW’2023) is a combination of 29 workshops organized by the ISPRS Working Groups active in areas of interest of ISPRS. The Geospatial Week 2023 is held from 2–7 September 2023, at the Semiramis Cairo Hotel, on the magnificent river Nile, Egypt, and is convened by the Arab Academy for Science, Technology & Maritime Transport (AASTMT) acting as the local organizer. The GSW’2023 is the first in Africa and the Middle East. The conference was chaired by Prof. Naser EL-Sheimy and Prof. Ismail Abdelgafhar (President of AASTMT) as the Honorary Chair and host of the conference under the Auspices of H.E. Professor Mostafa Madbouly, Prime Minister of the Arab Republic of Egypt. The GSW’2023 is the fifth edition, after, Antalya Turkey in 2013, La Grande Motte France in 2015, Wuhan China in 2017, and Enschede The Netherlands in 2019. The following 29 workshops provide state of the art and future trends in geospatial, sensors, photogrammetry, remote sensing, and spatial information sciences technologies in their applications in many industries and economic sectors: Cultural Heritage Visualization and Virtual Restoration SpACE – Spectral Remote Sensing in the era of AI, Cloud and Edge Computing Youth Presentation Forum Openness in Geospatial and Remote Sensing Precision GNSS: Technology Advances and Applications for Navigation and Mapping Photogrammetric 3D Reconstruction for Geo-Applications (PhotoGA 2023) Geospatial Data Analytics for Physical Geography Impact Assessment on Environment, Health andSociety Intelligent Systems in Sensor Web and Internet of Things Underwater Mapping Workshop: Geospatial techniques for underwater documentation, mapping andmonitoring SO&C: Sensor orientation and calibration for mapping and navigation purposes Smart Forests – Forest ecosystem assessment and monitoring using Remote Sensing, ArtificialIntelligence, and Robotics Satellite Remote Sensing and Its Applications Advanced Data Preparation and Data Management for Geospatial and Remote Sensing Scenarios Laser Scanning 2023 NGC of AV: Navigation, Guidance and Control of Autonomous Vehicles ISSDQ 2023- Artificial Intelligence and Uncertainty Modeling in Spatial Analysis Semantics3D – Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences GeoHB 2023: Geo-Spatial Computing for Understanding Human Behaviours GI4SDGs: The Geospatial Information and SDG Nexus SARcon 2023 – SAR constellations and applications Digital Construction CrowdMapping: Crowdsourcing for Global Mapping Indoor 3D IAMS – Intelligent and autonomous mapping systems AI-PC: AI-based Point Cloud and Image Understanding UAV-based mapping with imaging and LiDAR systems: challenges, data processing, and applications 3DS Smart Cities – 3D Sensing for Smart Cities Robotics for Mapping – SLAM approaches for mobile mapping and robot intelligence MMT and HD Maps – Mobile Mapping Technologies and HD Maps Many of the workshops are part of well-established series of workshops convened in the past. They cover topics like UAV for mapping, laser scanning, Semantics3D, SARcon, ISSDQ hyperspectral imaging, and crowd sourcing and collaborative mapping with applications ranging from indoor mapping, smart forest and smart cities to global mapping studies and planetary mapping. The conference also includes for the first-time few workshops organized by other working groups from sister organizations and the ISPRS youth forum. In total 172 full papers and 423 extended abstracts were submitted by authors from 68 countries. 1400 reviews have been delivered by 283 reviewers. A total of 137 full papers have been accepted for the volume X-1/W1-2023 of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Annals). Another 260 papers are published in volume XLVIII-1/W2-2023 of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Archives). The editors would like to thank all contributing authors, reviewers and all workshop organizers for their role in preparing and organizing the Geospatial Week 2023. Thanks to their contributions, we can offer an excessive and varying collection in the Annals and the Archives. The editors would like to extend their great appreciation to the President of AASTMT Prof. Ismail Abdel Ghafar for his great support in hosting the ISPRS GSW’2023. We also would like to extend our thanks to the members of the local organizing committee for great effort in making GSW’2023 conference a success, namely, Prof. Amr Ali Hanafi, Prof. Ahmed Shaker, Research Eng. Ayman Elzagh, Dr. Mengchi Ai, Dr. Adel Moussa, Dr. Yiran Luo, Mr. Yasser Elrashidi, Eng. Nagy K. Aly, Prof. Amira Zaki, Dr. Radwa Osman, Eng. Mohamed ElQurashi and Mr. Mohamed Abdallah Mahmoud. We hope you enjoy reading the proceedings. Editor: Prof. Naser El-Sheimy, Geospatial Week Director 2023, General Chair (University of Calgary, Canada)Co-Editors, The Arab Academy for Science, Technology & Maritime Transport:Prof. Alaa Abdelwahed Abdelbary, Chair Local Organizing CommitteeProf. Nashwa El-BendaryProf. Yahya Mohasseb

Preface: ISPRS Geospatial Week 2023

Abstract. The International Society for Photogrammetry and Remote Sensing (ISPRS) Geospatial Week 2023 (GSW’2023) is a combination of 29 workshops organized by the ISPRS Working Groups active in areas of interest of ISPRS. The Geospatial Week 2023 is held from 2–7 September 2023, at the Semiramis Cairo Hotel, on the magnificent river Nile, Egypt, and is convened by the Arab Academy for Science, Technology & Maritime Transport (AASTMT) acting as the local organizer. The GSW’2023 is the first in Africa and the Middle East. The conference was chaired by Prof. Naser EL-Sheimy and Prof. Ismail Abdelgafhar (President of AASTMT) as the Honorary Chair and host of the conference under the Auspices of H.E. Professor Mostafa Madbouly, Prime Minister of the Arab Republic of Egypt. The GSW’2023 is the fifth edition, after, Antalya Turkey in 2013, La Grande Motte France in 2015, Wuhan China in 2017, and Enschede The Netherlands in 2019. The following 29 workshops provide state of the art and future trends in geospatial, sensors, photogrammetry, remote sensing, and spatial information sciences technologies in their applications in many industries and economic sectors: Cultural Heritage Visualization and Virtual Restoration SpACE – Spectral Remote Sensing in the era of AI, Cloud and Edge Computing Youth Presentation Forum Openness in Geospatial and Remote Sensing Precision GNSS: Technology Advances and Applications for Navigation and Mapping Photogrammetric 3D Reconstruction for Geo-Applications (PhotoGA 2023) Geospatial Data Analytics for Physical Geography Impact Assessment on Environment, Health andSociety Intelligent Systems in Sensor Web and Internet of Things Underwater Mapping Workshop: Geospatial techniques for underwater documentation, mapping andmonitoring SO&C: Sensor orientation and calibration for mapping and navigation purposes Smart Forests – Forest ecosystem assessment and monitoring using Remote Sensing, ArtificialIntelligence, and Robotics Satellite Remote Sensing and Its Applications Advanced Data Preparation and Data Management for Geospatial and Remote Sensing Scenarios Laser Scanning 2023 NGC of AV: Navigation, Guidance and Control of Autonomous Vehicles ISSDQ 2023- Artificial Intelligence and Uncertainty Modeling in Spatial Analysis Semantics3D – Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences GeoHB 2023: Geo-Spatial Computing for Understanding Human Behaviours GI4SDGs: The Geospatial Information and SDG Nexus SARcon 2023 – SAR constellations and applications Digital Construction CrowdMapping: Crowdsourcing for Global Mapping Indoor 3D IAMS – Intelligent and autonomous mapping systems AI-PC: AI-based Point Cloud and Image Understanding UAV-based mapping with imaging and LiDAR systems: challenges, data processing, and applications 3DS Smart Cities – 3D Sensing for Smart Cities Robotics for Mapping – SLAM approaches for mobile mapping and robot intelligence MMT and HD Maps – Mobile Mapping Technologies and HD Maps Many of the workshops are part of well-established series of workshops convened in the past. They cover topics like UAV for mapping, laser scanning, Semantics3D, SARcon, ISSDQ hyperspectral imaging, and crowd sourcing and collaborative mapping with applications ranging from indoor mapping, smart forest and smart cities to global mapping studies and planetary mapping. The conference also includes for the first-time few workshops organized by other working groups from sister organizations and the ISPRS youth forum. In total 172 full papers and 423 extended abstracts were submitted by authors from 68 countries. 1400 reviews have been delivered by 283 reviewers. A total of 137 full papers have been accepted for the volume X-1/W1-2023 of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Annals). Another 260 papers are published in volume XLVIII-1/W2-2023 of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Archives). The editors would like to thank all contributing authors, reviewers and all workshop organizers for their role in preparing and organizing the Geospatial Week 2023. Thanks to their contributions, we can offer an excessive and varying collection in the Annals and the Archives. The editors would like to extend their great appreciation to the President of AASTMT Prof. Ismail Abdel Ghafar for his great support in hosting the ISPRS GSW’2023. We also would like to extend our thanks to the members of the local organizing committee for great effort in making GSW’2023 conference a success, namely, Prof. Amr Ali Hanafi, Prof. Ahmed Shaker, Research Eng. Ayman Elzagh, Dr. Mengchi Ai, Dr. Adel Moussa, Dr. Yiran Luo, Mr. Yasser Elrashidi, Eng. Nagy K. Aly, Prof. Amira Zaki, Dr. Radwa Osman, Eng. Mohamed ElQurashi and Mr. Mohamed Abdallah Mahmoud. We hope you enjoy reading the proceedings. Editor: Prof. Naser El-Sheimy, Geospatial Week Director 2023, General Chair (University of Calgary, Canada)Co-Editors, The Arab Academy for Science, Technology & Maritime Transport:Prof. Alaa Abdelwahed Abdelbary, Chair Local Organizing CommitteeProf. Nashwa El-BendaryProf. Yahya Mohasseb

CNN and Transformer Fusion for Remote Sensing Image Semantic Segmentation

Semantic segmentation of remote sensing images has been widely used in environmental protection, geological disaster discovery, and natural resource assessment. With the rapid development of deep learning, convolutional neural networks (CNNs) have dominated semantic segmentation, relying on their powerful local information extraction capabilities. Due to the locality of convolution operation, it can be challenging to obtain global context information directly. However, Transformer has excellent potential in global information modeling. This paper proposes a new hybrid convolutional and Transformer semantic segmentation model called CTFuse, which uses a multi-scale convolutional attention module in the convolutional part. CTFuse is a serial structure composed of a CNN and a Transformer. It first uses convolution to extract small-size target information and then uses Transformer to embed large-size ground target information. Subsequently, we propose a spatial and channel attention module in convolution to enhance the representation ability for global information and local features. In addition, we also propose a spatial and channel attention module in Transformer to improve the ability to capture detailed information. Finally, compared to other models used in the experiments, our CTFuse achieves state-of-the-art results on the International Society of Photogrammetry and Remote Sensing (ISPRS) Vaihingen and ISPRS Potsdam datasets.

ICSF: An Improved Cloth Simulation Filtering Algorithm for Airborne LiDAR Data Based on Morphological Operations

Ground filtering is an essential step in airborne light detection and ranging (LiDAR) data processing in various applications. The cloth simulation filtering (CSF) algorithm has gained popularity because of its ease of use advantage. However, CSF has limitations in topographically and environmentally complex areas. Therefore, an improved CSF (ICSF) algorithm was developed in this study. ICSF uses morphological closing operations to initialize the cloth, and estimates the cloth rigidness for providing a more accurate reference terrain in various terrain characteristics. Moreover, terrain-adaptive height difference thresholds are developed for better filtering of airborne LiDAR point clouds. The performance of ICSF was assessed using International Society for Photogrammetry and Remote Sensing urban and rural samples and Open Topography forested samples. Results showed that ICSF can improve the filtering accuracy of CSF in the samples with various terrain and non-ground object characteristics, while maintaining the ease of use advantage of CSF. In urban and rural samples, ICSF obtained an average total error of 4.03% and outperformed another eight reference algorithms in terms of accuracy and robustness. In forested samples, ICSF produced more accuracy than the well-known filtering algorithms (including the maximum slope, progressive morphology, and cloth simulation filtering algorithms), and performed better with respect to the preservation of steep slopes and discontinuities and vegetation removal. Thus, the proposed algorithm can be used as an efficient tool for LiDAR data processing.

DMAU-Net: An Attention-Based Multiscale Max-Pooling Dense Network for the Semantic Segmentation in VHR Remote-Sensing Images

High-resolution remote-sensing images cover more feature information, including texture, structure, shape, and other geometric details, while the relationships among target features are more complex. These factors make it more complicated for classical convolutional neural networks to obtain ideal results when performing a feature classification on remote-sensing images. To address this issue, we proposed an attention-based multiscale max-pooling dense network (DMAU-Net), which is based on U-Net for ground object classification. The network is designed with an integrated max-pooling module that incorporates dense connections in the encoder part to enhance the quality of the feature map, and thus improve the feature-extraction capability of the network. Equally, in the decoding, we introduce the Efficient Channel Attention (ECA) module, which can strengthen the effective features and suppress the irrelevant information. To validate the ground object classification performance of the multi-pooling integration network proposed in this paper, we conducted experiments on the Vaihingen and Potsdam datasets provided by the International Society for Photogrammetry and Remote Sensing (ISPRS). We compared DMAU-Net with other mainstream semantic segmentation models. The experimental results show that the DMAU-Net proposed in this paper effectively improves the accuracy of the feature classification of high-resolution remote-sensing images. The feature boundaries obtained by DMAU-Net are clear and regionally complete, enhancing the ability to optimize the edges of features.

IMPROVED TORNADO METHOD FOR GROUND POINT FILTERING FROM LIDAR POINT CLOUDS

Abstract. In this paper, an improved Tornado method for filtering LiDAR (Light Detection and Ranging) point clouds is presented. The original method uses a vertical cone with a downward vertex and an upward base to remove the points within it as non-ground points. The remaining points are ground points. The cone moves on the ground surface over the entire region of the point cloud. In this work, the regions of the objects are predicted by extracting the vertical features that have points in the vertical plane or vertical column. Therefore, the tornado method is only used in regions that contain objects. In addition, our improved method uses a specific height for a tornado to reduce the Type I error in mountainous areas. Also, a cylinder surrounding the cone is used to reduce the distance calculations between the cone and the point cloud. The results show that this method is very effective and fast compared to the original method. It also has promising results for the Type I error. In addition, this method was tested on the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets and produced outstanding results. The results show that this method achieves high filtering accuracy. Moreover, the proposed method achieves an overall average error of 6.83%, which is lower than most other methods.

An improved progressive triangular irregular network densification filtering algorithm for airborne LiDAR data

Airborne lidar is a technology for mapping surface spatial information and has been widely used in many areas of geospatial information disciplines. The filtering process of removing non-ground points has always been the focus of research. PTD (Progressive Triangular Irregular Network Densification) filtering algorithm is a widely used filtering algorithm for airborne lidar data. However, this algorithm has shortcomings in retaining ground points in steep areas, leading to large type Ⅰ errors. Therefore, this paper proposes an improved PTD algorithm. The improvement is the addition of the seed points filtering. Specifically, after the potential seed points are obtained by the progressive morphological filter, the seed points filtering is performed on it to remove the non-ground points, so that the obtained seed points are more accurate. The benchmark dataset of ISPRS (International Society for Photogrammetry and Remote Sensing) Working Group III is used to assess the proposed method. Results show that the method is effective in decreasing type Ⅰ error in steep areas. Comparing with the classic PTD algorithm, the type Ⅰ error and total error are decreased by 8.46% and 5.06% respectively. In addition, the proposed method shows a great advantage in computational efficiency, that is eight times more efficient than the classic PTD algorithm.

BIBED-Seg: Block-in-Block Edge Detection Network for Guiding Semantic Segmentation Task of High-Resolution Remote Sensing Images

Edge optimization of semantic segmentation results is a challenging issue in remote sensing image processing. This article proposes a semantic segmentation model guided by a block-in-block edge detection network named BIBED-Seg. This is a two-stage semantic segmentation model, where edges are extracted first and then segmented. We do two key works: The first work is edge detection, and we present BIBED, a block-in-block edge detection network, to extract the accurate boundary features. Here, the edge detection of multiscale feature fusion is first realized by creating the block-in-block residual network structure and devising the multilevel loss function. Second, we add the channel and spatial attention module into the residual structure to improve high-resolution remote sensing images' boundary positioning and detection accuracy by focusing on their channel and spatial dimensions. Finally, we evaluate our method on International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen data sets and obtain ODS F-measure of 0.6671 and 0.7432, higher than other excellent edge detection methods. The second work is two-stage segmentation. First, the proposed BIBED is individually pretrained, and subsequently, the pretrained model is introduced into the entire segmentation network to extract boundary features. In the second segmentation stage, the edge detection network is used to constrain semantic segmentation results by loss cycles and feature bootstrapping. Our best model obtains the OA of 90.2%, 87.7%, and 81.5%, the IOU of 76.0%, 69.6%, and 61.3% on the ISPRS and WHDLD datasets, respectively.

Edge Detection Guide Network for Semantic Segmentation of Remote-Sensing Images

The acquisition of high-resolution satellite and airborne remote sensing images has been significantly simplified due to the rapid development of sensor technology. Several practical applications of high-resolution remote sensing images (HRRSIs) are based on semantic segmentation. However, single-modal HRRSIs are difficult to classify accurately in the complex situation of some scene objects; therefore, the semantic segmentation of multi-source information fusion is gaining popularity. The inherent difference between multimodal features and the semantic gap between multi-level features typically affect the performance of existing multi-mode fusion methods. We propose a multimodal fusion network based on edge detection to address these issues. This method aids multimodal information fusion by utilizing spatial information contained in the boundary. An edge detection guide module is included in the feature extraction stage to realize the boundary information through the fusion of details and semantics between high-level and low-level features. The boundary information is extended into the well-designed multimodal adaptive fusion block (MAFB) to obtain the multimodal fusion features. Furthermore, a residual adaptive fusion block (RAFB) and a spatial position module (SPM) in the feature decoding stage have been designed to fuse multi-level features from the standpoint of local and global dependence. We compared our method to several state-of-the-art (SOTA) methods using the International Society for Photogrammetry and Remote Sensing’s (ISPRS) Vaihingen and Potsdam datasets. The final results demonstrate that our method achieves excellent performance.

Roof Segmentation From Airborne LiDAR Using Octree-Based Hybrid Region Growing and Boundary Neighborhood Verification Voting

Building roof segmentation is a key step in the process of three-dimensional (3D) building reconstruction using airborne Light Detection and Ranging (LiDAR) point cloud data. Voxel-based region growing is one of the most widely used methods to segment planes because of its high efficiency and easy implementation, but it is easy to omit roof planes due to the unreasonable voxel size and the complex roof structures. In addition, boundaries between adjacent roof planes are inaccurate. To solve the issues, a roof segmentation method using octree-based hybrid region growing and boundary neighborhood verification voting is proposed. First, an octree-based voxelization is performed on the raw building points to generate two basic units: planar voxels and individual points (i.e., points that are not in the planar voxels). Then, the hybrid region growing is conducted on these two basic units to segment coarse roof planes. A parameter-free boundary neighborhood verification voting strategy is used to assign the boundary points to the correct roof planes by verifying the neighborhoods of the boundary points and using reliable neighborhood information. Experimental results of four datasets, including two datasets provided by the International Society for Photogrammetry and Remote Sensing (ISPRS), and two high-density datasets provided by OpenTopography, verify that roof planes can be successfully segmented by the proposed method with over 96.8% completeness and a minimum 93.2% correctness. In addition, boundary points are assigned to the correct roof planes by the neighborhood verification voting strategy. Thus, the segmented roof planes can be used in various applications.

PCL–PTD Net: Parallel Cross-Learning-Based Pixel Transferred Deconvolutional Network for Building Extraction in Dense Building Areas With Shadow

Urban building segmentation from remote sensed imageries is challenging because there usually exists a variety of building features. Furthermore, very high spatial resolution imagery can provide many details of the urban building, such as styles, small gaps among buildings, building shadows, etc. Hence, satisfactory accuracy in detecting and extracting urban features from highly detailed images still remains. Deep learning semantic segmentation using baseline networks works well on building extraction; however, their ability in building extraction in shadows area, unclear building feature, and narrow gaps among buildings in dense building zone is still limited. In this article, we propose parallel cross-learning-based pixel transferred deconvolutional network (PCL–PTD net), and then is used to segment urban buildings from aerial photographs. The proposed method is evaluated and intercompared with traditional baseline networks. In PCL–PTD net, it is composed of parallel network, cross-learning functions, residual unit in encoder part, and PTD in decoder part. The performance is applied to three datasets (Inria aerial dataset, international society for photogrammetry and remote sensing Potsdam dataset, and UAV building dataset), to evaluate its accuracy and robustness. As a result, we found that PCL–PTD net can improve learning capacities of the supervised learning model in differentiating buildings in dense area and extracting buildings covered by shadows. As compared to the baseline networks, we found that proposed network shows superior performance compared to all eight networks (SegNet, U-net, pyramid scene parsing network, PixelDCL, DeeplabV3+, U-Net++, context feature enhancement networ, and improved ResU-Net). The experiments on three datasets also demonstrate the ability of proposed framework and indicating its performance.

Dual-Range Context Aggregation for Efficient Semantic Segmentation in Remote Sensing Images

Although introducing self-attention mechanisms is beneficial to establish long-range dependencies and explore global context information in the task of remote sensing image semantic segmentation, it results in expensive computation and large memory cost. In this letter, we address this dilemma by proposing a lightweight dual-range context aggregation network (LDCANet) for efficient remote sensing image semantic segmentation. First, a dual-range context aggregation module (DCAM) is designed to aggregate the local features and the global semantic context acquired by convolutions and self-attention, respectively, where self-attention is implemented easily by applying two cascaded linear layers to reduce the computational complexity. Furthermore, a simple and lightweight decoder is employed to combine information from different levels, in which a multilayer perceptron (MLP)-based efficient linear block (ELB) is proposed to yield a strong and efficient representation. Experiments conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen dataset and the Gaofen Image dataset (GID) prove that our LDCANet achieves an excellent trade-off between segmentation accuracy and computational efficiency. In particular, our method achieves 74.12% mean intersection over union (mIoU) on the ISPRS Vaihingen dataset and 61.42% mIoU on the GID with only 4.98-M parameter size.

CTMFNet: CNN and Transformer Multiscale Fusion Network of Remote Sensing Urban Scene Imagery

Semantic segmentation of remotely sensed urban scene images is widely demanded in areas such as land cover mapping, urban change detection, and environmental protection. With the development of deep learning, methods based on convolutional neural networks (CNNs) have been dominant due to their powerful ability to represent hierarchical feature information. However, the limitations of the convolution operation itself limit the network’s ability to extract global contextual information. With the successful use of transformer in computer vision in recent years, transformer has shown great potential for modeling global contextual information. However, transformer is not sufficiently capable of capturing local detailed information. In this article, to explore the potential of the joint CNN and transformer mechanism for semantic segmentation of remotely sensed urban scenes, we propose a CNN and transformer multiscale fusion network (CTMFNet) based on encoding–decoding for urban scene understanding. To couple local–global context information more efficiently, we designed a dual backbone attention fusion module (DAFM) to couple the local and global context information of the dual-branch encoder. In addition, to bridge the semantic gap between scales, we built a multi-layer dense connectivity network (MDCN) as our decoder. The MDCN enables the full flow of semantic information between multiple scales to be fused with each other through upsampling and residual connectivity. We conducted extensive subjective and objective comparison experiments and ablation experiments on both the International Society of Photogrammetry and Remote Sensing (ISPRS) Vaihingen and ISPRS Potsdam datasets. Numerous experimental results have proven the superiority of our method compared to currently popular methods.

Unsupervised Domain Adaptation Augmented by Mutually Boosted Attention for Semantic Segmentation of VHR Remote Sensing Images

This work investigates unsupervised domain adaptation (UDA)-based semantic segmentation of very high-resolution (VHR) remote sensing (RS) images from different domains. Most existing UDA methods resort to generative adversarial networks (GANs) to cope with the domain shift problem caused by the discrepancies across different domains. However, these GAN-based UDA methods directly align two domains in the appearance, latent, or output space based on convolutional neural networks (CNNs), making them ineffective in exploiting long-range dependencies across the high-level feature maps derived from different domains. Unfortunately, such high-level features play an essential role in characterizing RS images with complex content. To circumvent this obstacle, a mutually boosted attention transformer (MBATrans) is proposed to capture cross-domain dependencies of semantic feature representations in this work. Compared with conventional UDA methods, MBATrans can significantly reduce domain discrepancies by capturing transferable features using global attention. More specifically, MBATrans utilizes a novel mutually boosted attention (MBA) module to align cross-domain feature maps while enhancing domain-general features. Furthermore, a novel GAN-based network with improved discriminative capability is devised by integrating an additional discriminator to learn domain-specific features. Extensive experiments on two large-scale VHR RS datasets, namely, International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam and Vaihingen, confirm the superior performance of the proposed MBATrans-augmented GAN (MBATA-GAN) architecture. The source code in this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sstary/SSRS</uri> .

Multilevel Heterogeneous Domain Adaptation Method for Remote Sensing Image Segmentation

Due to more abundant data sources, more various objects of interest, and more time-consuming annotations, there is a large amount of out-of-distribution (OOD) data in the remote sensing field, on which the performance of high-accuracy image segmentation models trained under ideal experimental conditions generally degrades dramatically. Domain adaptation (DA) consequently comes into being, which aims to learn the predictor for the label-scarce target domain of interest with the help of the label-sufficient source domain in the presence of the distribution difference, namely, domain shift, between the two domains. However, the off-the-shelf DA methods for image segmentation not only struggle to cope with the more complex domain shift problems in remote sensing imagery but also almost cannot process heterogeneous data directly without information loss. While the current heterogeneous DA methods mostly still rely on some supervision information from the target domain, which is typically inaccessible in the real world. To overcome these drawbacks, we propose the multilevel heterogeneous unsupervised DA (UDA) method, termed MHDA, which unifies the instance-level DA based on cycle consistency, the feature-level DA based on contrastive learning, and the decision-level DA based on task consistency into a framework to more effectively handle the complex domain shift and heterogeneous data. After that, extensive DA experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) dataset, the BigCity dataset constructed by ourselves, and the Wuhan University (WHU) dataset, to explore the effect of each module in MHDA, the necessity of heterogeneous DA, and the effectiveness of multilevel DA. And the results demonstrate that MHDA can achieve superior performance on the remote sensing image segmentation task, compared with several state-of-the-art DA methods.