Selecting Optimal Combination of Data Channels for Semantic Segmentation in City Information Modelling (CIM)

Yuanzhi Cai,Fangyu Guo,Hong Huang,Lei Fan,Kaiyang Wang,Cheng Zhang

doi:10.3390/rs13071367

Abstract

Over the last decade, a 3D reconstruction technique has been developed to present the latest as-is information for various objects and build the city information models. Meanwhile, deep learning based approaches are employed to add semantic information to the models. Studies have proved that the accuracy of the model could be improved by combining multiple data channels (e.g., XYZ, Intensity, D, and RGB). Nevertheless, the redundant data channels in large-scale datasets may cause high computation cost and time during data processing. Few researchers have addressed the question of which combination of channels is optimal in terms of overall accuracy (OA) and mean intersection over union (mIoU). Therefore, a framework is proposed to explore an efficient data fusion approach for semantic segmentation by selecting an optimal combination of data channels. In the framework, a total of 13 channel combinations are investigated to pre-process data and the encoder-to-decoder structure is utilized for network permutations. A case study is carried out to investigate the efficiency of the proposed approach by adopting a city-level benchmark dataset and applying nine networks. It is found that the combination of IRGB channels provide the best OA performance, while IRGBD channels provide the best mIoU performance.

Highlights

Over the last decade, the concept of city information modelling (CIM) has received a growing interest in many fields, such as surveying engineering and civil engineering [1].Generally, CIM provides valuable benefits for stakeholders, including enhancing the public management process and establishing an intelligent digital platform to store, control, and understand big data
According to the comprehensive survey proposed by Guo et al [17], point cloud semantic segmentation approaches in the deep learning (DL) framework can be divided in three paradigms: Projection-based, point-based, and discretization-based
An efficient data fusion approach is proposed in this article by exploring the optimal combination of data channels

Summary

Introduction

The concept of city information modelling (CIM) has received a growing interest in many fields, such as surveying engineering and civil engineering [1]. Several studies in the remote sensing application have indicated the importance of selecting an optimal combination of data channels regarding multispectral datasets. Bhuiyan et al [11] experimented with testing the optimal three-channel combination in model prediction using very high spatial resolution (VHSR) multispectral (MS) satellite images Their findings emphasized the importance of considering input MS channels and the careful selection of optimal channels of DL network predictions for mapping applications. Alshawabkeh [14] developed a novel dataset to evaluate the feasibility of combined LiDAR data and images for object segmentation by integrating RGB-D channels (i.e., color and depth information). The present paper aims to explore a simple optimal combination of data channels based on their semantic segmentation performance in the urban scenario.

Paradigms for Semantic Segmentation

Study Materials

Experiment Arrangement

Findings

Discussion

Conclusions