Urban 3D segmentation and modelling from street view images and LiDAR point clouds

Pouria Babahajiani,Joni-Kristian Kämäräinen,Lixin Fan,Moncef Gabbouj

doi:10.1007/s00138-017-0845-3

Pouria Babahajiani, Joni-Kristian Kämäräinen + Show 2 more

https://doi.org/10.1007/s00138-017-0845-3

Copy DOI

Abstract

3D urban maps with semantic labels and metric information are not only essential for the next generation robots such autonomous vehicles and city drones, but also help to visualize and augment local environment in mobile user applications. The machine vision challenge is to generate accurate urban maps from existing data with minimal manual annotation. In this work, we propose a novel methodology that takes GPS registered LiDAR (Light Detection And Ranging) point clouds and street view images as inputs and creates semantic labels for the 3D points clouds using a hybrid of rule-based parsing and learning-based labelling that combine point cloud and photometric features. The rule-based parsing boosts segmentation of simple and large structures such as street surfaces and building facades that span almost 75% of the point cloud data. For more complex structures, such as cars, trees and pedestrians, we adopt boosted decision trees that exploit both structure (LiDAR) and photometric (street view) features. We provide qualitative examples of our methodology in 3D visualization where we construct parametric graphical models from labelled data and in 2D image segmentation where 3D labels are back projected to the street view images. In quantitative evaluation we report classification accuracy and computing times and compare results to competing methods with three popular databases: NAVTEQ True, Paris-Rue-Madame and TLS (terrestrial laser scanned) Velodyne.

Highlights

Automatic segmentation and labelling of urban point cloud data is challenging due to a number of data specific challenges
Our method is made efficient by combining fast rule-based processing for building and street surface segmentation and super-voxel-based feature extraction and classification for remaining map elements. – We propose two back ends for semantically labelled urban 3D map data that exemplify two important applications: (i) 3D urban map visualization and (ii) semantic segmentation of 2D street view images by backprojection of the 3D labels. – Parameters of the different processing stages have clear physical and intuitive meaning, and they are easy to set for novel data or optimize by cross-validation over certain ranges
It is noteworthy that urban 3D segmentation has been investigated for stereo-generated point clouds [32], but there noise level is orders of magnitude higher and we focus on high-quality Light Detection And Ranging (LiDAR) data

Summary

Introduction

Automatic segmentation and labelling of urban point cloud data is challenging due to a number of data specific challenges. The rule-based method can typically label 70–80% of the point cloud data and rulebased methods are more than 6× faster than the otherwise efficient boosted decision trees [17]. The final step improves the visual quality of the semantic 3D models output from our processing pipeline, especially for those sparse and incomplete point clouds corresponding to small objects. Another application of our method is semantic segmentation of street view images which is achieved by backprojecting the semantic labels of the point cloud points to the corresponding street view images. Contributions Preliminary results on components of our processing pipeline have been reported in [2,3], and in this work we make the following novel contributions:

Methods

Findings

Discussion

Conclusion