Fine-Level Semantic Labeling of Large-Scale 3D Model by Active Learning

Abstract

Semantic labeling of 3D models has been a challenging task in recent years. Due to the various categories and shapes of 3D objects in different scenes, it is hard to develop a versatile method suitable for most scenes. In this paper, we propose an Active Learning based method to tackle the problem. The proposed method takes a 3D mesh model generated from images using SfM and MVS, as well as the calibrated images, as the input, and outputs a semantic mesh model in which each facet takes a fine-level semantic label. Starting with a small annotated image set, we progressively fine-tune a Convolutional Neural Network (CNN) with the ever-enlarging annotated image set for image semantic segmentation. In each iteration, by back-projecting the pixel labels to the 3D model and fusing them in 3D space, a semantic 3D model is generated. The semantic 3D model functions as a supervisor to select a batch of worthy images for annotation to boost the performance of the CNN in next iteration. This process iterates until the label assignment of the 3D model becomes steady. By making full use of the 3D geometric information, the proposed method could significantly reduce the annotation cost without losing the labeling quality of 3D models. Experimental results of fine-level labeling on two large-scale ancient Chinese architectures demonstrate the effectiveness of the proposed method.