2D freehand sketch labeling using CNN and CRF

Abstract

Accurate and fast sketch segmentation and labeling is a hard task, since sketches have much fewer features than natural images. This paper proposes a novel hybrid approach for fast automatic sketch labeling, which is based on convolutional neural network (CNN) and conditional random field (CRF). Firstly, we design a CNN for stroke classification. The CNN is equipped with larger first layer filters and larger pooling, which is suitable for extracting descriptive features from strokes. Secondly, we integrate each stroke with its host sketch to construct a more informative input for the CNN model. Finally, we leverage the spatio-temporal relations among strokes in the same sketch to create a connected graph, based on which we apply a CRF model to further refine the result of the CNN. We evaluate our method on two public benchmark datasets. Experimental results demonstrate that our method achieves the state-of-the-art level on both accuracy and runtime.