DeepSketch 3

Abstract

Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets), state-of-the-art in the field of sketch recognition, to address several applications of automatic sketch processing: complete and partial sketch recognition, sketch retrieval using query-by-example (QbE), and sketch-based image retrieval (SBIR) i.e the retrieval of images using a QbE paradigm but where the query is a sketch. We first focus on improving sketch recognition. For this purpose we compare different ConvNet architectures, training paradigms and data fusion schemes. This enabled us to outperform previous state-of-the-art in two large scale benchmarks for sketch classification. We achieved a mean average accuracy of 79.18% for the TU-Berlin sketch benchmark and 93.02% for the sketchy database. For partial sketch recognition, we were able to produce a system that achieves a mean average accuracy of 52.58% with only 40% of the strokes. We then conduct a comprehensive study of ConvNets features to enhance sketch retrieval and image retrieval, using a kNN similarity search paradigm in the ConvNet feature space. For the sketch retrieval tasks, we compare the performance obtained with features extracted from various depths (ConvNet layers) using one of the best performing model from the previous work. For the sketch-based image retrieval (SBIR), a sketch query is used to retrieve images of objects that belong to the same category, or even with a shape and pose close to the sketch query. The main challenge in the field of SBIR is to obtain efficient cross-domain features for sketch-image similarity measure. For this, besides comparing features extracted from different depth, we additionally compare different training approaches (some novel) for the ConvNets applied to sketches and images. Eventually, our best SBIR system achieves state-of-the-art results on the sketchy database (close to 40% recall at k = 1).