A multi-layer deep fusion convolutional neural network for sketch based image retrieval

Abstract

Abstract The purpose of this paper is to introduce a new approach for the free-hand sketch representation in the sketch based image retrieval (SBIR), where the sketches are treated as the queries to search for the natural photos in the natural image dataset. This task is known as an extremely challenging work for 3 main reasons: (i) sketches show a highly abstract visual appearance versus natural photos, fewer context can be extracted as descriptors using the existing methods, (ii) for the same object, different people provide widely different sketches, making sketch-photo matching harder, (iii) mapping the sketches and photos into a common domain is also a challenging task. In this paper, we address the cross-domain question using a strategy of mapping sketches and natural photos in multiple layers. For the first time, we introduce a multi-layer deep CNNs framework to train the multi-layer representation of free hand sketches and natural photos. We use Flickr15k dataset as benchmark for the retrieval and show that our learned representation significantly outperformances both hand-crafted features as well as deep features trained by sketches or photos.