Multi-granularity Association Learning for On-the-fly Fine-grained Sketch-based Image Retrieval

Abstract

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a specific photo from a given query sketch. However, its widespread applicability is limited because it is difficult for most people to draw a complete sketch, and the drawing process is often time consuming. In this study, we aim to retrieve the target photo from an partial sketch with the least number of strokes possible; the method is referred to as on-the-fly FG-SBIR (Bhunia et al., 2020), in which the retrieval begins after each stroke of the drawing. We consider that a significant correlation exists between these incomplete sketches in the sketch-drawing episodes of each photo. We propose a multi-granularity-association-learning method that further optimizes the embedding space of all incomplete sketches to learn an efficient joint-embedding space. Specifically, based on the integrity of the sketch, a complete sketch episode can be divided into several stages, each of which corresponds to a simple linear-mapping layer. Furthermore, our framework guides the vector space representation of the current sketch to approximate that with its later sketches. In this manner, the retrieval performance of a sketch with fewer strokes can approach that of a sketch with more strokes. We conducted experiments that included more realistic challenges, and our method achieved superior early-retrieval efficiency over the state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch-retrieval datasets.