Prior semantic-embedding representation learning for on-the-fly FG-SBIR

Abstract

In fine-grained sketch-based image retrieval (FG-SBIR) framework, sketch drawing is a time-consuming and skill-intensive process that greatly restricts its practical application. Our study aims to enhance the early-stage retrieval efficiency by utilizing partial sketches with a minimum number of strokes in the on-the-fly FG-SBIR framework. Nevertheless, partial sketches with few strokes usually lack global information and consist only of sparse local details. This lack of detail results in ambiguous semantic content and indistinct differences among images, which consequently impairs early-stage retrieval. To acquire an efficient representation for these sketches, we propose a multi-granularity feature representation model that utilizes prior knowledge embedding to deal with the sketch drawing process. In the first stage,we design a triplet network to learn the joint a coarse representation embedding space that is common between the complete sketch and the image. In the second stage, We enhanced the semantic representation of sketches by utilizing prior knowledge, which aids in supplementing missing fine-grained features. This was achieved by formalizing the prior knowledge and integrating it into the feature vector obtained in the initial stage, thereby acquiring the final representation of a subset of sketches. Experiments conducted on two public datasets indicate that our method outperforms the state-of-the-art baseline during the early retrieval phase. In practical applications, our approach has also achieved good results.