SketchHelper: Real-Time Stroke Guidance for Freehand Sketch Retrieval

Abstract

Text-based retrieval systems have been popular, but content-based retrieval systems have gained widespread acceptance in recent years to directly retrieve diverse media based on their visual content, such as color, texture, and shape. Among many content-based retrieval systems, sketch-based media retrieval systems have attracted attention recently with the proliferation of tablet PCs and smart mobile devices. Sketch-based retrieval requires the user to draw a freehand sketch query, but freehand drawing can be challenging for those with limited drawing skills. This degrades retrieval performance, since successful retrieval depends on the quality of the sketch query image drawn by the user. To address this issue, we propose a real-time stroke guidance for freehand sketch retrieval that continuously displays next-stroke shadow sketches on the canvas based on the user's step-by-step partial strokes. We train a stroke guidance network that learns the mapping between the step-wise stroke relations to predict the user's next stroke. The proposed stroke guidance for freehand sketch retrieval system runs on a five step next-stroke prediction model that identifies candidate next-stroke sketches from a database of millions of sketches. The system retrieves variable number of sketch object classes at different drawing stages. During the initial sketching stage, diverse drawing possibilities are covered by retrieving multiple sketch classes; as the sketching progresses, the intended sketch class is narrowed down to one. Deep binary hashing is employed for efficient similarity matching of relevant next-stroke sketches. We extend the Google QuickDraw dataset to create a five step sketch stroke database. Qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed system, which can be utilized for drawing guidance, tracing, and sketch retrieval. Tracing refers to the act of copying the shadowed line of a guiding image by drawing over its lines.