Fine-grained Sketch-based Image Retrieval Research Articles

Prior semantic-embedding representation learning for on-the-fly FG-SBIR

In fine-grained sketch-based image retrieval (FG-SBIR) framework, sketch drawing is a time-consuming and skill-intensive process that greatly restricts its practical application. Our study aims to enhance the early-stage retrieval efficiency by utilizing partial sketches with a minimum number of strokes in the on-the-fly FG-SBIR framework. Nevertheless, partial sketches with few strokes usually lack global information and consist only of sparse local details. This lack of detail results in ambiguous semantic content and indistinct differences among images, which consequently impairs early-stage retrieval. To acquire an efficient representation for these sketches, we propose a multi-granularity feature representation model that utilizes prior knowledge embedding to deal with the sketch drawing process. In the first stage,we design a triplet network to learn the joint a coarse representation embedding space that is common between the complete sketch and the image. In the second stage, We enhanced the semantic representation of sketches by utilizing prior knowledge, which aids in supplementing missing fine-grained features. This was achieved by formalizing the prior knowledge and integrating it into the feature vector obtained in the initial stage, thereby acquiring the final representation of a subset of sketches. Experiments conducted on two public datasets indicate that our method outperforms the state-of-the-art baseline during the early retrieval phase. In practical applications, our approach has also achieved good results.

Bi-LSTM Sequence Modeling for On-The-Fly Fine-Grained Sketch-Based Image Retrieval

Bi-LSTM Sequence Modeling for On-The-Fly Fine-Grained Sketch-Based Image Retrieval

Expansion window local alignment weighted network for fine-grained sketch-based image retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) is a worthwhile task, which can be useful in many scenarios like recommendation systems, receiving a great deal of attention. In this study, we analyze challenges faced in FG-SBIR and propose a novel Expansion Window Local Alignment Weighted Network (EWLAW-Net). Specifically, it contains two main components: the Expansion Window Local Alignment module (EWLA) and the Local Weighted Fusion module (LWF). The EWLA module adopts an expansion window mechanism to align local features extracted from the backbone with the same semantic meaning between photos and sketches. The LWF module assigns weights to each local feature of the sketch after evaluating their importance and fuses them to calculate the similarity between the sketch and photos for retrieval. Experiments are conducted on five datasets and the results demonstrate the effectiveness of the proposed method.

Multi-feature fusion for fine-grained sketch-based image retrieval

Multi-feature fusion for fine-grained sketch-based image retrieval

DLI-Net: Dual Local Interaction Network for Fine-Grained Sketch-Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) is considered an ideal method of image retrieval due to the rich and easily accessible characteristics of sketches. It aims to find the most similar photo from the photo gallery based on the input sketch. Most previous works follow the paradigm that extracting global feature first and then projecting the features of sketch and photo to unified embedding feature space using triplet loss. However, the global feature is not appropriate for extracting the crucial fine-grained information. Based on this principle, we propose a Dual Local Interaction Network (DLI-Net). DLI-Net explores an effective and efficient way to utilize local features for FG-SBIR. Specifically, we first propose a Local Feature Extractor to extract mid-level local features. Then, in response to the problems brought by local features, we propose a Dual Interaction Module, which contains Self Interaction Module and Cross Interaction Module. Self Interaction Module speeds up retrieval by eliminating the redundant local features of background. Cross Interaction Module solves the spatial misalignment by making the sketches interact with photos. Extensive experiments on six commonly used datasets show that our DLI-Net outperforms state-of-the-art competitors by a significant margin with a reasonable retrieval speed. Moreover, to the best of our knowledge, DLI-Net is the first model that beats humans on all six datasets. Besides, DLI-Net also performs best on cross-category fine-grained sketch-based image retrieval task, which further demonstrates local features are more appropriate for FG-SBIR.

Multi-granularity Association Learning for On-the-fly Fine-grained Sketch-based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a specific photo from a given query sketch. However, its widespread applicability is limited because it is difficult for most people to draw a complete sketch, and the drawing process is often time consuming. In this study, we aim to retrieve the target photo from an partial sketch with the least number of strokes possible; the method is referred to as on-the-fly FG-SBIR (Bhunia et al., 2020), in which the retrieval begins after each stroke of the drawing. We consider that a significant correlation exists between these incomplete sketches in the sketch-drawing episodes of each photo. We propose a multi-granularity-association-learning method that further optimizes the embedding space of all incomplete sketches to learn an efficient joint-embedding space. Specifically, based on the integrity of the sketch, a complete sketch episode can be divided into several stages, each of which corresponds to a simple linear-mapping layer. Furthermore, our framework guides the vector space representation of the current sketch to approximate that with its later sketches. In this manner, the retrieval performance of a sketch with fewer strokes can approach that of a sketch with more strokes. We conducted experiments that included more realistic challenges, and our method achieved superior early-retrieval efficiency over the state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch-retrieval datasets.

Three-way enhanced part-aware network for fine-grained sketch-based image retrieval

Three-way enhanced part-aware network for fine-grained sketch-based image retrieval

A deformable CNN-based triplet model for fine-grained sketch-based image retrieval

With the popularity of electronic touch-screen and pressure sensing devices, fine-grained sketch based image retrieval (FG-SBIR) has become a research hotspot. In this paper, we stress the core problems of FG-SBIR: a. how to reduce the difference between the non-homogenous of heterogeneous media, and b. how to improve the distinguishability of sketch features. Specifically, a sketch generation model is first proposed to replace the conventional pre-processing of roughly extracting image edges, moreover, this model can alleviate the dilemma of sketch data scarcity. We then construct a novel FG-SBIR model which takes advantage of deformable convolutional neural network while taking into consideration of semantic attributes together. In addition, we build a fine-grained clothing sketch-image dataset, which has rich attribute annotations, for the first time. Extensive experiments exhibit that our proposed model achieves a better performance in improving the retrieval accuracy over the state-of-the-art baselines.

AE-Net: Fine-grained sketch-based image retrieval via attention-enhanced network

In this paper, we investigate the task of Fine-grained Sketch-based Image Retrieval (FG-SBIR), which uses hand-drawn sketches as input queries to retrieve the relevant images at the fine-grained instance level. The sketches and images come from different modalities, thus the similarity computation needs to consider both fine-grained and cross-modal characteristics. Existing solutions only focus on fine-grained details or spatial contexts, while ignoring the channel context and spatial sequence information. To mitigate such challenging problems, we propose a novel deep FG-SBIR model, which aims at inferring attention maps along channel dimension and spatial dimension, improving modules of channel attention and spatial attention, and exploring Transformer to enhance the model’s ability for constructing and understanding spatial sequence information. We focus not only on the correlation information between two modalities of sketch and image, but also on the discrimination information inside the single modality. Mutual Loss is especially proposed to enhance the traditional triplet loss, and promote the internal discrimination ability of the model on a single modality. Extensive experiments show that our AE-Net obtains promising results on Sketchy, which is the largest public dataset available for FG-SBIR at present.

Dark-Aware Network For Fine-Grained Sketch-Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) is an emerging topic in high-level computer vision. Among existing methods, edge-only information based ones are convenient to use but incompetent in distinguishing among ambiguous samples. On the other hand, even though the methods using additional color information can significantly improve the retrieval performance, they are not convenient for practical use as the increase of sketch complexity. This paper fills the gap between these two kinds of methods by proposing a novel method, namely dark-aware sketch-based image retrieval (DA-SBIR), which incorporates the dark region information of images into FG-SBIR without a sacrifice of convenience. Our model consists of two branches to process edge structure and dark region, respectively. Specifically, instead of using real color, DA-SBIR only requires some additional free-hand black sketch to represent the dark region. Besides, a Split Generative Adversarial Network is introduced to automatically split a sketch into edge-structure-only and dark-region-only sketches. Moreover, we build a new clothes dataset, SJTU-Cloth, with more ambiguous samples for FG-SBIR. Experimental results on the QMUL-Shoe dataset and our SJTU-Cloth dataset show that our approach achieves consistent improvements over state-of-the-arts. Code and dataset are available at https://github.com/y2242794082/SBIR.git.

Fine-Grained Instance-Level Sketch-Based Image Retrieval

The problem of fine-grained sketch-based image retrieval (FG-SBIR) is defined and investigated in this paper. In FG-SBIR, free-hand human sketch images are used as queries to retrieve photo images containing the same object instances. It is thus a cross-domain (sketch to photo) instance-level retrieval task. It is an extremely challenging problem because (i) visual comparisons and matching need to be executed under large domain gap, i.e., from black and white line drawing sketches to colour photos; (ii) it requires to capture the fine-grained (dis)similarities of sketches and photo images while free-hand sketches drawn by different people present different levels of deformation and expressive interpretation; and (iii) annotated cross-domain fine-grained SBIR datasets are scarce, challenging many state-of-the-art machine learning techniques, particularly those based on deep learning. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based object instance retrieval application. Specifically, a new large-scale FG-SBIR database is introduced which is carefully designed to reflect the real-world application scenarios. A deep cross-domain matching model is then formulated to solve the intrinsic drawing style variability, large domain gap issues, and capture instance-level discriminative features. It distinguishes itself by a carefully designed attention module. Extensive experiments on the new dataset demonstrate the effectiveness of the proposed model and validate the need for a rigorous definition of the FG-SBIR problem and collecting suitable datasets.

Sketch-based image retrieval via CAT loss with elastic net regularization

Fine-grained sketch-based image retrieval (FG-SBIR) is an important problem that uses free-hand human sketch as queries to perform instance-level retrieval of photos. Human sketches are generally highly abstract and iconic, which makes FG-SBIR a challenging task. Existing FG-SBIR approaches using triplet loss with \begin{document}$ \ell_2 $\end{document} regularization or higher-order energy function to conduct retrieval performance, which neglect the feature gap between different domains (sketches, photos) and need to select the weight layer matrix. This yields high computational complexity. In this paper, we define a new CAT loss function with elastic net regularization based on attention model. It can close the feature gap between different subnetworks and embody the sparsity of the sketches. Experiments demonstrate that the proposed approach is competitive with state-of-the-art methods.

Deep cascaded cross-modal correlation learning for fine-grained sketch-based image retrieval

Fine-grained Sketch-based Image Retrieval (FG-SBIR), which utilizes hand-drawn sketches to search the target object images, has recently drawn much attention. It is a challenging task because sketches and images belong to different modalities and sketches are highly abstract and ambiguous. Existing solutions to this problem either focus on visual comparisons between sketches and images and ignore the multimodal characteristics of annotated images, or treat the retrieval as a one-time process. In this paper, we formulate FG-SBIR as a coarse-to-fine process, and propose a Deep Cascaded Cross-modal Ranking Model (DCCRM) that can exploit all the beneficial multimodal information in sketches and annotated images and improve both the retrieval efficiency and the top-K ranked effectiveness. Our goal concentrates on constructing deep representations for sketches, images, and descriptions, and learning the optimized deep correlations across such different domains. Thus for a given query sketch, its relevant images with fine-grained instance-level similarities in a specific category can be returned, and the strict requirement of the instance-level retrieval for FG-SBIR is satisfied. Very positive results have been obtained in our experiments by using a large quantity of public data.

A survey on freehand sketch recognition and retrieval

With the development of digital devices and pressure sensing equipment, research into freehand sketches from touch-screen interfaces has increased significantly in recent years. As such, we provide the first comprehensive survey of recognition tasks based on sketch generation, freehand sketch classification, sketch-based image retrieval (SBIR), fine-grained sketch-based image retrieval (FG-SBIR), and sketch-based 3D shape image retrieval. Specifically, SBIR and FG-SBIR were the main focus of the survey. Primary technologies and benchmark datasets related to all sketch-based recognition topics are also discussed, along with future trends for this promising technology.

Gradually focused fine-grained sketch-based image retrieval.

This paper focuses on fine-grained image retrieval based on sketches. Sketches capture detailed information, but their highly abstract nature makes visual comparisons with images more difficult. In spite of the fact that the existing models take into account the fine-grained details, they can not accurately highlight the distinctive local features and ignore the correlation between features. To solve this problem, we design a gradually focused bilinear attention model to extract detailed information more effectively. Specifically, the attention model is to accurately focus on representative local positions, and then use the weighted bilinear coding to find more discriminative feature representations. Finally, the global triplet loss function is used to avoid oversampling or undersampling. The experimental results show that the proposed method outperforms the state-of-the-art sketch-based image retrieval methods.

Toward Deep Universal Sketch Perceptual Grouper.

Human free-hand sketches provide the useful data for studying human perceptual grouping, where the grouping principles such as the Gestalt laws of grouping are naturally in play during both the perception and sketching stages. In this paper, we make the first attempt to develop a universal sketch perceptual grouper. That is, a grouper that can be applied to sketches of any category created with any drawing style and ability, to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to achieving this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping dataset to date, consisting of 20 000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep perceptual grouping model learned with both generative and discriminative losses. The generative loss improves the generalization ability of the model, while the discriminative loss guarantees both local and global grouping consistency. Extensive experiments demonstrate that the proposed grouper significantly outperforms the state-of-the-art competitors. In addition, we show that our grouper is useful for a number of sketch analysis tasks, including sketch semantic segmentation, synthesis, and fine-grained sketch-based image retrieval.

Better freehand sketch synthesis for sketch-based image retrieval: Beyond image edges

Abstract With the rapid development of electronic touch screen and pressure-sensing devices, research on freehand sketches has become a hotspot in recent years. In this paper, we first propose a new freehand sketch generation model (FHS-GAN), which is based on the deep architecture of dual generative adversarial nets (GANs). We construct a model which utilize the deep convolutional neural network (CNN) and GAN to produce freehand sketches. We then propose an improved deep CNN model as a validated network, which is based on Faster R-CNN, to measure the similarity of real sketches and generated freehand sketches by FHS-GAN, and we test the improved model using the produced sketches with two large sketch datasets. The experiments show that the proposed FHS-GAN framework achieves state-of-the-art results in comparison with other baseline models. Furthermore, the generated sketches can be used for other sketch recognition tasks, such as in a pre-processing step for application in sketch-based image retrieval (SBIR) and fine-grained sketch-based image retrieval (FG-SBIR). Overall, our FHS-GAN model is important for the development of freehand sketches.

A survey of sketch-based image retrieval

Sketch-based image retrieval (SBIR) has been studied since the early 1990s and has drawn more and more interest recently. Yet, a comprehensive review of the SBIR field is still absent. This survey tries to fill in this gap by reviewing the representative papers studying the SBIR problem. More importantly, this survey tries to answer two important questions which are generally not well discussed: what are the objectives of SBIR, and what is the general methodology of SBIR? The reviewed papers are organized in a chronological way and analyzed by answering these two important questions. As a novel trend, fine-grained SBIR has become the main topic for the recent research. The discussion on it is also integrated. From this survey, we hope that different perspectives can be observed, common values can be discovered and new ideas can be inspired.

Cross-modal subspace learning for fine-grained sketch-based image retrieval

Sketch-based image retrieval (SBIR) is challenging due to the inherent domain-gap between sketch and photo. Compared with pixel-perfect depictions of photos, sketches are iconic renderings of the real world with highly abstract. Therefore, matching sketch and photo directly using low-level visual clues are insufficient, since a common low-level subspace that traverses semantically across the two modalities is non-trivial to establish. Most existing SBIR studies do not directly tackle this cross-modal problem. This naturally motivates us to explore the effectiveness of cross-modal retrieval methods in SBIR, which have been applied in the image-text matching successfully. In this paper, we introduce and compare a series of state-of-the-art cross-modal subspace learning methods and benchmark them on two recently released fine-grained SBIR datasets. Through thorough examination of the experimental results, we have demonstrated that the subspace learning can effectively model the sketch-photo domain-gap. In addition we draw a few key insights to drive future research. © 2017 Elsevier B.V.

Synergistic Instance-Level Subspace Alignment for Fine-Grained Sketch-Based Image Retrieval

We study the problem of fine-grained sketch-based image retrieval. By performing instance-level (rather than category-level) retrieval, it embodies a timely and practical application, particularly with the ubiquitous availability of touchscreens. Three factors contribute to the challenging nature of the problem: 1) free-hand sketches are inherently abstract and iconic, making visual comparisons with photos difficult; 2) sketches and photos are in two different visual domains, i.e., black and white lines versus color pixels; and 3) fine-grained distinctions are especially challenging when executed across domain and abstraction-level. To address these challenges, we propose to bridge the image-sketch gap both at the high level via parts and attributes, as well as at the low level via introducing a new domain alignment method. More specifically, first, we contribute a data set with 304 photos and 912 sketches, where each sketch and image is annotated with its semantic parts and associated part-level attributes. With the help of this data set, second, we investigate how strongly supervised deformable part-based models can be learned that subsequently enable automatic detection of part-level attributes, and provide pose-aligned sketch-image comparisons. To reduce the sketch-image gap when comparing low-level features, third, we also propose a novel method for instance-level domain-alignment that exploits both subspace and instance-level cues to better align the domains. Finally, fourth, these are combined in a matching framework integrating aligned low-level features, mid-level geometric structure, and high-level semantic attributes. Extensive experiments conducted on our new data set demonstrate effectiveness of the proposed method.