Sketch-based 3D Shape Retrieval Research Articles

D2GL: Dual-level dual-scale graph learning for sketch-based 3D shape retrieval

Sketch-based 3D shape retrieval (SBSR) is an active research area in the computer vision community, but it is still very challenging. One main reason is that existing deep learning-based methods usually treat sketches as 2D images, neglecting the sparsity and diversity. In this paper, we propose a novel Dual-level Dual-scale Graph Learning (D2GL) method to effectively enhance structural information and produce robust representations for sparse and diverse hand-drawn sketches. Specifically, in addition to the traditional branches for SBSR, we introduce a Dual-level Dual-scale Graph Self-attention (DLDS-GSA) as an auxiliary branch. DLDS-GSA further consists of two levels of encoders, i.e., a local structural encoder and a dual-scale global structural encoder, to capture both local discriminative and multi-scale global structures while minimizing the impact of various sketch drawing details. Comprehensive experiments on SHREC’13 and SHREC’14 datasets demonstrate the superiority of D2GL for SBSR, with extended experiments on PART-SHREC’14 confirming its generalization for unseen classes in SBSR.

Global semantics correlation transmitting and learning for sketch-based cross-domain visual retrieval

Sketch-based cross-domain visual data retrieval is the process of searching for images or 3D models using sketches as input. Achieving feature alignment is a significantly challenging task due to the high heterogeneity of cross-domain data. However, the alignment process faces significant challenges, such as domain gap, semantic gap, and knowledge gap. The existing methods adopt different ideas for sketch-based image and 3D shape retrieval tasks, one is domain alignment, and the other is semantic alignment. Technically, both tasks verify the accuracy of extracted features. Hence, we propose a method based on the global feature correlation and the feature similarity for multiple sketch-based cross-domain retrieval tasks. Specifically, the data from various modalities are fed into separate feature extractors to generate original features. Then, these features are projected to the shared subspace. Finally, domain consistency learning, semantic consistency learning, feature correlation learning and feature similarity learning are performed jointly to make the projected features modality-invariance. We evaluate our method on multiple benchmark datasets. Where the MAP in Sketchy, TU-Berlin, SHREC 2013 and SHREC 2014 are 0.466, 0.473, 0.860 and 0.816. The extensive experimental results demonstrate the superiority and generalization of the proposed method, compared to the state-of-the-art approaches. The in-depth analyses of various design choices are also provided to gain insight into the effectiveness of the proposed method. The outcomes of this research contribute to advancing the field of sketch-based cross-domain visual data retrieval and are expected to be applied to a variety of applications that require efficient retrieval of cross-domain domain data.

Retrieval-Specific View Learning for Sketch-to-Shape Retrieval

Sketch-based 3D shape retrieval (SBSR) can be approached by learning domain-invariant descriptors or ranking metrics from sketches and 2D view images of 3D shapes rendered through numerous viewpoints. However, determining the most appropriate viewpoints that convey discriminative geometric features to benefit the task of SBSR became an essential yet not fully explored area. Existing works extract 3D features from multi-view images observed through pre-defined viewpoints to match 2D sketches. Those methods, however, fail to dynamically select viewpoints by considering the SBSR task. In this work, we introduce a fully differentiable viewpoint learning paradigm driven by the downstream SBSR task, which supports the task-aware and sketch-dependent dynamic viewpoint determination process. We naturally integrate this task-specific and sketch-dependent viewpoint learning process into a meta-learning framework to develop a novel Dynamic Viewer (DV) module for SBSR. DV module comprises a Meta View Learner (MVL) block and a View Generator (VG) block. Specifically, as the first part of the DV module, the MVL block learns to initiate the necessary network parameters of the VG block. Then, the VG block that serves as the second part learns the best viewpoints to render 2D images. To learn the optimal viewpoints for SBSR, we further introduce a view mining loss that aims to maximize the similarity of feature-level information among rendered 2D views and the query sketch. Further, we adopt a variational autoencoder (VAE) to retrieve 3D shapes by setting the newly rendered images and query sketch as inputs. As evidenced by the comprehensive experimental results conducted on popular SBSR datasets, the proposed framework has been demonstrated to outperform recent methods in both category-level sketch-based and fine-grained SBSR.

Sketch-based 3D shape retrieval via teacher–student learning

One of the main difficulties of sketch-based 3D shape retrieval is the significant cross-modal difference between 2D sketches and 3D shapes. Most previous works adopt one-stage methods to directly learn the aligned common embedding space of sketches and shapes by a shared classifier. However, the intra-class difference of the sketch is more significant than the shape, harming the feature learning of 3D shapes when the two modalities are considered under the shared classifier. This issue harms the discrimination of the learned common embedding space. This paper proposes a novel two-stage method to learn a common aligned embedding space via teacher–student learning to address the issue. Specifically, we first employ a classification network to learn the discriminative features of shapes. The learned shape features are considered a teacher to guide the feature learning of sketches. Moreover, we design a guidance loss to achieve the feature transfer with semantic alignment. The proposed method achieves an effective, aligned cross-modal embedding space. Experiments on three public benchmark datasets prove the superiority of the proposed method over state-of-the-art methods.

Spatially aligned sketch-based fine-grained 3D shape retrieval

Spatially aligned sketch-based fine-grained 3D shape retrieval

HDA[formula omitted]L: Hierarchical Domain-Augmented Adaptive Learning for sketch-based 3D shape retrieval

The sketch-based 3D shape retrieval has been an active but challenging task for several decades. In this paper, we deeply analyze the challenges and propose a novel Hierarchical Domain-Augmented Adaptive Learning (HDA2L) for sketch-based 3D shape retrieval. The first notable challenge is the vast cross-modality discrepancies between sketches and 3D shapes. To this issue, the existing methods restrict the consistency of the final features by establishing a shared cross-domain loss but ignore the feature extraction process, resulting in a limited effect. We argue that the mutual information of the samples from the same class but different domains can provide an important cue to enhance common features captured in the feature extraction process. Thus, we design an Inter-Domain Augmented Network (Inter-DAN) by employing inter-domain feature correlation learning to capture cross-domain mutual information to learn augmented common global features for both sketches and 3D shapes. Another challenge is that the input sketch is various: it may be particularly abstract and contains only the overall outline of the target model, or it may be too sketchy and only contains some salient local regions of the target model. Though existing methods have demonstrated their capability to capture overall features, they always ignore the learning of local discriminative features and fail to adapt to the various changes of input sketches. To address this issue, we design an Intra-Domain Augmented Network (Intra-DAN) for sketches and 3D shapes, respectively, which learns augmented local discriminative features by adopting cascading cross-layer bilinear pooling operations. In addition, we design a Source-Agnostic Adversarial Network (SAAN) to accomplish the adaptive hierarchical domain features fusion, which forces the network to adaptively focus on more discriminative information from global features and local features and further adapt the diversity of the input sketches. The experiments on three benchmark datasets demonstrate that our method obtains superior retrieval performance than the state-of-the-art sketch-based 3D shape retrieval approaches.

SHREC’22 track: Sketch-based 3D shape retrieval in the wild

Sketch-based 3D shape retrieval (SBSR) is an important yet challenging task, which has drawn more and more attention in recent years. Existing approaches address the problem in a restricted setting, without appropriately simulating real application scenarios. To mimic the realistic setting, in this track, we adopt large-scale sketches drawn by amateurs of different levels of drawing skills, as well as a variety of 3D shapes including not only CAD models but also models scanned from real objects. We define two SBSR tasks and construct two benchmarks consisting of more than 46,000 CAD models, 1700 realistic models, and 145,000 sketches in total. Four teams participated in this track and submitted 15 runs for the two tasks, evaluated by 7 commonly-adopted metrics. We hope that, the benchmarks, the comparative results, and the open-sourced evaluation code will foster future research in this direction among the 3D object retrieval community.

Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval

Sketch-based 3D shape retrieval is a challenging task due to the large domain discrepancy between sketches and 3D shapes. Since existing methods are trained and evaluated on the same categories, they cannot effectively recognize the categories that have not been used during training. In this paper, we propose a novel domain disentangled generative adversarial network (DD-GAN) for zero-shot sketch-based 3D retrieval, which can retrieve the unseen categories that are not accessed during training. Specifically, we first generate domain-invariant features and domain-specific features by disentangling the learned features of sketches and 3D shapes, where the domain-invariant features are used to align with the corresponding word embeddings. Then, we develop a generative adversarial network that combines the domain-specific features of the seen categories with the aligned domain-invariant features to synthesize samples, where the synthesized samples of the unseen categories are generated by using the corresponding word embeddings. Finally, we use the synthesized samples of the unseen categories combined with the real samples of the seen categories to train the network for retrieval, so that the unseen categories can be recognized. In order to reduce the domain shift problem, we utilize unlabeled unseen samples to enhance the discrimination ability of the discriminator. With the discriminator distinguishing the generated samples from the unlabeled unseen samples, the generator can generate more realistic unseen samples. Extensive experiments on the SHREC'13 and SHREC'14 datasets show that our method significantly improves the retrieval performance of the unseen categories.

Hash Self-Attention End-to-End Network for Sketch-Based 3D Shape Retrieval

Hash Self-Attention End-to-End Network for Sketch-Based 3D Shape Retrieval

Sketch-based 3D Shape Retrieval Using Similarity Weighting between Multi-View

3D shape retrieval has always been a hot research topic in the field of computer vision, and the research goal is to perform fast and efficient retrieval to obtain 3D shapes that meet user needs. With the rapid development and popularization of touch screen devices, hand-drawn sketches have undoubtedly become the most convenient and user-friendly input form. However, the huge difference between the 3D shape and the 2D sketch is the main challenge that affects retrieval performance. In this paper, we propose a method of adding a sketch and view feature similarity comparison module during the training process to obtain the scores for the final feature descriptors under the premise of feature extraction of the 3D shape based on multi-view. Specifically, we render the 3D shape into 2D views from multiple different perspectives to represent the shape. Perform feature extraction on two types of inputs through two different networks, and design a similarity weighting module to calculate the scores of each view, so as to obtain the final descriptors. Finally, a final descriptor similarity metric network is trained based on contrastive loss. The experimental results on SHREC’13 dataset demonstrate the superiority and robustness of our method.

Uncertainty Learning for Noise Resistant Sketch-Based 3D Shape Retrieval.

Recently, sketch-based 3D shape retrieval has received growing attention in the community of computer graphics and computer vision. Most previous works focus on the problem of how to reduce the large cross-modality difference between 2D sketch and 3D shape data and make significant progress. Nevertheless, little attention has been paid to another important problem of how to deal with noise in the sketch data. For the first time, this work investigates the problem of noisy sketch data. It firstly provides qualitative and insightful analysis on the impact of noise, revealing that the noisy data are a key factor for unsatisfactory retrieval performance, as they cause severe over fitting and impair feature learning. Thus, the issue is worthy of serious treatment. Then, we propose to estimate sketch noise as data uncertainty, motivated by existing ideas that model data uncertainty with a distributional representation. We present methods with simple network structure and loss functions. They achieve strong results and establish new state-of-the-art on two benchmarks. Comprehensive experiment results, ablation studies, and insightful analysis validate the effectiveness of our methods, revealing that sketch feature learning with uncertainty is crucial for noise resistant sketch based 3D shape retrieval.

Toward Fine-Grained Sketch-Based 3D Shape Retrieval.

In this paper we study, for the first time, the problem of fine-grained sketch-based 3D shape retrieval. We advocate the use of sketches as a fine-grained input modality to retrieve 3D shapes at instance-level - e.g., given a sketch of a chair, we set out to retrieve a specific chair from a gallery of all chairs. Fine-grained sketch-based 3D shape retrieval (FG-SBSR) has not been possible till now due to a lack of datasets that exhibit one-to-one sketch-3D correspondences. The first key contribution of this paper is two new datasets, consisting a total of 4,680 sketch-3D pairings from two object categories. Even with the datasets, FG-SBSR is still highly challenging because (i) the inherent domain gap between 2D sketch and 3D shape is large, and (ii) retrieval needs to be conducted at the instance level instead of the coarse category level matching as in traditional SBSR. Thus, the second contribution of the paper is the first cross-modal deep embedding model for FG-SBSR, which specifically tackles the unique challenges presented by this new problem. Core to the deep embedding model is a novel cross-modal view attention module which automatically computes the optimal combination of 2D projections of a 3D shape given a query sketch.

Editorial issue 31.2

This issue contains three papers. In the first paper, M. Hassaballah, Abdelraheem M. Aly, and Ahmad Abdelnaim, from South Valley University Qena, Egypt, propose for the Interactive simulations of fluids flow to obtain the stable pressure from modifications in the source term of pressure Poisson equation (PPE). The particles disorders are solved using a shifting technique with the current treatment of source term in PPE. In addition, the dummy boundary particles are used for the rigid boundary treatment. Performance of the stabilized ISPH model is tested on various numerical simulations with largely distorted free surface including liquid sloshing problems, fluid–fluid, fluid–structure interactions and dam-break over horizontal and inclined plane. To extend the applicability of the stabilized ISPH model, the postprocess including visual realism with the highly rendering scheme is coupled. In the second paper, Xianxuan Lin, Wang Xun, Jinyu Li, Bailin Yang, Kaili Zhang, Tianxiang Wei, from Zhejiang Gongshang University in Hangzhou, China, and Frederick W. B. Li, from Durham University, UK, propose a color migration framework for home scene images. It picks the coloring from a template image and transforms such coloring to a home scene image through a simple interaction. Their framework comprises three main parts. First, they carry out an interactive segmentation to divide an image into local regions and extract their corresponding colors. Second, they generate a matching color table by sampling the template image according to the color structure of the original home scene image. Finally, they transform colors from the matching color table to the target home scene image with boundary transition maintained. Experiment results demonstrated their framework can produce aesthetically attractive home scene coloring. In the last paper, Xiaoyu Cui, Ruifan Cai, Xiangjun Tang, Xiaogang Jin, from Zhejiang University, in Hangzhou, China, and Zhigang Deng, from University of Houston, Texas, United States, propose a novel shape-constrained fireworks simulation method with rich textures in a helmet-mounted display virtual environment using sketched feature lines as input. Their approach first retrieves an object from a three-dimensional (3D) model database using a sketch-based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, they introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, they introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through their experiments, they demonstrate that their approach can produce better results than state-of-the-art approaches.

Deep point-to-subspace metric learning for sketch-based 3D shape retrieval

One key issue in managing a large scale 3D shape dataset is to identify an effective way to retrieve a shape-of-interest. The sketch-based query, which enjoys the flexibility in representing the user’s intention, has received growing interests in recent years due to the popularization of the touchscreen technology. Essentially, the sketch depicts an abstraction of a shape in a certain view while the shape contains the full 3D information. Matching between them is a cross-modality retrieval problem, and the state-of-the-art solution is to project the sketch and the 3D shape into a common space with which the cross-modality similarity can be calculated by the feature similarity/distance within. However, for a given query, only part of the viewpoints of the 3D shape is representative. Thus, blindly projecting a 3D shape into a feature vector without considering what is the query will inevitably bring query-unrepresentative information. To handle this issue, in this work we propose a Deep Point-to-Subspace Metric Learning (DPSML) framework to project a sketch into a feature vector and a 3D shape into a subspace spanned by a few selected basis feature vectors. The similarity between them is defined as the distance between the query feature vector and its closest point in the subspace by solving an optimization problem on the fly. Note that, the closest point is query-adaptive and can reflect the viewpoint information that is representative to the given query. To efficiently learn such a deep model, we formulate it as a classification problem with a special classifier design. To reduce the redundancy of 3D shapes, we also introduce a Representative-View Selection (RVS) module to select the most representative views of a 3D shape. By conducting extensive experiments on various datasets, we show that the proposed method can achieve superior performance over its competitive baseline methods and attain the state-of-the-art performance.

Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval.

How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Deep Correlated Metric Learning for Sketch-based 3D Shape Retrieval

The explosive growth of 3D models has led to the pressing demand for an efficient searching system. Traditional model-based search is usually not convenient, since people don't always have 3D model available by side. The sketch-based 3D shape retrieval is a promising candidate due to its simpleness and efficiency. The main challenge for sketch-based 3D shape retrieval is the discrepancy across different domains. In the paper, we propose a novel deep correlated metric learning (DCML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCML trains two distinct deep neural networks (one for each domain) jointly with one loss, which learns two deep nonlinear transformations to map features from both domains into a nonlinear feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the transfered space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features at least a predefined margin within each domain, while the correlation loss focuses on minimizing the distribution discrepancy across different domains. Our proposed method is evaluated on SHREC 2013 and 2014 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval

Sketch-based 3D shape retrieval, which returns a set of relevant 3D shapes based on users' input sketch queries, has been receiving increasing attentions in both graphics community and vision community. In this work, we address the sketch-based 3D shape retrieval problem with a novel Cross-Domain Neural Networks (CDNN) approach, which is further extended to Pyramid Cross-Domain Neural Networks (PCDNN) by cooperating with a hierarchical structure. In order to alleviate the discrepancies between sketch features and 3D shape features, a neural network pair that forces identical representations at the target layer for instances of the same class is trained for sketches and 3D shapes respectively. By constructing cross-domain neural networks at multiple pyramid levels, a many-to-one relationship is established between a 3D shape feature and sketch features extracted from different scales. We evaluate the effectiveness of both CDNN and PCDNN approach on the extended large-scale SHREC 2014 benchmark and compare with some other well established methods. Experimental results suggest that both CDNN and PCDNN can outperform state-of-the-art performance, where PCDNN can further improve CDNN when employing a hierarchical structure.

A comparison of methods for sketch-based 3D shape retrieval

Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. To foster this research area, two Shape Retrieval Contest (SHREC) tracks on this topic have been organized by us in 2012 and 2013 based on a small-scale and large-scale benchmarks, respectively. Six and five (nine in total) distinct sketch-based 3D shape retrieval methods have competed each other in these two contests, respectively. To measure and compare the performance of the top participating and other existing promising sketch-based 3D shape retrieval methods and solicit the state-of-the-art approaches, we perform a more comprehensive comparison of fifteen best (four top participating algorithms and eleven additional state-of-the-art methods) retrieval methods by completing the evaluation of each method on both benchmarks. The benchmarks, results, and evaluation tools for the two tracks are publicly available on our websites [1,2].