Sketch-based 3D Model Research Articles

Sequential learning for sketch-based 3D model retrieval

Sequential learning for sketch-based 3D model retrieval

End-to-End Sketch-3D Model Retrieval with Spatiotemporal Information Joint Embedding

<p indent=0mm>The existing sketch-based 3D model retrieval methods often regard data as static input, and utilize convolutional neural network to extract features, which ignore the dynamic attributes of input data, resulting in partial loss of useful information and not ideal retrieval effect. In order to solve this problem, a spatiotemporal information joint embedding-based end-to-end sketch-3D model retrieval method is proposed. Firstly, the sketch is represented as a dynamic drawing sequence, which reflects the temporal information contained in the drawing process; meanwhile, the 3D model is represented as a multi-view sequence to reflect the position relationship between views. Secondly, an end-to-end dual-stream network including static spatial feature extraction and dynamic time series feature extraction is constructed. Combined with triplet central metric learning, a joint spatiotemporal feature embedding of cross domain data is established to fully capture the static and dynamic features contained in sketches and 3D models, and reduce the difference between cross domain. Finally, experiments are carried out on the standard public data sets SHREC2013 and SHREC2014. Compared with the existing work, the accuracy rate is higher, which verifies the feasibility and effectiveness of the proposed algorithm.

3D sketching for 3D object retrieval

Sketching provides the most natural way to provide a visual search query for visual object search. However, how to draw 3D sketches in a three-dimensional space and how to use a hand-drawn 3D sketch to search similar 3D models are not only interesting and novel, but also challenging research topics. In this paper, we try to answer them by initiating a novel study on 3D sketching and build a 3D sketching system which allows users to freely draw 3D sketches in the air and demonstrate its promising potentials in related applications such as collecting 3D sketch data and conducting 3D sketch-based 3D model retrieval. By utilizing the 3D sketching system, we collect a 3D sketch dataset, build a 3D sketch-based 3D model retrieval benchmark, and organize a Eurographics Shape Retrieval Contest (SHREC) track on 3D sketch-based shape retrieval based on the benchmark. We investigate 3D sketch and model matching problems and propose a novel 3D sketch-based model retrieval algorithm CNN-SBR based on Convolutional Neural Networks (CNNs) and achieve the best performance in the SHREC track. We wish that the 3D sketching system, the 3D sketch-based model retrieval benchmark, and the proposed 3D sketch-based model retrieval algorithm CNN-SBR will further promote sketch-based shape retrieval and its applications. We have made all of these publicly available on the project homepage: http://orca.st.usm.edu/~bli/SBR16/project.html .

Cross-Domain Correspondence for Sketch-Based 3D Model Retrieval Using Convolutional Neural Network and Manifold Ranking

Due to the huge difference in the representation of sketches and 3D models, sketch-based 3D model retrieval is a challenging problem in the areas of graphics and computer vision. Some state-of-the-art approaches usually extract features from 2D sketches and produce multiple projection views of 3D models, and then select one view of 3D models to match sketch. It's hard to find “the best view” and views from different perspectives of a 3D model may be completely different. Other methods apply learning features to retrieve 3D models based on 2D sketch. However, sketches are abstract images and are usually drawn subjectively. It is difficult to be learned accurately. To address these problems, we propose cross-domain correspondence method for sketch-based 3D model retrieval based on manifold ranking. Specifically, we first extract learning features of sketches and 3D models by two-parts CNN structures. Subsequently, we generate cross-domain undirected graphs using learning features and semantic labels to create correspondence between sketches and 3D models. Finally, the retrieval results are computed by manifold ranking. Experimental results on SHREC 13 and SHREC 14 datasets show the superior performance in all 7 standard metrics, compared to the state-of-the-art approaches.

A Novel Sketch-Based Medical 3D Model Retrieval Approach by Multi-Scale Weighted Gabor Feature Fusion and IP-HOG Feature Extraction

A Novel Sketch-Based Medical 3D Model Retrieval Approach by Multi-Scale Weighted Gabor Feature Fusion and IP-HOG Feature Extraction

Deep Common Semantic Space Embedding for Sketch-Based 3D Model Retrieval.

Sketch-based 3D model retrieval has become an important research topic in many applications, such as computer graphics and computer-aided design. Although sketches and 3D models have huge interdomain visual perception discrepancies, and sketches of the same object have remarkable intradomain visual perception diversity, the 3D models and sketches of the same class share common semantic content. Motivated by these findings, we propose a novel approach for sketch-based 3D model retrieval by constructing a deep common semantic space embedding using triplet network. First, a common data space is constructed by representing every 3D model as a group of views. Second, a common modality space is generated by translating views to sketches according to cross entropy evaluation. Third, a common semantic space embedding for two domains is learned based on a triplet network. Finally, based on the learned features of sketches and 3D models, four kinds of distance metrics between sketches and 3D models are designed, and sketch-based 3D model retrieval results are achieved. The experimental results using the Shape Retrieval Contest (SHREC) 2013 and SHREC 2014 datasets reveal the superiority of our proposed method over state-of-the-art methods.

A novel sketch-based 3D model retrieval approach based on skeleton

&lt;span&gt;Since the skeleton represents the topology structure of the query sketch and 2D views of 3D model, this paper proposes a novel sketch-based 3D model retrieval algorithm which utilizes skeleton characteristics as the features to describe the object shape. Firstly, we propose advanced skeleton strength map (ASSM) algorithm to create the skeleton which computes the skeleton strength map by isotropic diffusion on the gradient vector field, selects critical points from the skeleton strength map and connects them by Kruskal's algorithm. Then, we propose histogram feature comparison algorithm which adopts the radii of the disks at skeleton points and the lengths of skeleton branches to extract the histogram feature, and compare the similarity between two skeletons using the histogram feature matrix of skeleton endpoints. Experiment results demonstrate that our approach which combines these two algorithms significantly outperforms several leading sketch-based retrieval approaches.&lt;/span&gt;

Local features and manifold ranking coupled method for sketch-based 3D model retrieval

3D model retrieval can benefit many downstream virtual reality applications. In this paper, we propose a new sketch-based 3D model retrieval framework by coupling local features and manifold ranking. At technical fronts, we exploit spatial pyramids based local structures to facilitate the efficient construction of feature descriptors. Meanwhile, we propose an improved manifold ranking method, wherein all the categories between arbitrary model pairs will be taken into account. Since the smooth and detail-preserving line drawings of 3D model are important for sketch-based 3D model retrieval, the Difference of Gaussians (DoG) method is employed to extract the line drawings over the projected depth images of 3D model, and Bezier Curve is then adopted to further optimize the extracted line drawing. On that basis, we develop a 3D model retrieval engine to verify our method. We have conducted extensive experiments over various public benchmarks, and have made comprehensive comparisons with some state-of-the-art 3D retrieval methods. All the evaluation results based on the widely-used indicators prove the superiority of our method in accuracy, reliability, robustness, and versatility.

Sketch-Based 3D Model Retrieval Using Attributes

With the rapid growth of available 3D models on the Internet, how to retrieve 3D models based on hand-drawn sketch retrieval are becoming increasingly important. This article proposes a new sketch-based 3D model retrieval approach. This approach is different from current methods that make use of low-level visual features to capture the search intention of users. The proposed method uses two kinds of semantic attributes, including pre-defined attributes and latent attributes. Specifically, pre-defined attributes are defined manually which can provide prior knowledge about different sketch categories and latent-attributes are more discriminative which can differentiate sketch categories at a finer level. Therefore, these semantic attributes can provide a more descriptive and discriminative meaningful representation than low-level feature descriptors. The experiment results demonstrate that this proposed method can achieve superior performance over previously proposed sketch-based 3D model retrieval methods.

SVM: Sketch-based 3D Retrieval Application Using Classification Method

With the rapid development of 3D technology, the demand to use and retrieve the 3D model is becoming more and more urgent. In this case, it is more and more import and necessary that sketch-based 3D model retrieval. The user provides a hand-drawn sketch, then the system can be provided the possible list of models, then the user can select the need model from this list. In this paper, it’s proposed a framework of using SVM classification methods and K-means cluster algorithm. In preprocess part; the adaptive thinning algorithm is used to process the input sketch. In the offline part, the model is projected to multi-view images, and then SVM classifier has been used to classify these images. Moreover, we extract the images features, and use K-means algorithm to cluster these features. Finally, the feature dictionary will be acquired. In online process part, it only needs to extract the input sketch feature, and computes the distance between dictionary and their features. Besides, the experiment is realized to verify the feasibility of the approach. Finally, it was compared with other’s approaches; the result shows that the approach is viable and robustness.

A new sketch-based 3D model retrieval method by using composite features

With the rapid growth of available 3D models in various areas, effective methods to search 3D models are becoming increasingly important. In this paper, we propose a new method for sketch-based 3D model retrieval. Different from current methods that make use of either global or local features, the proposed method uses composite features combining global and local features extracted from representative 2D views of 3D models. The global features, shape strings, represent exterior boundary shape of the views and the local features, improved Pyramid of Histograms of Orientation Gradients (iPHOG), represent their interior details. Specifically, a global feature based filtering step is adopted to select more relevant candidate models to the query sketch and a local feature based process is used to refine chosen candidates. To evaluate the performance of the proposed method with that of other previous ones, we conducted a series of experiments on public standard 3D model databases. Experimental results are presented and indicate the effectiveness of the new approach for sketch-based 3D model retrieval.

Rapid three-dimensional scene modeling by sketch retrieval and auto-arrangement

The existing three-dimensional (3D) object layout methods are focused mainly on indoor scenes and they are limited for outdoor applications. In this study, we propose a data-driven method for outdoor scene modeling by using fast retrieval and automatic optimization layout techniques. Unlike the current methods, we first employ an improved manifold ranking algorithm in the sketch-based 3D model retrieval stage, which achieves higher accuracy. Next, according to the particular properties of outdoor architectures, specialized constraints are then proposed to define an energy function, which meets both the functional and aesthetic requirements. Finally, we cast the auto-arrangement as a combinatorial optimization problem, which we solve using an optimization algorithm. In contrast to the earlier version of this method, which was presented at Cyberworlds 2016, this extended version combines simulated annealing and particle swarm optimization algorithms, which have the advantages of rapid convergence and avoiding becoming trapped by local minima. Our experimental results demonstrate that the proposed method is more intuitive and effective for modeling 3D scenes, and can be employed in the actual development of game scenes.

Sketch-based 3D model retrieval utilizing adaptive view clustering and semantic information

Searching for relevant 3D models based on hand-drawn sketches is both intuitive and important for many applications, such as sketch-based 3D modeling and recognition, human computer interaction, 3D animation, game design, and etc. In this paper, our target is to significantly improve the current sketch-based 3D retrieval performance in terms of both accuracy and efficiency. We propose a new sketch-based 3D model retrieval framework by utilizing adaptive view clustering and semantic information. It first utilizes a proposed viewpoint entropy-based 3D information complexity measurement to guide adaptive view clustering of a 3D model to shortlist a set of representative sample views for 2D-3D comparison. To bridge the gap between the query sketches and the target models, we then incorporate a novel semantic sketch-based search approach to further improve the retrieval performance. Experimental results on several latest benchmarks have evidently demonstrated our significant improvement in retrieval performance.

View selection for sketch-based 3D model retrieval using visual part shape description

Hand drawings are imprints of shapes in human's mind. How a human expresses a shape is a consequence of how he or she visualizes it. A query-by-sketch 3D object retrieval application is closely tied to this concept from two aspects. First, describing sketches must involve elements in a figure that matter most to a human. Second, representative 2D projection of target 3D objects should be limited to the canonical from a human cognition perspective. We advocate for these two rules by presenting a new approach for sketch-based 3D object retrieval that describes a 2D shape by visual protruding parts of its silhouette. Furthermore, we present a list of candidate 2D projections that represent canonical views of a 3D object. The general rule is that humans would visually avoid part occlusion and symmetry. We quantify extent of part occlusion of projected silhouettes of 3D objects by skeletal length computations. Sorting projected views in decreasing order of skeletal lengths gives access to a subset of best representative views. We experimentally show how views that cause misinterpretation and mismatching can be detected according to part occlusion criteria. We also propose criteria for locating side, off axis, or asymmetric views.

A novel sketch-based 3D model retrieval method by integrating skeleton graph and contour feature

With the rapid growth of 3D models on the Web, effective methods to retrieve appropriate 3D models are becoming crucial. In this paper, we propose a novel sketch-based 3D model retrieval approach which integrates the skeleton graph and contour feature. Skeleton graph represents the topology structure of the query sketch as well as representative 2D views of 3D models, and contour feature captures their detailed information. Moreover, A hierarchical coarse-to-fine searching strategy is employed in this method. Firstly, a skeleton graph based matching method is used to eliminate those 3D models which are dissimilar to the query sketch. Then, the selected candidate models are refined by measuring the similarity of their contour feature. In addition, we adopt a graph transduction method to improve the ranking order of the retrieved 3D models. The proposed method was tested on the public sketch benchmarks and compared with other leading sketch-based 3D model retrieval methods. The experiment results demonstrate that our approach can accurately retrieve desired 3D models for users and has a superior performance to other state-of-the-art approaches.

A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries

Large-scale 3D shape retrieval has become an important research direction in content-based 3D shape retrieval. To promote this research area, two Shape Retrieval Contest (SHREC) tracks on large scale comprehensive and sketch-based 3D model retrieval have been organized by us in 2014. Both tracks were based on a unified large-scale benchmark that supports multimodal queries (3D models and sketches). This benchmark contains 13680 sketches and 8987 3D models, divided into 171 distinct classes. It was compiled to be a superset of existing benchmarks and presents a new challenge to retrieval methods as it comprises generic models as well as domain-specific model types. Twelve and six distinct 3D shape retrieval methods have competed with each other in these two contests, respectively. To measure and compare the performance of the participating and other promising Query-by-Model or Query-by-Sketch 3D shape retrieval methods and to solicit state-of-the-art approaches, we perform a more comprehensive comparison of twenty-six (eighteen originally participating algorithms and eight additional state-of-the-art or new) retrieval methods by evaluating them on the common benchmark. The benchmark, results, and evaluation tools are publicly available at our websites (http://www.itl.nist.gov/iad/vug/sharp/contest/2014/Generic3D/, 2014, http://www.itl.nist.gov/iad/vug/sharp/contest/2014/SBR/, 2014).

Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval

We study the problem of sketch-based 3D model retrieval, and propose a solution powered by a new query-to-model distance metric and a powerful feature descriptor based on the bag-of-features framework. The main idea of the proposed query-to-model distance metric is to represent a query sketch using a compact set of sample views (called basic views) of each model, and to rank the models in ascending order of the representation errors. To better differentiate between relevant and irrelevant models, the representation is constrained to be essentially a combination of basic views with similar viewpoints. In another aspect, we propose a mid-level descriptor (called BOF-JESC) which robustly characterizes the edge information within junction-centered patches, to extract the salient shape features from sketches or model views. The combination of the query-to-model distance metric and the BOF-JESC descriptor achieves effective results on two latest benchmark datasets.

A new sketch-based 3D model retrieval approach by using global and local features

With the rapid growth of available 3D models, fast retrieval of suitable 3D models has become a crucial task for industrial applications. This paper proposes a novel sketch-based 3D model retrieval approach which utilizes both global feature-based and local feature-based techniques. Unlike current approaches which use either global or local features, as well as do not take into account semantic relations between local features, we extract these two kinds of feature information from the representative 2D views of 3D models that can facilitate semantic description and retrieval for 3D models. Global features represent the gross exterior boundary shape information, and local features describe the interior details by compact visual words. Specifically, an improved bag-of-features method is provided to extract local features and their latent semantic relations. In addition, an efficient two-stage matching strategy is used to measure the distance between the query sketch and 3D models for selection and refinement. Experiment results demonstrate that our approach which combines these two kinds of complementary features significantly outperforms several state-of-the-art approaches.

Sketch-based 3D model retrieval by incorporating 2D-3D alignment

Sketch-based 3D model retrieval is very important for applications such as 3D modeling and recognition. In this paper, a sketch-based retrieval algorithm is proposed based on a 3D model feature named View Context and 2D relative shape context matching. To enhance the accuracy of 2D sketch-3D model correspondence as well as the retrieval performance, we propose to align a 3D model with a query 2D sketch before measuring their distance. First, we efficiently select some candidate views from a set of densely sampled views of the 3D model to align the sketch and the model based on their View Context similarities. Then, we compute the more accurate relative shape context distance between the sketch and every candidate view, and regard the minimum one as the sketch-model distance. To speed up retrieval, we precompute the View Context and relative shape context features of the sample views of all the 3D models in the database. Comparative and evaluative experiments based on hand-drawn and standard line drawing sketches demonstrate the effectiveness and robustness of our approach and it significantly outperforms several latest sketch-based retrieval algorithms.

Sketch-based 3D model retrieval using compressive sensing classification

A sketch-based 3D model retrieval methodology by using compressive sensing (CS) is presented. The approach to search and automatically return a set of 3D mesh models from a large database consists of histogram of oriented gradient feature extraction from projected images of 3D models, and CS based classification technique. Experimental results show that the proposed 3D model retrieval is very efficient to search for the 3D models from user-drawn sketches including the variation of shape, pose, and view-points.