Input Sketch Research Articles

Sketch2CAD

We present a sketch-based CAD modeling system, where users create objects incrementally by sketching the desired shape edits, which our system automatically translates to CAD operations. Our approach is motivated by the close similarities between the steps industrial designers follow to draw 3D shapes, and the operations CAD modeling systems offer to create similar shapes. To overcome the strong ambiguity with parsing 2D sketches, we observe that in a sketching sequence, each step makes sense and can be interpreted in the context of what has been drawn before. In our system, this context corresponds to a partial CAD model, inferred in the previous steps, which we feed along with the input sketch to a deep neural network in charge of interpreting how the model should be modified by that sketch. Our deep network architecture then recognizes the intended CAD operation and segments the sketch accordingly, such that a subsequent optimization estimates the parameters of the operation that best fit the segmented sketch strokes. Since there exists no datasets of paired sketching and CAD modeling sequences, we train our system by generating synthetic sequences of CAD operations that we render as line drawings. We present a proof of concept realization of our algorithm supporting four frequently used CAD operations. Using our system, participants are able to quickly model a large and diverse set of objects, demonstrating Sketch2CAD to be an alternate way of interacting with current CAD modeling systems.

DeepFaceDrawing

Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch. We take a local-to-global approach. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

Kinetic Sculpture Design Using the Dynamic Cage

Kinetic sculptures are attractive for periodic dynamic deformation effects. Designing such a sculpture is not easy considering the holistic shape versus many part details. To provide an easier design process, we propose a top-down framework for kinetic sculpture design. The system converts several input sketch curves to a dynamic cage, which shows the deformation of the sculpture’s holistic shape in motion. The user can manipulate the sculpture’s shape by dragging on the cage at any moment in motion. After that, a group of skeletons is generated from the cage. The user can adjust their shapes for detailed unit model generation. Experimental results show that with our system, designers can concentrate on the shape and motion effects, significantly promoting the orientation and speed of design.

Computational LEGO technic design

We introduce a method to automatically compute LEGO Technic models from user input sketches, optionally with motion annotations. The generated models resemble the input sketches with coherently-connected bricks and simple layouts, while respecting the intended symmetry and mechanical properties expressed in the inputs. This complex computational assembly problem involves an immense search space, and a much richer brick set and connection mechanisms than regular LEGO. To address it, we first comprehensively model the brick properties and connection mechanisms, then formulate the construction requirements into an objective function, accounting for faithfulness to input sketch, model simplicity, and structural integrity. Next, we model the problem as a sketch cover, where we iteratively refine a random initial layout to cover the input sketch, while guided by the objective. At last, we provide a working system to analyze the balance, stress, and assemblability of the generated model. To evaluate our method, we compared it with four baselines and professional designs by a LEGO expert, demonstrating the superiority of our automatic designs. Also, we recruited several users to try our system, employed it to create models of varying forms and complexities, and physically built most of them.

IMPLEMENTING SKETCH-BASED IMAGE RETRIEVAL SYSTEM USING EDGE DETECTION AND SIFT ALGORITHM

Throughout the past few years, several researchers have introduced various methods and various algorithms for a precise and dependable sketch-based image retrieval system. In this paper, a proposed sketch-based image retrieval system is introduced. The framework goes over two phases: creating the sketch dataset phase and implementing SIFT (scale invariant feature transform) algorithm. The sketch dataset was created by selecting 100 colored image passed through canny edge detection operator. The system tends to enter a line-based/hand-drawing sketch and applies the SIFT algorithm to match between the input sketch and all sketches in the dataset. SIFT is one of the main efficient algorithms that are used to make description and matching, since it works on large keypoints. This system retrieves images depending on sketch image, and the result of matching will retrieve images that are approximate the entered sketch. The proposed system is assessed according to the measures that are utilized in detection, description and matching grounds, which are precision, recall and accuracy measures. The system showed (96 %) accuracy for line based sketches and (84%) for hand drawing where the detection was identical.

3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction

A framework is proposed for facilitating the exploration process during the early design phase through computational sketch synthesis and interactive 3D reconstruction. In that phase, designers concentrate on developing concepts through numerous alternatives. Therefore, they constantly sketch so that they can rapidly visualize their ideas. Recently, the design industry has attempted to streamline the design process by implementing 3D model generation in the early design phase so that ideas may be more thoroughly explored, thus improving concept and final design conformance; however, efficiency issues have arisen. In this study, a 3D computational sketch synthesis framework was developed comprising two major components. First, a robust method was proposed to synthesize design alternatives by interpolating an input sketch with sketches in a database so that unvisited combinations may be explored. Secondly, a novel interactive 3D model reconstruction method was developed to facilitate the shape transition of design elements so that designers can quickly evaluate the potential of a large number of design variations. Finally, an interface for design refinement was developed so that designs may be embodied by sketching over the 3D model. To test the proposed methodology, expert designers were recruited for a validation experiment with two conditions followed up by in-depth interviews. In the first condition, the participants were asked to sketch based on a design brief in their current working manner. In the second condition, they were asked to create designs using the proposed framework. It was tested whether there was a difference in the design outcomes. It was demonstrated that the proposed framework resulted in more satisfactory and higher-quality designs and generated design alternatives faster and in greater quantities. All participants agreed that the framework could be useful in the early design phase and responded that the proposed system provides more design inspiration than traditional design methods. Most importantly, it was demonstrated that the proposed framework could enhance the reevaluation potential of design concepts and assist in making better-informed design decisions.

Computational LEGO technic design

We introduce a method to automatically compute LEGO Technic models from user input sketches, optionally with motion annotations. The generated models resemble the input sketches with coherently-connected bricks and simple layouts, while respecting the intended symmetry and mechanical properties expressed in the inputs. This complex computational assembly problem involves an immense search space, and a much richer brick set and connection mechanisms than regular LEGO. To address it, we first comprehensively model the brick properties and connection mechanisms, then formulate the construction requirements into an objective function, accounting for faithfulness to input sketch, model simplicity, and structural integrity. Next, we model the problem as a sketch cover, where we iteratively refine a random initial layout to cover the input sketch, while guided by the objective. At last, we provide a working system to analyze the balance, stress, and assemblability of the generated model. To evaluate our method, we compared it with four baselines and professional designs by a LEGO expert, demonstrating the superiority of our automatic designs. Also, we recruited several users to try our system, employed it to create models of varying forms and complexities, and physically built most of them.

Component-based face sketch recognition using an enhanced evolutionary optimizer

The main aim of this work is to develop a component-based face sketch recognition model. The proposed model adopts an enhanced evolutionary optimizer (EEO) to perform the task of face sketched components localization. EEO is applied to an unknown input sketch to make an automatic localization for its components i.e. eyes, nose, and mouth. After that, HOG features are extracted, and cosine similarity measure is computed to find the best components location. EEO integrates Q-learning algorithm with the simulated annealing (SA) algorithm as a single mode. The Q-learning algorithm is used to control the execution of SA parameters i.e. temperature and the mutation rate at run time. The proposed approach was evaluated on three face sketch recognition benchmark problem which are LFW, AR, and CUHK. The experimental results show that EEO significantly outperform SA as well as other well-known meta-heuristic optimization algorithms such as PSO, Harmony, and MVO.

A Sketch Recognition Algorithm Based on Bayesian Network and Convolution Neural Network

Most of the existing sketch recognition algorithms are used to restrict the user’s drawing habits to achieve the stroke grouping and recognition. In order to solve the problem, a new sketch recognition algorithm based on Bayesian network and convolution neural network (CNN) is proposed. First, the input sketch is processed by Gaussian low-pass filter and a smoother stroke can be obtained. The stroke of continuous input is divided, then the Bayesian network and CNN are performed on stroke recognition respectively. The recognition result of Bayesian network is adopted when the reliability of stroke is larger than the threshold, otherwise recognition result of CNN will be adopted. The experiment result shows that the proposed algorithm is effective in circuit symbol recognition. The recognition rate was achieved 80.34% in the drawing process, and the final recognition rate was achieved 93.48%.

Research on Image Generation and Style Transfer Algorithm Based on Deep Learning

Aiming at the current process of artistic creation and animation creation, there are a lot of repeated manual operations in the process of conversion from sketch to the stylized image. This paper presented a solution based on a deep learning framework to realize image generation and style transfer. The method first used the conditional generation to resist the network, optimizes the loss function of the training mapping relationship, and generated the actual image from the input sketch. Then, by defining and optimizing the perceptual loss function of the style transfer model, the style features are extracted from the image, thereby forming the actual The conversion between images and stylized art images. Experiments show that this method can greatly reduce the work of coloring and converting with different artistic effects, and achieve the purpose of transforming simple stick figures into actual object images.

User Interface Code Retrieval: A Novel Visual-Representation-Aware Approach

Code reuse has been perceived to be an effective tool for user interface development that is known to be complex and messy. A fundamental problem in user interface code reuse is how to effectively retrieve working code from existing code repositories, which renders functional interfaces similar to a programmer's input sketch. Existing generic code retrieval techniques are normally inadequate for user interface code due to its associated visual information. In this paper, we put forward a novel approach that retrieves matching candidate code of an interface sketch based on their visual-representation trees. To better capture sub-region similarities, we propose to decompose a visual-representation tree into an ordered list of paths (or sequences). We then devise a novel algorithm to measure the similarity between two sequences by leveraging various visual information and another algorithm to calculate the similarity of two ordered lists by considering the relative positions of different widgets. To evaluate our solution, we carefully design an experiment to generate the similarity scores of 6,750 user interface pairs based on real users' judgement. Based on this ground truth, we show that our method is able to generate accurate similarity scores, outperforming several state-of-the-art competitors.

DeepSketch2Face

Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.

SVM: Sketch-based 3D Retrieval Application Using Classification Method

With the rapid development of 3D technology, the demand to use and retrieve the 3D model is becoming more and more urgent. In this case, it is more and more import and necessary that sketch-based 3D model retrieval. The user provides a hand-drawn sketch, then the system can be provided the possible list of models, then the user can select the need model from this list. In this paper, it’s proposed a framework of using SVM classification methods and K-means cluster algorithm. In preprocess part; the adaptive thinning algorithm is used to process the input sketch. In the offline part, the model is projected to multi-view images, and then SVM classifier has been used to classify these images. Moreover, we extract the images features, and use K-means algorithm to cluster these features. Finally, the feature dictionary will be acquired. In online process part, it only needs to extract the input sketch feature, and computes the distance between dictionary and their features. Besides, the experiment is realized to verify the feasibility of the approach. Finally, it was compared with other’s approaches; the result shows that the approach is viable and robustness.

A sketch-based semantic retrieval approach for 3D CAD models

During the new product development process, reusing the existing CAD models could avoid designing from scratch and decrease human cost. With the advent of big data, how to rapidly and efficiently find out suitable 3D CAD models for design reuse is taken more attention. Currently the sketch-based retrieval approach makes search more convenient, but its accuracy is not high enough; on the other hand, the semantic-based retrieval approach fully utilizes high level semantic information, and makes search much closer to engineers’ intent. However, effectively extracting and representing semantic information from data sets is difficult. Aiming at these problems, we proposed a sketch-based semantic retrieval approach for reusing 3D CAD models. Firstly a fine granularity semantic descriptor is designed for representing 3D CAD models; Secondly, several heuristic rules are adopted to recognize 3D features from 2D sketch, and the correspondences between 3D feature and 2D loops are built; Finally, semantic and shape similarity measurements are combined together to match the input sketch to 3D CAD models. Hence the retrieval accuracy is improved. A sketch-based prototype system is developed. Experimental results validate the feasibility and effectiveness of our proposed approach.

Swarmic autopoiesis and computational creativity

ABSTRACTIn this paper two swarm intelligence algorithms are used, the first leading the “attention” of the swarm and the latter responsible for the tracing mechanism. The attention mechanism is coordinated by agents of Stochastic Diffusion Search where they selectively attend to areas of a digital canvas (with line drawings) which contains (sharper) corners. Once the swarm's attention is drawn to the line of interest with a sharp corner, the corresponding line segment is fed into the tracing algorithm, Dispersive Flies Optimisation which “consumes” the input in order to generate a “swarmic sketch” of the input line. The sketching process is the result of the “flies” leaving traces of their movements on the digital canvas which are then revisited repeatedly in an attempt to re-sketch the traces they left. This cyclic process is then introduced in the context of autopoiesis, where the philosophical aspects of the autopoietic artist are discussed. The autopoetic artist is described in two modalities: gluttonous and contented. In the Gluttonous Autopoietic Artist mode, by iteratively focussing on areas-of-rich-complexity, as the decoding process of the input sketch unfolds, it leads to a less complex structure which ultimately results in an empty canvas; therein reifying the artwork's “death”. In the Contented Autopoietic Artist mode, by refocussing the autopoietic artist's reflections on “meaning” onto different constitutive elements, and modifying her reconstitution, different behaviours of autopoietic creativity can be induced and therefore, the autopoietic processes become less likely to fade away and more open-ended in their creative endeavour.

Shape Synthesis from Sketches via Procedural Models and Convolutional Networks.

Procedural modeling techniques can produce high quality visual content through complex rule sets. However, controlling the outputs of these techniques for design purposes is often notoriously difficult for users due to the large number of parameters involved in these rule sets and also their non-linear relationship to the resulting content. To circumvent this problem, we present a sketch-based approach to procedural modeling. Given an approximate and abstract hand-drawn 2D sketch provided by a user, our algorithm automatically computes a set of procedural model parameters, which in turn yield multiple, detailed output shapes that resemble the user's input sketch. The user can then select an output shape, or further modify the sketch to explore alternative ones. At the heart of our approach is a deep Convolutional Neural Network (CNN) that is trained to map sketches to procedural model parameters. The network is trained by large amounts of automatically generated synthetic line drawings. By using an intuitive medium, i.e., freehand sketching as input, users are set free from manually adjusting procedural model parameters, yet they are still able to create high quality content. We demonstrate the accuracy and efficacy of our method in a variety of procedural modeling scenarios including design of man-made and organic shapes.

Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval

Sketch-based 3D shape retrieval, which returns a set of relevant 3D shapes based on users' input sketch queries, has been receiving increasing attentions in both graphics community and vision community. In this work, we address the sketch-based 3D shape retrieval problem with a novel Cross-Domain Neural Networks (CDNN) approach, which is further extended to Pyramid Cross-Domain Neural Networks (PCDNN) by cooperating with a hierarchical structure. In order to alleviate the discrepancies between sketch features and 3D shape features, a neural network pair that forces identical representations at the target layer for instances of the same class is trained for sketches and 3D shapes respectively. By constructing cross-domain neural networks at multiple pyramid levels, a many-to-one relationship is established between a 3D shape feature and sketch features extracted from different scales. We evaluate the effectiveness of both CDNN and PCDNN approach on the extended large-scale SHREC 2014 benchmark and compare with some other well established methods. Experimental results suggest that both CDNN and PCDNN can outperform state-of-the-art performance, where PCDNN can further improve CDNN when employing a hierarchical structure.

A fast approach for perceptually-based fitting strokes into elliptical arcs

Fitting elliptical arcs to strokes of an input sketch is discussed. We describe an approach which automatically combines existing algorithms to get a balance of speed and precision. For measuring precision, we introduce fast metrics which are based on perceptual criteria and are tolerant of sketching imperfections. We return a likelihood estimate based on these metrics rather than deterministic yes/no result, in order that the approach can be used in higher-level collaborative-decision recognition flows.

Data-driven segmentation and labeling of freehand sketches

We present a data-driven approach to derive part-level segmentation and labeling of free-hand sketches, which depict single objects with multiple parts. Our method performs segmentation and labeling simultaneously, by inferring a structure that best fits the input sketch, through selecting and connecting 3D components in the database. The problem is formulated using Mixed Integer Programming, which optimizes over both the local fitness of the selected components and the global plausibility of the connected structure. Evaluations show that our algorithm is significantly better than the straightforward approaches based on direct retrieval or part assembly, and can effectively handle challenging variations in the sketch.

Modeling 3D animals from a side-view sketch

Using 2D contour sketches as input is an attractive solution for easing the creation of 3D models. This paper tackles the problem of creating 3D models of animals from a single, side-view sketch. We use the a priori assumptions of smoothness and structural symmetry of the animal about the sagittal plane to inform the 3D reconstruction. Our contributions include methods for identifying and inferring the contours of shape parts from the input sketch, a method for identifying the hierarchy of these structural parts including the detection of approximate symmetric pairs, and a hierarchical algorithm for positioning and blending these parts into a consistent 3D implicit-surface-based model. We validate this pipeline by showing that a number of plausible animal shapes can be automatically constructed from a single sketch.▪