2D Sketches Research Articles

Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches

In this paper, for the first time, we investigate the problem of generating 3D shapes from professional 2D sketches via deep learning. We target sketches done by professional artists, as these sketches are likely to contain more details than the ones produced by novices, and thus the reconstruction from such sketches poses a higher demand on the level of detail in the reconstructed models. This is importantly different to previous work, where the training and testing was conducted on either synthetic sketches or sketches done by novices. Novices sketches often depict shapes that are physically unrealistic, while models trained with synthetic sketches could not cope with the level of abstraction and style found in real sketches. To address this problem, we collected the first large-scale dataset of professional sketches, where each sketch is paired with a reference 3D shape, with a total of 1,500 professional sketches collected across 500 3D shapes. The dataset is available at http://sketchx.ai/downloads/. We introduce two bespoke designs within a deep adversarial network to tackle the imprecision of human sketches and the unique figure/ground ambiguity problem inherent to sketch-based reconstruction. We show that existing 3D shapes generation methods designed for images fail to be naively applied to our problem, and demonstrate the effectiveness of our method both qualitatively and quantitatively.

Immersive 3D sketching tools: Implications for visual thinking and communication

Paper-based (PB) sketching involves the challenge of representing three-dimensional (3D) shapes on two-dimensional (2D) surfaces. The recent generation of virtual reality (VR) sketching tools offer a way to overcome this challenge. These immersive 3D sketching environments permit the rapid construction of freehand stroke-based 3D models in 3D space while replicating the immediate experience of PB sketching. To explore the potential advantages of VR sketching in visual thinking and visual communication, we conducted investigations with sixteen architectural students engaged in PB and VR sketching tasks. We observed their visualization behavior during VR sketching and their behavior in transitioning between PB and VR sketching. The participants experiences of the two media were also recorded in semi-structured interviews and questionnaires. Our observations show that immersive 3D sketching is a unique form of visual representation that facilitates the rapid and flexible creation of large and detailed (but inaccurate) 3D computer models. It is a multimodal medium that supports visual thinking and communication behaviors associated with PB sketching, CAD modeling, physical model-making and gesturing, all within the same space. This unique combination enables users to engage in visual thinking and visual communication activities in ways that cannot presently be achieved with any other single representation technique.

3D sketching for 3D object retrieval

Sketching provides the most natural way to provide a visual search query for visual object search. However, how to draw 3D sketches in a three-dimensional space and how to use a hand-drawn 3D sketch to search similar 3D models are not only interesting and novel, but also challenging research topics. In this paper, we try to answer them by initiating a novel study on 3D sketching and build a 3D sketching system which allows users to freely draw 3D sketches in the air and demonstrate its promising potentials in related applications such as collecting 3D sketch data and conducting 3D sketch-based 3D model retrieval. By utilizing the 3D sketching system, we collect a 3D sketch dataset, build a 3D sketch-based 3D model retrieval benchmark, and organize a Eurographics Shape Retrieval Contest (SHREC) track on 3D sketch-based shape retrieval based on the benchmark. We investigate 3D sketch and model matching problems and propose a novel 3D sketch-based model retrieval algorithm CNN-SBR based on Convolutional Neural Networks (CNNs) and achieve the best performance in the SHREC track. We wish that the 3D sketching system, the 3D sketch-based model retrieval benchmark, and the proposed 3D sketch-based model retrieval algorithm CNN-SBR will further promote sketch-based shape retrieval and its applications. We have made all of these publicly available on the project homepage: http://orca.st.usm.edu/~bli/SBR16/project.html .

Stroke-Hover Intent Recognition for Mid-Air Curve Drawing Using Multi-Point Skeletal Trajectories

Abstract Drawing curves is a fundamental task in mid-air interactive applications such as 3D sketching, geometric modeling, hand-writing recognition, and authentication. Existing research in mid-air drawing is solely focused on determining what the user drew assuming that the intended curve is segmented from the continuous user-generated trajectory. In this work, our aim is to address the complementary problem: to determine when the user actually intended to draw without the use of any prescribed gestures or hand-held controllers (e.g., Wii remote, HTC Vive). In our previously published work, we demonstrated that in mid-air drawing tasks, not only it is possible to statistically learn drawing intent from hand motion, but it is also perceived to be more natural by users. Our idea was to simply classify each instance of hand trajectories as either a stroke or a hover. Our current work investigates new representations of the users’ motion beyond a single point (such as a tracked palm) to richer multi-point trajectories obtained with other skeletal joints such as wrist and elbow. We trained several binary classifiers on five such trajectory representations obtained from 3D drawing data from 25 users using a hand tracking device. We compare these representations and the corresponding classifiers for predicting user intent for mid-air drawing. Our extended approach resulted in improved prediction accuracy (mean: 80.17%, min: 79.92%, max: 91.30%) with respect to our earlier work (mean: 76.75%, min: 74.23%, max: 84.01%).

Conceptual Modeling in Product Design within Virtual Reality Environments

Digitalization has already permeated most of the design activities, but in spite of this, the generation of visual representations of concepts in the product design domain still relies on analog tools in real-world contexts. Despite immersive 3D technologies, such as Virtual Reality, have become widely available and affordable, most designers still make use of pencils and paper sheets, or their digital counterparts, to sketch their initial ideas on 2D supports. This study aims at investigating the reasons behind the mismatch between the rapid growth of immersive technologies and their scarce adoption in the conceptual design activities. Based on the analysis of the state of the art, a classification of the approaches proposing new ways to conduct conceptual representation of products has been drawn. The geometry representation, i.e. parametric or polygonal, and the interaction methods have been taken as metrics to categorize previous works. Weak connections between the modeling paradigm implemented and the interaction methods, lack of spatial faithfulness, ergonomic concerns and the need for quantitative metrics to compare objectively the data resulting from different testing sessions across the various studies are the main issues identified. In order to get concrete evidence of such thoughts, an experimental session has been devised with users from different backgrounds. They were asked to make conceptual sketches of a computer mouse in a traditional fashion, i.e. with pen and paper, and using two off-the-shelf Virtual Reality applications, based on 3D sketching and 3D sculpting respectively. The results are discussed qualitatively by visually comparing the sketches made by the testers, enriched by information deducted by surveying the users before and after the experiments. By comparing the sketches made by each user with the three procedures, preliminary results indicate that VR systems don't bring dramatic improvements compared to traditional 2D sketching tools. Furthermore, despite being enjoyable, VR systems caused physical fatigue, which is a problem that basically does not affect 2D sketching. Despite the size of the sample cannot provide statistical evidence, the outcomes provided good indications about the technology readiness level of Virtual Reality as a conceptual design tool, paving the way for future research directions.

Cross-domain retrieving sketch and shape using cycle CNNs

In this paper, we present a deep learning approach for cross-domain retrieval of 3D shape and 2D sketch image. Cross-domain retrieval has received significant attention to flexibly find information across different modalities of data. Effective measuring the similarity between different modalities of data is the key of cross-domain retrieval. Different modalities such as shape and sketch have imbalanced and complementary relationships, which contain unequal amount of information when describing the same semantics. Existing methods based on deep learning networks mostly construct one common space for different modalities, and these nets usually loss exclusive modality-specific characteristics.To address this problem, we propose a novel Cycle CNNs to estimate the cross-domain mapping between the space of 3D shape descriptors and the one of 2D sketch features. First, we employ the existing networks to construct independent feature spaces for each modality. For each feature space, modality-specific properties within one modality are fully exploited. Next, we use the designed Cycle CNNs to learn the mapping function between different feature spaces. This network can capture the mapping relationship between 3D shape feature space and 2D sketch feature domain. Finally, we use the explored mapping between the feature spaces of different modalities to perform cross-domain retrieval. We demonstrate a variety of promising results, where our method achieves better retrieval accuracy than existing state-of-the-art approaches.

The 3D virtual drawing mobile application based on augmented reality using AR-Framework

This study discussed the visualization system of 3D sketching model based on mobile Augmented Reality (AR) which supports information needed in real environment. The mobile application was made with adding virtual 3D objects to the 2D drawing by applying technology of AR. The visual system assists those who don’t have any specialized skill to be able to draw easily, because 3D model augmented over the drawing through camera of smart phone was added without any marker. This method is the basic combination between dynamic virtual object with the real environment, which nowadays has been developed as the mix reality. As the result on this research, the application can detect the real environment and mark it as a marker for the drawing-place as well. It will become the strategic future research of mix reality or marker less AR, because with this development, virtual world should be more attractive because it has been presented with immersive mixing to the real world.

Cross-Domain Correspondence for Sketch-Based 3D Model Retrieval Using Convolutional Neural Network and Manifold Ranking

Due to the huge difference in the representation of sketches and 3D models, sketch-based 3D model retrieval is a challenging problem in the areas of graphics and computer vision. Some state-of-the-art approaches usually extract features from 2D sketches and produce multiple projection views of 3D models, and then select one view of 3D models to match sketch. It's hard to find “the best view” and views from different perspectives of a 3D model may be completely different. Other methods apply learning features to retrieve 3D models based on 2D sketch. However, sketches are abstract images and are usually drawn subjectively. It is difficult to be learned accurately. To address these problems, we propose cross-domain correspondence method for sketch-based 3D model retrieval based on manifold ranking. Specifically, we first extract learning features of sketches and 3D models by two-parts CNN structures. Subsequently, we generate cross-domain undirected graphs using learning features and semantic labels to create correspondence between sketches and 3D models. Finally, the retrieval results are computed by manifold ranking. Experimental results on SHREC 13 and SHREC 14 datasets show the superior performance in all 7 standard metrics, compared to the state-of-the-art approaches.

A 3D shape generative method for aesthetic product design

This work describes a generative method for the exploration of product shapes in the conceptual design stage. The method is based on three concepts: the notion of grammars to capture product appearance, the implementation of sketching transformation rules to produce design variations and the use of a parametric modeller to build shapes. We represent product solutions as 3D sketches using combinations of basic shapes arranged in simple and schematic product structures. This procedure allows creating many varied configurations with a minimal number of shapes, and facilitates the adaptation of the generative model to different products. The performance of the method is demonstrated through several examples from the literature.

From 2D Sketches to Photo-Realistic Images using Generative Adversarial Networks

With increasing technological advancements, there is a need for automation in this ever-evolving world. This may result in improved efficiency, faster work and enhanced capabilities. Sketch-to-image translation is an image processing application that can be used as a helping hand in a variety of fields. One of these is the utilization of Generative Adversarial Networks to guide edges to photographs, with the assistance of image generators and discriminators who work connected to produce realistic images. We have also incorporated Histogram of Oriented Gradients (HOG) as a feature/image descriptor. The HOG technique counts the gradient orientations to differentiate the target and the background. Support Vector Machine (SVM) is the classifier used for classification. This HOG and SVM model can be improved, altered and executed as multi-program software.

3D Computational Sketch Synthesis Framework: Assisting Design Exploration Through Generating Variations of User Input Sketch and Interactive 3D Model Reconstruction

A framework is proposed for facilitating the exploration process during the early design phase through computational sketch synthesis and interactive 3D reconstruction. In that phase, designers concentrate on developing concepts through numerous alternatives. Therefore, they constantly sketch so that they can rapidly visualize their ideas. Recently, the design industry has attempted to streamline the design process by implementing 3D model generation in the early design phase so that ideas may be more thoroughly explored, thus improving concept and final design conformance; however, efficiency issues have arisen. In this study, a 3D computational sketch synthesis framework was developed comprising two major components. First, a robust method was proposed to synthesize design alternatives by interpolating an input sketch with sketches in a database so that unvisited combinations may be explored. Secondly, a novel interactive 3D model reconstruction method was developed to facilitate the shape transition of design elements so that designers can quickly evaluate the potential of a large number of design variations. Finally, an interface for design refinement was developed so that designs may be embodied by sketching over the 3D model. To test the proposed methodology, expert designers were recruited for a validation experiment with two conditions followed up by in-depth interviews. In the first condition, the participants were asked to sketch based on a design brief in their current working manner. In the second condition, they were asked to create designs using the proposed framework. It was tested whether there was a difference in the design outcomes. It was demonstrated that the proposed framework resulted in more satisfactory and higher-quality designs and generated design alternatives faster and in greater quantities. All participants agreed that the framework could be useful in the early design phase and responded that the proposed system provides more design inspiration than traditional design methods. Most importantly, it was demonstrated that the proposed framework could enhance the reevaluation potential of design concepts and assist in making better-informed design decisions.

Exterior Electric Hatchback Crossover Car Design with Strong, Vicious and Sharp Concept

Identity of design of an electric car is necessary to show electrical technology and added values for consumers. Design of electic cars today, in the meaning of appearance and image, are judged by consumers as wierd and ugly compared to design of internal combustion engine (ICE) cars. The distinguished factor of electric car compared to ICE car is the technology that provides environment friendly and fuel energy crisis solution. Symbols of electric car can actually be build to be an interesting form and shape, such as smart, lightweight, efficient and clean. This research aims to design a small-medium electric car with 4-passenger car for young urban family consumer. The methods are consecutively: kind of car defining, 2D sketching, ergonomic or anthopometrical study, engineering and passenger package study, 3D styling and part design. The concept of form has adapted by keywords: strong, vicious, and sharp. Definition of ergonomic study based on Indonesian people with percentile of 5% women and 95% of men. Definition of engineering package based on Li-ion battery with size of 24 kWh placed under floor and single electric motor package placed on the front with front wheel system. The design result is a small medium crossover car with dimensions of wheelbase of 2500 mm, height of 1534 mm, front overhang of 858 mm and rear overhang of 723 mm. Design character of strong, vicious and sharp are transformed by small sharp headlamp and rear lamp; small grille with LED lines; smooth sculpting line on entire body; deep sculpting line on side body, firm and bold form of front and rear fender. The design outputs are 3D model of exterior, 3D mode; of body parts, 3D animation and physical scale model

A Smart Substation Training Platform based on Primary Equipment Simulator

A training scheme based on combination of virtual substation primary equipment real secondary devices is proposed for smart substation accident management. The primary equipment simulator is implemented through communication between 3D cartoon server and simulated switch controller. Electronic current and voltage transformer is simulated by EMTDC transient model and FPGA+DSP terminal controller. A smart substation on-site training environment is established for the substation O&M personnel.

Computational peeling art design

Some artists peel citrus fruits into a variety of elegant 2D shapes, depicting animals, plants, and cartoons. It is a creative art form, called Citrus Peeling Art. This art form follows the conservation principle, i.e., each shape must be created using one entire peel. Central to this art is finding optimal cut lines so that the citruses can be cut and unfolded into the desired shapes. However, it is extremely difficult for users to imagine and generate cuts for their desired shapes. To this end, we present a computational method for citrus peeling art designs. Our key insight is that instead of solving the difficult cut generation problem, we map a designed input shape onto a citrus in an attempt to cover the entire citrus and use the mapped boundary to generate the cut paths. Sometimes, a mapped shape is unable to completely cover a citrus. Consequently, we have developed five customized ways of interaction that are used to rectify the input shape so that it is suitable for citrus peeling art. The mapping process and user interactions are iteratively conducted to satisfy a user's design intentions. A large number of experiments, including a formative user study, demonstrate the capability and practicability of our method for peeling art design and construction.

Associating 2D Sketch Information with 3D CAD Models for VR/AR Viewing During Bridge Maintenance Process

Virtual reality (VR), augmented reality (AR), and mixed reality technologies are utilized at various stages of product lifecycle. For products with long lifecycles such as bridges and dams, the maintenance and inspection stages are very important to keep the product safe and well-functioning. One of the advantages of VR/AR is the ability to add important information such as past inspection data. Past inspection information is summarized in a document consisting of the 2D sketches of bridge degradation drawings. However, this degradation sketch is in 2D, and it has no correspondence with the 3D world. In this study, we propose a method to associate important information of 2D sketches with a 3D industry foundation classes (IFC) model, which is a standardized computer aided design model. To display a VR image of a bridge during the inspection process, the proposed method is applied to the 3D IFC model of the bridge and 2D degradation sketch of the inspection report.

Combining voxel and normal predictions for multi-view 3D sketching

Recent works on data-driven sketch-based modeling use either voxel grids or normal/depth maps as geometric representations compatible with convolutional neural networks. While voxel grids can represent complete objects – including parts not visible in the sketches – their memory consumption restricts them to low-resolution predictions. In contrast, a single normal or depth map can capture fine details, but multiple maps from different viewpoints need to be predicted and fused to produce a closed surface. We propose to combine these two representations to address their respective shortcomings in the context of a multi-view sketch-based modeling system. Our method predicts a voxel grid common to all the input sketches, along with one normal map per sketch. We then use the voxel grid as a support for normal map fusion by optimizing its extracted surface such that it is consistent with the re-projected normals, while being as piecewise-smooth as possible overall. We compare our method with a recent voxel prediction system, demonstrating improved recovery of sharp features over a variety of man-made objects.

A 3D Virtual Sketching System Using NURBS Surfaces and Leap Motion Controller

A 3D sketching system is presented for transforming an ideation to an actual CAD model using Human Computer Interaction (HCI). The system uses a Leap motion controller to capture human gestures, and translated them by a recognition model. The pinch and pull gestures are used to represent actions for flexible manipulations on control points coordinates of NURBS surfaces. The continuous motion of the designers hands is converted to morphological/geometrical and mathematical values. Both cases of single and dual hands are developed, offering the designer a variety of resolution tools to convey their ideas and creativity.

Deep Common Semantic Space Embedding for Sketch-Based 3D Model Retrieval.

Sketch-based 3D model retrieval has become an important research topic in many applications, such as computer graphics and computer-aided design. Although sketches and 3D models have huge interdomain visual perception discrepancies, and sketches of the same object have remarkable intradomain visual perception diversity, the 3D models and sketches of the same class share common semantic content. Motivated by these findings, we propose a novel approach for sketch-based 3D model retrieval by constructing a deep common semantic space embedding using triplet network. First, a common data space is constructed by representing every 3D model as a group of views. Second, a common modality space is generated by translating views to sketches according to cross entropy evaluation. Third, a common semantic space embedding for two domains is learned based on a triplet network. Finally, based on the learned features of sketches and 3D models, four kinds of distance metrics between sketches and 3D models are designed, and sketch-based 3D model retrieval results are achieved. The experimental results using the Shape Retrieval Contest (SHREC) 2013 and SHREC 2014 datasets reveal the superiority of our proposed method over state-of-the-art methods.

Animate! Breathing new life into an old pedagogy: Human embryology teaching reenvisioned

The teaching of human embryology is as ancient as that of human anatomy itself. Over the centuries up to the present time, embryology pedagogy has evolved from enlisting the dissection of fetal cadavers and use of two‐dimensional (2D) sketches and diagrams, to the modern technologies of 2D and 3D animations, CT scans of embryos and fetuses with 3D rendering of the images, and the use of ultrasound imaging. These techniques seek to balance accurate portrayal of embryonic structures with the cognitive load associated with learning the concepts presented using these images, and their value as learning adjuncts for students. This paper discusses the rationale, pedagogical basis and implementation of narrated 2D animations of embryonic development, which present the early stages of human life in accessible cartoon‐like sequences. In response to the demand from curriculum leaders at the University of British Columbia (UBC) medical school for high‐quality, simplified yet accurate rendition of early embryonic development, we have developed the first of a series of 2D animations, which presents key embryonic processes during the first 12 weeks of life contemporaneously. First and second year MD students at UBC will have access to these animations as adjuncts to the embryology components of the MD Undergraduate pre‐clerkship program. The animation was developed using Adobe Animate® software during approximately 300 hours of programming. Audio design, recording and editing were completed with Adobe Audition®, After Effects® and Adobe Premiere Pro®. In evaluating the animation, students rated it highly for visual appeal and helpfulness to learn basic embryology, while providing useful comments on how it may be improved.Support or Funding InformationSouthern Medical Program, UBC Faculty of MedicineThis abstract is from the Experimental Biology 2019 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

Innovation through practice

PurposeThrough the study of visualizations, virtual worlds and information exchange, the purpose of this paper is to reveal the complex connections between technology and the work of design and construction. The authors apply the sociotechnical view of technology and the ramifications this view has on successful use of technology in design and construction.Design/methodology/approachThis is a discussion paper reviewing over a decade of research that connects three streams of research on architecture, engineering and construction (AEC) teams as these teams grappled with adapting work practices to new technologies and the opportunities these technologies promised.FindingsFrom studies of design and construction practices with building information modeling and energy modeling, the authors show that given the constructed nature of models and the loose coupling of project teams, these team organizational practices need to mirror the modeling requirements. Second, looking at distributed teams, whose interaction is mediated by technology, the authors argue that virtual world visualizations enhance discovery, while distributed AEC teams also need more traditional forms of 2D abstraction, sketching and gestures to support integrated design dialogue. Finally, in information exchange research, the authors found that models and data have their own logic and structure and, as such, require creativity and ingenuity to exchange data across systems. Taken together, these streams of research suggest that process innovation is brought about by people developing new practices.Originality/valueIn this paper, the authors argue that technology alone does not change practice. People who modify practices with and through technology create process innovation.