Sketching System Research Articles

SketchMetaFace: A Learning-Based Sketching Interface for High-Fidelity 3D Character Face Modeling.

Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency.

DynBioSketch: A tool for sketching dynamic visual summaries in biology, and its application to infection phenomena

Having simple methods of illustration is essential to scientific thinking. To complement the abstract sketches regularly used in cell biology, we propose DynBioSketch, an easy-to-use digital modeling and animation tool, enabling biologists to resort to less simplified representations when necessary without having to call professional artists. DynBioSketch is an interactive sketching system dedicated to the design and communication of biological phenomena at the cellular scale that can be illustrated in a few minutes of animation. Our model integrates 3D modeling, pattern-based design of 3D shape distributions, and sketch-based animation. These elements can be combined to create complex scenarios such as the infection phenomenon on which we focus, allowing a narrative design adapted to communication between researchers or educational applications in biology. Our results, along with a user study conducted with biology researchers, highlight the potential of DynBioSketch in enabling the direct design of dynamic visual summaries that convey relevant information, as shown in our infection case study. By bridging the gap between abstract representations used by experts and more illustrative depictions, DynBioSketch opens a new avenue for communicating biological concepts.

Stylus and Gesture Asymmetric Interaction for Fast and Precise Sketching in Virtual Reality

This research investigates fast and precise Virtual Reality (VR) sketching methods with different tool-based asymmetric interfaces. In traditional real-world drawing, artists commonly employ an asymmetric interaction system where each hand holds different tools, facilitating diverse and nuanced artistic expressions. However, in virtual reality (VR), users are typically limited to using identical tools in both hands for drawing. To bridge this gap, we aim to introduce specifically designed tools in VR that replicate the varied tool configurations found in the real world. Hence, we developed a VR sketching system supporting three hybrid input techniques using a standard VR controller, a VR stylus, or a data glove. We conducted a formal user study consisting of an internal comparative experiment with four conditions and three tasks to compare three asymmetric input methods with each other and with a traditional symmetric controller-based solution based on questionnaires and performance evaluations. The results showed that in contrast to symmetric dual VR controller interfaces, the asymmetric input with gestures significantly reduced task completion times while maintaining good usability and input accuracy with a low task workload. This shows the value of asymmetric input methods for VR sketching. We also found that the overall user experience could be further improved by optimizing the tracking stability of the data glove and the VR stylus.

Using genetic algorithm for drawing path planning in a robotic arm pencil sketching system

The purpose of this study is to implement a robotic arm pencil sketching system through a novel drawing path planning method. Firstly, to enhance the craftsmanship and uniqueness of the artwork, the original image is transformed into an animated style using AnimeGANv2. After image processing, the image is divided into boundary and background layers for drawing path planning. In the realm of path planning, traditional methods often involve manual path design by artists, and only recently have automated path planning methods been proposed. However, these recent approaches still have some limitations. For instance, a pursuit of efficiency can lead to overly uniform strokes, making the artwork resemble a print rather than a genuine piece of art. Therefore, this study introduces a new path-planning approach based on the traveling salesman problem (TSP), utilising various algorithms for path-planning experiments. Different stroke effects generated by distinct algorithms are applied to appropriate areas. The aim is to let the generated artworks exhibit the technological appeal of algorithms while retaining the distinctive characteristics of pencil sketches. For robotic arm control, a force sensor is integrated onto the arm, and a proportional-integral-derivative (PID) controller is employed to monitor and precisely control the drawing force. Experimental results demonstrate that artworks created using genetic algorithms to generate drawing paths exhibit distinct colour-layering effects, emphasising the characteristics of pencil strokes. In comparison with previous methods, this approach effectively addresses the issue of overly uniform and rigid strokes in past techniques.

Race and statistics in facial recognition: Producing types, physical attributes, and genealogies.

Principal component analysis (PCA) is a common statistical procedure. In forensics, it is used in facial recognition technologies and composite sketching systems. PCA is especially helpful in contexts with high facial diversity, which is often translated as racial diversity. In these settings, researchers use PCA to define a 'normal face' and organize the rest of the available facial diversity based on their resemblance to or difference from that norm. In this way, the use of PCA introduces an 'ontology of the normal' in which expectations about how a normal face should look are corroborated by statistical calculations of normality. I argue that the use of PCA can lead to a statistical reification of racial stereotypes that informs recognition practices. I discuss current and historical cases in which PCA is used: one of face perception theorization ('face space theory') and two of technology development (the 'eigenfaces' facial recognition algorithm and the 'EvoFIT' composite sketching system). In each, PCA aligns facial normality with racial expectations, and instrumentalizes race in specific ways: as a type, physical attribute, or genealogy. This analysis of PCA does two things. First, it opens the black box of facial recognition to uncover how stereotypes and intuitions about normality become part of theories and technologies of facial recognition. Second, it explains why racial categorizations remain central in contemporary identification technologies and other forensic practices.

Rapid design of articulated objects

Designing articulated objects is challenging because, unlike with static objects, it requires complex decisions to be made regarding the form, parts, rig, poses, and motion. We present a novel 3D sketching system for rapidly authoring concepts of articulated objects for the early stages of design, when designers make such decisions. Compared to existing CAD software, which focuses on slowly but elaborately producing models consisting of precise surfaces and volumes, our system focuses on quickly but roughly producing models consisting of key curves through a small set of coherent pen and multi-touch gestures. We found that professional designers could easily learn and use our system and author compelling concepts in a short time, showing that 3D sketching can be extended to designing articulated objects and is generally applicable in film, animation, game, and product design.

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

We present DeepSketchHair, a deep learning based tool for modeling of 3D hair from 2D sketches. Given a 3D bust model as reference, our sketching system takes as input a user-drawn sketch (consisting of hair contour and a few strokes indicating the hair growing direction within a hair region), and automatically generates a 3D hair model, matching the input sketch. The key enablers of our system are three carefully designed neural networks, namely, S2ONet, which converts an input sketch to a dense 2D hair orientation field; O2VNet, which maps the 2D orientation field to a 3D vector field; and V2VNet, which updates the 3D vector field with respect to the new sketches, enabling hair editing with additional sketches in new views. All the three networks are trained with synthetic data generated from a 3D hairstyle database. We demonstrate the effectiveness and expressiveness of our tool using a variety of hairstyles and also compare our method with prior art.

Edge Detection Algorithm Used in a 3D Sketching & Photogrammetry System

A clear 3D representation is vital to prototyping in design. 3D modeling is a popular approach to design structures in 3D, resulting in accurate models that can be viewed from multiple perspectives. However, 3D modeling is too costly and rigid for designers because it has a steep learning curve. Existing heavy-weight 3D software packages are usually not suitable for ideation. Therefore, we consider using sketches to convey ideas in 3D as a more natural way for prototyping. Even if most sketches are loose and 2D, people can still identify 3D structures from them. However, most sketching tools only support drawing in 2D, motivating us to find a solution to bridging 2D sketches and 3D structures. To fill the gap, we propose a 3D sketching system that facilitates sketch creation in 3D. Users can create a sketch in 3D and view it from multiple perspectives, without having to create multiple 2D sketches. This application also supports reconstructing a point cloud from images taken from existing structures. We designed a novel edge detection algorithm derived from the Hough transform to extract borders as strokes from a point cloud reconstructed from captured images. This includes fitting multiple planes to the point cloud and creating strokes by projecting points to the planes. We also present real-world use cases to evaluate our system. Our application allows designers to prototype more rapidly, opening a gate to more user-friendly ideation processes.

3D sketching for 3D object retrieval

Sketching provides the most natural way to provide a visual search query for visual object search. However, how to draw 3D sketches in a three-dimensional space and how to use a hand-drawn 3D sketch to search similar 3D models are not only interesting and novel, but also challenging research topics. In this paper, we try to answer them by initiating a novel study on 3D sketching and build a 3D sketching system which allows users to freely draw 3D sketches in the air and demonstrate its promising potentials in related applications such as collecting 3D sketch data and conducting 3D sketch-based 3D model retrieval. By utilizing the 3D sketching system, we collect a 3D sketch dataset, build a 3D sketch-based 3D model retrieval benchmark, and organize a Eurographics Shape Retrieval Contest (SHREC) track on 3D sketch-based shape retrieval based on the benchmark. We investigate 3D sketch and model matching problems and propose a novel 3D sketch-based model retrieval algorithm CNN-SBR based on Convolutional Neural Networks (CNNs) and achieve the best performance in the SHREC track. We wish that the 3D sketching system, the 3D sketch-based model retrieval benchmark, and the proposed 3D sketch-based model retrieval algorithm CNN-SBR will further promote sketch-based shape retrieval and its applications. We have made all of these publicly available on the project homepage: http://orca.st.usm.edu/~bli/SBR16/project.html .

Designing fine-grained interactions for automation in air traffic control

Our work aims to explore novel approaches to the challenge of designing the interaction between people and automation. Through a case study within the domain of air traffic control, we focus on designing fine-grained human–automation interactions. We design a concept and develop an interactive lo-fi prototype of an assisted sketching system to enable air traffic controllers to interact with automation in a fine-grained manner and to externalize mental images. Assisted sketching seems to offer a possible way to communicate different degrees of predictive certainty using visual cues and interaction. Our insights further suggest that externalization through assisted sketching could encourage exploration of future scenarios, and support communication and collaboration between air traffic controllers and between air traffic controllers and pilots. The explorative benefits for the individual decision-making process might be more evident in situations where air traffic controllers have more time for reflection, for example during planning or debriefing and in educational settings.

A 3D Virtual Sketching System Using NURBS Surfaces and Leap Motion Controller

A 3D sketching system is presented for transforming an ideation to an actual CAD model using Human Computer Interaction (HCI). The system uses a Leap motion controller to capture human gestures, and translated them by a recognition model. The pinch and pull gestures are used to represent actions for flexible manipulations on control points coordinates of NURBS surfaces. The continuous motion of the designers hands is converted to morphological/geometrical and mathematical values. Both cases of single and dual hands are developed, offering the designer a variety of resolution tools to convey their ideas and creativity.

EdSketch: execution-driven sketching for Java

Sketching is a synthesis approach that allows users to provide high-level insights into a synthesis problem and let synthesis tools complete low-level details. Users write sketches—partial programs that have “holes” and provide test assertions as the correctness criteria. The sketching techniques fill the holes with code fragments such that the complete program satisfies all test assertions. Traditional techniques translate the sketching problem to propositional satisfiability formulas and leverage SAT solvers to generate programs with the desired functionality. While effective for a range of small well-defined domains, such translation-based approaches have a key limitation when applying to real applications: They require either translating all relevant libraries that are invoked directly or indirectly by the given sketch or creating models of those libraries, which requires much manual effort. This paper introduces execution-driven sketching, a novel approach for synthesizing Java programs with on-demand candidate generation. The key novelty of our work is to leverage runtime behavior to prune a large amount of search space. EdSketch explores the actual program behaviors in the presence of libraries and sketches small parts of real-world applications, which may use complex constructs of modern languages, such as reflection, native calls and File I/O. We further leverage a set of pruning strategies based on Java syntax to expedite the synthesis process. EdSketch embodies our approach in two forms: a stateful search based on the Java PathFinder model checker; and a stateless search based on re-execution inspired by the VeriSoft model checker. Experimental results show that EdSketch can complete some sketches that contain complex constructs in the presence of libraries, recursive procedures and advanced features like reflection. Without translating to SAT, EdSketch ’s performance compares well with the SAT-based Sketch system for a range of small but complex data structure subjects.

CaricatureShop: Personalized and Photorealistic Caricature Sketching.

In this paper, we propose the first sketching system for interactively personalized and photorealistic face caricaturing. Input an image of a human face, the users can create caricature photos by manipulating its facial feature curves. Our system first performs exaggeration on the recovered 3D face model, which is conducted by assigning the laplacian of each vertex a scaling factor according to the edited sketches. The mapping between 2D sketches and the vertex-wise scaling field is constructed by a novel deep learning architecture. Our approach allows outputting different exaggerations when applying the same sketching on different input figures in term of their different geometric characteristics, which makes the generated results "personalized". With the obtained 3D caricature model, two images are generated, one obtained by applying 2D warping guided by the underlying 3D mesh deformation and the other obtained by re-rendering the deformed 3D textured model. These two images are then seamlessly integrated to produce our final output. Due to the severe stretching of meshes, the rendered texture is of blurry appearances. A deep learning approach is exploited to infer the missing details for enhancing these blurry regions. Moreover, a relighting operation is invented to further improve the photorealism of the result. These further make our results "photorealistic". The qualitative experiment results validated the efficiency of our sketching system.

Embodied sound design

Abstract Embodied sound design is a process of sound creation that involves the designer’s vocal apparatus and gestures. The possibilities of vocal sketching were investigated by means of an art installation. An artist–designer interpreted several vocal self-portraits and rendered the corresponding synthetic sketches by using physics-based and concatenative sound synthesis. Both synthesis techniques afforded a broad range of artificial sound objects, from concrete to abstract, all derived from natural vocalisations. The vocal-to-synthetic transformation process was then automated in SEeD, a tool allowing to set and play interactively with physics- or corpus-based sound models. The voice-driven process and tool, developed and evaluated through design exercises, show how an embodied sound sketching system can work in supporting the externalisation of sonic concepts.

Quantifying Collaboration with a Co-Creative Drawing Agent

This article describes a new technique for quantifying creative collaboration and applies it to the user study evaluation of a co-creative drawing agent. We present a cognitive framework called creative sense-making that provides a new method to visualize and quantify the interaction dynamics of creative collaboration, for example, the rhythm of interaction, style of turn taking, and the manner in which participants are mutually making sense of a situation. The creative sense-making framework includes a qualitative coding technique, interaction coding software, an analysis method, and the cognitive theory behind these applications. This framework and analysis method are applied to empirical studies of the Drawing Apprentice collaborative sketching system to compare human collaboration with a co-creative AI agent vs. a Wizard of Oz setup. The analysis demonstrates how the proposed technique can be used to analyze interaction data using continuous functions (e.g., integrations and moving averages) to measure and evaluate how collaborations unfold through time.

DeepSketch2Face

Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.

Sketching in three dimensions: A beautification scheme

AbstractPrimary among all the activities involved in conceptual design is freehand sketching. There have been significant efforts in recent years to enable digital design methods that leverage humans’ sketching skills. Conventional sketch-based digital interfaces are built on two-dimensional touch-based devices like sketchers and drawing pads. The transition from two-dimensional to three-dimensional (3-D) digital sketch interfaces represents the latest trend in developing new interfaces that embody intuitiveness and human–human interaction characteristics. In this paper, we outline a novel screenless 3-D sketching system. The system uses a noncontact depth-sensing RGB-D camera for user input. Only depth information (no RGB information) is used in the framework. The system tracks the user's palm during the sketching process and converts the data into a 3-D sketch. As the generated data is noisy, making sense of what is sketched is facilitated through a beautification process that is suited to 3-D sketches. To evaluate the performance of the system and the beautification scheme, user studies were performed on multiple participants for both single-stroke and multistroke sketching scenarios.

Controlled direct 3D sketching with haptic and motion constraints

Present work concerns intuitive generation of 3D sketches with intrinsic support for space constraints and articulation for multi-component product conceptualisation. The issues of lack of precision, stability and control in direct creation of 3D strokes in air, arising out of inadequacy of the proprioceptive feedback alone, are mitigated in this work with inclusion of stereoscopic display and haptics. Investigations based on paper-pencil analogy for haptic rendering revealed that pen-tilt-based method is more effective. For spatial conformity, surface and volumetric constraints have been studied using the novel reactive and field-based haptic rendering schemes. Methodologies for automatic perceptual grouping of strokes enabled interactive exploration of motion in articulated 3D concept sketches. Experiments with designers using the proposed 3D sketching system established that modelling of constraints is the central issue in 3D sketching. For more immersive experience, eye-movement coordinated realistic stereo-rendering and advanced interfaces for large-scale design scenarios are being pursued.

Sketching Folds

We present the first sketch-based modeling method for developable surfaces with pre-designed folds, such as garments or leather products. The main challenge we address for building folded surfaces from sketches is that silhouette strokes on the sketch correspond to discontinuous sets of non-planar curves on the 3D model. We introduce a new zippering algorithm for progressively identifying silhouette edges on the model and tying them to silhouette strokes. Our solution ensures that the strokes are fully covered and optimally sampled by the model. This new method, interleaved with developability optimization steps, is implemented in a multiview sketching system where the user can sketch the contours of internal folds in addition to the usual silhouettes, borders, and seam lines. All strokes are interpreted as hard constraints, while developability is only optimized. The developability error map we provide then enables users to add local seams or darts where needed and progressively improve their design. This makes our method robust, even to coarse input for which no fully developable solution exists.

A Smart Pad Assisted Hybrid Sketch System for the Conceptual Design of Vehicles

In this paper, a sketching system that uses a tablet computer and a large display connected to a 3D modeling server is proposed for the conceptual modeling stage of vehicles. The plane sketch interface can be conveniently used to set the volume and proportion of a vehicle in conceptual design, and the spatial sketch interface can be used for 3D model modification efficiently. Using a tablet computer, which can be easily carried and used to implement the plane input, the advantages of both sketch interfaces can be exploited. The modeling server connected to the large display shows complicated 3D models, traces, and spatial movement using sensors embedded in the tablet computer, and makes curve-based modeling and deformation possible by the use of the touch screen input. A system has been developed to maintain the plane sketch interface that is familiar to designers. The 3D spatial modeling is implemented using a method that involves selecting cross-planes of a 3D model by moving the tablet computer and applying it to the 3D model to modify the curves of the cross-plane shown on the tablet display.