Sketch-based System Research Articles

CreatureShop: Interactive 3D Character Modeling and Texturing From a Single Color Drawing.

Creating 3D shapes from 2D drawings is an important problem with applications in content creation for computer animation and virtual reality. We introduce a new sketch-based system, CreatureShop, that enables amateurs to create high-quality textured 3D character models from 2D drawings with ease and efficiency. CreatureShop takes an input bitmap drawing of a character (such as an animal or other creature), depicted from an arbitrary descriptive pose and viewpoint, and creates a 3D shape with plausible geometric details and textures from a small number of user annotations on the 2D drawing. Our key contributions are a novel oblique view modeling method, a set of systematic approaches for producing plausible textures on the invisible or occluded parts of the 3D character (as viewed from the direction of the input drawing), and a user-friendly interactive system. We validate our system and methods by creating numerous 3D characters from various drawings, and compare our results with related works to show the advantages of our method. We perform a user study to evaluate the usability of our system, which demonstrates that our system is a practical and efficient approach to create fully-textured 3D character models for novice users.

Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch.

The freeform architectural modeling process often involves two important stages: concept design and digital modeling. In the first stage, architects usually sketch the overall 3D shape and the panel layout on a physical or digital paper briefly. In the second stage, a digital 3D model is created using the sketch as a reference. The digital model needs to incorporate geometric requirements for its components, such as the planarity of panels due to consideration of construction costs, which can make the modeling process more challenging. In this work, we present a novel sketch-based system to bridge the concept design and digital modeling of freeform roof-like shapes represented as planar quadrilateral (PQ) meshes. Our system allows the user to sketch the surface boundary and contour lines under axonometric projection and supports the sketching of occluded regions. In addition, the user can sketch feature lines to provide directional guidance to the PQ mesh layout. Given the 2D sketch input, we propose a deep neural network to infer in real-time the underlying surface shape along with a dense conjugate direction field, both of which are used to extract the final PQ mesh. To train and validate our network, we generate a large synthetic dataset that mimics architect sketching of freeform quadrilateral patches. The effectiveness and usability of our system are demonstrated with quantitative and qualitative evaluation as well as user studies.

Draw and Listen! A Sketch-Based System for Music Inpainting

In this work we implement an interactive system for filling in missing measures in a monophonic music piece. This system takes a user’s hand-drawn curves as input and generates a melody whose rhythm and pitch contour match with the curves. Contrary to previous interactive music inpainting work, users of the proposed system do not need to understand the music notation; they just need a rough idea of the shape of the melody and draw it out. This system is implemented under the variational auto-encoder framework and is enabled by a proposed melody disentanglement scheme to disentangle relative pitch, relative rhythm and musical context. We also create a web-based graphical user interface to facilitate the user interaction. We evaluate the system on a commonly used Irish folk song dataset. Objective and subjective evaluations show that this novel interaction is intuitive and effective for melody inpainting, and the proposed neural approach outperforms two baselines we developed based on previous work, in terms of musicality and fidelity to the user’s input.

A sketch-based system for modeling 3D objects: Applications to taxonomy.

We present a sketch-based system for modeling 3D objects with multiple contours as overlaid structures, allowing users to infer perceivable symmetries and occluded parts of the model prior to its automatic 3D reconstruction. Many approaches for sketch-based interfaces and modeling focus on the final drawing and 3D reconstruction of solid objects, exploring, as inputs, contours; however, when the subject consists of multiple drawings as overlaid structures, as usual in species descriptions, these approaches have to roughly infer or discard occluded parts. Unlike previous sketch-based techniques, we explore a set of 2D visual effects to enhance the visual perception of users while sketching multiple overlaid objects in single-view. The proposed system was applied to general simple zoological and botanical sketches as well as to published descriptions of species of Diptera, demonstrating its usefulness for taxonomy.

Interactive example-palettes for discrete element texture synthesis

Textures composed of individual discrete elements are found in everything from human-made glass-tilings to forests and tropical coral. We propose an interactive sketch-based system for synthesizing scenes consisting of many discrete element textures. We have implemented an example-palette, a design window where a user can use our sketch-based tools to create discrete element textures and then paint those textures into a scene or back into the example-palette to create new textures. Our interactive sketch-based tools use a new and fast region-growing algorithm that iteratively synthesizes new elements around previously synthesized elements. To support discrete element textures with different scales in the same output, we parameterize our region-growing algorithm on a per-element basis. Our method is capable of synthesizing structured and stochastic example discrete element textures. We explore applications of our system for building virtual worlds (such as for video games) and for sketch-based modeling.

Automatic analysis and sketch-based retrieval of architectural floor plans

The contribution of this article is twofold. First, we propose a sketch-based system, namely the a.SCatch system, for querying a floor plan repository. Second, for generating such a repository, a novel complete system for floor plan analysis is presented. The latter system extracts the semantics from existing floor plans. We introduce novel preprocessing methods, e.g., the differentiation between thick, medium, and thin lines and the removal of components outside the convex hull of the outer walls. Especially, the use of the convex hull increases the performance of the final system. The a.SCatch system enables the user to easily access knowledge from past projects. The user searches for semantically similar floor plans just by drawing parts of the new plan. An algorithm extracts the semantic structure sketched by the architect on DFKI’s Touch &Write table. Finally, the extracted structures are compared using the graph matching, and the most similar one is retrieved. In our experiments for floor plan analysis on a reference data set we compare our approach to other approaches available in the literature. We show that our floor plan analysis system outperforms previous systems. Also the performance of the sketch recognition system is quite high. Overall, the performance of the floor plan analysis, as well as of the retrieval, is already good for the use in practice.

A sketch-based system for highway design with user-specified regions of influence

To create traffic simulations of high visual-fidelity, each component part needs to be designed and modelled in great detail. Roads can be created manually, but this can become a time-consuming and laborious process when modelling large-scale networks. Therefore, automated techniques for generating road networks efficiently and effectively, is highly desirable in both urban-planning and entertainment industries. In this paper we present a novel sketch-based tool to semi-automate the design, creation, and visualisation of road networks across both flat and undulating terrains. Our tool is guided by input sketches and a combination of prioritised constraints, including the curvature of roads, their inclination, and the volume of ground that would be displaced during construction. We introduce ‘Influence Regions’ which are user-specified areas of the terrain that influence the path of the roads generated, and are used to attract or repel roads to/from certain obstacles or designated areas, such as forestation, listed buildings, marshland, etc. A user study is conducted to evaluate the usability of the system and the quality of roads generated in a diverse range of scenarios. The results indicate that our system is both user-friendly and able to produce roads that are true to the user's intention.

An Interactive Local Flattening Operator to Support Digital Investigations on Artwork Surfaces

Analyzing either high-frequency shape detail or any other 2D fields (scalar or vector) embedded over a 3D geometry is a complex task, since detaching the detail from the overall shape can be tricky. An alternative approach is to move to the 2D space, resolving shape reasoning to easier image processing techniques. In this paper we propose a novel framework for the analysis of 2D information distributed over 3D geometry, based on a locally smooth parametrization technique that allows us to treat local 3D data in terms of image content. The proposed approach has been implemented as a sketch-based system that allows to design with a few gestures a set of (possibly overlapping) parameterizations of rectangular portions of the surface. We demonstrate that, due to the locality of the parametrization, the distortion is under an acceptable threshold, while discontinuities can be avoided since the parametrized geometry is always homeomorphic to a disk. We show the effectiveness of the proposed technique to solve specific Cultural Heritage (CH) tasks: the analysis of chisel marks over the surface of a unfinished sculpture and the local comparison of multiple photographs mapped over the surface of an artwork. For this very difficult task, we believe that our framework and the corresponding tool are the first steps toward a computer-based shape reasoning system, able to support CH scholars with a medium they are more used to.

KnitSketch: A Sketch Pad for Conceptual Design of 2D Garment Patterns

In this paper, we present a new sketch-based system - KnitSketch, to improve the efficiency of process planning for knitting garments at an early design stage. The KnitSketch system utilizes sketching interface with the pen-paper metaphor and users only need to draw outlines of different parts of the garment. Based on sketching understanding, the system automatically makes reasonable geometric inferences about the process-planning data of the garment. The system is designed for nonprofessional users and can design diverse garment styles by freehand drawings. The contributions of this work include contextual extraction of reusable data from sketches, a MDG structure for sketch beautification, and an integrated system with natural expression and effective communication that reduces the cognitive load of human beings. User experience shows that the proposed system helps designers focus on the task instead of the designing tools, and thus improves the efficiency and productivity of human beings.

Contour-driven Sumi-e rendering of real photos

We propose a sketch-based system for rendering oriental brush strokes on complex shapes. While previous research has focused on methods for converting user-specified trajectories into oriental ink painting (Sumi-e) strokes, we propose an approach that takes as input the contours of complex shapes, estimates automatically the sizes of the brush footprints, their orientations, and the brush trajectory. Then, it renders them into oriental ink paintings. The approach performs in three steps; first complex shapes are decomposed into elementary shapes that can be rendered with a single brush stroke. Second, the brush trajectory estimation is formulated as the minimization of an energy function that measures the quality of the trajectory. Finally, the estimated trajectories are rendered into brush strokes by mapping on them footprint textures scanned from real images. We demonstrate that the proposed contour-driven approach is particularly suitable for converting real images into Sumi-e paintings with minimum interaction. To this end we develop a system where images are either interactively or automatically segmented into elementary shapes and converted into Sumi-e paintings. For the automatic segmentation, we observe that the strokes in Sumi-e paintings are nearly parallel to the edge tangent flow (ETF). We propose a segmentation procedure that is based on clustering the curves obtained by integrating the ETF of real images. Experiments on complex shapes show that the proposed contour-based approach produces a large variety of complex strokes compared to trajectory-based approaches.

Geo-Mobile Query-by-Sketch

The advent of wireless technology, such as cellphones, PDAs, tablet PCs, and sub-notebooks, allows transferring portions of traditional, desktop-based GIS technology to mobile environments. This paper introduces Geo-Mobile Query-by-Sketch, a sketch-based spatial querying system for mobile GIS environments that combines techniques for spatial querying with mobile technologies. The system implements an adaptive client-server architecture, which copes with restrictions of mobile environments, such as fluctuating bandwidth and frequent disconnections. The core concept analysed is the mobile sketch, a multi-representation data structure of a sketched scene, which enables an adaptation strategy that is tailored to the available transmission rates. We analyse the transmission cost of Geo-Mobile Query-by-Sketch and develop a protocol that optimises the adaptation level in order to guarantee quality of service.