Freehand Sketches Research Articles

DeepFaceReshaping: Interactive deep face reshaping via landmark manipulation

Deep generative models allow the synthesis of realistic human faces from freehand sketches or semantic maps. However, although they are flexible, sketches and semantic maps provide too much freedom for manipulation, and thus, are not easy for novice users to control. In this study, we present DeepFaceReshaping, a novel landmark-based deep generative framework for interactive face reshaping. To edit the shape of a face realistically by manipulating a small number of face landmarks, we employ neural shape deformation to reshape individual face components. Furthermore, we propose a novel Transformer-based partial refinement network to synthesize the reshaped face components conditioned on the edited landmarks, and fuse the components to generate the entire face using a local-to-global approach. In this manner, we limit possible reshaping effects within a feasible component-based face space. Thus, our interface is intuitive even for novice users, as confirmed by a user study. Our experiments demonstrate that our method outperforms traditional warping-based approaches and recent deep generative techniques.

Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-Made Objects.

We present a novel freehand sketch beautification method, which takes as input a freely drawn sketch of a man-made object and automatically beautifies it both geometrically and structurally. Beautifying a sketch is challenging because of its highly abstract and heavily diverse drawing manner. Existing methods are usually confined to their limited training samples and thus cannot beautify freely drawn sketches with both geometric and structural variations. To address this challenge, we adopt a divide-and-combine strategy. Specifically, we first parse an input sketch into semantic components, beautify individual components by a learned part beautification module based on part-level implicit manifolds, and then reassemble the beautified components through a structure beautification module. With this strategy, our method can go beyond the training samples and handle novel freehand sketches. We demonstrate the effectiveness of our system with extensive experiments and a perceptual study.

Neighborhood housing design using fifth generation manifesto towards nearly zero energy urban developments

Abstract Since cities are growing faster than ever before, this paper traces directly that the fifth generation of sustainable city dimensions is not a prosperous idea or a welfare mode of sustainability but a vital effective way to adapt to Climate change, provide energy production, increase carbon reduction, and enhance pedestrian-occupant thermal comfort. It is a systematic method to alleviate urban canyon temperatures, maximize shading in summer, enhance energy production, and minimize heat transfer. It shows better results than designing apartment buildings with parcel planning with its four wall surfaces absorbing heat all day long. Early from schematic design, the emphasis is based on freehand sketching, adjusting urban aspect ratio, courtyard housing closure ratio, compactness degree, and microclimatic orientation, then adding a layer of humanization with green corridors, and tree species numerically modeled based on microclimate with courtyarded passive housing clusters, moving towards detailed design execution with SCEB bearing wall construction having the majority of its mixture of clay which enhances thermal mass insulation, climate adaptation, indoor thermal comfort, and a green job opportunity. Afterward adding energy plus BIPV on the whole constructed microclimatic-based roof to twist the neighborhood into a power bank saving energy consumption, and to work as a double roof shading. It shows on the other urbanism coin side walkability, sequential visual order, urban identity, and architectural character. Numerical outdoor-indoor coupled simulations took place using Envi-met and DesignBuilder while meteorological and biometeorological parameters were measured. The new recipe ignites a sense of harmonizing results in enhancing environmental sustainability measures.

Text-guided image-to-sketch diffusion models

Recently, with the continuous advancement of deep learning techniques, research on sketch synthesis has been progressing. However, existing methods still face challenges in generating human-like freehand sketches from real-world natural images at both object and scene levels. To address this, we propose SketchDiffusion, a text-guided freehand sketch synthesis method based on conditional stable diffusion. In SketchDiffusion, we design a novel image enhancing module to efficiently extract high-quality image features. Moreover, we utilize additional guidance from global and local features extracted by a U-shaped diffusion guidance network to control the noise addition and denoising process of the diffusion model, thereby significantly improving controllability and performance in freehand sketch synthesis. Beyond the model architecture, we leverage the designed BLIP-based text generation method to create 70,280 text prompts for foreground, background, and panorama sketch synthesis in the extensive SketchyCOCO dataset, thereby improving the overall effectiveness of model training. Compared to the state-of-the-art methods, our proposed SketchDiffusion has shown an average improvement of over 16.4%, 16.75%, and 12.8% on three quantitative metrics (sketch recognition, sketch-based retrieval, and user perceptual study), respectively. Furthermore, our approach not only excels in synthesizing freehand sketches containing multiple abstract objects but also has multiple applications in supporting human–computer interaction.

To draw or not to draw? Evaluating the impact of computer-aided drawing tools on productivity in architectural design

This empirical study investigates the architectural design process, focusing on the influence of computer-aided drawing (CAD) tools during the initial design stages. Given the increasing reliance on CAD tools in contemporary architectural practice, the primary objective is to assess quantitatively how CAD tool adoption affects design productivity. The research was conducted in two phases, utilizing the protocol analysis method. In the first phase, we introduced a novel approach by having third-year architecture students complete two design assignments: the first using traditional freehand sketches and the second employing CAD tools. This unique method allowed us to capture their design actions and thoughts using the Concurrent verbalization technique. In the second phase, a meticulous comparison of the two sets of design protocols was undertaken. The analysis yielded surprising and counterintuitive results. Contrary to common beliefs, the use of CAD tools during early design phases can lead to unintended outcomes. Despite their digital precision and convenience, these tools may inadvertently impede cognitive productivity. Designers employing CAD tools showed signs of hesitation and indecision, resulting in prolonged design timelines and diminished ideational productivity. These findings have profound implications for architectural education and practice. They underscore the necessity for a balanced approach to integrating CAD tools into design processes. Educators and professionals must carefully consider the potential impact of these tools on the creative process, prompting a reassessment of their role in architectural design curricula and workflows. Future research holds great promise in exploring strategies to optimize CAD tool usage to mitigate their disruptive effects on early-stage architectural design. Additionally, investigating the role of training and interface design in enhancing the synergy between designers and digital tools offers exciting and fruitful research directions. In summary, this study highlights the complex relationship between CAD tools and cognitive productivity in architectural design, prompting a reexamination of their role in shaping the design landscape.

SMFS‐GAN: Style‐Guided Multi‐class Freehand Sketch‐to‐Image Synthesis

AbstractFreehand sketch‐to‐image (S2I) is a challenging task due to the individualized lines and the random shape of freehand sketches. The multi‐class freehand sketch‐to‐image synthesis task, in turn, presents new challenges for this research area. This task requires not only the consideration of the problems posed by freehand sketches but also the analysis of multi‐class domain differences in the conditions of a single model. However, existing methods often have difficulty learning domain differences between multiple classes, and cannot generate controllable and appropriate textures while maintaining shape stability. In this paper, we propose a style‐guided multi‐class freehand sketch‐to‐image synthesis model, SMFS‐GAN, which can be trained using only unpaired data. To this end, we introduce a contrast‐based style encoder that optimizes the network's perception of domain disparities by explicitly modelling the differences between classes and thus extracting style information across domains. Further, to optimize the fine‐grained texture of the generated results and the shape consistency with freehand sketches, we propose a local texture refinement discriminator and a Shape Constraint Module, respectively. In addition, to address the imbalance of data classes in the QMUL‐Sketch dataset, we add 6K images by drawing manually and obtain QMUL‐Sketch+ dataset. Extensive experiments on SketchyCOCO Object dataset, QMUL‐Sketch+ dataset and Pseudosketches dataset demonstrate the effectiveness as well as the superiority of our proposed method.

Winding Number Features for Vector Sketch Colorization

AbstractVector sketch software (e.g. Adobe Illustrator, Inkscape) and touch‐interactive technologies have long aided artists in the creation of resolution‐independent digital drawings that mimic the unconstrained nature of freehand sketches. However, artist intent behind stroke topology is often ambiguous, complicating traditional segmentation tasks such as coloring. For inspiration, we turn to the winding number, a classic geometric property of interest for binary segmentation in the presence of boundary data. Its direct application for multi‐region segmentation poses two main challenges: (1) strokes may not be consistently oriented to best identify perceptually salient regions; (2) for interior strokes there is no “correct” orientation, as either choice better distinguishes one of two neighboring regions. Thus, we form a harmonic feature space from multiple winding number fields and perform segmentation via Voronoi/power diagrams in this domain. Our perspective allows both for automatic fill region detection and for a semi‐automatic framework that naturally incorporates user hints and interactive sculpting of results, unlike competing automatic methods. Our method is agnostic to curve orientation and gracefully handles varying gap sizes in the sketch boundary, outperforming state‐of‐the‐art colorization methods on these “gappy” inputs. Moreover, it inherits the ability of winding numbers to specify “fuzzy” boundaries, leading to simple strategies for color diffusion and single‐parameter‐driven growing and shrinking of regions.

3Doodle: Compact Abstraction of Objects with 3D Strokes

While free-hand sketching has long served as an efficient representation to convey characteristics of an object, they are often subjective, deviating significantly from realistic representations. Moreover, sketches are not consistent for arbitrary viewpoints, making it hard to catch 3D shapes. We propose 3Dooole, generating descriptive and view-consistent sketch images given multi-view images of the target object. Our method is based on the idea that a set of 3D strokes can efficiently represent 3D structural information and render view-consistent 2D sketches. We express 2D sketches as a union of view-independent and view-dependent components. 3D cubic Bézier curves indicate view-independent 3D feature lines, while contours of superquadrics express a smooth outline of the volume of varying viewpoints. Our pipeline directly optimizes the parameters of 3D stroke primitives to minimize perceptual losses in a fully differentiable manner. The resulting sparse set of 3D strokes can be rendered as abstract sketches containing essential 3D characteristic shapes of various objects. We demonstrate that 3Doodle can faithfully express concepts of the original images compared with recent sketch generation approaches. 1

Design And Development of Sketch Based Image Retrieval Using Deep Learning

In this cutting edge, the common wrong doing rate is expanding day-by-day and to manage up with this the criminal divisions as well ought to discover ways in which would speed up the by and large preparation and offer assistance in bringing one to justice. In response to rising crime rates, law enforcement agencies are turning to advanced algorithms capable of matching freehand sketches with images in databases. These algorithms, utilizing sophisticated feature extraction techniques and deep learning models, significantly enhance identification accuracy. By leveraging Sketch based image retrieval technology, investigations are expedited, leading to quicker suspect apprehensions and resolution of criminal cases. This results in improved public safety and justice outcomes, as well as more efficient law enforcement practices overall.

Enhance Sketch Recognition’s Explainability via Semantic Component-Level Parsing

Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.

Tacit knowledge of places. Experiences from the post-agrarian countryside

This article is framed by a broader discussion on the new meanings conferred by rural areas, and the construction of rurality recently undertaken by human geographers. The aim of the research presented here has been to identify perceptions of the countryside in Poland, at a time when a fundamental transition from agrarian to post-agrarian communities has already taken place. After a briefreview of the literature and a presentation of the contemporary conditions of rural development in Poland, the paper presents the results of the study conducted in 22 pre-selected villages in Poland. The social research involved semi-structured interviews and freehand sketches exemplifying tacit knowledge about these places, i.e. the symbolic meaning attributed by people to their living spaces. Results of this research show that uncovering this knowledge generates a variety of information useful in interpreting contemporary rurality as shaped by dynamic change in Poland’s economy and social life.

Deep3DSketch-im: rapid high-fidelity AI 3D model generation by single freehand sketches

Deep3DSketch-im: rapid high-fidelity AI 3D model generation by single freehand sketches

Cross-Modal Pixel-and-Stroke representation aligning networks for free-hand sketch recognition

We consider the cross-modal alignment problem for free-hand sketch. Given a sequence of stroke and a rasterized image, the objective is to enhance the performance of sketch recognition through cross-modal interactions. Existing works mostly employ simple weighted adding and concatenation for late fusion, or shallow attention layers for cross-modal alignment. Due to the high heterogeneity between sketch modalities, these methods do not capture meaningful feature representations sufficiently. In this paper, we propose a sketch recognition framework CMPS for aligning Cross-Modal Pixel-and-Stroke representation, which includes novel components, namely the Semantic-Temporal Alignment Rasterization (STAR) and Pixel-Stroke Alignment (PSA) module. STAR aligns stroke with image at the semantic and temporal levels during the rasterization preprocessing phase by utilizing color variations in the RGB space for sketch. PSA, through its pre-alignment and post-alignment, learns how to align semantic connections at both pixel and stroke levels, capturing cross-modal dependencies, rather than relying on shallow matrix operations for interaction. Additionally, we introduce a concise stroke processing network called StrokeFormer. It extracts two hierarchical features, i.e., point-level and stroke-level, based on the formation mechanism of sketch. StrokeFormer outperforms most RNN-based and CNN-based models by a significant margin. Our experimental results demonstrate that proposed CMPS achieves new state-of-the-art performance on the Google QuickDraw-414 K dataset and TU-Berlin dataset. The code is available at https://github.com/WoodratTradeCo/CMPS.

A Diagnostic Analysis of Observational Sketching: Examples from the University of Latvia

The importance of freehand sketching is being updated and revised at a time when sketching by hand is, in many cases, being replaced by sketching with digital technologies. In addition, in Latvia, since the reform of the general education curriculum, sketching has been included in the new primary education curriculum, which requires primary school teachers to have experience and understanding of sketching. Freehand sketching is also part of the curriculum for future designers’ education. Researchers at the University of Latvia developed a task, criteria, and a description of the assessment levels (rubric) for sketching from an image to assess the initial preparedness and observational sketching skills of students on design and primary school education teachers’ programmes. The conclusion was that students’ sketching skills could be developed and extended by encouraging the use of different technical approaches and means of expression, as well as by practising the accuracy of observation.

DeepPortraitDrawing: Generating human body images from freehand sketches

Various methods for generating realistic images of objects and human faces from freehand sketches have been explored. However, generating realistic human body images from sketches is still a challenging problem. It is, first because of the sensitivity to human shapes, second because of the complexity of human images caused by body shape and pose changes, and third because of the domain gap between realistic images and freehand sketches. In this work, we present DeepPortraitDrawing, a deep generative framework for converting roughly drawn sketches to realistic human body images. To encode complicated body shapes under various poses, we take a local-to-global approach. Locally, we employ semantic part auto-encoders to construct part-level shape spaces, which are useful for refining the geometry of an input pre-segmented hand-drawn sketch. Globally, we employ a cascaded spatial transformer network to refine the structure of body parts by adjusting their spatial locations and relative proportions. Finally, we use a style-based generator as the global synthesis network for the sketch-to-image translation task which is modulated by segmentation maps for semantic preservation. Extensive experiments have shown that given roughly sketched human portraits, our method produces more realistic images than the state-of-the-art sketch-to-image synthesis techniques.

Level Perspective

Abstract: Level Perspectives or so-called (Parallel and Angular Perspectives) appear when the camera is set level. In other words, the center line of vision is level. Such perspective drawings appear to converge in one or two vanishing points, in accordance with the position of the object with respect to the observer. Thus, the verticals will always remain vertical in Level Perspective. Those who study perspective drawing are familiar with only one method that involves the Station Point to define the verticals in perspective. However, horizontal structures, such as, patterns on the floor or ceiling, or any top projection would be hard to draw by applying the conventional method. My approach is using a method appropriate to the line-directions in the top view or the plan. Thus, there is one method for each of the skew lines, vertical lines and horizontal lines in the plan. Each method is simple as it only consists of two or three lines. The conventional method also consists of two lines which is appropriate for all vertical edges, but it can’t be applied to level structures. Interior Designs are hard to do using the conventional method. That is the reason to use freehand sketches in interior designs. However, it will be seen how easy to do interior perspectives using my methods.

Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval

Sketch-based image retrieval (SBIR) is an attractive research area where freehand sketches are used as queries to retrieve relevant images. Existing solutions have advanced the task to the challenging zero-shot setting (ZS-SBIR), where the trained models are tested on new classes without seen data. However, they are prone to overfitting under a realistic scenario when the test data includes both seen and unseen classes. In this paper, we study generalized ZS-SBIR (GZS-SBIR) and propose a novel semi-transductive learning paradigm. Transductive learning is performed on the image modality to explore the potential data distribution within unseen classes, and zero-shot learning is performed on the sketch modality sharing the learned knowledge through a semi-heterogeneous architecture. A hybrid metric learning strategy is proposed to establish semantics-aware ranking property and calibrate the joint embedding space. Extensive experiments are conducted on two large-scale benchmarks and four evaluation metrics. The results show that our method is superior over the state-of-the-art competitors in the challenging GZS-SBIR task.

Interactive pose and shape editing with simple sketches from different viewing angles

The challenge of generating and creating humanoid 3D models is nowadays omnipresent in animation films, video games, and many other applications. To enhance and speed up this costly process, we propose an interactive method to infer 3D human poses and shapes from simple free-hand sketches. In contrast to previous work, our sketches are easy and fast to draw even for inexperienced users. Reconstructing the human 3D pose and shape from a single sketch is difficult due to the perspective projection resulting in ambiguous representations. We approach this challenge by adding a novel multi-view sketch interface for human pose estimation. Our simple sketch style allows for fast and intuitive, iterative refinement of not only the pose but also the shape using simple 2D interactions. The pose is inferred from a pre-trained neural network that is trained on a self-generated data set of sketches, derived by sampling 3D STAR models and rendering them in a sketch style. We conduct a user study to evaluate the accuracy, intuitiveness, and usability of our system in a real-world scenario. With our novel multi-view input system, we can effectively resolve ambiguities inherent to single sketch approaches while still delivering a fast and intuitive user experience.

Applications of augmented reality and virtual reality on computer-assisted teaching for analytical sketching of architectural scene and construction

ABSTRACT Analytical sketching is a method to document an inspired understanding from direct observations and analysis of an architectural environment through freehand sketching. A course dedicated to teaching analytical sketching has encountered problems of how to integrate different teaching activities involving in-class lectures and on-site practices. A mobile learning application (Architectural Sketching) was previously developed to bring the in-class instruction on hand to the field. However, how to assist in the development of key sketching ability of interrelating conceptual architectural configurations and their perceptual spatial experiences through mobile learning remains unexplored. This study proceeds from prior action research and reports the experimental implementation of augmented reality and virtual reality to a mobile learning application. To teach architectural scene sketching, live demonstrations of on-site sketching were provided in different interactive display methods of viewing as regular video, interactive 180-degree video on handheld devices, and 180-degree VR video with VR headset. For architectural construction, augmented reality and 3D holograms, along with 3D animations of geometrical analysis, were used to create digital learning tools. Preference rating and preference scales were established from students’ responses. The results reveal that the user-friendly interactivity and functional realism enhance the mobile learning applications to assist on-site sketching training.

3D model retrieval based on interactive attention CNN and multiple features.

3D (three-dimensional) models are widely applied in our daily life, such as mechanical manufacture, games, biochemistry, art, virtual reality, and etc. With the exponential growth of 3D models on web and in model library, there is an increasing need to retrieve the desired model accurately according to freehand sketch. Researchers are focusing on applying machine learning technology to 3D model retrieval. In this article, we combine semantic feature, shape distribution features and gist feature to retrieve 3D model based on interactive attention convolutional neural networks (CNN). The purpose is to improve the accuracy of 3D model retrieval. Firstly, 2D (two-dimensional) views are extracted from 3D model at six different angles and converted into line drawings. Secondly, interactive attention module is embedded into CNN to extract semantic features, which adds data interaction between two CNN layers. Interactive attention CNN extracts effective features from 2D views. Gist algorithm and 2D shape distribution (SD) algorithm are used to extract global features. Thirdly, Euclidean distance is adopted to calculate the similarity of semantic feature, the similarity of gist feature and the similarity of shape distribution feature between sketch and 2D view. Then, the weighted sum of three similarities is used to compute the similarity between sketch and 2D view for retrieving 3D model. It solves the problem that low accuracy of 3D model retrieval is caused by the poor extraction of semantic features. Nearest neighbor (NN), first tier (FT), second tier (ST), F-measure (E(F)), and discounted cumulated gain (DCG) are used to evaluate the performance of 3D model retrieval. Experiments are conducted on ModelNet40 and results show that the proposed method is better than others. The proposed method is feasible in 3D model retrieval.