Freehand Sketches Research Articles

Light-SRNet: a lightweight dual-attention feature fusion network for hand-drawn sketch recognition

Free-hand sketches play a vital role in graphically portraying ideas and concepts in image recognition systems. Most recently proposed learning-based sketch recognition methods have achieved marked progress in recognition accuracy, but they rarely optimize the use of the sparsity features of sketch images. Although several attention-based sketch recognition models have been presented, they endure complex computations and large model sizes. To address these challenges, we present a lightweight convolutional neural network called Light-SRNet based on a dual-attention mechanism to improve the accuracy of sketch recognition while retaining its lightweight nature. In the proposed model, we introduced both the spatial and channel attention mechanisms into the feature extraction network to highlight more discriminative feature representations to enhance its powerful sketch recognition ability. We compared the proposed Light-SRNet with its competitors on the TU-Berlin dataset, Sketchy dataset, and QuickDrawExtended dataset. Extensive experimental results show that Light-SRNet achieves a recognition accuracy of 73.14%, which is comparable to other similar sketch recognition techniques, while requiring only about a quarter of the model parameters.

Deep Learning for Free-Hand Sketch: A Survey.

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community.

Stroke-based semantic segmentation for scene-level free-hand sketches

Stroke-based semantic segmentation for scene-level free-hand sketches

From the mind to the sheet, through the hand. Topicality of freehand sketching in the architectural project.

Nell’epoca in cui la transazione digitale sta sempre più investendo tutti gli ambiti della società, consapevoli del fatto che tale processo è già in atto, da tempo, nell’ambito della definizione, rappresentazione e gestione del progetto architettonico, si ritiene necessario mantenere l’attenzione sulla centralità (non solo formativa) del ruolo del disegno a mano libera, con particolare riferimento allo schizzo, nella fase del processo creativo iniziale del progetto di architettura. La lettura di alcuni elaborati esemplificativi prodotti da figure di spicco quali Pier Luigi Nervi e Renzo Piano, pur nella ovvia diversità degli approcci e degli esiti architettonici, rende evidente il ruolo imprescindibile di questo momento in cui il concept del progetto viene elaborato e affinato in un dialogo intimo che vede nel disegno a mano libera lo strumento ancora oggi primario ed insostituibile per la definizione delle linee e delle caratteristiche del progetto della futura architettura.

Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval.

Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) aims at searching corresponding natural images with the given free-hand sketches, under the more realistic and challenging scenario of Zero-Shot Learning (ZSL). Prior works concentrate much on aligning the sketch and image feature representations while ignoring the explicit learning of heterogeneous feature extractors to make themselves capable of aligning multi-modal features, with the expense of deteriorating the transferability from seen categories to unseen ones. To address this issue, we propose a novel Transferable Coupled Network (TCN) to effectively improve network transferability, with the constraint of soft weight-sharing among heterogeneous convolutional layers to capture similar geometric patterns, e.g., contours of sketches and images. Based on this, we further introduce and validate a general criterion to deal with multi-modal zero-shot learning, i.e., utilizing coupled modules for mining modality-common knowledge while independent modules for learning modality-specific information. Moreover, we elaborate a simple but effective semantic metric to integrate local metric learning and global semantic constraint into a unified formula to significantly boost the performance. Extensive experiments on three popular large-scale datasets show that our proposed approach outperforms state-of-the-art methods to a remarkable extent: by more than 12% on Sketchy, 2% on TU-Berlin and 6% on QuickDraw datasets in terms of retrieval accuracy. The project page is available at: https://haowang1992.github.io/publication/TCN.

DifferSketching

Multiple sketch datasets have been proposed to understand how people draw 3D objects. However, such datasets are often of small scale and cover a small set of objects or categories. In addition, these datasets contain freehand sketches mostly from expert users, making it difficult to compare the drawings by expert and novice users, while such comparisons are critical in informing more effective sketch-based interfaces for either user groups. These observations motivate us to analyze how differently people with and without adequate drawing skills sketch 3D objects. We invited 70 novice users and 38 expert users to sketch 136 3D objects, which were presented as 362 images rendered from multiple views. This leads to a new dataset of 3,620 freehand multi-view sketches, which are registered with their corresponding 3D objects under certain views. Our dataset is an order of magnitude larger than the existing datasets. We analyze the collected data at three levels, i.e., sketch-level, stroke-level, and pixel-level, under both spatial and temporal characteristics, and within and across groups of creators. We found that the drawings by professionals and novices show significant differences at stroke-level, both intrinsically and extrinsically. We demonstrate the usefulness of our dataset in two applications: (i) freehand-style sketch synthesis, and (ii) posing it as a potential benchmark for sketch-based 3D reconstruction. Our dataset and code are available at https://chufengxiao.github.io/DifferSketching/.

The perceived value of human-AI collaboration in early shape exploration: An exploratory assessment.

As a vital element of early shape exploration, divergence can be time-consuming and challenging, with iterative cycles where idea fixation and creative blocks must be overcome for fuzzy ideas to be fully expanded and understood. Despite interesting tools that have been developed for this purpose, some important challenges remain, as it appears that many designers still prefer simple freehand sketching and tend to defer the use of computational tools to later stages. This work presents an exploratory assessment of the perceived value of a new tool, Shapi, developed to assist early shape exploration by addressing some of the pitfalls reported in the literature. Shapi is envisioned as an autonomous assistant that provides local and global shape variations in the form of rough sketches based on an initial human sketch and interactive cycles. These shape variations are What-If scenarios and cognitive facilitators that may spark new ideas or enable a deeper understanding of the shape and the identification of interesting patterns. Shapi’s capabilities are explored in a diverse set of case studies with different purposes: nine implementations in industrial design, three in graphic design, and five with open-ended artistic purposes. These implementations are then used in a survey about initial perceived value in which the majority gave high ratings in terms of exploration (75.5% ≥ 4 out of 5), interpretation (83.7% ≥ 4), adaptation (77.6% ≥ 4), value (73.5% ≥ 4), creativity (69.4% ≥ 4), and general interest in the tool (79.6% ≥ 4). This work brings insight into promising functionalities, opportunities, and risks in the intersection between artificial intelligence, design, and art.

Data-Driven Sketch Beautification With Neural Feature Representation.

This article presents a data-driven approach for beautifying freehand sketches. Our key premise is that the artist-drawn vector can be used to sketch visually appealing shapes, such as local shapes with a clean appearance and better global visual properties (e.g., symmetry). However, these merits may not apply to all object categories. In this article, we use a neural network to represent local and global merits across different object categories to design our beautification method. First, we match sample points between input sketches and the collected vector shapes using the extracted feature representations. Then, we design an optimization problem to ensure resemblance between the deformed sketch and vector shape in the representation space while preserving the semantic meaning and style of the original sketch. Finally, we demonstrate our method on sketches across different shape categories.

A study of deep single sketch-based modeling: View/style invariance, sparsity and latent space disentanglement

Deep image-based modeling has received a lot of attention in recent years. Sketch-based modeling in particular has gained popularity given the ubiquitous nature of touchscreen devices. In this paper, we (i) study and compare diverse single-image reconstruction methods on sketch input, comparing the different 3D shape representations: multi-view, voxel- and point-cloud-based, mesh-based and implicit ones; and (ii) analyze the main challenges and requirements of sketch-based modeling systems. We introduce the regression loss and provide two variants of its formulation for the two most promising 3D shape representations: point clouds and signed distance functions. We show that this loss can increase general reconstruction accuracy, and the view- and style-robustness of the reconstruction methods. Moreover, we demonstrate that this loss can benefit the disentanglement of latent space to view-invariant and view-specific information, resulting in further improved performance. To address the figure-ground ambiguity typical for sparse freehand sketches, we propose a two-branch architecture that exploits sparse user labeling. We hope that our work will inform future research on sketch-based modeling.

Zero-shot sketch-based image retrieval with structure-aware asymmetric disentanglement

The goal of Sketch-Based Image Retrieval (SBIR) is using free-hand sketches to retrieve images of the same category from a natural image gallery. However, SBIR requires all test categories to be seen during training, which cannot be guaranteed in real-world applications. So we investigate more challenging Zero-Shot SBIR (ZS-SBIR), in which test categories do not appear in the training stage. After realizing that sketches mainly contain structure information while images contain additional appearance information, we attempt to achieve structure-aware retrieval via asymmetric disentanglement. For this purpose, we propose our STRucture-aware Asymmetric Disentanglement (STRAD) method, in which image features are disentangled into structure features and appearance features while sketch features are only projected to structure space. Through disentangling structure and appearance space, bi-directional domain translation is performed between the sketch domain and the image domain. Extensive experiments demonstrate that our STRAD method remarkably outperforms state-of-the-art methods on three large-scale benchmark datasets.

On-Site Architectural Drawing with Hand-held Mobile Instructions

The advances in computer-aided design tools have enabled design visualization and realization to become more efficient and effective. However, these fast-growing digital technologies are also gradually reducing the presence of hand drawings in architectural education. This leads to a reduction in the ability to be inspired from the direct observation of the architectural environment through on-site freehand sketching. This study aims to implement digital technology as a teaching aid to retrieve these lost abilities. Analytical drawing is a method that encourages thinking before drawing, laying out the invisible underlying structure, and finalizing it with a visible appearance. This method remains an effective way of three-dimensional visual thinking. Accordingly, this study presents an interactive smartphone application that brings computer-assisted instructions into mobile learning. Promising responses from students revealed that using digital technology as a teaching aid can help to retrieve the lost abilities of visual thinking through on-site sketching.

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs.

Sketch-based image retrieval (SBIR) is a long-standing research topic in computer vision. Existing methods mainly focus on category-level or instance-level image retrieval. This paper investigates the fine-grained scene-level SBIR problem where a free-hand sketch depicting a scene is used to retrieve desired images. This problem is useful yet challenging mainly because of two entangled facts: 1) achieving an effective representation of the input query data and scene-level images is difficult as it requires to model the information across multiple modalities such as object layout, relative size and visual appearances, and 2) there is a great domain gap between the query sketch input and target images. We present SceneSketcher-v2, a Graph Convolutional Network (GCN) based architecture to address these challenges. SceneSketcher-v2 employs a carefully designed graph convolution network to fuse the multi-modality information in the query sketch and target images and uses a triplet training process and end-to-end training manner to alleviate the domain gap. Extensive experiments demonstrate SceneSketcher-v2 outperforms state-of-the-art scene-level SBIR models with a significant margin.

Efficient Facial Recognition Authentication Using Edge and Density Variant Sketch Generator

Image translation plays a significant role in realistic image synthesis, entertainment tasks such as editing and colorization, and security including personal identification. In Edge GAN, the major contribution is attribute guided vector that enables high visual quality content generation. This research study proposes automatic face image realism from freehand sketches based on Edge GAN. We propose a density variant image synthesis model, allowing the input sketch to encompass face features with minute details. The density level is projected into non-latent space, having a linear controlled function parameter. This assists the user to appropriately devise the variant densities of facial sketches and image synthesis. Composite data set of Large Scale CelebFaces Attributes (ClebA), Labelled Faces in the Wild (LFWH), Chinese University of Hong Kong (CHUK), and self-generated Asian images are used to evaluate the proposed approach. The solution is validated to have the capability for generating realistic face images through quantitative and qualitative results and human evaluation.

SketchHairSalon

Recent deep generative models allow real-time generation of hair images from sketch inputs. Existing solutions often require a user-provided binary mask to specify a target hair shape. This not only costs users extra labor but also fails to capture complicated hair boundaries. Those solutions usually encode hair structures via orientation maps, which, however, are not very effective to encode complex structures. We observe that colored hair sketches already implicitly define target hair shapes as well as hair appearance and are more flexible to depict hair structures than orientation maps. Based on these observations, we present SketchHairSalon , a two-stage framework for generating realistic hair images directly from freehand sketches depicting desired hair structure and appearance. At the first stage, we train a network to predict a hair matte from an input hair sketch, with an optional set of non-hair strokes. At the second stage, another network is trained to synthesize the structure and appearance of hair images from the input sketch and the generated matte. To make the networks in the two stages aware of long-term dependency of strokes, we apply self-attention modules to them. To train these networks, we present a new dataset containing thousands of annotated hair sketch-image pairs and corresponding hair mattes. Two efficient methods for sketch completion are proposed to automatically complete repetitive braided parts and hair strokes, respectively, thus reducing the workload of users. Based on the trained networks and the two sketch completion strategies, we build an intuitive interface to allow even novice users to design visually pleasing hair images exhibiting various hair structures and appearance via freehand sketches. The qualitative and quantitative evaluations show the advantages of the proposed system over the existing or alternative solutions.

Freehand-Sketched Part Recognition Using VGG-CapsNet

<p indent=0mm>To solve the problem that the existing CAD system is difficult to match the corresponding parts accurately through the freehand sketch in the conceptual design, a recognition model (VGG-CapsNet) for freehand sketch of part is proposed, which combining the pre-trained network (VGG) and capsule network (CapsNet). Five designers are recruited to sketch parts, and build 23 kinds of freehand sketch of parts including standard parts and non-standard parts. The between-group experiment and within-group experiment are designed, and then the recognition models of VGG-CapsNet are constructed respectively. The recognition results of the VGG-CapsNet models are compared with the rVGG-13 models and the rCNN-13 models. The experimental results show that the mean accuracy of VGG-CapsNet model is higher than the other two models, which provides technical support for the retrieval and reuse of part design knowledge.

Guidance Cleaning Network for Sketch-Based 3D Shape Retrieval

The sketch-based query for 3D shapes has drawn growing attention from the academic community due to its flexibility and accessibility. However, most recent works focus on reducing the cross-modality discrepancy between 2D sketches and 3D shapes, which neglect the influence caused by noise information in some low-quality free-hand sketches. To address this issue, we proposed a novel network to decrease the impact in two ways: 1) an attention weighting module to detect the noisy samples by a self-attention mechanism; 2) a data cleaning module to clear up low-quality sketches according to a ranking regularization. Experiments on two widely-used datasets demonstrate the effectiveness of our method.

Simulating Remote Support for Mathematical Perseverance Through a Digital Sketching Application

This exploratory study showed how a digital sketching application helped keep 4th-grade students engaged with challenging mathematics tasks and support their perseverance to learn mathematics conceptually. The application is currently being developed as a tool based on digital assignments for which students freehand sketch visual representations to solve conceptual fractions tasks. Participants engaged with a simulation of the application and received personalized and conceptual feedback based on their sketching mistakes. Our findings showed that participants were able to leverage such feedback to persevere with the task and make mathematical progress, even at moments when they were most challenged and frustrated.

Methodological Notes on Modelling Reconstructions After Literary Architecture

The manifold relationships between text and image are investigated by analysing attempts at a three-dimensional reconstruction of the Palace of Love and Venus described in Giovan Battista Marino’s Adonis. From text analysis and iconographical precedents to free-hand sketching and three-dimensional modelling, the article describes the progressive translation of some verses of the poem into a consistent system of drawings and models, which, besides revealing some hidden features of text, addresses the implicit limits of the ineffable quality of Marino’s literary space.

Segmentation of Online Freehand Sketching Based on Speed Feature

We present a novel method to extract speed feature points for segmenting hand-drawn strokes into geometric primitives. The method consists of three steps. Firstly, the input strokes are classified into uniform and nonuniform speed strokes, representing a stroke drawn at relatively constant or uneven speeds, respectively. Then, a sharpening filter is used to enhance the peak features of the uniform speed strokes. Finally, a three-threshold technique that uses the average speed of the pen and its upper and lower deviations is used to extract speed feature points of strokes. We integrate the proposed method into our freehand sketch recognition (FSR) system to improve its robustness to support multiprimitive strokes. Through a user study with 8 participants, we demonstrate that the proposed method achieves higher segmentation efficiency in finding speed feature points than the existing method based on a single speed threshold.

Multigraph Transformer for Free-Hand Sketch Recognition

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with convolutional neural networks (CNNs) or the temporal sequential property with recurrent neural networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel graph neural network (GNN), the multigraph transformer (MGT), for learning representations of sketches from multiple graphs, which simultaneously capture global and local geometric stroke structures as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: 1) achieves a small recognition gap to the CNN-based performance upper bound (72.80% versus 74.22%) and infers faster than the CNN competitors and 2) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.