Freehand Sketches Research Articles

Freehand Sketching Portrait Recognition with Least Square CycleGAN

Freehand sketching portrait recognition refers to the recognition of sketched portraits and face photos drawn by artists. Existing research mainly involves inputting a given real portrait and converting it into a similar sketch portrait, and then matching the face with the sketch portrait in the real portrait database. This paper starts from the idea of inputting a sketch image to identify the real portrait in the database, focuses on the research based on the method of portrait synthesis, and introduces the existing methods of sketch portrait identification. We use the CycleGAN improved by least squares to achieve the translation from sketch to portrait, and finally PCA was used to complete face matching. The results show that LS_CycleGAN has certain advantages. Compared with other methods, its synthesis results are most close to the portrait. The average score of SSIM and PSNR is 0.844 and 17.331, and the average success rate of recognition is 88.2% for Rank10.

Sketch-specific data augmentation for freehand sketch recognition

Sketch recognition remains a significant challenge due to the limited training data and the substantial intra-class variance of freehand sketches for the same object. Conventional methods for this task often rely on the availability of the temporal order of sketch strokes, additional cues acquired from different modalities and supervised augmentation of sketch datasets with real images, which also limit the applicability and feasibility of these methods in real scenarios.In this paper, we propose a novel sketch-specific data augmentation (SSDA) method that leverages the quantity and quality of the sketches automatically. From the aspect of quantity, we introduce a Bezier pivot based deformation (BPD) strategy to enrich the training data. Towards quality improvement, we present a mean stroke reconstruction (MSR) approach to generate a set of novel types of sketches with smaller intra-class variances. Both of these solutions are unrestricted from any multi-source data and temporal cues of sketches. Furthermore, we show that some recent deep convolutional neural network models that are trained on generic classes of real images can be better choices than most of the elaborate architectures that are designed explicitly for sketch recognition. As SSDA can be integrated with any convolutional neural networks, it has a distinct advantage over the existing methods. Our extensive experimental evaluations demonstrate that the proposed method achieves the state-of-the-art results (84.27%) on the TU-Berlin dataset, outperforming the human performance by a remarkable 11.17% increase. Finally, more experiments show the practical value of our approach for the task of sketch-based image retrieval.

Controllable stroke-based sketch synthesis from a self-organized latent space

Learning to synthesize free-hand sketches controllably according to specified categories and sketching styles is a challenging task, due to the lack of training data with category labels and style labels. One choice to control the synthesis is by self-organizing a latent coding space to preserve the similarity of structural patterns of the observed data. A practical way is introducing a Gaussian mixture prior over the latent codes, where each Gaussian component represents a specific categorical or stylistic pattern. As a result, we can generate sketches by sampling the latent variables from the Gaussian components or continuously manipulating the latent representations by interpolation. To achieve robust controllable sketch synthesis, it is critical to determine an appropriate Gaussian number. An underestimated Gaussian number cannot fully represent all the sketch patterns, i.e., some clusters have to contain sketches with more than one pattern. An overestimated one introduces redundant components, usually representing a chaotic collection of sketches with diverse patterns featured by other components. Both cases disturb pattern clustering over the coding space and make the internal code generation difficult to control for specific patterns. However, the Gaussian number is unavailable in this unsupervised task. In this paper, we present Rival Penalized Competitive Learning pixel to sequence (RPCL-pix2seq) to automatically determine the Gaussian number. Both quantitative and qualitative experimental results show RPCL-pix2seq can partition the codes for the sketches into an approximate stable number of clusters. Hence, we are able to do synthesis reasoning over the latent space, generating novel but reasonable sketches which neither appear in the training dataset nor exist in real life.

AI-Based Topology Optimization of Freehand Sketches

In order to minimize the effort involved in conceptualizing lightweight designs, this paper presents a novel approach to topology optimization that helps designers to enhance their initial sketches for a lightweight structure. The presented tool can interpret photographs of design sketches and proposes optimized flows of forces. At the core of this tool is an artificial neural network being trained to generate structures matching the sketched load cases. The findings include an intuitive means of communication between designers and an artificial intelligence tool, ways for implementing the tool, and the appropriate means of structuring and training the neural network.

Research on Recognition Effect of DSCN Network Structure in Hand-Drawn Sketch.

With the rapid development of image recognition technology, freehand sketch recognition has attracted more and more attention. How to achieve good recognition effect in the absence of color and texture information is the key to the development of freehand sketch recognition. Traditional nonlearning classical models are highly dependent on manual selection features. To solve this problem, a neural network sketch recognition method based on DSCN structure is proposed in this paper. Firstly, the stroke sequence of the sketch is drawn; then, the feature is extracted according to the stroke sequence combined with neural network, and the extracted image features are used as the input of the model to construct the time relationship between different image features. Through the control experiment on TU-Berlin dataset, the results show that, compared with the traditional nonlearning methods, HOG-SVM, SIFT-Fisher Vector, MKL-SVM, and FV-SP, the recognition accuracy of DSCN network is improved by 15.8%, 10.3%, 6.0%, and 2.9%, respectively. Compared with the classical deep learning model, Alex-Net, the recognition accuracy is improved by 5.6%. The above results show that the DSCN network proposed in this paper has strong ability of feature extraction and nonlinear expression and can effectively improve the recognition accuracy of hand-painted sketches after introducing the stroke order.

Dark-Aware Network For Fine-Grained Sketch-Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) is an emerging topic in high-level computer vision. Among existing methods, edge-only information based ones are convenient to use but incompetent in distinguishing among ambiguous samples. On the other hand, even though the methods using additional color information can significantly improve the retrieval performance, they are not convenient for practical use as the increase of sketch complexity. This paper fills the gap between these two kinds of methods by proposing a novel method, namely dark-aware sketch-based image retrieval (DA-SBIR), which incorporates the dark region information of images into FG-SBIR without a sacrifice of convenience. Our model consists of two branches to process edge structure and dark region, respectively. Specifically, instead of using real color, DA-SBIR only requires some additional free-hand black sketch to represent the dark region. Besides, a Split Generative Adversarial Network is introduced to automatically split a sketch into edge-structure-only and dark-region-only sketches. Moreover, we build a new clothes dataset, SJTU-Cloth, with more ambiguous samples for FG-SBIR. Experimental results on the QMUL-Shoe dataset and our SJTU-Cloth dataset show that our approach achieves consistent improvements over state-of-the-arts. Code and dataset are available at https://github.com/y2242794082/SBIR.git.

Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation

Categorizing free-hand human sketches has profound implications in applications such as human computer interaction and image retrieval. The task is non-trivial due to the iconic nature of sketches, signified by large variances in both appearance and structure when compared with photographs. Despite recent advances made by deep learning methods, the requirement of a large training set is commonly imposed making them impractical for real-world applications where training sketches are cumbersome to obtain - sketches have to be hand-drawn one by one other than crawled freely on the Internet. In this work, we aim to delve further into the data scarcity problem of sketch-related research, by proposing a few-shot sketch classification framework. The model is based on a co-regularized embedding algorithm where common/shareable parts of learned human sketches are exploited, thereby can embed query sketch into a co-regularized sparse representation space for few-shot classification. A new dataset of 8,000 part-level annotated sketches of 100 categories is also proposed to facilitate future research. Experiment shows that our approach can achieve an 5-way one-shot classification accuracy of 85%, and 20-way one-shot at 51%.

Volumetric lung nodule segmentation in thoracic CT scan using freehand sketch

The authors presented a novel computerised scheme to segment pulmonary nodules using freehand sketch. Here, freehand sketching is considered to identify the location of a nodule and it serves as a natural way and also it provides inferring adaptive information i.e. size, density, texture and mass centre etc. The proposed scheme includes two phases. In first phase using the freehand sketch analysis the multi seed points to select RoI. In the second phase nodule volumetric extraction is done using geometric modelling and implicit surface reconstruction for volumetric analysis. Spherical bins are used for ray triangle intersections and then local implicit surface fitting and blending method for surface reconstruction and depiction. The performance of the proposed scheme is assessed by accuracy and consistency using 112 CT examinations from LIDC. The IoU and ASD were used to assess the discrepancy between proposed method and inter observer agreement in the proposed approach. In estimating the reproducibility, the discrepancy in proposed scheme and the manual contouring by the expert is observed to be on an average of 0.13 ± 0.07 mm and 3.04 ± 1.7 mm respectively. The experiment shows that, the proposed scheme performs reasonably well and demonstrate merit of freehand sketch.

Deep Semantic Parsing of Freehand Sketches With Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning

In this paper, we propose a novel deep framework for part-level semantic parsing of freehand sketches, which makes three main contributions that are experimentally shown to have substantial practical merit. First, we propose a homogeneous transformation method to address the problem of domain adaptation. For the task of sketch parsing, there is no available data of labeled freehand sketches that can be directly used for model training. An alternative solution is to learn from datasets of real image parsing, while the domain adaptation is an inevitable problem. Unlike existing methods that utilize the edge maps of real images to approximate freehand sketches, the proposed homogeneous transformation method transforms the data from domains of real images and freehand sketches into a homogeneous space to minimize the semantic gap. Second, we design a soft-weighted loss function as guidance for the training process, which gives attention to both the ambiguous label boundary and class imbalance. Third, we present a staged learning strategy to improve the parsing performance of the trained model, which takes advantage of the shared information and specific characteristic from different sketch categories. Extensive experimental results demonstrate the effectiveness of the above three methods. Specifically, to evaluate the generalization ability of our homogeneous transformation method, additional experiments for the task of sketch-based image retrieval are conducted on the QMUL FG-SBIR dataset. Finally, by integrating the proposed three methods into a unified framework of deep semantic sketch parsing (DeepSSP), we achieve the state-of-the-art on the public SketchParse dataset.

Fine-Grained Instance-Level Sketch-Based Image Retrieval

The problem of fine-grained sketch-based image retrieval (FG-SBIR) is defined and investigated in this paper. In FG-SBIR, free-hand human sketch images are used as queries to retrieve photo images containing the same object instances. It is thus a cross-domain (sketch to photo) instance-level retrieval task. It is an extremely challenging problem because (i) visual comparisons and matching need to be executed under large domain gap, i.e., from black and white line drawing sketches to colour photos; (ii) it requires to capture the fine-grained (dis)similarities of sketches and photo images while free-hand sketches drawn by different people present different levels of deformation and expressive interpretation; and (iii) annotated cross-domain fine-grained SBIR datasets are scarce, challenging many state-of-the-art machine learning techniques, particularly those based on deep learning. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based object instance retrieval application. Specifically, a new large-scale FG-SBIR database is introduced which is carefully designed to reflect the real-world application scenarios. A deep cross-domain matching model is then formulated to solve the intrinsic drawing style variability, large domain gap issues, and capture instance-level discriminative features. It distinguishes itself by a carefully designed attention module. Extensive experiments on the new dataset demonstrate the effectiveness of the proposed model and validate the need for a rigorous definition of the FG-SBIR problem and collecting suitable datasets.

Leveraging Free-Hand Sketches for Potential Screening of PTSD

Post-traumatic stress disorder (PTSD) negatively influences a person's ability to cope and increases psychiatric morbidity. The existing diagnostic tools of PTSD are often difficult to administer within marginalized communities due to language and cultural barriers, lack of skilled clinicians, and stigma around disclosing traumatic experiences. We present an initial proof of concept for a novel, low-cost, and creative method to screen the potential cases of PTSD based on free-hand sketches within three different communities in Bangladesh: Rohingya refugees (n = 44), slum-dwellers (n = 35), and engineering students (n = 85). Due to the low overhead and nonverbal nature of sketching, our proposed method potentially overcomes communication and resource barriers. Using corner and edge detection algorithms, we extracted three features (number of corners, number and average length of strokes) from the images of free-hand sketches. We used these features along with sketch themes, participants' gender and group to train multiple logistic regression models for potentially screening PTSD (accuracy: 82.9-87.9%). We improved the accuracy (99.29%) by integrating EEG data with sketch features in a Random Forest model for the refugee population. Our proposed initial assessment method of PTSD based on sketches could potentially be integrated with phones and EEG headsets, making it widely accessible to the underrepresented communities.

DeepFaceDrawing

Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch. We take a local-to-global approach. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

A New Car Concept Developed with Stylistic Design Engineering (SDE)

In this work, a structured design method, the Stylistic Design Engineering (SDE), is applied for the construction of a new minivan car, in particular a new city car, which we will call FIAT 600 Omega. The SDE, or Stylistic Design Engineering, is a structured engineering method for carrying out automotive design projects. The SDE method consists of six different phases: (1) Analysis of stylistic trends; (2) Sketches; (3) 2D Computer Aided Design (CAD) drawings; (4) 3D CAD models; (5) Rendering; (6) Solid stylistic model (also called style maquette). This project deals with the external redesign of the Fiat 600 multiple, a small minivan which was very successful in the 1950s and 1960s. SDE is a methodology consisting of various technologies and design methodologies that will be further explained in detail, such as the Pininfarina method, the Quality Function Deployment (QFD) method, Benchmarking (BM), and Top Flop Analysis (TPA). The work was organized according to the different phases. Initially, the Fiat style was studied, in particular the style of the FIAT 600 MULTI PURPOUSE VEHICLE (MPV). This step is essential to better understand the characteristics of the brand and also the main characteristics carried out over the decades. Then we moved on to the freehand sketching phase, based on what we learned in the previous phase of the study. When a satisfactory shape was found for the new car, by analyzing and discarding the different proposals of the various types of style, we proceeded to the evaluation of the proportions and dimensions through two-dimensional drawings and finally we obtained the three-dimensional shape of the new car thanks to 3D CAD software and rendering software. Many advantages in the industrial world SDE takes together with its development. In fact, until the early 2000s, car design and styling was considered quite a craft activity, not a technical or scientific one, mostly based on the great capability of famous car designers and masters, just like Giugiaro, Zagato, Bertone, Pininfarina, Stephenson, Bangle, etc. Then, thanks to the industrial activity of Eng. Lorenzo Ramacciotti, former CEO of Pininfarina Spa and Mechanical Engineer, and also thanks to the academic studies developed at ALMA MATER STUDIORUM University of Bologna, SDE became the object of attention, because it is able to systematize the car design process and reduce costs. With SDE, a good design research or an industrial product development team can complete a car design project, also without the presence of a mentor. Car Design Process finally becomes with SDE a scientific method; Car Design becomes with SDE an industrial method. Industrial needs are nice products made in a short time; SDE is structured to attend these issues. Industrial challenges follow innovation, in shape and functionality; SDE is able to recognize innovation. Industrial benefits can be reached with SDE, ensuring beautiful aesthetic projects are realized systematically and with low costs.

Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval

Sketch based image retrieval (SBIR), which uses free-hand sketches to search the images containing similar objects/scenes, is attracting more and more attentions as sketches could be got more easily with the development of touch devices. However, this task is difficult as the huge differences between sketches and images. In this paper, we propose a cross-domain representation learning framework to reduce these differences for SBIR. This framework aims to transfer sketches to images with the information learned both in the sketch domain and image domain by the proposed domain migration generative adversarial network (DMGAN). Furthermore, to reduce the representation gap between the generated images and natural images, a similarity learning network (SLN) is also proposed with the new designed loss function incorporating semantic information. Extensive experiments have been done from different aspects, including comparison with state-of-the-art methods. The results show that the proposed DMGAN and SLN really work for SBIR.

SketchDesc: Learning Local Sketch Descriptors for Multi-View Correspondence

In this article, we study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object and predict as output the semantic correspondence among the sketches. This problem is challenging since the visual features of corresponding points at different views can be very different. To this end, we take a deep learning approach and learn a novel local sketch descriptor from data. We contribute a training dataset by generating the pixel-level correspondence for the multi-view line drawings synthesized from 3D shapes. To handle the sparsity and ambiguity of sketches, we design a novel multi-branch neural network that integrates a patch-based representation and a multi-scale strategy to learn the pixel-level correspondence among multi-view sketches. We demonstrate the effectiveness of our proposed approach with extensive experiments on hand-drawn sketches and multi-view line drawings rendered from multiple 3D shape datasets.

Tolerance Zone-Based Grouping Method for Online Multiple Overtracing Freehand Sketches

Multiple overtracing strokes are common drawing behaviors in freehand sketching; that is, additional strokes are often drawn repeatedly over the existing ones to add more details. This paper proposes a method based on stroke-tolerance zones to group multiple overtraced strokes which are drawn to express a 2D primitive, aiming to convert online freehand sketches into 2D line drawings, which is a base for further 3D reconstruction. Firstly, after the user inputs a new stroke, a tolerance zone around the stroke is constructed by reference to its polygonal approximation points obtained from the stroke preprocessing. Then, the input strokes are divided into stroke groups, each representing a primitive through the stroke grouping process based on the overtraced ratio of two strokes. At last, each stroke group is fitted into one or more 2D geometric primitives including line segments, polylines, ellipses, and arcs. The proposed method groups two strokes together based on their screen-space proximity directly instead of classifying and fitting them firstly, so that it can group strokes of arbitrary shapes. A sketch-recognition prototype system has been implemented to test the effectiveness of the proposed method. The results showed that the proposed method could support online multiple overtracing freehand sketching with no limitation on drawing sequence, but it only deals with strokes with relatively high overtraced ratio.

Deep Self-Supervised Representation Learning for Free-Hand Sketch

In this paper, we tackle for the first time, the problem of self-supervised representation learning for free-hand sketches. This importantly addresses a common problem faced by the sketch community – that annotated supervisory data are difficult to obtain. This problem is very challenging in which sketches are highly abstract and subject to different drawing styles, making existing solutions tailored for photos unsuitable. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs: (i) we propose a set of pretext tasks specifically designed for sketches that mimic different drawing styles, and (ii) we further exploit the use of the textual convolution network (TCN) together with the convolutional neural network (CNN) in a dual-branch architecture for sketch feature learning, as means to accommodate the sequential stroke nature of sketches. We demonstrate the superiority of our sketch-specific designs through two sketch-related applications (retrieval and recognition) on a million-scale sketch dataset, and show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods, and significantly narrows the performance gap between with supervised representation learning. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> PyTorch code of this work is available at <uri>https://github.com/zzz1515151/self-supervised_learning_sketch</uri>. </fn>

Sketch based modeling and editing via shape space exploration

This paper proposes a framework that takes a user provided free-hand sketch as input and realistic 3D shape as result. Specifically, the proposed approach retrieves 3D shape based on a sketch, and then a deformation procedure is carried out for a further consistency with the provided sketch. In the retrieval stage, a locality preserving view selection scheme is proposed, which suggests views that are mostly possible to be views when creating a sketch. The proposed criterion predicates the sketch views accurately while significantly reduces the number of views that need to be rendered. In the deformation stage, retrieved shapes are modified according to sketch. However, it is a tremendously difficult job as free-hand sketches always contain various kinds of drawing errors such as stroke jittering and asymmetry. Extracting plausible deformation from sketch while discarding undesirable drawing errors is difficult. To address these issues, we obtain the plausible deformation contained in sketch by exploring a shape space trained by a collection of shapes in the same category. Furthermore, we also develop a user interface with two different intuitive editing modes based on the shape space established. Experimental results show that more consistent 3D shapes with sketches can be obtained by our method. We also illustrate that our proposed method provides easy and smart ways for both a heuristic and a purposeful shape editing.

The neurophysiological activations of mechanical engineers and industrial designers while designing and problem-solving

Abstract This paper presents results from an experiment using electroencephalography to measure neurophysiological activations of mechanical engineers and industrial designers when designing and problem-solving. In this study, we adopted and then extended the tasks described in a previous functional magnetic resonance imaging study reported in the literature. The block experiment consists of a sequence of three tasks: problem-solving, basic design and open design using a physical interface. The block is preceded by a familiarizing pre-task and then extended to a fourth open design task using free-hand sketching. This paper presents the neurophysiological results from 36 experimental sessions of mechanical engineers and industrial designers. Results indicate significant differences in activations between the problem-solving and the open design tasks. The paper focuses on the two prototypical tasks of problem-solving layout and open design sketching and presents results for both aggregate and temporal activations across participants within each domain and across domains.

Sketch-based image retrieval via CAT loss with elastic net regularization

Fine-grained sketch-based image retrieval (FG-SBIR) is an important problem that uses free-hand human sketch as queries to perform instance-level retrieval of photos. Human sketches are generally highly abstract and iconic, which makes FG-SBIR a challenging task. Existing FG-SBIR approaches using triplet loss with \begin{document}$ \ell_2 $\end{document} regularization or higher-order energy function to conduct retrieval performance, which neglect the feature gap between different domains (sketches, photos) and need to select the weight layer matrix. This yields high computational complexity. In this paper, we define a new CAT loss function with elastic net regularization based on attention model. It can close the feature gap between different subnetworks and embody the sparsity of the sketches. Experiments demonstrate that the proposed approach is competitive with state-of-the-art methods.