Input Sketches Research Articles

Faceify- Human Face Sketch to Real

The Human Face Sketch to Real Image Application using deep learning presents the development of an innovative application designed to convert human face sketches into realistic image using advanced deep learning techniques. The primary objective of this research is to bridge the gap between manual sketching and digital realism, offering a powerful tool for artists, law enforcement, and digital content creators. The proposed system leverages generative adversarial networks (GANs), particularly a variant known as the Sketch-to-Image GAN (SI-GAN), which is optimizes for translating sparse and abstract sketch lines into high-fidelity facial representations. The architecture consists of a generator network that refines sketches into realistic images, and a discriminator network that distinguishes between generated images and real photographs, continuously enhancing the model’s output. To ensure high accuracy and detail preservation, the model is trained on a diverse dataset comprising thousands of paired sketch and real face images. Advanced techniques such as attention mechanisms and multi-scale feature extraction are incorporated to maintain the integrity of facial features and textures. Additionally, a user friendly interface has been developed, allowing users to input sketches and receive high-quality realistic images in real-time. Evaluation metrics, including structural similarity index (SSIM) and Frechet Inception Distance (FID), Indicate that the proposed model outperforms existing state-of-the-art methods in both quality and realism. Keywords: Image Translation, Conditional Generative Adversarial Networks (CGANs), Deep Learning, Mobile Application Development, Backend Integration, Face Sketch, Photorealistic Image.

Research on the Integration and Innovation of Artificial Intelligence Technology in Modern Brand Design

Abstract In the context of the artificial intelligence era, the application of artificial intelligence technology in modern brand design is increasingly widespread and is profoundly changing the design creation in this field. The purpose of this paper is to propose an intelligent image acquisition and automated brand design technology that utilizes AI technology to assist with brand design. The image acquisition and retrieval process of the online CBIR system is established, the average precision mean is used as the evaluation index of image retrieval, and the BoF model is constructed to realize RootSIFT feature extraction and image acquisition. Meanwhile, MultiStyleLogo GAN, a multi-style generative adversarial network, is designed to intelligently automate random output and generate brand design solutions through input sketches and style semantic descriptions to improve designers’ design efficiency. This paper utilizes technology to redesign the brand of A green plum wine in a province in China. Except for the pattern and font indicators, the average value of the newly designed brand image in each evaluation index of aesthetic design is greater than 4. The average values of the image aesthetics, cultural connotation, and emotional resonance indicators in the evaluation of brand narrative reach 4.25, 4.24, and 4.15, respectively, which achieve a high consumer satisfaction level.

Transforming Sketches into Realistic Images: Leveraging Machine Learning and Image Processing for Enhanced Architectural Visualization

This article presents a novel approach for transforming architectural sketches into realistic images through the utilization of machine learning and image processing techniques. The proposed method leverages the Stable Diffusion model, a deep learning framework specifically designed for text-to-image generation. By integrating image processing algorithms into the workflow, the model gains a better understanding of the input sketches, resulting in visually coherent and meaningful output images. The study explores the application of the Stable Diffusion model in the context of architectural design, showcasing its potential to enhance the visualization process and support designers in generating accurate and compelling representations. The efficacy of the method is evaluated through qualitative assessment, demonstrating its effectiveness in bridging the gap between initial sketches and photorealistic renderings. This research contributes to the growing body of knowledge on the integration of machine learning and image processing in architecture, providing insights and practical implications for architects, design professionals and researchers in the field.

TreeSketchNet: From Sketch to 3D Tree Parameters Generation

Three-dimensional (3D) modeling of non-linear objects from stylized sketches is a challenge even for computer graphics experts. The extrapolation of object parameters from a stylized sketch is a very complex and cumbersome task. In the present study, we propose a broker system that can transform a stylized sketch of a tree into a complete 3D model by mediating between a modeler and a 3D modeling software. The input sketches do not need to be accurate or detailed: They must only contain a rudimentary outline of the tree that the modeler wishes to 3D model. Our approach is based on a well-defined Deep Neural Network architecture, called TreeSketchNet (TSN), based on convolutions and capable of generating Weber and Penn [1995] parameters from a simple sketch of a tree. These parameters are then interpreted by the modeling software, which generates the 3D model of the tree pictured in the sketch. The training dataset consists of synthetically generated sketches that are associated with Weber–Penn parameters, generated by a dedicated Blender modeling software add-on. The accuracy of the proposed method is demonstrated by testing the TSN with synthetic and hand-made sketches. Finally, we provide a qualitative analysis of our results, by evaluating the coherence of the predicted parameters with several distinguishing features.

HDA[formula omitted]L: Hierarchical Domain-Augmented Adaptive Learning for sketch-based 3D shape retrieval

The sketch-based 3D shape retrieval has been an active but challenging task for several decades. In this paper, we deeply analyze the challenges and propose a novel Hierarchical Domain-Augmented Adaptive Learning (HDA2L) for sketch-based 3D shape retrieval. The first notable challenge is the vast cross-modality discrepancies between sketches and 3D shapes. To this issue, the existing methods restrict the consistency of the final features by establishing a shared cross-domain loss but ignore the feature extraction process, resulting in a limited effect. We argue that the mutual information of the samples from the same class but different domains can provide an important cue to enhance common features captured in the feature extraction process. Thus, we design an Inter-Domain Augmented Network (Inter-DAN) by employing inter-domain feature correlation learning to capture cross-domain mutual information to learn augmented common global features for both sketches and 3D shapes. Another challenge is that the input sketch is various: it may be particularly abstract and contains only the overall outline of the target model, or it may be too sketchy and only contains some salient local regions of the target model. Though existing methods have demonstrated their capability to capture overall features, they always ignore the learning of local discriminative features and fail to adapt to the various changes of input sketches. To address this issue, we design an Intra-Domain Augmented Network (Intra-DAN) for sketches and 3D shapes, respectively, which learns augmented local discriminative features by adopting cascading cross-layer bilinear pooling operations. In addition, we design a Source-Agnostic Adversarial Network (SAAN) to accomplish the adaptive hierarchical domain features fusion, which forces the network to adaptively focus on more discriminative information from global features and local features and further adapt the diversity of the input sketches. The experiments on three benchmark datasets demonstrate that our method obtains superior retrieval performance than the state-of-the-art sketch-based 3D shape retrieval approaches.

Sketch2Photo: Synthesizing photo-realistic images from sketches via global contexts

Sketch-to-image synthesis aims to generate realistic images that match the input sketches or edge maps exactly. Most known sketch-to-image synthesis methods use various generative adversarial networks (GANs) that are trained with numerous pairs of sketches and real images. Because of the convolution locality, the low-level layers of the generators in these GANs lack global perception ability, causing feature maps derived from them easily to overlook global cues. Since the global receptive field is crucial for acquiring the non-local structures and features of sketches, the absence of global contexts will impact the generation of high-quality images. Some recent models turn to self-attention to construct global dependencies. However, they are not viable for large feature maps for the quadratic computational complexity concerning the size of feature maps. To address these problems, in this work, we propose Sketch2Photo — a new image synthesis approach that can capture global contexts as well as local features to generate photo-realistic images from weak or partial sketches or edge maps. We employ fast Fourier convolution (FFC) residual blocks to create global receptive fields in the bottom layers of the network and incorporate Swin Transformer block (STB) units to obtain long-range global contexts for large-size feature maps efficiently. We also present an improved spatial attention pooling (ISAP) module to relax the strict alignment requirements between incomplete sketches and generated images. Quantitative and qualitative experiments on multiple public datasets demonstrate the superiority of the proposed approach over many other sketch-to-image synthesis methods. The project code is available at https://github.com/hengliusky/Skecth2Photo.

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

Sketching is an intuitive and simple way to depict sciences with various object form and appearance characteristics. In the past few years, widely available touchscreen devices have increasingly made sketch-based human-AI co-creation applications popular. One key issue of sketch-oriented interaction is to prepare input sketches efficiently by non-professionals because it is usually difficult and time-consuming to draw an ideal sketch with appropriate outlines and rich details, especially for novice users with no sketching skills. Thus, sketching brings great obstacles for sketch applications in daily life. On the other hand, hand-drawn sketches are scarce and hard to collect. Given the fact that there are several large-scale sketch datasets providing sketch data resources, but they usually have a limited number of objects and categories in sketch, and do not support users to collect new sketch materials according to their personal preferences. In addition, few sketch-related applications support the reuse of existing sketch elements. Thus, knowing how to extract sketches from existing drawings and effectively re-use them in interactive scene sketch composition will provide an elegant way for sketch-based image retrieval (SBIR) applications, which are widely used in various touch screen devices. In this study, we first conduct a study on current SBIR to better understand the main requirements and challenges in sketch-oriented applications. Then we develop the SketchMaker as an interactive sketch extraction and composition system to help users generate scene sketches via reusing object sketches in existing scene sketches with minimal manual intervention. Moreover, we demonstrate how SBIR improves from composited scene sketches to verify the performance of our interactive sketch processing system. We also include a sketch-based video localization task as an alternative application of our sketch composition scheme. Our pilot study shows that our system is effective and efficient, and provides a way to promote practical applications of sketches.

Data-Driven Sketch Beautification With Neural Feature Representation.

This article presents a data-driven approach for beautifying freehand sketches. Our key premise is that the artist-drawn vector can be used to sketch visually appealing shapes, such as local shapes with a clean appearance and better global visual properties (e.g., symmetry). However, these merits may not apply to all object categories. In this article, we use a neural network to represent local and global merits across different object categories to design our beautification method. First, we match sample points between input sketches and the collected vector shapes using the extracted feature representations. Then, we design an optimization problem to ensure resemblance between the deformed sketch and vector shape in the representation space while preserving the semantic meaning and style of the original sketch. Finally, we demonstrate our method on sketches across different shape categories.

Interaction Fields: Intuitive Sketch‐based Steering Behaviors for Crowd Simulation

AbstractThe real‐time simulation of human crowds has many applications. In a typical crowd simulation, each person ('agent') in the crowd moves towards a goal while adhering to local constraints. Many algorithms exist for specific local ‘steering’ tasks such as collision avoidance or group behavior. However, these do not easily extend to completely new types of behavior, such as circling around another agent or hiding behind an obstacle. They also tend to focus purely on an agent's velocity without explicitly controlling its orientation. This paper presents a novel sketch‐based method for modelling and simulating many steering behaviors for agents in a crowd. Central to this is the concept of aninteraction field(IF): a vector field that describes the velocities or orientations that agents should use around a given ‘source’ agent or obstacle. An IF can also change dynamically according to parameters, such as the walking speed of the source agent. IFs can be easily combined with other aspects of crowd simulation, such as collision avoidance. Using an implementation of IFs in a real‐time crowd simulation framework, we demonstrate the capabilities of IFs in various scenarios. This includes game‐like scenarios where the crowd responds to a user‐controlled avatar. We also present an interactive tool that computes an IF based on input sketches. This IF editor lets users intuitively and quickly design new types of behavior, without the need for programming extra behavioral rules. We thoroughly evaluate the efficacy of the IF editor through a user study, which demonstrates that our method enables non‐expert users to easily enrich any agent‐based crowd simulation with new agent interactions.

Multi-Density Sketch-to-Image Translation Network

Sketch-to-image (S2I) translation plays an important role in image synthesis and manipulation tasks, such as photo editing and colorization. Some specific S2I translations, including sketch-to-photo and sketch-to-painting, can be used as powerful tools in the art design industry. However, previous methods only support S2I translation with a single level of density, which gives less flexibility to users for controlling the input sketches. In this work, we propose the first multi-level density sketch-to-image translation framework, which allows the input sketch to cover a wide range from rough object outlines to microstructures. Moreover, to tackle the problem of noncontinuous representation of multi-level density input sketches, we project the density level into a continuous latent space, which can then be linearly controlled by a parameter. This allows users to conveniently control the densities of input sketches and the generation of images. Moreover, our method has been successfully verified on various datasets for different applications, including face editing, multi-modal sketch-to-photo translation, and anime colorization, providing coarse-to-fine levels of controls to these applications.

Cali-sketch: Stroke calibration and completion for high-quality face image generation from human-like sketches

Image generation has received increasing attention because of its wide application in security and entertainment. Sketch-based face generation brings more fun and better quality of image generation due to supervised interaction. However, when a sketch poorly aligned with the true face is given as input, existing supervised image-to-image translation methods often cannot generate acceptable photo-realistic face images. To address this problem, in this paper we propose Cali-Sketch, a human-like-sketch to photo-realistic-image generation method. Cali-Sketch explicitly models stroke calibration and image generation using two constituent networks: a Stroke Calibration Network (SCN), which calibrates strokes of facial features and enriches facial details while preserving the original intent features; and an Image Synthesis Network (ISN), which translates the calibrated and enriched sketches to photo-realistic face images. In this way, we manage to decouple a difficult cross-domain translation problem into two easier steps. Extensive experiments verify that the face photos generated by Cali-Sketch are both photo-realistic and faithful to the input sketches, compared with state-of-the-art methods.

System for Inception of Images From An Under Drawing

Catching the criminal on the basis of eyewitness description sketches which are generated by any software or hand drawn this technique becomes useful when there is deficiency of evidence. Recognising the sketches with face photos to find the results. A facial recognition is a technology capable of verifying a person by using deep learning is the part of data science and this creating more importance in law enforcement agencies. Deep learning generally used for recognising application like voice recognition, face recognition, audio and video recognition. In this system, we are working larger dataset because deep learning not useful for small amount of data. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

Interactive liquid splash modeling by user sketches

Splashing is one of the most fascinating liquid phenomena in the real world and it is favored by artists to create stunning visual effects, both statically and dynamically. Unfortunately, the generation of complex and specialized liquid splashes is a challenging task and often requires considerable time and effort. In this paper, we present a novel system that synthesizes realistic liquid splashes from simple user sketch input. Our system adopts a conditional generative adversarial network (cGAN) trained with physics-based simulation data to produce raw liquid splash models from input sketches, and then applies model refinement processes to further improve their small-scale details. The system considers not only the trajectory of every user stroke, but also its speed, which makes the splash model simulation-ready with its underlying 3D flow. Compared with simulation-based modeling techniques through trials and errors, our system offers flexibility, convenience and intuition in liquid splash design and editing. We evaluate the usability and the efficiency of our system in an immersive virtual reality environment. Thanks to this system, an amateur user can now generate a variety of realistic liquid splashes in just a few minutes.

Multi-Domain Recognition of Hand-Drawn Diagrams Using Hierarchical Parsing

This paper presents an approach for the recognition of multi-domain hand-drawn diagrams, which exploits Sketch Grammars (SkGs) to model the symbols’ shape and the abstract syntax of diagrammatic notations. The recognition systems automatically generated from SkGs process the input sketches according to the following phases: the user’ strokes are first segmented and interpreted as primitive shapes, then by exploiting the domain context they are clustered into symbols of the domain and, finally, an interpretation of whole diagram is given. The main contribution of this paper is an efficient model of parsing suitable for both interactive and non-interactive sketch-based interfaces, configurable to different domains, and able to exploit contextual information for improving recognition accuracy and solving interpretation ambiguities. The proposed approach was evaluated in the domain of UML class diagrams obtaining good results in terms of recognition accuracy and usability.

DeepFaceDrawing

Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch. We take a local-to-global approach. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

Staged Sketch-to-Image Synthesis via Semi-supervised Generative Adversarial Networks

Sketch-based image synthesis is a challenging problem in computer graphics and vision. Existing approaches either require exact edge maps or rely on the retrieval of existing photographs, which limits their applications in real-world scenarios. Accordingly in this work, we propose a staged semi-supervised generative adversarial networks based method for sketch-to-image synthesis, which can directly generate realistic images from novice sketches. More specifically, we first adopt a conditional generative adversarial network (CGAN) to extract class-wise representations from unpaired images. These class-wise representations are then exploited and incorporated with another CGAN, which are used to generate realistic images from sketches. By incorporating the class-wise representations, our method can leverage both the general class information from unpaired images and the targeted object information from input sketches. Additionally, this network architecture also enables us to take full advantage of widely available unpaired images and learn more accurate class representations. Extensive experiments demonstrate, compared with state-of-the-art image translation methods, our approach can achieve more promising results and synthesize images with significantly better Inception Scores and Frechet Inception Distance.

Sketch-guided Deep Portrait Generation

Generating a realistic human class image from a sketch is a unique and challenging problem considering that the human body has a complex structure that must be preserved. Additionally, input sketches often lack important details that are crucial in the generation process, hence making the problem more complicated. In this article, we present an effective method for synthesizing realistic images from human sketches. Our framework incorporates human poses corresponding to locations of key semantic components (e.g., arm, eyes, nose), seeing that its a strong prior for generating human class images. Our sketch-image synthesis framework consists of three stages: semantic keypoint extraction, coarse image generation, and image refinement. First, we extract the semantic keypoints using Part Affinity Fields (PAFs) and a convolutional autoencoder. Then, we integrate the sketch with semantic keypoints to generate a coarse image of a human. Finally, in the image refinement stage, the coarse image is enhanced by a Generative Adversarial Network (GAN) that adopts an architecture carefully designed to avoid checkerboard artifacts and to generate photo-realistic results. We evaluate our method on 6,300 sketch-image pairs and show that our proposed method generates realistic images and compares favorably against state-of-the-art image synthesis methods.

Multistroke Grouping of Online Freehand Axonometric Sketches for Mechanical Models

Multistroke drawing occurs frequently in conceptual design sketches; however, it is almost unsupported by the current sketch-based user interfaces. We proposed a sketch recognition system based on the multistroke primitive grouping method. Based on grouping the strokes that lie within the mutual boundaries between adjacent regions, we create line drawings from online freehand axonometric sketches of mechanical models. First, closed regions and their boundary bands of the sketch were extracted. Then, the strokes that cross the boundary bands of two or more closed regions are segmented, and the strokes that lie within the intersection of two adjacent boundary bands are grouped. Finally, grouped strokes are simplified into a new single stroke and then fitted as a geometric primitive; thus, the input sketches are recognized to the line drawings. We developed a prototype of the sketch recognition system to evaluate the proposed method. The results showed that the input sketches are simplified into the accurate line drawings efficiently. The proposed method can be applied to both multistroke overtracing and nonovertracing sketches.

Reconstructing 3D Shapes from Multiple Sketches using Direct Shape Optimization.

3D shape reconstruction from multiple hand-drawn sketches is an intriguing way to 3D shape modeling. Currently, state-of-the-art methods employ neural networks to learn a mapping from multiple sketches from arbitrary view angles to a 3D voxel grid. Because of the cubic complexity of 3D voxel grids, however, neural networks are hard to train and limited to low resolution reconstructions, which leads to a lack of geometric detail and low accuracy. To resolve this issue, we propose to reconstruct 3D shapes from multiple sketches using direct shape optimization (DSO), which does not involve deep learning models for direct voxel-based 3D shape generation. Specifically, we first leverage a conditional generative adversarial network (CGAN) to translate each sketch into an attenuance image that captures the predicted geometry from a given viewpoint. Then, DSO minimizes a project-and-compare loss to reconstruct the 3D shape such that it matches the predicted attenuance images from the view angles of all input sketches. Based on this, we further propose a progressive update approach to handle inconsistencies among a few hand-drawn sketches for the same 3D shape. Our experimental results show that our method significantly outperforms the state-of-the-art methods under widely used benchmarks and produces intuitive results in an interactive application.

Sketch-a-Segmenter: Sketch-based Photo Segmenter Generation.

Given pixel-level annotated data, traditional photo segmentation techniques have achieved promising results. However, these photo segmentation models can only identify objects in categories for which data annotation and training have been carried out. This limitation has inspired recent work on few-shot and zero-shot learning for image segmentation. In this paper, we show the value of sketch for photo segmentation, in particular as a transferable representation to describe a concept to be segmented. We show, for the first time, that it is possible to generate a photo-segmentation model of a novel category using just a single sketch and furthermore exploit the unique fine-grained characteristics of sketch to produce more detailed segmentation. More specifically, we propose a sketch-based photo segmentation method that takes sketch as input and synthesizes the weights required for a neural network to segment the corresponding region of a given photo. Our framework can be applied at both the category-level and the instance-level, and fine-grained input sketches provide more accurate segmentation in the latter. This framework generalizes across categories via sketch and thus provides an alternative to zero-shot learning when segmenting a photo from a category without annotated training data. To investigate the instance-level relationship across sketch and photo, we create the SketchySeg dataset which contains segmentation annotations for photos corresponding to paired sketches in the Sketchy Dataset.