Hand-drawn Sketches Research Articles

REBOR: A new sketch-based 3d object retrieval framework using retina inspired features

With the rapid development of data science and modeling engineering, the capacity of cyberspace has significantly expanded which enables the online storage of increasing number of 3D models. Hence, the development of effective and efficient approaches to search 3D models is becoming increasingly important and urgent. In this paper, we propose a new sketch-based 3D retrieval framework named REBOR under the inspiration of retina which is not only consistent with human perception sensitivity but also simplifies the requirement of retrieval query by enabling hand-drawn sketch. The feature extraction process incorporates human visual system by simulating the ganglion perceptive mechanism in retina. Support Vector Machine is used to classify the query sketches and is further optimized by means of an global optimization algorithm so as to acquire optimal results automatically. Experiments are done on the database generated by ourselves with 15 categories of 3D objects, and the results indicate the effectiveness of REBOR in terms of retrieval accuracy.

Generative adversarial learning for detail-preserving face sketch synthesis

Face sketch synthesis aims to generate a face sketch image from a corresponding photo image and has wide applications in law enforcement and digital entertainment. Despite the remarkable achievements that have been made in face sketch synthesis, most existing works pay main attention to the facial content transfer, at the expense of facial detail information. In this paper, we present a new generative adversarial learning framework to focus on detail preservation for realistic face sketch synthesis. Specifically, the high-resolution network is modified as generator to transform a face image from photograph to sketch domain. Except for the common adversarial loss, we design a detail loss to force the synthesized face sketch images have proximate details to its corresponding photo images. In addition, the style loss is adopted to restrain the synthesized face sketch images have vivid sketch style as the hand-drawn sketch images. Experimental results demonstrate that the proposed approach achieves superior performance, compared to state-of-the-art approaches, both on visual perception and objective evaluation. Specifically, this study indicated the higher FSIM values (0.7345 and 0.7080) and Scoot values (0.5317 and 0.5091) than most comparison methods on the CUFS and CUFSF datasets, respectively.

Forensic face sketch recognition

Crime has taken a drastic revolution, and so does the demand for enhancing the security of the forensic files and records. Nowadays, there is an increased requirement to use technological measures in the field of crime as well for identification, detection and recognition of the suspects. To deal with safety and security biometric recognition is necessary. One of the most common biometric techniques is face recognition, as the face is one of the most convenient and reliable way of identification. On the contrary, hand-drawn sketches and sketchers are limited in number although sketches are drawn in almost every criminal case. To come to the rescue, face recognition using a biometric mechanism using computer vision helps in solving the case in special circumstances where the suspects use masks to hide his face or some features. Suspect can conveniently hide the face like nose, eyes, lips, face colour, etc. but the outliner features of face cannot be duplicated. Face sketches have essential information about the face structure and topology and to deal with such precision, there is a requirement of intervention with some assisted computer mechanisms like Deep Convolutional Neural Network which helps in recognizing specific features.

Controllable Sketch-to-Image Translation for Robust Face Synthesis.

In this paper, we propose a novel controllable sketch-to-image translation framework that allows users to interactively and robustly synthesize and edit face images with hand-drawn sketches. Inspired by the coarse-to-fine painting process of human artists, we propose a novel dilation-based sketch refinement method to refine sketches at varied coarse levels without the need for real sketch training data. We further investigate multi-level refinement that enables users to flexibly define how "reliable" the input sketch should be considered for the final output through a refinement level control parameter, which helps balance between the realism of the output and its structural consistency with the input sketch. It is realized by leveraging scale-aware style transfer to model and adjust the style features of sketches at different coarse levels. Moreover, advanced user controllability in terms of the editing region control, facial attribute editing, and spatially non-uniform refinement is further explored for fine-grained and semantic editing. We demonstrate the effectiveness of the proposed method in terms of visual quality and user controllability through extensive experiments including qualitative and quantitative comparison with state-of-the-art methods, ablation studies and various applications.

Angular-Driven Feedback Restoration Networks for Imperfect Sketch Recognition.

Automatic hand-drawn sketch recognition is an important task in computer vision. However, the vast majority of prior works focus on exploring the power of deep learning to achieve better accuracy on complete and clean sketch images, and thus fail to achieve satisfactory performance when applied to incomplete or destroyed sketch images. To address this problem, we first develop two datasets that contain different levels of scrawl and incomplete sketches. Then, we propose an angular-driven feedback restoration network (ADFRNet), which first detects the imperfect parts of a sketch and then refines them into high quality images, to boost the performance of sketch recognition. By introducing a novel "feedback restoration loop" to deliver information between the middle stages, the proposed model can improve the quality of generated sketch images while avoiding the extra memory cost associated with popular cascading generation schemes. In addition, we also employ a novel angular-based loss function to guide the refinement of sketch images and learn a powerful discriminator in the angular space. Extensive experiments conducted on the proposed imperfect sketch datasets demonstrate that the proposed model is able to efficiently improve the quality of sketch images and achieve superior performance over the current state-of-the-art methods.

Time and Continuity: An Ancient Inn Endures

Time and Continuity:An Ancient Inn Endures David H. Lynn (bio) Smiling and clearly nervous, Caroline Cheffers steps through the low threshold and down onto the floor of the Malt House. Only on special occasions does she typically draw the bolts on this great and dark room for airing and, if necessary, heating. And now the reason, the special occasion, does emerge immediately behind her: Queen Elizabeth II, serene and regal, pocketbook in the crook of her elbow, steps down as well and makes her way toward the video camera. Even on the small computer screen, I recognize the passage from which they've just emerged. It seems unlikely. Narrow, dustladen surely, even after a thorough wash, the route is filled with the moldy residue of centuries. It extends between the Malt House and a stone cellar where casks of various ales reside. This is the Bridge Inn's inner and most ancient terrain, a route typically frequented only by staff. Yet Her Majesty is smiling in perfect order, seemingly indifferent to primeval grime. Her predecessor, the original Elizabeth, no doubt exhibited a similar sangfroid. That would have been [End Page 276] some four hundred years ago, when this inn's timbers were already timeworn and weathered. I'm studying the video from four-thousand miles away and nearly two decades after the Queen paid her call. Friends have long regaled me with stories of the Royal Visit, but it has only just occurred to me that with a little searching online, I might be able to conjure a visual record. As it happens, my wife Wendy and I are about to return to Topsham, the small town in Devon, for another academic year. As so often in such a moment, I'm worried about all that may have changed in my absence. I'm aware, too, of course, of my own trajectory of change and of the memories and stories that go with it. Clutching her black patent-leather handbag paired with sensible shoes and gloves, the Queen continues to beam as if reviewing a parade. Clearly, she has long understood that every gesture, every wince will be noted. Now the moment arrives for her to approach the bar that stands just inside the main door. At his post stands Norman Cheffers, Caroline's father, ready to greet his sovereign. White-haired, jovial, and a bit portly, Norman alone of those present actually belongs to Elizabeth's generation. As a token of his fealty, the patriarch presents her with a small brown bottle of ale. Even if the label's not quite visible, I know it features 101 in bold print and a hand-drawn sketch of the old inn. Dark and rich, the special ale has been brewed to hallowed recipe; it also celebrates the century and more his family have served as custodians here. Norman's is the third generation, though family tradition has seen actual management pass from mother to daughter. Too often have the menfolk wandered off to fight their wars or to ply their trades or simply to dawdle. It's Caroline who sees to the day-to-day operations, with her mother Phyllis having passed some years ago. Norman is beaming, his magnificently bulbous nose rampant. Accepting the tribute, his Queen hefts the bottle appreciatively in [End Page 277] her gloves. She is gracious and deliberate. Yet I sense her momentary conundrum. What is to be done? What if the gift is bobbled? What if it explodes? Her diplomatic mien—her poker face—gives nothing away. Yet as soon as courtesy allows, the monarch conveys her trophy to a discreet adjutant. Elizabeth's teeth remain fixed and visible, her thin, firm lips ever smiling. If some twenty years earlier, in 1976, my own introduction to English pubs took place right here at the Bridge, so too does this occasion represent the Queen's first official visit to such an establishment. But of course, it's no accident. In the wake of the roiling debacle of Princess Diana's life and sudden death, Buckingham Palace has scripted this event as part of a new initiative. Once again, it has been left to Elizabeth, the...

Development of the Ontario Forest Resources Inventory: a historical review

The Forest Resources Inventory provides the base layer to support decision-making for a variety of stakeholders in Ontario, Canada. The inventory is used to identify areas of high economic potential, endangered species habitat, and recreational opportunities. As technology evolves, the inventory creation process has changed. From fully manual efforts conducted in the era before planes or roads to hand-drawn sketches made aboard early aircraft, the early inventory program was designed to provide a rough context of timber availability. As society changed, more was asked of the inventory, and ecological and biological features were added. With a land area larger than many European countries to inventory, the creation of an accurate forest inventory in Ontario is a technological and logistical challenge. Although the acquisition technology and strategy have changed over time, the goal of providing an improved understanding into the extent and distribution of natural resources in the province of Ontario has not changed. This review aims to provide a comprehensive look at the history of these inventory efforts in Ontario, as well as provide some context for why the inventory contains the attributes it does today.

Optimal forgetting: Semantic compression of episodic memories.

It has extensively been documented that human memory exhibits a wide range of systematic distortions, which have been associated with resource constraints. Resource constraints on memory can be formalised in the normative framework of lossy compression, however traditional lossy compression algorithms result in qualitatively different distortions to those found in experiments with humans. We argue that the form of distortions is characteristic of relying on a generative model adapted to the environment for compression. We show that this semantic compression framework can provide a unifying explanation of a wide variety of memory phenomena. We harness recent advances in learning deep generative models, that yield powerful tools to approximate generative models of complex data. We use three datasets, chess games, natural text, and hand-drawn sketches, to demonstrate the effects of semantic compression on memory performance. Our model accounts for memory distortions related to domain expertise, gist-based distortions, contextual effects, and delayed recall.

Attention-Net: An Ensemble Sketch Recognition Approach Using Vector Images

For the past few decades, machines have replaced humans in several disciplines. However, machine cognition still lags behind the human capabilities. We address the machines’ ability to recognize human drawn sketches in this work. Visual representations, such as sketches have long been a medium of communication for humans. For artificially intelligent systems to effectively immerse in interactive environments, it is required that machines understand such notations. The abstract nature and varied artistic styling of these sketches make automatic recognition of drawings more challenging than other areas of image classification. In this article, we use sketches represented as a sequence of strokes, i.e., as vector images, to effectively capture the long-term temporal dependencies in hand-drawn sketches. The proposed approach combines the self-attention capabilities of Transformers while effectively utilizing the long-term temporal dependencies through temporal convolution networks (TCNs) for sketch recognition. The confidence scores obtained from the two techniques are combined using triangular-norm (T-norm). Attention heat maps are plotted to isolate the discriminating parts of a sketch that contribute to sketch classification. The extensive quantitative and qualitative evaluation confirms that the proposed network performs favorably against state-of-the-art techniques.

Sverre Fehn’s Brussels and Venice pavilions: the process of becoming different

The early career of Sverre Fehn, the influential Norwegian architect, is marked by the designs of pavilions which were constructed shortly after each other for two major competitions: The Norwegian Pavilion for the World Exposition in Brussels (1956–1958) and The Nordic Pavilion at the Giardini site of the Venice Biennale (1958–1962). This paper studies the process which led from design to construction for both pavilions, identifies structural changes during that process, and investigates the causes and effects of such changes. A study of Fehn’s hand-drawn sketches and photos of models, made throughout the design and construction stages, reveals that despite functional and formal similarities the two pavilions’ development process and the changes which they underwent are noticeably different. The pursuit of ‘abstract space’ during the competition stage for the Brussels Pavilion was tested by considerations of construction only later on, creating a critical tension in the project. Not only did the organisation of structural components change significantly during the design process but the logic of construction underwent a complete transformation. Nevertheless, this transition between different iterations of the Brussels Pavilion suggested an alternative approach for the Venice Pavilion in which the initial thought of construction became a projection, rather than an abstract imitation of built reality. The spatial intention and construction-related considerations of the Venice Pavilion were addressed by a ‘synergetic’ structural principle, implicit in the competition scheme. This unprecedented feature significantly enlarged the ‘space of possibility’ of the roof structure, which provided for adaptation to indeterminacies without compromising the initial design intentions. Changes in the design of the Brussels and Venice Pavilions, therefore, differ in the ways in which they evolved from design to completion: sequential and abrupt, vs. gradual and continuous. This sheds light on a critical shift in the work of an influential architect and the way he approached design indeterminacies, rather than forcing them into submission.

StyleGuide: Zero-Shot Sketch-Based Image Retrieval Using Style-Guided Image Generation

The goal of zero-shot sketch-based image retrieval is to retrieve relevant images from a search set against a hand-drawn sketch query, which belongs to a class, previously unseen by the model. The knowledge gap between such unseen and seen classes along with the domain-gap between the query and search-set makes the problem extremely challenging. In this work, we address this problem by proposing a novel retrieval methodology, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StyleGuide</i> using style-guided <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fake</i> -image generation. In addition, we further study the scenario of generalized zero-shot sketch-based image retrieval, where the search set contains images from both seen and unseen categories. Specifically, we propose a detection approach for unseen class samples in the search-set, based on pre-computed seen class-prototypes, to obtain a refined search-set for a particular unseen-class query. Thus, the query sketch needs to be compared only to those image data which are more likely to belong to the unseen classes, resulting in improved retrieval performance. Extensive experiments on two large-scale sketch-image datasets, Sketchy extended and TU-Berlin show that the proposed approach performs better or comparable to the state-of-the-art for ZS-SBIR and gives significant improvements over the state-of-the-art for generalized ZS-SBIR.

Self-evaluative Scientific Modeling in an Outreach Gene Technology Laboratory

The integration of scientific modeling into science teaching is key to the development of students’ understanding of complex scientific phenomena, such as genetics. With this in mind, we conducted an introductory hands-on module during an outreach gene technology laboratory on the structure of DNA. Our module examined the influence of two model evaluation variants on cognitive achievement: Evaluation 1, based on students’ hand-drawn sketches of DNA models and two open questions, and evaluation 2, based on students’ own evaluations of their models in comparison to a commercially available DNA model. We subsequently subdivided our sample (N = 296) into modellers-1 (n = 151) and modellers-2 (n = 145). Analyses of cognitive achievement revealed that modellers-2 achieved higher scores than modellers-1. In both cases, low achievers, in particular, benefitted from participation. Assessment of modellers-2 self-evaluation sheets revealed differences between self-evaluation and independent reassessment, as non-existent model features were tagged as correct whereas existent features were not identified. Correlation analyses between the models’ assessment scores and cognitive achievement revealed small-to-medium correlations. Consequently, our evaluation-2 phase impacted students’ performance in overall and model-related cognitive achievement, attesting to the value of our module as a means to integrate real scientific practices into science teaching. Although it may increase the workload for science teachers, we find that the potential scientific modeling holds as an inquiry-based learning strategy is worth the effort.

Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-Based Image Retrieval

Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketch-image pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned paired cycle-consistent generative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch of the generative adversarial network maps the visual information from sketch and image to a common semantic space via adversarial training. Each of these branches maintains cycle consistency that only requires supervision at the category level, and avoids the need of aligned sketch-image pairs. A classification criteria on the generators’ outputs ensures the visual to semantic space mapping to be class-specific. Furthermore, we propose to combine textual and hierarchical side information via an auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in any-shot SBIR performance over the state-of-the-art on the extended version of the challenging Sketchy, TU-Berlin and QuickDraw datasets.

RETRACTED ARTICLE: Automated Hand-drawn sketches retrieval and recognition using regularized Particle Swarm Optimization based deep convolutional neural network

One of the most popular and rising research area of image processing is free hand-drawn sketch recognition and its retrieval. Enlarger number of methods is introduced to retrieve the sketch images but it made few complexity issues and their performance often degraded. So, in this paper, we proposed an effective method of Regularized Particle Swarm Optimization Based Deep Convolutional Neural Network (RPSO-DCNN) algorithm to retrieve the performance of free hand-drawn sketches. In feature extraction, the Regularized Particle Swarm Optimization (RPSO) model that aim is to produce an optimal evolutionary deep learning result. Therefore, the free hand-drawn sketch image classification and its retrieval are performed by Support Vector Machine and Levenshtein distance-based fuzzy k-nearest neighbour (L-FkNN) algorithms. Hence, this work can bring in communication between human and computer. Experimentally, the simulation work of the proposed RPSO-DCNN model is implemented in the running software of MATLAB. The sketch images are chosen from the TU-Berlin dataset, Sketch dataset, SHREC13 dataset, Flickr dataset and Sketchy dataset. Aiming is to facilitate the performance of the proposed RPSO-DCNN model with various kinds of state of art methods such as H-CNN, Fuzzy, CNN, MARQS and TCVD. The experimental result demonstrates that, the proposed RPSO-DCNN accomplish the optimal accuracy with different state-of-art methods.

Computer-assisted terrain sketch mapping that considers the geomorphological features in a loess landform

In geography, a terrain sketch map is necessary to understand the features and internal structures of a landscape due to its ability to depict key information in a geographical scene using as few lines as possible. Previous computer-drawn sketch maps have focused on the artistic effect rather than on depicting terrain features and landform structures, and thus, they differ considerably from hand-drawn sketch maps by geographers or geologists. This study develops a DEM-based method for terrain sketch mapping that considers the typical feature descriptions of a loess landform and the visual hierarchy expression in line with the law of visual perspective. The method was tested with experiments on two landforms of Chinese Loess Plateau, based on digital elevation models (DEM) with a horizontal resolution of 5 m. In the developed method, first, typical terrain features, including the visual outline, shoulder line, gully and flow lines, are extracted from the DEM. Second, the map is divided into three visual levels in accordance with the data extension and viewing point. Then, terrain feature lines are assigned to different visual levels. Finally, the visual outlines in the distant view are generalized following the law of visual perspective. Results are assessed through a questionnaire with specialists (experts) and students (non-experts). The sketch map was able to characterise loess landforms, and is somewhat similar to traditional hand-drawn maps. The generalisation method realises the near and distant view characteristics of a sketch map, which are detailed and simplified, respectively. The results of the questionnaire also showed that our method presents terrain morphology and geographical scene more accurately and reliably than a hand-drawn sketch map.

Mental Retrieval of Remote Sensing Images via Adversarial Sketch-Image Feature Learning

Searching the targets of interest in large-scale remote sensing images is a fundamental problem, which becomes a very challenging issue when there is no relevant example at hand but a mental picture in mind. Hand-drawn sketch as a precise and convenient expression of the mental picture makes sketch-based remote sensing image retrieval (SBRSIR) an ideal choice to cope with this issue. However, the accuracy of SBRSIR algorithm, which is critical to effective retrieval, is still far behind classic query-by-example image retrieval. Two central limiting factors for this performance gap are: 1) the lack of effective cross-domain representations for bridging the domain gap between sketches and remote sensing images and 2) the absence of large sketch/remote sensing image data sets for developing, evaluating, and comparing the SBRSIR approaches. In this article, we first develop a novel SBRSIR model to learn a deep joint embedding space with discriminative losses, where adversarial training is used for the embedding space to learn domain-invariant representations. Then, we contribute a sketch/remote sensing image data set specifically for SBRSIR and provide a benchmark for subsequent researchers. Extensive experiments on the data set and large scene images demonstrate the effectiveness and superiority of the method for both the seen and unseen categories.

SketchDesc: Learning Local Sketch Descriptors for Multi-View Correspondence

In this article, we study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object and predict as output the semantic correspondence among the sketches. This problem is challenging since the visual features of corresponding points at different views can be very different. To this end, we take a deep learning approach and learn a novel local sketch descriptor from data. We contribute a training dataset by generating the pixel-level correspondence for the multi-view line drawings synthesized from 3D shapes. To handle the sparsity and ambiguity of sketches, we design a novel multi-branch neural network that integrates a patch-based representation and a multi-scale strategy to learn the pixel-level correspondence among multi-view sketches. We demonstrate the effectiveness of our proposed approach with extensive experiments on hand-drawn sketches and multi-view line drawings rendered from multiple 3D shape datasets.

To Link or Not to Link: Clause Chaining in Japanese Narratives.

Japanese is a clause chaining language, in which a sentence is formed by linking two or more clauses in a series (chain), using different forms of the verb in “non-final” clauses within the chain from the “final” clauses that end the chain. Since the verb comes at the end of a clause in Japanese, speakers must decide by the time they reach the verb at end of one clause whether or not to connect it to the following clause. This study analyzes the narratives of Japanese children and adults with the dual goal of discovering why narrators decide either to link the clause they are producing to the following clause or not to link, i.e., to end the clause chain. Stories were elicited from 60 3- to 7-year-old children and 10 adults, who performed two tasks: (1) telling the story depicted in a hand-drawn cartoon, and (2) viewing a 7-min video and then recounting the plot from memory. The data were analyzed qualitatively and quantitatively. The qualitative analysis examines story length and prompting, length of clause chains, variety of linking connectives, semantic relations—such as simultaneity, temporal sequence, and causality—between clauses in chains, shifts of subject from one story character to another (switch-reference), and contexts in stories where narrators end clause chains. For the quantitative analysis, a multifactorial, mixed-effects model was fit to determine which of several potential predictors have a significant effect on the probability that narrators will link clauses. The model is significant, though weak in discriminatory power. Among the significant effects, simultaneous relations between events increase the probability that narrators will link clauses, while changing the subject referent and reaching the end of a narrative unit, e.g., an episode, increase the probability that narrators will end their clause chain. There are no significant age effects: the children respond in the same way to the predictors as the adults. The qualitative and quantitative results of the study are interpreted with respect to the close relation between grammar (clause chains) and discourse (narrative structure) as well as the cognitive process of producing clause chains during narration.

Reconstructing 3D Shapes from Multiple Sketches using Direct Shape Optimization.

3D shape reconstruction from multiple hand-drawn sketches is an intriguing way to 3D shape modeling. Currently, state-of-the-art methods employ neural networks to learn a mapping from multiple sketches from arbitrary view angles to a 3D voxel grid. Because of the cubic complexity of 3D voxel grids, however, neural networks are hard to train and limited to low resolution reconstructions, which leads to a lack of geometric detail and low accuracy. To resolve this issue, we propose to reconstruct 3D shapes from multiple sketches using direct shape optimization (DSO), which does not involve deep learning models for direct voxel-based 3D shape generation. Specifically, we first leverage a conditional generative adversarial network (CGAN) to translate each sketch into an attenuance image that captures the predicted geometry from a given viewpoint. Then, DSO minimizes a project-and-compare loss to reconstruct the 3D shape such that it matches the predicted attenuance images from the view angles of all input sketches. Based on this, we further propose a progressive update approach to handle inconsistencies among a few hand-drawn sketches for the same 3D shape. Our experimental results show that our method significantly outperforms the state-of-the-art methods under widely used benchmarks and produces intuitive results in an interactive application.

LayoutGAN: Synthesizing Graphic Layouts With Vector-Wireframe Adversarial Networks.

Layout is important for graphic design and scene generation. We propose a novel Generative Adversarial Network, called LayoutGAN, that synthesizes layouts by modeling geometric relations of different types of 2D elements. The generator of LayoutGAN takes as input a set of randomly-placed 2D graphic elements, represented by vectors and uses self-attention modules to refine their labels and geometric parameters jointly to produce a realistic layout. Accurate alignment is critical for good layouts. We, thus, propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNN-based discriminator is used to optimize the layouts in image space. We validate the effectiveness of LayoutGAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation, tangram graphic design, mobile app layout design, and webpage layout optimization from hand-drawn sketches.