Hand-drawn Sketches Research Articles

StyleWe: Towards Style Fusion in Generative Fashion Design with Efficient Federated AI

Collaboration can amalgamate diverse ideas, styles, and visual elements, fostering creativity and innovation among different designers. In collaborative design, sketches play a pivotal role as a means of expressing design creativity. However, designers often tend to not openly share these meticulously crafted sketches. This phenomenon of data island in the design area hinders its digital transformation under the third wave of AI. In this paper, we introduce a Federated Generative Artificial Intelligence Clothing system, namely StyleWe, employing federated learning to aid in sketch design. StyleWe is committed to establishing an ecosystem wherein designers can exchange sketch styles among themselves. Through StyleWe, designers can generate sketches that incorporate various designers' styles from their peers, drawing inspiration from collaboration without the need for data disclosure or upload. Extensive performance evaluations and user studies indicate that our StyleWe system can produce multi-styled sketches of comparable quality to human-designed ones while significantly enhancing efficiency compared to hand-drawn sketches.

IDENTIFEYE: Sketch-Based Facial Analysis in Forensics

Traditional forensic methods, like hand-drawn sketches, often fall short in quickly identifying suspects, making the process both slow and resource-intensive. This paper introduces an innovative application designed to streamline this task. Through a simple drag-and-drop interface, users can upload sketches and match them with entries in police databases, significantly boosting the speed and accuracy of suspect recognition. By integrating deep learning and cloud technology, this tool builds on existing algorithms to deliver higher precision and ease of use. Earlier methods have had trouble aligning sketches with photographs, especially when orientations differ. Our application overcomes these issues, merging the art of sketching with modern technology to enhance forensic work. Testing reveals promising accuracy, though ongoing improvements are needed to fully meet the demands of law enforcement. This research aims to offer a real-world solution, tackling past limitations and making suspect identification faster and more efficient for investigators. Keywords: Forensic identification, Suspect sketching, Deep learning, Recognition software, Law enforcement tools, Cloud technology.

Boosting cross-modal retrieval in remote sensing via a novel unified attention network

With the rapid advent and abundance of remote sensing data in different modalities, cross-modal retrieval tasks have gained importance in the research community. Cross-modal retrieval belongs to the research paradigm in which the query is of one modality and the retrieved output is of the other modality. In this paper, the remote sensing (RS) data modalities considered are the earth observation optical data (aerial photos) and the corresponding hand-drawn sketches. The main challenge of the cross-modal retrieval research objective for optical remote sensing images and the corresponding sketches is the distribution gap between the shared embedding space of the modalities. Prior attempts to resolve this issue have not yielded satisfactory outcomes regarding accurately retrieving cross-modal sketch-image RS data. The state-of-the-art architectures used conventional convolutional architectures, which focused on local pixel-wise information about the modalities to be retrieved. This limits the interaction between the sketch texture and the corresponding image, making these models susceptible to overfitting datasets with particular scenarios. To circumvent this limitation, we suggest establishing multi-modal correspondence using a novel architecture of the combined self and cross-attention algorithms, SPCA-Net to minimize the modality gap by employing attention mechanisms for the query and other modalities. Efficient cross-modal retrieval is achieved through the suggested attention architecture, which empirically emphasizes the global information of the relevant query modality and bridges the domain gap through a unique pairwise cross-attention network. In addition to the novel architecture, this paper introduces a unique loss function, label-specific supervised contrastive loss, tailored to the intricacies of the task and to enhance the discriminative power of the learned embeddings. Extensive evaluations are conducted on two sketch-image remote sensing datasets, Earth-on-Canvas and RSketch. Under the same experimental conditions, the performance metrics of our proposed model beat the state-of-the-art architectures by significant margins of 16.7%, 18.9%, 33.7%, and 40.9% correspondingly.

GPT-4 Vision: Multi-Modal Evolution of ChatGPT and Potential Role in Radiology.

GPT-4 Vision (GPT-4V) represents a significant advancement in multimodal artificial intelligence, enabling text generation from images without specialized training. This marks the transformation of ChatGPT as a large language model (LLM) into GPT-4's promised large multimodal model (LMM). As these AI models continue to advance, they may enhance radiology workflow and aid with decision support. This technical note explores potential GPT-4V applications in radiology and evaluates performance for sample tasks.GPT-4V capabilities were tested using images from the web, personal and institutional teaching files, and hand-drawn sketches. Prompts evaluated scientific figure analysis, radiologic image reporting, image comparison, handwriting interpretation, sketch-to-code, and artistic expression. In this limited demonstration of GPT-4V's capabilities, it showed promise in classifying images, counting entities, comparing images, and deciphering handwriting and sketches. However, it exhibited limitations in detecting some fractures, discerning a change in size of lesions, accurately interpreting complex diagrams, and consistently characterizing radiologic findings. Artistic expression responses were coherent. WhileGPT-4V may eventually assist with tasks related to radiology, current reliability gaps highlight the need for continued training and improvement before consideration for any medical use by the general public and ultimately clinical integration. Future iterations could enable a virtual assistant to discuss findings, improve reports, extract data from images, provide decision support based on guidelines, white papers, and appropriateness criteria. Human expertise remain essential for safe practice and partnerships between physicians, researchers, and technology leaders are necessary to safeguard against risks like bias and privacy concerns.

SketchDream: Sketch-based Text-To-3D Generation and Editing

Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.

D2GL: Dual-level dual-scale graph learning for sketch-based 3D shape retrieval

Sketch-based 3D shape retrieval (SBSR) is an active research area in the computer vision community, but it is still very challenging. One main reason is that existing deep learning-based methods usually treat sketches as 2D images, neglecting the sparsity and diversity. In this paper, we propose a novel Dual-level Dual-scale Graph Learning (D2GL) method to effectively enhance structural information and produce robust representations for sparse and diverse hand-drawn sketches. Specifically, in addition to the traditional branches for SBSR, we introduce a Dual-level Dual-scale Graph Self-attention (DLDS-GSA) as an auxiliary branch. DLDS-GSA further consists of two levels of encoders, i.e., a local structural encoder and a dual-scale global structural encoder, to capture both local discriminative and multi-scale global structures while minimizing the impact of various sketch drawing details. Comprehensive experiments on SHREC’13 and SHREC’14 datasets demonstrate the superiority of D2GL for SBSR, with extended experiments on PART-SHREC’14 confirming its generalization for unseen classes in SBSR.

SketchCleanGAN: A generative network to enhance and correct query sketches for improving 3D CAD model retrieval systems

Given an input query, a search and retrieval system fetches relevant information from a dataset. In the Engineering domain, such a system is beneficial for tasks such as design reuse. A two-dimensional (2D) sketch is more conducive for an end user to give as a query than a three-dimensional (3D) object. Such query sketches, nevertheless, will inevitably contain defects like incomplete lines, mesh lines, overdrawn areas, missing areas, etc. Since a retrieval system’s results are only as good as the query, it is necessary to improve the query sketches.In this paper, the problem of transforming a defective CAD sketch into a defect-free sketch is addressed using Generative Adversarial Networks (GANs), which, to the best of our knowledge, has not been investigated before. We first create a dataset of 534 hand-drawn sketches by tracing the boundaries of images of CAD models. We then pair the corrected sketches with their corresponding defective sketches and use them for training a C-WGAN (Conditional Wasserstein Generative Adversarial Network), called SketchCleanGAN. We model the transformation from defective to defect-free sketch as a factorization of the defective input sketch and then translate it to the space of defect-free sketch. We propose a three-branch strategy to this problem. Ablation studies and comparisons with other state-of-the-art techniques demonstrate the efficacy of the proposed technique. Additionally, we also contribute to a dataset of around 58000 improved sketches using the proposed framework.

A Biomimetic Sketch-Based Form Finding Tool

Sketches play a fundamental role in the design disciplines, serving as crucial representations for ideation, problem-solving, and communication. In the realm of architecture, sketches encapsulate the evolution of ideas from conceptualization to construction. Hand-drawn sketches, characterized by their open-ended, ambiguous nature and rapid production, stand out as indispensable tools in bridging the gap between abstract concepts and tangible designs, guiding the progression from early design stages to final product realization. However, despite their significant potential and pivotal role in the design process, hand sketches have often been overlooked and swiftly abandoned in the ongoing discourse surrounding traditional versus digital design methodologies, particularly with the widespread integration of computer technologies. This study endeavors to unlock the wealth of information embedded within hand sketches, spanning from initial design concepts to intricate manufacturing details, through the utilization of a hybrid digital form-finding tool. By employing swarm algorithms in the quest for form, it is anticipated that the boundaries of conceptual ideas delineated by hand sketches will be expanded. This is facilitated by an algorithm developed in Processing using the Java coding language, complemented by an intuitive interface. The research journey commences with a comprehensive literature review encompassing biomimetics, sketching techniques, and tools for transitioning hand sketches into digital realms. Subsequently, a thorough elucidation of the algorithm, crafted within the Processing environment, is provided.The efficacy of the tool is assessed through experimentation involving adjustments of various parameters on identical sketches, as well as application to seven architectural sketches representing built designs, with subsequent interpretation of the outcomes. It is posited that the adaptability of the algorithm's core logic, coupled with the development-friendly environment of Processing, holds immense potential for empowering designers to steer sketches in desired directions through tailored enhancements.

Virtual Reality Scenario Analysis of Art Design Taking into Account Interactive Digital Media Pattern Generation Technology

In art design, 3D printing technology is crucial, and more and more creators conceive scenes using 3D modeling software to get a three-dimensional and beautiful work. Due to the large amount of noise and redundant points in the raw data collected during the modeling process, the generation speed and rendering effect of 3D models are reduced. Given the above problems, the study designed an interactive 3D lightweight modeling system based on the combination of hand-drawn sketching and laser 3D scanning based on the streamlined algorithm. The experimental results showed that when the hand-drawn speed was 300, the number of triangular slices, model size, and time required to generate the model of the hand-drawn sketching model based on the streamlined algorithm were reduced by 67.39%, 65.48%, and 63.79%, respectively. In the real-time point cloud data streamlining process of the laser 3D scanning model, the point cloud data reduction ratio and the streamlining goodness index of the point cloud streamlining algorithm are 71.99% and 3.06%, respectively. The system performance is robust, and the data processing speed and rendering effect are good.

Precision sketching with de-aging networks in forensics

Addressing the intricacies of facial aging in forensic facial recognition, traditional sketch portraits often fall short in precision. This study introduces a pioneering system that seamlessly integrates a de-aging module and a sketch generator module to overcome the limitations inherent in existing methodologies. The de-aging module utilizes a deepfake-based neural network to rejuvenate facial features, while the sketch generator module leverages a pix2pix-based Generative Adversarial Network (GAN) for the generation of lifelike sketches. Comprehensive evaluations on the CUHK and AR datasets underscore the system’s superior efficiency. Significantly, comprehensive testing reveals marked enhancements in realism during the training process, demonstrated by notable reductions in Frechet Inception Distance (FID) scores (41.7 for CUHK, 60.2 for AR), augmented Structural Similarity Index (SSIM) values (0.789 for CUHK, 0.692 for AR), and improved Peak Signal-to-Noise Ratio (PSNR) metrics (20.26 for CUHK, 19.42 for AR). These findings underscore substantial advancements in the accuracy and reliability of facial recognition applications. Importantly, the system, proficient in handling diverse facial characteristics across gender, race, and culture, produces both composite and hand-drawn sketches, surpassing the capabilities of current state-of-the-art methods. This research emphasizes the transformative potential arising from the integration of de-aging networks with sketch generation, particularly for age-invariant forensic applications, and highlights the ongoing necessity for innovative developments in de-aging technology with broader societal and technological implications.

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Effectively and efficiently retrieving images from remote-sensing databases is a critical challenge in the realm of remote-sensing big data. Utilizing hand-drawn sketches as retrieval inputs offers intuitive and user-friendly advantages, yet the potential of multi-level feature integration from sketches remains underexplored, leading to suboptimal retrieval performance. To address this gap, our study introduces a novel zero-shot, sketch-based retrieval method for remote-sensing images, leveraging multi-level feature extraction, self-attention-guided tokenization and filtering, and cross-modality attention update. This approach employs only vision information and does not require semantic knowledge concerning the sketch and image. It starts by employing multi-level self-attention guided feature extraction to tokenize the query sketches, as well as self-attention feature extraction to tokenize the candidate images. It then employs cross-attention mechanisms to establish token correspondence between these two modalities, facilitating the computation of sketch-to-image similarity. Our method significantly outperforms existing sketch-based remote-sensing image retrieval techniques, as evidenced by tests on multiple datasets. Notably, it also exhibits robust zero-shot learning capabilities in handling unseen categories and strong domain adaptation capabilities in handling unseen novel remote-sensing data. The method’s scalability can be further enhanced by the pre-calculation of retrieval tokens for all candidate images in a database. This research underscores the significant potential of multi-level, attention-guided tokenization in cross-modal remote-sensing image retrieval. For broader accessibility and research facilitation, we have made the code and dataset used in this study publicly available online.

A Computer Vision Framework for Structural Analysis of Hand-Drawn Engineering Sketches.

Structural engineers are often required to draw two-dimensional engineering sketches for quick structural analysis, either by hand calculation or using analysis software. However, calculation by hand is slow and error-prone, and the manual conversion of a hand-drawn sketch into a virtual model is tedious and time-consuming. This paper presents a complete and autonomous framework for converting a hand-drawn engineering sketch into an analyzed structural model using a camera and computer vision. In this framework, a computer vision object detection stage initially extracts information about the raw features in the image of the beam diagram. Next, a computer vision number-reading model transcribes any handwritten numerals appearing in the image. Then, feature association models are applied to characterize the relationships among the detected features in order to build a comprehensive structural model. Finally, the structural model generated is analyzed using OpenSees. In the system presented, the object detection model achieves a mean average precision of 99.1%, the number-reading model achieves an accuracy of 99.0%, and the models in the feature association stage achieve accuracies ranging from 95.1% to 99.5%. Overall, the tool analyzes 45.0% of images entirely correctly and the remaining 55.0% of images partially correctly. The proposed framework holds promise for other types of structural sketches, such as trusses and frames. Moreover, it can be a valuable tool for structural engineers that is capable of improving the efficiency, safety, and sustainability of future construction projects.

SketchTrans: Disentangled Prototype Learning With Transformer for Sketch-Photo Recognition.

Matching hand-drawn sketches with photos (a.k.a sketch-photo recognition or re-identification) faces the information asymmetry challenge due to the abstract nature of the sketch modality. Existing works tend to learn shared embedding spaces with CNN models by discarding the appearance cues for photo images or introducing GAN for sketch-photo synthesis. The former unavoidably loses discriminability, while the latter contains ineffaceable generation noise. In this paper, we start the first attempt to design an information-aligned sketch transformer (SketchTrans +) via cross-modal disentangled prototype learning, while the transformer has shown great promise for discriminative visual modelling. Specifically, we design an asymmetric disentanglement scheme with a dynamic updatable auxiliary sketch (A-sketch) to align the modality representations without sacrificing information. The asymmetric disentanglement decomposes the photo representations into sketch-relevant and sketch-irrelevant cues, transferring sketch-irrelevant knowledge into the sketch modality to compensate for the missing information. Moreover, considering the feature discrepancy between the two modalities, we present a modality-aware prototype contrastive learning method that mines representative modality-sharing information using the modality-aware prototypes rather than the original feature representations. Extensive experiments on category- and instance-level sketch-based datasets validate the superiority of our proposed method under various metrics.

Sketch2Prototype: rapid conceptual design exploration and prototyping with generative AI

AbstractSketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback.

Hand drawing-based daylight analysis using deep learning and augmented reality

The utilization of artificial intelligence (AI) in daylight analysis has recently experienced an increase. Its capabilities have been demonstrated through different studies. While using AI for daytime simulation can reduce time, it also presents significant challenges. Whether we utilize the conventional approach of simulating daylight or employ artificial intelligence, the process of going from hand drawing to obtaining the daylight analysis result involves intricate procedures associated with modeling, programming, and integrating AI models. This paper introduces an efficient approach to rapidly predict daylight performance directly from the user's hand-drawn sketches. The core idea is to leverage a deep learning model for predicting directly from hand drawing and augmented reality (AR) to seamlessly display results in an AR environment, overlaying them on the original drawings. This method not only increases user engagement and accessibility but also eliminates the dependency on traditional daylight simulation software and workflow. Artificial neural network (ANN) was exploited using data derived from 3D building models and Useful Daylight Illuminance simulations created by Rhino3D and Honeybee. The color code method, various imaging, and data processing techniques were implemented to ensure compatibility between the hand-drawn input and the simulation model. Unity3D is used as the development platform with the AR engine Vuforia. Notably, the ANN model yields promising results, with R2 ranging from 0.872 to 0.985. This is achieved by efficiently capturing and converting grid-based hand-drawn images into suitable input for the ANN. The obtained results demonstrate the feasibility and emphasize the concept's practicality.

Zero-shot sketch-based image retrieval via adaptive relation-aware metric learning

Retrieving natural images with the query sketches under the zero-shot scenario is known as zero-shot sketch-based image retrieval (ZS-SBIR). Most of the best-performing methods adapt the triplet loss to learn projections that map natural images and sketches to a latent embedding space. They nevertheless neglect the modality gap between the hand-drawn sketches and the photos and consider no difference between any two incorrect classes, which limits their performance in real use cases. Towards this end, we put forward a simple and effective model, which adopts relation-aware metric learning to suppress the modality gap between the sketches and the photos. We also propose an adaptive margin that utilizes each anchor in embedding space to improve clustering ability in metric learning. Extensive experiments on the Sketchy and TU-Berlin datasets show the dominant position of our proposed model over SOTA competitors.

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for ‘numbats.’ Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., “numbat digging in the ground.” In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of “difficult-to-name but easy-to-draw” objects and text describing “difficult-to-sketch but easy-to-verbalize” object's attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of ~2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNet (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at: https://vl2g.github.io/projects/cstbir.

DualSmoke: Sketch-based smoke illustration design with two-stage generative model

The dynamic effects of smoke are impressive in illustration design, but it is a troublesome and challenging issue for inexpert users to design smoke effects without domain knowledge of fluid simulations. In this work, we propose DualSmoke, a two-stage global-to-local generation framework for interactive smoke illustration design. In the global stage, the proposed approach utilizes fluid patterns to generate Lagrangian coherent structures from the user’s hand-drawn sketches. In the local stage, detailed flow patterns are obtained from the generated coherent structure. Finally, we apply a guiding force field to the smoke simulator to produce the desired smoke illustration. To construct the training dataset, DualSmoke generates flow patterns using finite-time Lyapunov exponents of the velocity fields. The synthetic sketch data are generated from the flow patterns by skeleton extraction. Our user study verifies that the proposed design interface can provide various smoke illustration designs with good user usability. Our code is available at https://github.com/shasph/DualSmoke.

A Sketch-texture Retrieval Framework using Perceptual Similarity

Sketch-based image retrieval is an important research topic in the field of image processing. Hand-drawn sketches consist only of contour lines, and lack detailed information such as color and textons. As a result, they differ significantly from color images in terms of image feature distribution, making sketch-based image retrieval a typical cross-domain retrieval problem. To solve this problem, we constructed a perceptual space consistent with both textures and sketches, and using perceptual similarity for sketch-based texture retrieval. To implement this approach, we first conduct a set of psychological experiments to analyze the similarity of visual perception of the textures, then we create a dataset of over a thousand hand-drawn sketches according to the textures. We proposed a layer-wise perceptual similarity learning method that integrates perceptual similarity, with which we trained a similarity prediction network to learn the perceptual similarity between hand-drawn sketches and natural texture images. The trained network can be used for perceptual similarity prediction and efficient retrieval. Our experimental results demonstrate the effectiveness of sketch-based texture retrieval using perceptual similarity.

Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Constraints.

Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance. It is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can also handle hand-drawn sketches. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.