Abstract Despite recent advances in multi-modal artificial intelligence (AI) technology, there remains a significant gap in their ability to be incorporated into complex design and engineering work. One such challenge relates to contexts where sketch-based inputs are desirable, due to the difficulty in recognizing freehand sketches or interpreting underlying human intent. To elucidate requirements for emerging sketch-based AI systems for complex design contexts, we consider an architectural design case study. Using a Wizard of Oz experimental paradigm, we substitute the “tool” with human agents and conduct a lab-based study in which professional architectural designers complete a design brief using this “tool.” Here, the human agents execute functions such as recognizing freely produced design plans and perspective drawings for downstream applications (e.g., generating inspirational images or high-quality renders). Observing the human agents performing the sketch recognition task, results demonstrate that agents not only rely on visible sketch elements (i.e., lines) and architectural drawing codes, but also on their memory of previous lines and their knowledge of the design brief to comprehend perceived lines. Agents gradually develop an understanding of the designed artifact, but also of the designer's intentions. These activities are crucial for the agent to obtain a functional model of the designed object, beyond a purely topological and geometric perception model. Insights about this human workflow bring new potential techniques of sketch recognition for design and engineering tasks, informing the inclusion of new resources within AI tools.

The evolving landscape of distributed user interfaces requires the prototyping stage also be distributed between users, tasks, platforms, and environments. To create a cohesive distribution of the user interface elements in such ecosystems, paired sketching has emerged as a collaborative design method that leverages multiple stakeholders’ strengths, including designers, developers, and end users, working in pairs. In the context of developer experience applied to paired sketching for distributed user interfaces, we decomposed a workflow into four disciplines according to the Software and Systems Process Engineering Meta-Model (SPEM) notation. First, we defined a protocol to deploy paired sketching of distributed user interfaces, supported by UbiSketch , a collaborative software environment tailored featuring sketch recognition and whiteboarding. Second, to evaluate paired sketching for engineering interactive systems, we conducted an experiment involving five pairs of stakeholders who sketched a distributed user interface for inside-the-vehicule interaction distributed on four platforms: smartphone, tablet, pen display, and tabletop. Empirical results from questionnaires, reactivity, intention, perceived satisfaction, and free comments, suggest a preference order in which the tabletop is ranked first, followed by the tablet, smartphone, and pen display. Based on these results, we discuss the potential of paired sketching for distributed user interfaces.

Through the use of deep learning algorithms and cloud infrastructure, including Amazon Web Services (AWS), this project seeks to improve the effectiveness and precision of forensic face sketch construction and recognition. In order to generate accurate composite sketches, the system makes use of pre-existing facial sketch datasets and databases to recommend pertinent facial features. Additionally, it incorporates a face sketch recognition module that helps identify suspects by comparing created sketches to databases maintained by law enforcement. The platform has strong security features like IP-based access control and MAC address filtering, and it shows excellent recognition accuracy. In order to facilitate wider applications in criminal identification and public safety, future improvements might incorporate integration with social media platforms and realtime video feeds.

This Real time Sketch Recognition System application features a Reactbased frontend for an interactive user interface with real-time display and input validation, coupled with a Node.js/Express backend that processes arithmetic operations through RESTful APIs, ensuring accurate calculations with proper error handling. The project demonstrates clean separation of concerns, with the frontend managing UI/UX elements and the backend handling computational logic, while its modular architecture supports potential enhancements like user authentication, calculation history, or advanced mathematical functions. Built with modern web technologies, it exemplifies fundamental full-stack development principles including API integration, state management, and responsive design, providing a scalable foundation for future expansion into more complex features or deployment scenarios.

SketchMLP: effectively utilize rasterized images and drawing sequences for sketch recognition

Global and Local Similarity Aggregation for Face Sketch Recognition

Text-guided image-to-sketch diffusion models

This paper deals with creative co-design between human and machine. It presents an alternative design method based on an emerging technology of sketch interpretation to support co-creation and collaborative creativity in architecture. This technology embraces spontaneity in design by generating inspirational images linked to the architect’s sketches. Our research aims to determine the benefits and challenges of this alternative instrumentation. We are developing a Wizard of Oz test method by immersing several designers in a studio instrumented by this human-machine co-creation technology. We analyze quantitatively and qualitatively the single-designer ideation activity of these subjects. We then investigate the integration of this co-creation instrumentation within the framework of a team design involving several architects. This confirms known benefits such as speeding-up and freeing-up of ideation and highlights the need for designers to evaluate sketched ideas by means of images simulating their real-life rendering, as well as the need for inspiration to materialize the premises of ideas that are still vague.

A sketch recognition method based on bi-modal model using cooperative learning paradigm

Face sketch synthesis (FSS) is considered as an image-to-image translation problem, where a face sketch is generated from an input face photo. FSS plays a vital role in video/image surveillance-based law enforcement. In this paper, motivated by the recent success of generative adversarial networks (GAN), we consider conditional GAN (cGAN) to approach the problem of face sketch synthesis. However, despite the powerful cGAN model’s ability to generate fine textures, low-quality inputs characterized by the facial sketches drawn by artists cannot offer realistic and faithful details and have unknown degradation due to the drawing process, while high-quality references are inaccessible or even unexistent. In this context, we propose an approach based on Generative Reference Prior (GRP) to improve the synthesized face sketch perception. Our proposed model, that we call cGAN-GRP, leverages diverse and rich priors encapsulated in a pre-trained face GAN for generating high-quality facial sketch synthesis. Extensive experiments on publicly available face databases using facial sketch recognition rate and image quality assessment metrics as criteria demonstrate the effectiveness of our proposed model compared to several state-of-the-art methods.

Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.

Deep learning is one of the most progressive technologies around the world. With its profits of high flexibility and less economic consumption on development, more and more people are attracted to the industry and strive to improve its environment. As two of the biggest branch incorporated with such a novel and robust technology, automatic plotting, and sketch recognition have developed simultaneously. In this paper, various experiments and designs are listed to display the application of the technology. The paper indicated the possibility of such a combination through these tests and experiments. Because of the introduction of deep learning, these fields, such as facial recognition, and medicine, reduced the challenges, solved the potential risk, and paved the path for smoother improvement. Beyond the fact that deep learning still already achieves so much work, the success of in these technologies also helps its growth such as remedy the fact of limited data. It indicates its widespread usage in different areas, suggesting a confirmed future of exploiting.

This research provides a strategy for enhancing the precision of face sketch identification through adversarial sketch-photo transformation. The approach uses a generative adversarial network (GAN) to learn to convert sketches into photographs, which may subsequently be utilized to enhance the precision of face sketch identification. The suggested method is evaluated in comparison to state-of-the-art face sketch recognition and synthesis techniques, such as sketchy GAN, similarity-preserving GAN (SPGAN), and super-resolution GAN (SRGAN). Possible domains of use for the proposed adversarial sketch-photo transformation approach include law enforcement, where reliable face sketch recognition is essential for the identification of suspects. The suggested approach can be generalized to various contexts, such as the creation of creative photographs from drawings or the conversion of pictures between modalities. The suggested method outperforms state-of-the-art face sketch recognition and synthesis techniques, confirming the usefulness of adversarial learning in this context. Our method is highly efficient for photo-sketch synthesis, with a structural similarity index (SSIM) of 0.65 on The Chinese University of Hong Kong dataset and 0.70 on the custom-generated dataset.

Cross-Modal Pixel-and-Stroke representation aligning networks for free-hand sketch recognition

In this paper, we aim to propose a novel methodology that leverages the shearlet transform as a pre-layer

With tech taking over most of the life process in recent times, primary education still lacks interactive processes using technology in the process of “Learning by Doing”. Most of the Ed. techs offer visual learning with no or near-to-zero input by children. Tinker Hunt can be a possible and one of its kind of interactive learning application used to tackle the same. A rough sketching input by children can yield results over the internet in the form of pictures, detailed information, etc. Tinker Hunt is created using Google’s QuickDraw Dataset, using hundreds of sketch patterns for multiple categories of icon-level doodles, collected from hundreds of individuals that are recognized by CNN trained model. Also, the app is tested for high accuracy in sketch recognition and categorization. Key Word: Machine Learning; QuickDraw Dataset; Interactive Learning; Convolutional Neural Network; Artificial Neural Network

A sketch is a black-and-white, 2-D graphical representation of an object and contains fewer visual details as compared to a colored image. Despite fewer details, humans can recognize a sketch and its context very efficiently and consistently across languages, cultures, and age groups, but it is a difficult task for computers to recognize such low-detail sketches and get context out of them. With the tremendous increase in popularity of IoT devices such as smartphones and smart cameras, etc., it has become more critical to recognize free hand-drawn sketches in computer vision and human-computer interaction in order to build a successful artificial intelligence of things (AIoT) system that can first recognize the sketches and then understand the context of multiple drawings. Earlier models which addressed this problem are scale-invariant feature transform (SIFT) and bag-of-words (BoW). Both SIFT and BoW used hand-crafted features and scale-invariant algorithms to address this issue. But these models are complex and time-consuming due to the manual process of features setup. The deep neural networks (DNNs) performed well with object recognition on many large-scale datasets such as ImageNet and CIFAR-10. However, the DDN approach cannot be carried out for hand-drawn sketches problems. The reason is that the data source is images, and all sketches in the images are, for example, 'birds' instead of their specific category (e.g., 'sparrow'). Some deep learning approaches for sketch recognition problems exist in the literature, but the results are not promising because there is still room for improvement. This article proposed a convolutional neural network (CNN) architecture called Sketch-DeepNet for the sketch recognition task. The proposed Sketch-DeepNet architecture used the TU-Berlin dataset for classification. The experimental results show that the proposed method beats the performance of the state-of-the-art sketch classification methods. The proposed model achieved 95.05% accuracy as compared to existing models DeformNet (62.6%), Sketch-DNN (72.2%), Sketch-a-Net (77.95%), SketchNet (80.42%), Thinning-DNN (74.3%), CNN-PCA-SVM (72.5%), Hybrid-CNN (84.42%), and human recognition accuracy of 73% on the TU-Berlin dataset.

There are many pre-trained deep learning-based face recognition models developed in the literature, such as FaceNet, ArcFace, VGG-Face, and DeepFace. However, performing transfer learning of these models for handling face sketch recognition is not applicable due to the challenge of limited sketch datasets (single sketch per subject). One promising solution to mitigate this issue is by using optimization algorithms, which will perform a fine-tuning and fitting of these models for the face sketch problem. Specifically, this research introduces an enhanced optimizer that will evolve these models by performing automatic weightage/fine-tuning of the generated feature vector guided by the recognition accuracy of the training data. The following are the key contributions to this work: (i) this paper introduces a novel Smart Switching Slime Mold Algorithm (S2SMA), which has been improved by embedding several search operations and control rules; (ii) the proposed S2SMA aims to fine-tune the pre-trained deep learning models in order to improve the accuracy of the face sketch recognition problem; and (iii) the proposed S2SMA makes simultaneous fine-tuning of multiple pre-trained deep learning models toward further improving the recognition accuracy of the face sketch problem. The performance of the S2SMA has been evaluated on two face sketch databases, which are XM2VTS and CUFSF, and on CEC’s 2010 large-scale benchmark. In addition, the outcomes were compared to several variations of the SMA and related optimization techniques. The numerical results demonstrated that the improved optimizer obtained a higher level of fitness value as well as better face sketch recognition accuracy. The statistical data demonstrate that S2SMA significantly outperforms other optimization techniques with a rapid convergence curve.

Process models play an important role in various software engineering contexts. Among others, they are used to capture business-related requirements and provide the basis for the development of process-oriented applications in low-code/no-code settings. To support modelers in creating, checking, and maintaining process models, dedicated tools are available. While these tools are generally considered as indispensable to capture process models for their later use, the initial version of a process model is often sketched on a whiteboard or a piece of paper. This has been found to have great advantages, especially with respect to communication and collaboration. It, however, also creates the need to subsequently transform the model sketch into a digital counterpart that can be further processed by modeling and analysis tools. Therefore, to automate this task, various so-called sketch recognition approaches have been defined in the past. Yet, these existing approaches are too limited for use in practice, since they, for instance, require sketches to be created on a digital device or do not address the recognition of edges or textual labels. Against this background, we use this paper to introduce Sketch2Process, the first end-to-end sketch recognition approach for process models captured using BPMN. Sketch2Process uses a neural network-based architecture to recognize the shapes, edges, and textual labels of highly expressive process models, covering 25 types of BPMN elements. To train and evaluate our approach, we created a dataset consisting of 704 hand-drawn and manually annotated BPMN models. Our experiments demonstrate that our approach is highly accurate and consistently outperforms the state of the art.

Free-hand sketches play a vital role in graphically portraying ideas and concepts in image recognition systems. Most recently proposed learning-based sketch recognition methods have achieved marked progress in recognition accuracy, but they rarely optimize the use of the sparsity features of sketch images. Although several attention-based sketch recognition models have been presented, they endure complex computations and large model sizes. To address these challenges, we present a lightweight convolutional neural network called Light-SRNet based on a dual-attention mechanism to improve the accuracy of sketch recognition while retaining its lightweight nature. In the proposed model, we introduced both the spatial and channel attention mechanisms into the feature extraction network to highlight more discriminative feature representations to enhance its powerful sketch recognition ability. We compared the proposed Light-SRNet with its competitors on the TU-Berlin dataset, Sketchy dataset, and QuickDrawExtended dataset. Extensive experimental results show that Light-SRNet achieves a recognition accuracy of 73.14%, which is comparable to other similar sketch recognition techniques, while requiring only about a quarter of the model parameters.