2D Sketches Research Articles

Retrieval-Specific View Learning for Sketch-to-Shape Retrieval

Sketch-based 3D shape retrieval (SBSR) can be approached by learning domain-invariant descriptors or ranking metrics from sketches and 2D view images of 3D shapes rendered through numerous viewpoints. However, determining the most appropriate viewpoints that convey discriminative geometric features to benefit the task of SBSR became an essential yet not fully explored area. Existing works extract 3D features from multi-view images observed through pre-defined viewpoints to match 2D sketches. Those methods, however, fail to dynamically select viewpoints by considering the SBSR task. In this work, we introduce a fully differentiable viewpoint learning paradigm driven by the downstream SBSR task, which supports the task-aware and sketch-dependent dynamic viewpoint determination process. We naturally integrate this task-specific and sketch-dependent viewpoint learning process into a meta-learning framework to develop a novel Dynamic Viewer (DV) module for SBSR. DV module comprises a Meta View Learner (MVL) block and a View Generator (VG) block. Specifically, as the first part of the DV module, the MVL block learns to initiate the necessary network parameters of the VG block. Then, the VG block that serves as the second part learns the best viewpoints to render 2D images. To learn the optimal viewpoints for SBSR, we further introduce a view mining loss that aims to maximize the similarity of feature-level information among rendered 2D views and the query sketch. Further, we adopt a variational autoencoder (VAE) to retrieve 3D shapes by setting the newly rendered images and query sketch as inputs. As evidenced by the comprehensive experimental results conducted on popular SBSR datasets, the proposed framework has been demonstrated to outperform recent methods in both category-level sketch-based and fine-grained SBSR.

Sketch-based 3D shape retrieval via teacher–student learning

One of the main difficulties of sketch-based 3D shape retrieval is the significant cross-modal difference between 2D sketches and 3D shapes. Most previous works adopt one-stage methods to directly learn the aligned common embedding space of sketches and shapes by a shared classifier. However, the intra-class difference of the sketch is more significant than the shape, harming the feature learning of 3D shapes when the two modalities are considered under the shared classifier. This issue harms the discrimination of the learned common embedding space. This paper proposes a novel two-stage method to learn a common aligned embedding space via teacher–student learning to address the issue. Specifically, we first employ a classification network to learn the discriminative features of shapes. The learned shape features are considered a teacher to guide the feature learning of sketches. Moreover, we design a guidance loss to achieve the feature transfer with semantic alignment. The proposed method achieves an effective, aligned cross-modal embedding space. Experiments on three public benchmark datasets prove the superiority of the proposed method over state-of-the-art methods.

I am a Genius! Influence of Virtually Embodying Leonardo da Vinci on Creative Performance.

Virtual reality (VR) provides users with the ability to substitute their physical appearance by embodying virtual characters (avatars) using head-mounted displays and motion-capture technologies. Previous research demonstrated that the sense of embodiment toward an avatar can impact user behavior and cognition. In this paper, we present an experiment designed to investigate whether embodying a well-known creative genius could enhance participants' creative performance. Following a preliminary online survey (N = 157) to select a famous character suited to the purpose of this study, we developed a VR application allowing participants to embody Leonardo da Vinci or a self-avatar. Self-avatars were approximately matched with participants in terms of skin tone and morphology. 40 participants took part in three tasks seamlessly integrated in a virtual workshop. The first task was based on a Guilford's Alternate Uses test (GAU) to assess participants' divergent abilities in terms of fluency and originality. The second task was based on a Remote Associates Test (RAT) to evaluate convergent abilities. Lastly, the third task consisted in designing potential alternative uses of an object displayed in the virtual environment using a 3D sketching tool. Participants embodying Leonardo da Vinci demonstrated significantly higher divergent thinking abilities, with a substantial difference in fluency between the groups. Conversely, participants embodying a self-avatar performed significantly better in the convergent thinking task. Taken together, these results promote the use of our virtual embodiment approach, especially in applications where divergent creativity plays an important role, such as design and innovation.

Central composite disigns for optimization of the energy factor in 3D printing

This study proposes an optimization strategy to analyze the trade-off between the conflicting objectives of minimizing energy use in 3D printing by fused deposition modeling. The motivation for this work is the need to optimize natural resources, finite in nature, in a more competitive industrial reality and increasingly focused on sustainability, another important point is that energy savings generate improvement in consumption raising organizational profit. The methodologies used were a brief review of the literature and response surface methodology in a CCD experiment. The modeling of the specimen took place through the CAD Fusion 360 software, its development began with the creation of a rectangular 2D sketch, obeying the parameters of 80 mm in its length and 10 mm in width, an Ender 3 printer, yellow PLA, was used following the guidelines set out in ISO 178. Objective of the research is to optimize the manufacturing process using fused deposition modeling, reducing energy consumption (kwh). A complete factorial design was used , as factors: the printing speed (X1), the printing density (X2), layer height (X3) and the layer width (X4), as a response of the experiment were adopted for the manufacturing process, energy (Y). The residue normality tests were performed, with a p-value of 0.170 > 0.05, showing that the data are normal, the VIF below 10 and R-sq (adj) is above 87.16%, the equation has the validated model.

GA‐Sketching: Shape Modeling from Multi‐View Sketching with Geometry‐Aligned Deep Implicit Functions

AbstractSketch‐based shape modeling aims to bridge the gap between 2D drawing and 3D modeling by providing an intuitive and accessible approach to create 3D shapes from 2D sketches. However, existing methods still suffer from limitations in reconstruction quality and multi‐view interaction friendliness, hindering their practical application. This paper proposes a faithful and user‐friendly iterative solution to tackle these limitations by learning geometry‐aligned deep implicit functions from one or multiple sketches. Our method lifts 2D sketches to volume‐based feature tensors, which align strongly with the output 3D shape, enabling accurate reconstruction and faithful editing. Such a geometry‐aligned feature encoding technique is well‐suited to iterative modeling since features from different viewpoints can be easily memorized or aggregated. Based on these advantages, we design a unified interactive system for sketch‐based shape modeling. It enables users to generate the desired geometry iteratively by drawing sketches from any number of viewpoints. In addition, it allows users to edit the generated surface by making a few local modifications. We demonstrate the effectiveness and practicality of our method with extensive experiments and user studies, where we found that our method outperformed existing methods in terms of accuracy, efficiency, and user satisfaction. The source code of this project is available at https://github.com/LordLiang/GA‐Sketching.

An empirical study of an MR-enhanced kinematic prototyping approach for articulated products

In the conceptual design and early embodiment design stages, an effective kinematic prototype can assist in detecting design errors, minimizing unnecessary parameter modifications, and optimizing subsequent resource deployment. Existing methods for creating prototypes, such as 2D sketches, 3D modelling, or analysis software, have their advantages in precise modelling and simulation. However, there is a lack of an efficient approach that can achieve both rapid 3D kinematic idea generation and early on-site interactive verification. To address these challenges, this research proposes a Mixed Reality (MR)-enhanced kinematic prototyping approach for articulated products to enable the rapid generation of 3D kinematic concepts, movement preview, and early interactions with users or the actual workspace. The proposed approach facilitates the quick and rough construction of geometry and kinematic structures through intuitive hand gestures and interactive movement preview within the MR environment. The usability of the proposed approach was evaluated by a comparison experiment with traditional sketch and CAD tools involving eight participants. It is verified that our proposed MR kinematic prototyping system could accelerate the kinematic prototype generation process, improve the user’s kinematic understanding, and facilitate on-site interactive verification in advance.

A methodology for 3D digital sketching practice in undergraduate architectural education

Paper-based sketches used in the first stage of architectural design have started to be replaced by digital sketches. Since many universities have switched to online education during the pandemic period, digital sketches have come to the fore in this period. It is seen that the architectural forms in the digital sketches are shaped according to the computer programme used, especially in undergraduate architecture education. This situation was identified as a problem in this study. As a solution to this, it is aimed to develop a method in order to develop the form-oriented creativity of the students in the computer environment. After the experimental case study, a questionnaire was conducted with the students to evaluate this method. It has been concluded that the exercise method proposed in this study can enable students to make digital sketches more easily.

Evaluating visual encoding quality of a mixed reality user interface for human–machine co-assembly in complex operational terrain

During human–machine collaboration in manufacturing activities, it is important to provide real-time annotations in the three-dimensional workspace for local workers who may lack relevant experience and knowledge. For example, in MR assembly, workers need to be alerted to avoid entering hazardous areas when manually replacing components. Recently, many researchers have explored various visual cues for expressing physical task progress information in the MR interface of intelligent systems. However, the relationship between the implantation of visual cues and the balance of interface cognition has not been well revealed, especially in tasks that require annotating hazardous areas in complex operational terrains. In this study, we developed a novel MR interface for an intelligent assembly system that supports local scene sharing based on dynamic 3D reconstruction, remote expert behavior intention recognition based on deep learning, and local personnel operational behavior visual feedback based on external bounding box. We compared the encoding results of the proposed MR interface with 3D annotations combined with 3D sketch cues (3DS), which combines 3D spatial cues (3DSC) and 3DS combined with adaptive cues (AVC), through a case study. We found that for physical tasks that require specific area annotations, 3D annotations with context (3DAC) can better improve the quality of manual work and regulate the cognitive load distribution of the MR interface more reasonably.

Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch.

The freeform architectural modeling process often involves two important stages: concept design and digital modeling. In the first stage, architects usually sketch the overall 3D shape and the panel layout on a physical or digital paper briefly. In the second stage, a digital 3D model is created using the sketch as a reference. The digital model needs to incorporate geometric requirements for its components, such as the planarity of panels due to consideration of construction costs, which can make the modeling process more challenging. In this work, we present a novel sketch-based system to bridge the concept design and digital modeling of freeform roof-like shapes represented as planar quadrilateral (PQ) meshes. Our system allows the user to sketch the surface boundary and contour lines under axonometric projection and supports the sketching of occluded regions. In addition, the user can sketch feature lines to provide directional guidance to the PQ mesh layout. Given the 2D sketch input, we propose a deep neural network to infer in real-time the underlying surface shape along with a dense conjugate direction field, both of which are used to extract the final PQ mesh. To train and validate our network, we generate a large synthetic dataset that mimics architect sketching of freeform quadrilateral patches. The effectiveness and usability of our system are demonstrated with quantitative and qualitative evaluation as well as user studies.

SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields

Realistic 3D facial generation based on Neural Radiance Fields (NeRFs) from 2D sketches benefits various applications. Despite the high realism of free-view rendering results of NeRFs, it is tedious and difficult for artists to achieve detailed 3D control and manipulation. Meanwhile, due to its conciseness and expressiveness, sketching has been widely used for 2D facial image generation and editing. Applying sketching to NeRFs is challenging due to the inherent uncertainty for 3D generation with 2D constraints, a significant gap in content richness when generating faces from sparse sketches, and potential inconsistencies for sequential multi-view editing given only 2D sketch inputs. To address these challenges, we present SketchFaceNeRF, a novel sketch-based 3D facial NeRF generation and editing method, to produce free-view photo-realistic images. To solve the challenge of sketch sparsity, we introduce a Sketch Tri-plane Prediction net to first inject the appearance into sketches, thus generating features given reference images to allow color and texture control. Such features are then lifted into compact 3D tri-planes to supplement the absent 3D information, which is important for improving robustness and faithfulness. However, during editing, consistency for unseen or unedited 3D regions is difficult to maintain due to limited spatial hints in sketches. We thus adopt a Mask Fusion module to transform free-view 2D masks (inferred from sketch editing operations) into the tri-plane space as 3D masks, which guide the fusion of the original and sketch-based generated faces to synthesize edited faces. We further design an optimization approach with a novel space loss to improve identity retention and editing faithfulness. Our pipeline enables users to flexibly manipulate faces from different viewpoints in 3D space, easily designing desirable facial models. Extensive experiments validate that our approach is superior to the state-of-the-art 2D sketch-based image generation and editing approaches in realism and faithfulness.

Locally Attentional SDF Diffusion for Controllable 3D Shape Generation

Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework --- locally attentional SDF diffusion , to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion , aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion , synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work.

Differential Operators on Sketches via Alpha Contours

A vector sketch is a popular and natural geometry representation depicting a 2D shape. When viewed from afar, the disconnected vector strokes of a sketch and the empty space around them visually merge into positive space and negative space , respectively. Positive and negative spaces are the key elements in the composition of a sketch and define what we perceive as the shape. Nevertheless, the notion of positive or negative space is mathematically ambiguous: While the strokes unambiguously indicate the interior or boundary of a 2D shape, the empty space may or may not belong to the shape's exterior. For standard discrete geometry representations, such as meshes or point clouds, some of the most robust pipelines rely on discretizations of differential operators, such as Laplace-Beltrami. Such discretizations are not available for vector sketches; defining them may enable numerous applications of classical methods on vector sketches. However, to do so, one needs to define the positive space of a vector sketch, or the sketch shape. Even though extracting this 2D sketch shape is mathematically ambiguous, we propose a robust algorithm, Alpha Contours , constructing its conservative estimate: a 2D shape containing all the input strokes, which lie in its interior or on its boundary, and aligning tightly to a sketch. This allows us to define popular differential operators on vector sketches, such as Laplacian and Steklov operators. We demonstrate that our construction enables robust tools for vector sketches, such as As-Rigid-As-Possible sketch deformation and functional maps between sketches, as well as solving partial differential equations on a vector sketch.

SketchCADGAN: A generative approach for completing partially drawn query sketches of engineering shapes to enhance retrieval system performance

Retrieval systems are commonly used to find relevant data in large datasets. In engineering, these systems are useful for locating specific engineering shapes in a large dataset of engineering components. When end users want to search for a shape, they prefer a two-dimensional (2D) sketch over a three-dimensional (3D) object. However, users lacking domain knowledge may struggle to generate a complete query sketch and provide a partially completed sketch instead. Retrieving relevant information from partially drawn sketches is difficult because they may have missing edges, partially drawn circles, holes, ovals, etc. Most retrieval systems compare the similarity between the query and items in the database, so incomplete sketches may be ineffective in finding the relevant information.To address this problem with incomplete sketches, we propose a new generative adversarial network called SketchCADGAN. This network uses a two-stage cascaded architecture, with the first network attempting to predict a CAD model image from an incomplete sketch and the second network using the CAD model image to predict a completed sketch. Both networks are trained together adversarially. Our approach is proven more effective than other advanced techniques through qualitative and quantitative comparisons. Furthermore, we present the results of the retrieval system using both partially drawn and completed sketches, and demonstrate that incorporating completed sketches from the suggested cascaded GAN architecture results in improved retrieval performance.

FMASketch: Freehand Mid-Air Sketching in AR

Sketching is a common way to depict design ideas freely. Augmented reality technology extends the human–computer interaction space from the screen to the real world, providing a new application space for 3D sketching. However, due to the difficulty of depth perception, it is difficult for sketching strokes to maintain good planar and spatial consistency, making drawing 3D sketches in AR challenging. In this article, we propose FMASketch—a method of drawing mid-air sketches similar to building blocks. The auxiliary surfaces can ensure that the strokes drawn by users can always stay on the same plane as drawing in 2D space. Gestures are used to stitch together multiple drawing planes, and the strokes on the planes form a 3D sketch, which is very similar to how 3D modeling is done on the desktop by drawing different views and assembling them. Experiments have verified the effectiveness of the auxiliary surface generation method and gesture operations in this article. The mid-air sketching method is easy to learn and provides a new idea for rapid modeling in AR scenarios.

How virtual reality is used in industrial maritime design processes: Two case studies

Virtual reality (VR) is increasingly integrated into maritime design processes, and there is a need to better understand how it can support ship designers in their work. Through mapping and interviews, we investigated how VR was used in two design projects in the maritime industry. We explored two questions: What type of design activities and conversations were facilitated by VR as a collaborative medium? When VR was not selected, what other media were used instead, and for what reasons? We found that despite access to VR, most design conversations were facilitated by emails containing screenshots, visual interpretations, and text annotations. VR was used on a few occasions, together with executable files and flythrough videos made with real-time rendered models generated in a game engine, and a web-based 3D model-sharing tool. The use of VR as a collaboration medium was enabled by a variety of design activities ranging from field studies to 2D sketching and 3D modeling. The cases show how VR needed to be integrated with other forms of communication and was seen as a supplement and not the main communication channel for design. The cases also show that the pipeline required for VR scenes creation enabled other forms of collaboration across different supports, such as web-based online collaboration tools.

Analytical solution for the dynamics and optimization of fractional Klein-Gordon equation: an application to quantum particle.

Klein-Gordon equation characterizes spin-particles through neutral charge field within quantum particle. In this context, fractionalized Klein-Gordon equation is investigated for the comparative analysis of the newly presented fractional differential techniques with non-singularity among kernels. The non-singular and non-local kernels of fractional differentiations have been employed on Klein-Gordon equation for the development of governing equation. The analytical solutions of Klein-Gordon equation have been traced out by fractional techniques by means of Laplace transforms and expressed in terms of series form and gamma function. The data analysis of fractionalized Klein-Gordon equation is observed for Pearson's correlation coefficient, probable error and regression analysis. For the sake of comparative analysis of fractional techniques, 2D sketch, 3D pie chart, contour surface with projection and 3D bar sketch have been depicted on the basis of embedded parameters. Our results suggest that varying frequency has reversal trends for quantum wave and de Broglie wave.

DEVELOPMENT OF AUTODESK INVENTOR PROFESSIONAL 2022 TEACHING MATERIALS IN COMPUTER-AIDED ENGINEERING-BASED MECHRONIC SYSTEMS COURSES AT THE MECHRONIC EXPERTISE PROGRAM AT SMKN 8 MALANG

The world of education in the 4.0 era is required to apply learning according to technology. One of them is the Mechatronics System Based on Computer Aided Engineering (SMBCAE) in the Mechatronics Expertise Program at SMKN 8 Malang which is required to be able to operate a professional Autodesk inventor to draw digitally. This trains students to solve design problems in a project. This study aims to develop teaching materials as study guides that are varied and contain materials including initial concepts, 2D sketches, 3D models, and assembly of the Autodesk Inventor 2022 application. The development of these teaching materials uses the Borg & Gall development model. The validation, which consisted of one media expert validator and two material expert validators, showed that the module was feasible to develop. In addition, based on the results of product trials conducted on 60 students, the results were 82.5%. Thus it can be categorized that the teaching material is very good and feasible to use

GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand.

3D sketching in virtual reality (VR) provides an immersive drawing experience for designs. However, due to the lack of depth perception cues in VR, scaffolding surfaces that constrain strokes to 2D are usually used as visual guides to reduce the difficulty of drawing accurate strokes. When the dominant hand is occupied by the pen tool, the efficiency of scaffolding-based sketching can be improved by using gesture input to reduce the idleness of the non-dominant hand. This paper presents GestureSurface, a bi-manual interface that uses non-dominant hand performing gestures to operate scaffolding and the other hand drawing with controller. We designed a set of non-dominant gestures to create and manipulate scaffolding surfaces, which are assembled by automatic combination based on five predefined primitive surfaces. We evaluated GestureSurface through a 20-person user study and found that the method of scaffolding-based sketching using non-dominant hand has the advantages of high efficiency and low fatigue.

A novel MR remote collaboration system using 3D spatial area cue and visual notification

In remote collaboration for emergency maintenance or training, since local workers may lack relevant experience and knowledge, it is very important to support remote experts to annotate 3D spatial areas for local workers in their workspace, such as risky spatial areas that need to be avoided for safe operation when assembling replacement parts. Recently, many researchers have studied various visual communication cues for expressing spatial information in Mixed Reality (MR) remote collaboration systems for physical tasks. However, the expression and transmission of 3D spatial area information have not been well explored, especially in tasks that need to annotate specific spatial areas in 3D space (such as risky 3D spatial areas). In the study, we developed a novel MR remote collaboration system that supports remote experts using 3D spatial area cues and visual notification to create Augmented Reality (AR) instructions for local workers on the shared 3D stereoscopic scene of the local workspace in the Virtual Reality (VR) space. We performed a user study to explore the effect of three interfaces on remote collaborative physical assembly tasks in the workspace with risky 3D spatial areas: 3D sketch cue (3DS) which is the baseline solution, 3D sketch cue combines the novel 3D spatial area cue (3DSA), and 3D sketch cue combines 3D spatial area cue and visual notification (3DSAN). We found the 3DSAN can significantly improve collaboration efficiency, reduce operation errors, and improve the usability and collaboration experience (especially the level of awareness), among industrial remote collaborative physical tasks requiring 3D spatial area reference.

Pictonaut: movie cartoonization using 3D human pose estimation and GANs

This article describes Pictonaut, a novel method to automatically synthetise animated shots from motion picture footage. Its results are editable (backgrounds, characters, lighting, etc.) with conventional 3D software, and they have the finish of professional 2D animation. Rather than addressing the challenge solely as an image translation problem, a hybrid approach combining multi-person 3D human pose estimation and GANs is taken. Sub-sampled video frames are processed with OpenPose and SMPLify-X to obtain the 3D parameters of the pose (body, hands and face expression) of all depicted characters. The captured parameters are retargeted into manually selected 3D models, cel shaded to mimic the style of a 2D cartoon. The results of sub-sampled frames are interpolated to generate a complete and smooth motion for all the characters. The background is cartoonized with a GAN. Qualitative evaluation shows that the approach is feasible, and a small dataset of synthetised shots obtained from real movie scenes is provided.