Face Sketch Research Articles

Backdoor Attack against Face Sketch Synthesis.

Deep neural networks (DNNs) are easily exposed to backdoor threats when training with poisoned training samples. Models using backdoor attack have normal performance for benign samples, and possess poor performance for poisoned samples manipulated with pre-defined trigger patterns. Currently, research on backdoor attacks focuses on image classification and object detection. In this article, we investigated backdoor attacks in facial sketch synthesis, which can be beneficial for many applications, such as animation production and assisting police in searching for suspects. Specifically, we propose a simple yet effective poison-only backdoor attack suitable for generation tasks. We demonstrate that when the backdoor is integrated into the target model via our attack, it can mislead the model to synthesize unacceptable sketches of any photos stamped with the trigger patterns. Extensive experiments are executed on the benchmark datasets. Specifically, the light strokes devised by our backdoor attack strategy can significantly decrease the perceptual quality. However, the FSIM score of light strokes is 68.21% on the CUFS dataset and the FSIM scores of pseudo-sketches generated by FCN, cGAN, and MDAL are 69.35%, 71.53%, and 72.75%, respectively. There is no big difference, which proves the effectiveness of the proposed backdoor attack method.

Sketch-Based Image Retrieval System for Criminal Records Using Deep Learning

Abstract: An overview to the Sketch Based Image Retrieval for Criminal Record where the user provides a sketch as input to the system to retrieve relevant images from the database. It is seen that traditional methods to draw the face sketch are still difficult and time consuming. This system is developed so that the identification of criminals is done faster than the traditional method. Therefore, the paper presents a simple and effective deep learning framework where user can create the sketch of the suspect and can be matched to the database to get the relevant criminal images. It mainly uses Histogram Oriented Graph, Support Vector Machine, Deep Convolutional Neural Networks machine learning algorithms for face landmarks estimation, feature extraction and pattern matching. SBIR proves to be more efficient and faster to process the criminal face in real-time, as time plays an important role in immediate action in the crime branch.

Sketch face recognition based on light semantic Transformer network

AbstractSketch face recognition has a wide range of applications in criminal investigation, but it remains a challenging task due to the small‐scale sample and the semantic deficiencies caused by cross‐modality differences. The authors propose a light semantic Transformer network to extract and model the semantic information of cross‐modality images. First, the authors employ a meta‐learning training strategy to obtain task‐related training samples to solve the small sample problem. Then to solve the contradiction between the high complexity of the Transformer and the small sample problem of sketch face recognition, the authors build the light semantic transformer network by proposing a hierarchical group linear transformation and introducing parameter sharing, which can extract highly discriminative semantic features on small–scale datasets. Finally, the authors propose a domain‐adaptive focal loss to reduce the cross‐modality differences between sketches and photos and improve the training effect of the light semantic Transformer network. Extensive experiments have shown that the features extracted by the proposed method have significant discriminative effects. The authors’ method improves the recognition rate by 7.6% on the UoM‐SGFSv2 dataset, and the recognition rate reaches 92.59% on the CUFSF dataset.

Face Sketch Creation and Identification

Face Sketch Creation and Identification

Fine-Tuning of Pre-Trained Deep Face Sketch Models Using Smart Switching Slime Mold Algorithm

There are many pre-trained deep learning-based face recognition models developed in the literature, such as FaceNet, ArcFace, VGG-Face, and DeepFace. However, performing transfer learning of these models for handling face sketch recognition is not applicable due to the challenge of limited sketch datasets (single sketch per subject). One promising solution to mitigate this issue is by using optimization algorithms, which will perform a fine-tuning and fitting of these models for the face sketch problem. Specifically, this research introduces an enhanced optimizer that will evolve these models by performing automatic weightage/fine-tuning of the generated feature vector guided by the recognition accuracy of the training data. The following are the key contributions to this work: (i) this paper introduces a novel Smart Switching Slime Mold Algorithm (S2SMA), which has been improved by embedding several search operations and control rules; (ii) the proposed S2SMA aims to fine-tune the pre-trained deep learning models in order to improve the accuracy of the face sketch recognition problem; and (iii) the proposed S2SMA makes simultaneous fine-tuning of multiple pre-trained deep learning models toward further improving the recognition accuracy of the face sketch problem. The performance of the S2SMA has been evaluated on two face sketch databases, which are XM2VTS and CUFSF, and on CEC’s 2010 large-scale benchmark. In addition, the outcomes were compared to several variations of the SMA and related optimization techniques. The numerical results demonstrated that the improved optimizer obtained a higher level of fitness value as well as better face sketch recognition accuracy. The statistical data demonstrate that S2SMA significantly outperforms other optimization techniques with a rapid convergence curve.

A novel sketch based face recognition in unconstrained video for criminal investigation

Face recognition in video surveillance helps to identify an individual by comparing facial features of given photograph or sketch with a video for criminal investigations. Generally, face sketch is used by the police when suspect’s photo is not available. Manual matching of facial sketch with suspect’s image in a long video is tedious and time-consuming task. To overcome these drawbacks, this paper proposes an accurate face recognition technique to recognize a person based on his sketch in an unconstrained video surveillance. In the proposed method, surveillance video and sketch of suspect is taken as an input. Firstly, input video is converted into frames and summarized using the proposed quality indexed three step cross search algorithm. Next, faces are detected by proposed modified Viola-Jones algorithm. Then, necessary features are selected using the proposed salp-cat optimization algorithm. Finally, these features are fused with scale-invariant feature transform (SIFT) features and Euclidean distance is computed between feature vectors of sketch and each face in a video. Face from the video having lowest Euclidean distance with query sketch is considered as suspect’s face. The proposed method’s performance is analyzed on Chokepoint dataset and the system works efficiently with 89.02% of precision, 91.25% of recall and 90.13% of F-measure.

Intelligent generation of Peking opera facial masks with deep learning frameworks

The production of traditional Peking opera facial masks often relies on hand painting by experienced painters, which restricts the inheritance and development of this intangible cultural heritage. Current research mainly focuses on the digital reconstruction and storage of existing Peking opera facial masks, while high-quality facial mask generation technology is still in an infancy stage. In this paper, different deep learning frameworks are improved for learning features of Peking opera facial masks and generating new masks, which can effectively promote the creative application of Peking opera facial masks. First, using different data enhancement methods, an improved Style Generative Adversarial Network-2 (StyleGAN2) can learn implicit and explicit features of Peking opera facial masks and automatically generate new facial masks. In addition, an image translation framework for joint cross-domain communication under weak supervision is used to translate face sketches and color reference maps to an intermediate feature domain, and then synthesize new facial masks through an image generation network. The experimental results show that the generated Peking opera facial masks have good local randomness and excellent visual quality.

Image Retrieval Using Convolutional Autoencoder, InfoGAN, and Vision Transformer Unsupervised Models

Query by Image Content (QBIC), subsequently known as Content-Based Image Retrieval (CBIR) systems, may offer a more advantageous solution in a variety of applications, including medical, meteorological, search by image, and others. Such systems primarily use similarity matching algorithms to compare image content to get their relevance from databases. They are essentially measuring the spatial distance between extracted visual features from a query image and their correspondence in the dataset. One of the most challenging query retrieval problems is Facial Sketched-Real Image Retrieval (FSRIR), which is content similarity matching based. These facial retrieving systems are employed in a variety of contexts, including criminal justice. The difficulties of retrieving such sorts come from the composition of the human face and its distinctive parts. In addition, the comparison between these images is made from two different domains. Besides, to our knowledge, there is a rare existence of large-scale facial datasets that can be used to evolve the performance of the retrieving system. The success of the retrieval process is governed by the method used to calculate similarity and the efficient representation of compared images. However, by effectively representing visual features, the main challenge-posing component of such approaches might be resolved. Hence, this paper has several contributions that fill the research gap in content-based similarity matching and retrieving as follows: 1) The first contribution is extending the Chinese University Face Sketch (CUFS) dataset by including augmented images, introducing to the community a novel dataset named Extended Sketched-Real Image Retrieval (ESRIR). The CUFS dataset has been extended from 100 images to include 53,000 facial sketches and 53,000 real facial images. 2) The paper’s second contribution is proposing three new algorithms for sketched-real image retrieving based on convolutional autoencoder, Infogan, and Vision Transformer unsupervised models for large datasets. 3) Furthermore, to meet the subjective demands of the users because of the prevalence of multiple query formats. The third contribution of this paper is to train and assess the proposed algorithms across two additional facial datasets of various image sorts. 4) Recently, the majority of people have preferred searching for brand logo images, but it may be tricky to separate certain brand logo features from their alternatives and even from other features in an image. Thus, the fourth contribution is to compare logo image retrieval performance based on visual features derived from each of the three suggested retrieving systems. 5) The paper also proposes cloud-based energy-saving and computational complexity approaches in large-scale datasets. Due to the ubiquity of touchscreen devices, users often make drawings based on their fantasies for certain object image searches. Thus, the proposed algorithms were tested and assessed on a tough dataset of doodle-scratched human artworks. They are also studied for a multi-category dataset to cover practically all possible image types and situations. The results are compared with the most recent algorithms found in the literature. The results show that the proposed approaches are outperforming the recent algorithms.

HiFiSketch: High Fidelity Face Photo-Sketch Synthesis and Manipulation.

With the rapid development of generative adversarial networks, face photo-sketch synthesis has achieved promising performance and playing an increasingly important role in law enforcement as well as entertainment. However, most of the existing methods only work under the condition of no interference, and lack of generalization ability in wild scenes. The fidelity of the images generated by the existing methods are insufficient, and the manipulation ability according to text description is unavailable. Directly applying existing text-based image manipulation methods on face photo-sketch scenario may lead to severe distortions due to the cross-domain challenges. Therefore, we propose a novel cross-domain face photo-sketch synthesis framework named HiFiSketch, a network that learns to adjust the weights of generators for high-fidelity synthesis and manipulation. It can realize the translation of images between the photo domain and the sketch domain, and modify results according to the text input in the meanwhile. We further propose a cross-domain loss function, which can effectively preserve facial details during face photo-sketch synthesis. Extensive experiments on four public face sketch datasets show the superiority of our method compared to existing methods. We further present text-based face photo-sketch manipulation and sequential face photo-sketch manipulation for the first time to demonstrate the effectiveness of our method on high fidelity face photo-sketch synthesis and manipulation.

Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative Adversarial Network With Graph Representation Learning.

Biphasic face photo-sketch synthesis has significant practical value in wide-ranging fields such as digital entertainment and law enforcement. Previous approaches directly generate the photo-sketch in a global view, they always suffer from the low quality of sketches and complex photograph variations, leading to unnatural and low-fidelity results. In this article, we propose a novel semantic-driven generative adversarial network to address the above issues, cooperating with graph representation learning. Considering that human faces have distinct spatial structures, we first inject class-wise semantic layouts into the generator to provide style-based spatial information for synthesized face photographs and sketches. In addition, to enhance the authenticity of details in generated faces, we construct two types of representational graphs via semantic parsing maps upon input faces, dubbed the intraclass semantic graph (IASG) and the interclass structure graph (IRSG). Specifically, the IASG effectively models the intraclass semantic correlations of each facial semantic component, thus producing realistic facial details. To preserve the generated faces being more structure-coordinated, the IRSG models interclass structural relations among every facial component by graph representation learning. To further enhance the perceptual quality of synthesized images, we present a biphasic interactive cycle training strategy by fully taking advantage of the multilevel feature consistency between the photograph and sketch. Extensive experiments demonstrate that our method outperforms the state-of-the-art competitors on the CUHK Face Sketch (CUFS) and CUHK Face Sketch FERET (CUFSF) datasets.

Explicitly Semantic Guidance for Face Sketch Attribute Recognition With Imbalanced Data

Explicitly Semantic Guidance for Face Sketch Attribute Recognition With Imbalanced Data

Text-to-Sketch Synthesis via Adversarial Network

In the past, sketches were a standard technique used for recognizing offenders and have remained a valuable tool for law enforcement and social security purposes. However, relying on eyewitness observations can lead to discrepancies in the depictions of the sketch, depending on the experience and skills of the sketch artist. With the emergence of modern technologies such as Generative Adversarial Networks (GANs), generating images using verbal and textual cues is now possible, resulting in more accurate sketch depictions. In this study, we propose an adversarial network that generates human facial sketches using such cues provided by an observer. Additionally, we have introduced an Inverse Gamma Correction Technique to improve the training and enhance the quality of the generated sketches. To evaluate the effectiveness of our proposed method, we conducted experiments and analyzed the results using the inception score and Frechet Inception Distance metrics. Our proposed method achieved an overall inception score of 1.438 ± 0.049 and a Frechet Inception Distance of 65.29, outperforming other state-of-the-art techniques.

Third Eye: Forensic Face Sketch and Fingerprint Recognition

In Forensic science, Face sketches are time consuming and are very limited to identify subjects. In this paper ,we will present an interface which will allow users to create a face sketch of the suspects or they can insert and verify their hand drawn forensic face sketch. After the construction of face sketch, it can automatically match with the faces in the database. Fingerprint of suspects is also verified in database. By using the face and fingerprint recognition we can retrieve the details of the suspects from aadhaar database. We can easily identify the suspect with the help of the aadhaar details from aadhaar database. Biometric Recognition is necessary for safety and security

Converting Real Human Avatar to Cartoon Avatar Utilizing CycleGAN

Cartoons are an important art style, which not only has a unique drawing effect but also reflects the character itself, which is gradually loved by people. With the development of image processing technology, people's research on image research is no longer limited to image recognition, target detection, and tracking, but also images In this paper, we use deep learning based image processing to generate cartoon caricatures of human faces. Therefore, this paper investigates the use of deep learning-based methods to learn face features and convert image styles while preserving the original content features, to automatically generate natural cartoon avatars. In this paper, we study a face cartoon generation method based on content invariance. In the task of image style conversion, the content is fused with different style features based on the invariance of content information, to achieve the style conversion.

Sketch face recognition based on domain adaptation scaled entropy meta-network

In recent years, sketch face recognition has a wide application in law enforcement agencies and criminals. Deep learning plays a crucial role in the recent developments of face recognition, however, it is challenging to employ deep learning methods for sketch face recognition due to insufficient face photo–sketch data. Moreover, compared to photos, sketches lack detailed texture, and there exists a domain gap between photos and sketches, hence, traditional homogeneous face recognition methods perform poorly in sketch face recognition. In this paper, a novel deep learning method termed Domain Adaptation Scaled Entropy Meta-Network (DASEMN) is proposed to tackle sketch face recognition tasks. Specifically, a meta-learning training strategy is designed to tackle the few-shot problem and improve the generalization ability of the network. Then, a generalized entropy loss termed scaled mean entropy loss is proposed to guide the network to extract discriminate features. Finally, a domain adaptation module is introduced in the training set to reduce the domain gap between the sketch domain and the photo domain. Experiments on UoM-SGFS and CUFSF sketch face databases show that the proposed method is superior to other sketch face recognition methods.

Hand-drawn face sketch recognition using rank-level fusion of image quality assessment metrics

Hand-drawn face sketch recognition using rank-level fusion of image quality assessment metrics

Face photo–sketch synthesis via intra-domain enhancement

Face photo–sketch synthesis plays an increasingly important role in the field of law enforcement and entertainment. However, most of the existing methods only focus on how to cross the modality differences between the photo domain and the sketch domain, while neglecting the image quality enhancement of the generated faces. In this paper, we propose a novel intra-domain enhancement (IDE) method for the face photo–sketch synthesis task. Our method is composed of two steps to cope with the inter-domain generation gap and the intra-domain quality defect respectively, i.e. inter-domain face generation and intra-domain face enhancement. In the inter-domain face generation stage, we apply a probabilistic graphical model which is capable of synthesizing the coarse face sketch from photos with low quality. In the intra-domain face enhancement stage, a Multi-Layer feature Aggregation (MLA) based Generative Adversarial Networks (GAN) is designed to enhance facial details in the sketches and further refine the synthesized image quality. Our first phase helps bridge the modality gap between photos and sketches and ensures the shape consistency between them. The second phase aims to improve the detail-quality and resolution of the synthesized sketches. Extensive experimental results on public face sketch databases outperform the state-of-the-art face sketch synthesis methods.

Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch

Historically, hand-drawn face sketches have been commonly used by Indonesia’s police force, especially to quickly describe a person’s facial features in searching for fugitives based on eyewitness testimony. Several studies have been performed, aiming to increase the effectiveness of the method, such as comparing the facial sketch with the all-points bulletin (DPO in Indonesian terminology) or generating a facial composite. However, making facial composites using an application takes quite a long time. Moreover, when these composites are directly compared to the DPO, the accuracy is insufficient, and thus, the technique requires further development. This study applies a conditional generative adversarial network (cGAN) to convert a face sketch image into a color face photo with an additional Total Variation (TV) term in the loss function to improve the visual quality of the resulting image. Furthermore, we apply a color correction to adjust the resulting skin tone similar to that of the ground truth. The face image dataset was collected from various sources matching Indonesian skin tone and facial features. We aim to provide a method for Indonesian face sketch-to-photo generation to visualize the facial features more accurately than the conventional method. This approach produces visually realistic photos from face sketches, as well as true skin tones.

Face Sketch Synthesis Using Regularized Broad Learning System.

There are two main categories of face sketch synthesis: data- and model-driven. The data-driven method synthesizes sketches from training photograph-sketch patches at the cost of detail loss. The model-driven method can preserve more details, but the mapping from photographs to sketches is a time-consuming training process, especially when the deep structures require to be refined. We propose a face sketch synthesis method via regularized broad learning system (RBLS). The broad learning-based system directly transforms photographs into sketches with rich details preserved. Also, the incremental learning scheme of broad learning system (BLS) ensures that our method easily increases feature mappings and remodels the network without retraining when the extracted feature mapping nodes are not sufficient. Besides, a Bayesian estimation-based regularization is introduced with the BLS to aid further feature selection and improve the generalization ability and robustness. Various experiments on the CUHK student data set and Aleix Robert (AR) data set demonstrated the effectiveness and efficiency of our RBLS method. Unlike existing methods, our method synthesizes high-quality face sketches much efficiently and greatly reduces computational complexity both in the training and test processes.

Heterogeneous Face Interpretable Disentangled Representation for Joint Face Recognition and Synthesis.

Heterogeneous faces are acquired with different sensors, which are closer to real-world scenarios and play an important role in the biometric security field. However, heterogeneous face analysis is still a challenging problem due to the large discrepancy between different modalities. Recent works either focus on designing a novel loss function or network architecture to directly extract modality-invariant features or synthesizing the same modality faces initially to decrease the modality gap. Yet, the former always lacks explicit interpretability, and the latter strategy inherently brings in synthesis bias. In this article, we explore to learn the plain interpretable representation for complex heterogeneous faces and simultaneously perform face recognition and synthesis tasks. We propose the heterogeneous face interpretable disentangled representation (HFIDR) that could explicitly interpret dimensions of face representation rather than simple mapping. Benefited from the interpretable structure, we further could extract latent identity information for cross-modality recognition and convert the modality factor to synthesize cross-modality faces. Moreover, we propose a multimodality heterogeneous face interpretable disentangled representation (M-HFIDR) to extend the basic approach suitable for the multimodality face recognition and synthesis. To evaluate the ability of generalization, we construct a novel large-scale face sketch data set. Experimental results on multiple heterogeneous face databases demonstrate the effectiveness of the proposed method.