Sketch-photo Recognition Research Articles

SketchTrans: Disentangled Prototype Learning With Transformer for Sketch-Photo Recognition.

Matching hand-drawn sketches with photos (a.k.a sketch-photo recognition or re-identification) faces the information asymmetry challenge due to the abstract nature of the sketch modality. Existing works tend to learn shared embedding spaces with CNN models by discarding the appearance cues for photo images or introducing GAN for sketch-photo synthesis. The former unavoidably loses discriminability, while the latter contains ineffaceable generation noise. In this paper, we start the first attempt to design an information-aligned sketch transformer (SketchTrans +) via cross-modal disentangled prototype learning, while the transformer has shown great promise for discriminative visual modelling. Specifically, we design an asymmetric disentanglement scheme with a dynamic updatable auxiliary sketch (A-sketch) to align the modality representations without sacrificing information. The asymmetric disentanglement decomposes the photo representations into sketch-relevant and sketch-irrelevant cues, transferring sketch-irrelevant knowledge into the sketch modality to compensate for the missing information. Moreover, considering the feature discrepancy between the two modalities, we present a modality-aware prototype contrastive learning method that mines representative modality-sharing information using the modality-aware prototypes rather than the original feature representations. Extensive experiments on category- and instance-level sketch-based datasets validate the superiority of our proposed method under various metrics.

Face photo-sketch recognition based on multi-directional line features projection

Face photo-sketch recognition based on multi-directional line features projection

Exploiting an Intermediate Latent Space between Photo and Sketch for Face Photo-Sketch Recognition.

The photo-sketch matching problem is challenging because the modality gap between a photo and a sketch is very large. This work features a novel approach to the use of an intermediate latent space between the two modalities that circumvents the problem of modality gap for face photo-sketch recognition. To set up a stable homogenous latent space between a photo and a sketch that is effective for matching, we utilize a bidirectional (photo → sketch and sketch → photo) collaborative synthesis network and equip the latent space with rich representation power. To provide rich representation power, we employ StyleGAN architectures, such as StyleGAN and StyleGAN2. The proposed latent space equipped with rich representation power enables us to conduct accurate matching because we can effectively align the distributions of the two modalities in the latent space. In addition, to resolve the problem of insufficient paired photo/sketch samples for training, we introduce a three-step training scheme. Extensive evaluation on a public composite face sketch database confirms superior performance of the proposed approach compared to existing state-of-the-art methods. The proposed methodology can be employed in matching other modality pairs.

Face recognition based on full convolutional neural network based on transfer learning model

Deep learning has achieved a great success in face recognition (FR), however, little work has been done to apply deep learning for face photo-sketch recognition. This paper proposes an adaptive scale local binary pattern extraction method for optical face features. The extracted features are classified by Gaussian process. The most authoritative optical face test set LFW is used to train the trained model. Test, the test accuracy is 98.7%. Finally, the face features extracted by this method and the face features extracted from the convolutional neural network method are adapted to sketch faces through transfer learning, and the results of the adaptation are compared and analyzed. Finally, the paper tested the open-source sketch face data set CUHK Face Sketch database(CUFS) using the multimedia experiment of the Chinese University of Hong Kong. The test result was 97.4%. The result was compared with the test results of traditional sketch face recognition methods. It was found that the method recognized High efficiency, it is worth promoting.

Identity-aware CycleGAN for face photo-sketch synthesis and recognition

Face photo-sketch synthesis and recognition has many applications in digital entertainment and law enforcement. Recently, generative adversarial networks (GANs) based methods have significantly improved the quality of image synthesis, but they have not explicitly considered the purpose of recognition. In this paper, we first propose an Identity-Aware CycleGAN (IACycleGAN) model that applies a new perceptual loss to supervise the image generation network. It improves CycleGAN on photo-sketch synthesis by paying more attention to the synthesis of key facial regions, such as eyes and nose, which are important for identity recognition. Furthermore, we develop a mutual optimization procedure between the synthesis model and the recognition model, which iteratively synthesizes better images by IACycleGAN and enhances the recognition model by the triplet loss of the generated and real samples. Extensive experiments are performed on both photo-to-sketch and sketch-to-photo tasks using the widely used CUFS and CUFSF databases. The results show that the proposed method performs better than several state-of-the-art methods in terms of both synthetic image quality and photo-sketch recognition accuracy.

Deep feature representation and ball-tree for face sketch recognition

Forensic face sketch-photo recognition attracts considerable interest in the law enforcement agencies. This paper proposes a new face sketch-photo recognition method based on the VGG deep feature and ball-tree searching algorithm. In this paper, the recognition performances by different feature layers of pretrained VGG-Face model are explored. In addition, to accelerate the matching speed, the ball-tree algorithm is adopted to search the nearest neighbors of query sketches from gallery photos. The experimental results on CUFS and IIIT-D datasets demonstrate the superiority of the proposed method compared with existing algorithms.

Forensic Face Photo-Sketch Recognition Using a Deep Learning-Based Architecture

Numerous methods that automatically identify subjects depicted in sketches as described by eyewitnesses have been implemented, but their performance often degrades when using real-world forensic sketches and extended galleries that mimic law enforcement mug-shot galleries. Moreover, little work has been done to apply deep learning for face photo-sketch recognition despite its success in numerous application domains including traditional face recognition. This is primarily due to the limited number of sketch images available, which are insufficient to robustly train large networks. This letter aims to tackle these issues with the following contributions: 1) a state-of-the-art model pre-trained for face photo recognition is tuned for face photo-sketch recognition by applying transfer learning, 2) a three-dimensional morphable model is used to synthesise new images and artificially expand the training data, allowing the network to prevent over-fitting and learn better features, 3) multiple synthetic sketches are also used in the testing stage to improve performance, and 4) fusion of the proposed method with a state-of-the-art algorithm is shown to further boost performance. An extensive evaluation of several popular and state-of-the-art algorithms is also performed using publicly available datasets, thereby serving as a benchmark for future algorithms. Compared to a leading method, the proposed framework is shown to reduce the error rate by 80.7% for viewed sketches and lowers the mean retrieval rank by 32.5% for real-world forensic sketches.

Face sketch-photo recognition using local gradient checksum: LGCS

A new approach for matching of face sketch images with face photo images and vice versa has been presented here. For the extraction of local edge features from both the sketch and photo images, a new local feature called local gradient checksum (LGCS) has been developed. LGCS is a modality reduction local edge feature on gradient domain. It is calculated as the summation of four pairs of gradient differences between two local pixels that are at 180° with each other. The Euclidean distance between query sketch and gallery of photos are measured depending on extracted LGCS features. To further improve the result, a multi-scale LGCS is proposed. A rank-1 accuracy of 100 % is achieved in a gallery of 606 photos consisting of CUHK, AR, and XM2VTS face dataset. The proposed face sketch-photo recognition system requires neither learning procedures nor training data. Further, the experiment is extended to test the robustness of the proposed algorithm on blurred, noisy and disguised sketches, as well as photos. Under those situations also, LGCS has outperformed center-symmetric local binary pattern, directional local extrema pattern and weber local descriptor feature extraction techniques.

A new approach for face recognition by sketches in photos

Face recognition by sketches in photos remains a challenging task. Unlike the existing sketch–photo recognition methods, which convert a photo into sketch and then perform the sketch–photo recognition through sketch–sketch recognition, this paper devotes to synthesizing a photo from the sketch and transforming the sketch–photo recognition to photo–photo recognition to achieve better performance in mixture pattern recognition. The contribution of this paper mainly focuses on two aspects: (1) in view of that there are no many research findings of sketch–photo recognition based on the pseudo-photo synthesis and the existing methods require a large set of training samples, which is nearly impossible to achieve for the high cost of sketch acquisition, we make use of embedded hidden Markov model (EHMM), which can learn the nonlinearity of sketch–photo pair with less training samples, to produce pseudo-photos in terms of sketches; and (2) photos and sketches are divided into patches and pseudo-photo is generated by combining pseudo-photo patches, which makes pseudo-photo more recognizable. Experimental results demonstrate that the newly proposed method is effective to identify face sketches in photo set.

Face Sketch Synthesis Algorithm Based on E-HMM and Selective Ensemble

Sketch synthesis plays an important role in face sketch-photo recognition system. In this manuscript, an automatic sketch synthesis algorithm is proposed based on embedded hidden Markov model (E-HMM) and selective ensemble strategy. First, the E-HMM is adopted to model the nonlinear relationship between a sketch and its corresponding photo. Then based on several learned models, a series of pseudo-sketches are generated for a given photo. Finally, these pseudo-sketches are fused together with selective ensemble strategy to synthesize a finer face pseudo-sketch. Experimental results illustrate that the proposed algorithm achieves satisfactory effect of sketch synthesis with a small set of face training samples.