Unsupervised self-attention lightweight photo-to-sketch synthesis with feature maps

Abstract

Face-sketch synthesis is important for gaining a clear portrait photo of suspects when solving crimes. Recent research has made a great process in self-attention generative adversarial networks. We propose a method of unsupervised learning in the synthesis of face sketch-to-photo using a new attention module. The method of processing on a small reference set of photo-sketch pairs adds to the attention module, a focus on the regions distinguishing photos from sketches on the basis of the feature maps obtained by the auxiliary classifier. Unlike previous attention-based methods, which cannot handle the geometric changes between domains, our model can translate images requiring holistic changes. At the same time, we reduce the layers of the discriminator according to different residual layers to optimize our network. With the proposed approach, we can train our networks using a small reference set of photo-sketch pairs together with a large number of face-photo datasets and more distinguishing facial-feature regions in the self-attention model. Experiments have shown the superiority of the proposed method to existing face sketch-to-photo synthesis models using fixed network architectures and hyper-parameters.