Self-discriminative cycle generative adversarial networks for face image translation

Abstract

Face cross-domain translation aims at mapping face images from one image domain to another. Common face image translation tasks include face photo-sketch and face photo-APDrawing cross-domain translation, which can be widely applied in real-world scenarios, such as criminal investigation, movie production, and digital entertainment. However, due to the limited face image pairs and the great gap of color and texture between different domains, face image cross-domain translation still faces many challenges. Existing methods usually produce blurring, artifacts, and structural distortion, leading to poor visualization quality. To tackle this problem, we propose a self-discriminative cycle generative adversarial network, in which the generator adopts an encoder-decoder structure and the corresponding discriminator is the encoder of the other generator in the reverse direction.In the self-discriminative manner, the encoder (i.e., discriminator) cleverly incorporates “True/False" semantic information and the sensitivity to pixel-level information, thereby enhancing the robustness and generalization ability of the generative model.Besides, we propose a novel omni-directional pixel-gradient loss. Thedesigned convolution kernel calculates the gradients of all directions around each pixel to extract the gradient information.Our model is motivated to effectively learn the continuous inter-pixel variation pattern by constraining the gradient information of generated images and ground-truth images to be consistent. The omni-directional pixel-gradient loss can be flexibly applied to other generative models and improve their performance.Extensive experiments show that the proposed framework can produce advanced results on the paired face photo-sketch datasets (CUFS, CUFSF) and the photo-APDrawing dataset (APDrawing). We further demonstrate the strong generalization ability of our model on real-world data and the excellent performance on unpaired face datasets.