Semi-supervised Cycle-GAN for face photo-sketch translation in the wild

Abstract

The performance of face photo-sketch translation has improved a lot thanks to deep neural networks. GAN based methods trained on paired images can produce high-quality results under laboratory settings. Such paired datasets are, however, often very small and lack diversity. Meanwhile, Cycle-GANs trained with unpaired photo-sketch datasets suffer from the steganography phenomenon, which makes them not effective to face photos in the wild. In this paper, we introduce a semi-supervised approach with a noise-injection strategy, named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we propose a pseudo sketch feature representation for each input photo composed from a small reference set of photo-sketch pairs, and use the resulting pseudo pairs to supervise a photo-to-sketch generator Gp2s. The outputs of Gp2s can in turn help to train a sketch-to-photo generator Gs2p in a self-supervised manner. This allows us to train Gp2s and Gs2p using a small reference set of photo-sketch pairs together with a large face photo dataset (without ground-truth sketches). For the second problem, we show that the simple noise-injection strategy works well to alleviate the steganography effect in SCG and helps to produce more reasonable sketch-to-photo results with less overfitting than fully supervised approaches. Experiments show that SCG achieves competitive performance on public benchmarks and superior results on photos in the wild.