PortraitNET: Photo-realistic portrait cartoon style transfer with self-supervised semantic supervision

Abstract

We propose a novel framework to transfer the portrait image into its correspondence with photo-realistic and cartoon style. The existing work on neural style transfer conducts impressive results on artistic style transfer; however, the lack of semantic clues will lead to the color artifacts in photo-realistic style transfer because of the complex background and noise issues. In this work, we re-define the semantics as the pixel motion field according to the color displacement between adjacent animation frames along the optical direction and initiatively propose the self-supervised semantic network (SSNet) to learn semantic maps without human inference or any priories. The SSNet shares parameters with the style transfer network; thus, the superior alternatives can preserve the semantic completeness in the styled image. To solve the content missing and blur problems common in NST, we propose the bilateral convolution block (B-block) and feature fusion strategy (F-block) for visual smoothness to meet the perceptive satisfaction. The ablation studies are provided to validate the effectiveness, and comparative experiments with the state-of-the-art baselines demonstrate the advantages of the proposed method.