Portrait style transfer using deep convolutional neural networks and facial segmentation

Abstract

When standard neural style transfer approaches are used in portrait style transfer, they often inappropriately apply textures and colours in different regions of the style portraits to the content portraits, leading to unsatisfied transfer results. This paper presents a portrait style transfer method to transfer the style of one image to another. It first proposes a combined segmentation method for the portrait parts, which segments both the style portrait and the content portrait into masks of seven parts automatically, including background, face, eyes, nose, eyebrows, mouth and foreground. These masks are extracted to capture elements of the styles for objects in the style image and to preserve the structure in the content portrait. This paper then proposes an augmented deep Convolutional Neural Network (CNN) framework for portrait style transfer. The masks of seven parts are added into a trained deep convolutional neural network as feature maps in certain selected layers in the augmented deep CNN model. An improved loss function is proposed for the training of the portrait style transfer. Results on various images show that our method outperforms the state-of-the-art style transfer techniques.