Structure-Guided Arbitrary Style Transfer for Artistic Image and Video

Abstract

Recently, neural style transfer has become a popular task in both academic research and industrial applications. Although the existing methods made great progress in terms of quality and efficiency, most of them mainly focus on extracting high-level features. Therefore, it is still challenging to display the hierarchical structure of the content image due to lack of texture information, which causes blurred boundaries and distortion of the stylized image. In this paper, a novel neural image and video style transfer scheme is proposed to suppress distortion and preserve the semantic content of the content image, which is capable of yielding satisfactory stylized images and videos of a variety of scenarios. We first propose to assemble a refine network into an auto-encoder framework to guide style transfer, which can ensure that the stylized image have diverse levels of details. Then, we introduce the global content loss and the local region structure loss to train the model and enhance the robustness of the model. In addition, in order to produce a high-quality stylized video, our method not only preserves the image structure, but also introduces a temporal consistency loss and a cycle-temporal loss to avoid temporal incoherence and motion blur as far as possible. Our approach is also friendly for photographic and exposed image and video style transfer. Both quantitative and qualitative evaluation demonstrated the effectiveness of our method.