Neural Style Palette: A Multimodal and Interactive Style Transfer From a Single Style Image

Abstract

Despite the myriad of attributes found in a single style image, existing neural style transfer methods produce outputs with limited variety–typically only a single realization of the style image. They also do not provide an easy way to control the stylization process, limiting the creative freedom of users. In this paper, we propose Neural Style Palette ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NSP</i> ), a method for interactively generating a variety of stylized images from only a single style input. Our approach allows human influence in the stylization process, a design inspired by Hybrid Human-Artificial Intelligence. Like a color palette, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NSP</i> enables a meaningful interaction by presenting a collection of sub-textures, which we also refer to as anchor styles, that act as a visual guide for the users. These anchor styles capture different attributes in the single style image that the users can creatively blend to create their desired realizations. To offer a diversified selection in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NSP</i> , we constrain the anchor styles to be distant from one another while maintaining faithfulness to the original style image. This is possible through our two proposed novel losses: a style-separation loss that encourages the sub-textures to be distinct and a unification loss to ensure that the sub-textures center around the original style while encouraging additional diversity. We perform several experiments to prove the effectiveness of our method and generalize to improve existing methods.