Soft multimodal style transfer via optimal transport

Abstract

The goal of arbitrary Neural Style Transfer (NST) is to transform a content image into a different style. Based on the assumption that styles can be represented by global statistics, the majority of NST methods seek to match global statistics for style transfer. These global statistics disregard the multimodal property of the style because distinct semantic parts of a style image may comprise various styles. Multimodal style transfer addresses this issue by extracting semantic patterns of style images through image clustering, obtaining multiple different sub-style components. The matching of the sub-style components and content features in this case is crucial because neighboring content features may be given various sub-style components. In this paper, we propose a novel assignment based on optimal transport to solve this issue. The proposed assignment theoretically guarantees that the distributions between the stylized image and the style image are similar, and it also permits more than one sub-style component to be assigned to each content feature. Based on this soft assignment, a weighted style transfer method is proposed to blend different stylized results, each of which is stylized by one sub-style component. With a more precise description of the image style, the proposed method achieves promising stylized results. Extensive experiments are conducted to demonstrate the superior effectiveness of the proposed method.