Uncorrelated feature encoding for faster image style transfer

Abstract

Recent image style transfer methods use a pre-trained convolutional neural network as their feature encoder. However, the pre-trained network is not optimal for image style transfer but rather for image classification. Furthermore, they require time-consuming feature alignment to consider the existing correlation among channels of the encoded feature map. In this paper, we propose an end-to-end learning method that optimizes both encoder and decoder networks for style transfer task and relieves the computational complexity of the existing correlation-aware feature alignment. First, we performed end-to-end learning that updates not only decoder but also encoder parameters for the task of image style transfer in the network training phase. Second, in addition to the previous style and content losses, we use uncorrelation loss, i.e., the total correlation coefficient among responses of encoder channels. Our uncorrelation loss allows the encoder network to generate a feature map of channels without correlation. Subsequently, our method results in faster forward processing with only a light-weighted transformer of correlation-unaware feature alignment. Moreover, our method drastically reduced the channel redundancy of the encoded feature during the network training process. This provides us a possibility to perform channel elimination with negligible degradation in generated style quality. Our method is applicable to multiple scaled style transfer by using the cascade network scheme and allows a user to control style strength through the usage of a content-style trade-off parameter.