The Intra-Class and Inter-Class Relationships in Style Transfer

Abstract

Neural style transfer, which has attracted great attention in both academic research and industrial engineering and demonstrated very exciting and remarkable results, is the technique of migrating the semantic content of one image to different artistic styles by using convolutional neural network (CNN). Recently, the Gram matrices used in the original and follow-up studies for style transfer were theoretically shown to be equivalent to minimizing a specific Maximum Mean Discrepancy (MMD). Since the Gram matrices are not a must for style transfer, how to design the proper process for aligning the neural activation between images to perform style transfer is an important problem. After careful analysis of some different algorithms for style loss construction, we discovered that some algorithms consider the relationships between different feature maps of a layer obtained from the CNN (inter-class relationships), while some do not (intra-class relationships). Surprisingly, the latter often show more details and finer strokes in the results. To further support our standpoint, we propose two new methods to perform style transfer: one takes inter-class relationships into account and the other does not, and conduct comparative experiments with existing methods. The experimental results verified our observation. Our proposed methods can achieve comparable perceptual quality yet with a lower complexity. We believe that our interpretation provides an effective design basis for designing style loss function for style transfer methods with different visual effects.