Multi-style transfer and fusion of image’s regions based on attention mechanism and instance segmentation

Abstract

Recently neural style transfer technology has made great progress in terms of generation effect and work efficiency, realizing multi-style fusion and fast stylization. However, most of existing methods rely on the additional semantic description network to maintain the semantic information of the original content image, which leads to high complexity of the stylization model; they usually process regional style transferring with manually labeled masks, resulting in unnatural connection between regions. To solve the above problems, we proposed an improved regional multi-style transfer scheme based on attention mechanism and instance segmentation. Firstly, a convolutional block attention module (CBAM) attention layer and a conditional instance normalization (CIN) layer are combined and imported into the stylization network so as to realize the multi-style transfer with preserving the original semantic information. Then, the YOLACT (you only look at coefficients) network and Poisson fusion algorithm are used for stylization design and synthesis of different regions. The experimental results indicate that the proposed method not only can make the generated artworks effectively maintain the original salience semantic and visual features of the content images, but also can realize the regional stylization with natural transition among the regions of the stylized works. Compared with existing methods, our method can better emphasize the theme of original contents and enrich the visual effects with more diverse styles.