Face photo-sketch synthesis based on multi-scale feature fusion

Abstract

The synthesis of face sketches from real face photos and its inverse process have a wide range of applications, such as digital entertainment and assisting in the investigation of criminal cases. However, due to the significant discrepancies in texture between photos and sketches, the transformation between them is still a challenging problem. Recently, the methods based on generative adversarial nets (GANs) have shown encouraging results in the transformation between images, especially the transformation from photos to sketches. However, most of them produce different deformations or blur in the key components of the face, which affects the realism of the synthetic image. To address this challenge, we propose a novel face photo-sketch synthesis algorithm based on multi-scale feature fusion to improve the structural integrity and texture fidelity of the synthetic image. Firstly, an encoder is utilized to extract the multi-scale coding features of the input image. Then the bottom coding features are passed by a dilated convolution module and transmitted to the decoder for decoding. In the decoding process, the decoding features of different scales and the coding features of corresponding scales are concatenated in the channel dimension to obtain the multi-scale coding and decoding fusion features. Finally, the coding and decoding fusion features of different scales are further fused at the output of the decoder and the final synthesis result is generated through a convolution layer. By combining the features of different scales in the encoding and decoding processes in the channel dimension at the same time, we can maintain good image structure and texture details and generate realistic face sketch/photo images. We verify the effectiveness of the proposed method in several challenging datasets. The quantitative and qualitative evaluation shows that the proposed method is superior to other state-of-the-art methods in generating face sketches (or photos) with high visual quality.