CUMTGAN: An instance-level controllable U-Net GAN for facial makeup transfer

Abstract

Instance-level makeup transfer refers to transferring a specific makeup style to a particular face without cosmetic, while keeping its facial structure. To solve the challenges such as face structure loss and uncontrollable shade of makeup transfer in the existing makeup transfer methods, in this paper, we propose an instance-level face makeup transfer algorithm with generative adversarial network (CUMTGAN, Controllable U-Net Makeup Transfer GAN). The network uses the results of face semantic segmentation to put on or remove makeup on specific areas such as eyes, lips, while constraining the parts such as image background. It uses the characteristics of the skip connections of the U-Net structure to achieve hierarchical feature fusion. A Superimposed Module is also proposed to provide a controllable shade of makeup for the generated images, making the control of the makeup more arbitrary. Especially, we propose a new quantitative evaluation method to evaluate facial structure and makeup style respectively. To our best knowledge, it is the first time to evaluate generated samples from two aspects. The experimental results on the MT dataset show that the proposed method can generate natural and visual makeup images and makeup removal images, which is superior to the baseline methods in facial structure and makeup style.