Controllable Image Generation with Semi-supervised Deep Learning and Deformable-Mean-Template Based Geometry-Appearance Disentanglement

Abstract

Typical deep-neural-network (DNN) based generative image models often (i) show limited ability to learn a disentangled latent representation, (ii) show limited controllability leading to undesirable side effects when manipulating selected attributes during image generation, and (iii) require large attribute-annotated training sets. We propose a generative DNN model for face images by explicitly disentangling geometry and appearance modeling to achieve selective controllability of the desired attributes with less side effects. To learn geometric variability, we leverage grayscale sketch representations to learn (i) a deformable mean template representing the population-mean face geometry and (ii) a generative model of deformations to model individual face-geometry variations, using dense image registration. We learn the appearance variability in a (color-image) space that we explicitly design by factoring out the geometric variability. We propose a variational formulation to enable semi-supervised learning when manually-annotated attributes are severely limited in the training set. Results on large datasets show that, compared to schemes using deformation models or variational learning, our method significantly improves face-image model fits and facial-feature controllability even with semi-supervised learning.