Neural 3D face rendering conditioned on 2D appearance via GAN disentanglement method

Abstract

Previewing the shaded output of 3D models has been a long-standing requirement in the field. Typically, this is achieved by applying common materials; however, this approach is often labor-intensive and can yield only rough results in the trial stage. Conventional 2D style transfer methods are unsuitable for 3D-to-2D cross-domain conversion, and they cannot accurately reflect the mesh’s geometry. Inspired by StyleGAN2’s related research, we propose a method for rendering 2D images of 3D face meshes directly controlled by a single 2D reference image, using GAN disentanglement. Our approach involves an input of a 3D mesh and a reference image, where encoders extract geometric features from the mesh and appearance features from the reference image. These features control the StyleGAN2 generator to obtain a generated image that preserves the 3D mesh’s geometry and the reference image’s appearance. Our experiments demonstrate that this method performs well in generating images while maintaining geometric consistency.