Reconstructing 3D Virtual Face with Eye Gaze from a Single Image

Abstract

Reconstructing 3D virtual face from a single image has a wide range of applications in virtual reality. Existing approaches synthesize plausible reconstructed virtual faces, however, eye gaze information is usually ignored, which is critical in human-computer interaction. In this paper, we propose to reconstruct 3D virtual face with eye gaze information from a single image. The main challenges lie in two aspects, one is the low reconstruction quality in the eye region, the other one is the lack of an efficient method to obtain precise eye gaze information. To address these problems, we decompose this task into two key steps, i.e., 3D face reconstruction with precise eye region and eye contact guided facial-rotation for eye gaze information. The first step is designed for precise eye region reconstruction through joint optimization on 3D face/eye shapes and textures. The second step consists of two parts: eye contact discriminator and automatic eye contact search algorithm via gradient-based optimization to perform both eye contact and gaze estimation simultaneously. Extensive experiments on different tasks demonstrate the significant gain of the proposed approach, achieving an MSE of (30%), an SSIM of (17.85%), and a PSNR of (8.4%). It also produces lower angular errors (63.01%) in the gaze estimation task compared with human annotations.