A General Method of Realistic Avatar Modeling and Driving for Head-Mounted Display Users

Abstract

The head-mounted displays (HMDs) provide immersive experiences in the virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this article, we develop a novel framework to reconstruct the user’s 3-D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3-D Morphable Model (3DMM). A self-supervised UV map generative adversarial network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face data set with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.