Real-Time Virtual Hair Try-on by 3D Face Tracking

Abstract

<p indent=0mm>Virtual hair try-on is becoming popular in many applications such as virtual social networking, try-on for online shopping, and hairstyle design. Most of existing systems are image-based where the user is not allowed to view the try-on effects from different perspectives and distances in real-time, which limits the usability of the system. A real-time virtual hair try-on system based on 3D face tracking is proposed, in this system, by using an ordinary web camera, the user is enabled to perform the virtual hair try-on in a way as s/he conducts the real hair try-on in front of a mirror. Firstly, the system realizes 3D face tracking, which combines the deep learning technique and the traditional optimization algorithm for determining the shape and pose of the 3D faces from the 2D video frames. Then, it automatically puts the 3D hair onto the 3D faces corresponding to the video frames with several pairs of user-specified points to wear hair. Finally, virtual-real rendering, which handles the occlusion of the 3D hair and renders it with the face in the 2D video frames, leads to the virtual-real effect of hair try-on. In addition, a simple but effective algorithm of recoloring the 3D hair is proposed so as to improve the effect of virtual hair try-on. In indoor and outdoor environments, extensive virtual try-on evaluations demonstrate the effectiveness and practicability of our system.