Real-time mouth tracking and 3D reconstruction

Abstract

We propose a novel method for real-time mouth tracking and 3D reconstruction. This method comprises two successive processing stages. In the first stage, an AdaBoost learning algorithm and a Kalman filter are used to detect and track the mouth region in real-time under a complex background. In the second stage, the resultant 2D position of the mouth is used to determine the region where the 3D shape is reconstructed by use of a digital fringe projection and modified Fourier transform method. The main contribution of this paper is the real-time dense 3D reconstruction of the mouth region, which can be useful in many applications, such as lip-reading, biometrics, 3D animation, etc.