Integrating Multiple Visual Cues for Robust Real-Time 3D Face Tracking

Abstract

3D face tracking is an important component for many computer vision applications. Most state-of-the-art tracking algorithms can be characterized as being either intensity- or feature-based. The intensity-based tracker relies on the brightness constraint while the feature-based tracker utilizes 2D local feature correspondences. In this paper, we propose a hybrid tracker for robust 3D face tracking. Instead of relying on single source of information, the hybrid tracker integrates feature correspondence and brightness constraints within a nonlinear optimization framework. The proposed method can track the 3D face pose reliably in real-time. We have conducted a series of evaluations to compare the performance of the proposed tracker with other state-of-the-art trackers. The experiments consist of synthetic sequences with simulation of different environmental factors, real sequences with estimated ground truth, and sequences from a real-world HCI application. The proposed tracker is shown to be superior in both accuracy and robustness.