3D-aided face recognition from videos

Abstract

The deployment of cameras for security control allows for video stream to be used as input for face recognition (FR). However, most state of the art FR SDKs are generally specifically tuned for dealing with frontal and neutral face images, whereas expression and pose variations, which typically occur in unconstrained settings, e.g., video images, are still major challenges for reliable FR. In this paper, we aim to endow the state of the art FR SDKs with the capabilities to recognize faces in videos. For this purpose, given a video sequence of a person, an extended 3D Morphable Model (3DMM) is used to generate a novel view of this person where the pose is rectified and the expression neutralized. We present a 3DMM fitting method specifically designed for videos to take into account the temporal properties, making use of multiple frames for fitting. Moreover, some constraints of smoothness are used to get a better estimation of its 3D shape and to separate its expression component from its identity component. Finally, we evaluate the proposed method on the Prison Break TV serial and demonstrate its effectiveness using a standard commercial FR SDK.