Using facial landmarks to assist the stereo matching in fringe projection based 3D face profilometry

Abstract

A three-dimensional (3D) face profilometry based on facial landmarks and a multi-view system is proposed. Theoretically, at least three fringe patterns are required in phase-shifting profilometry, but the stereo relative phase pair obtained by a spatial unwrapping algorithm cannot be used for stereo matching directly because the starting points of phase unwrapping are not on the same fringe period order, so it will result in incorrect disparity. We proposed an innovative method that utilizes facial landmarks as prior knowledge to adjust the relative phase pair to the same phase reference for accurate stereo matching through a phase adjustment algorithm to determine the accurate fringe order difference. The three-step phase-shifting patterns are projected in a looping sequence, and each three-adjacent fringe stereo pair is treated as a group to obtain one frame of a 3D face, which means that the 3D face output of our proposed method can achieve the same frame rate as the camera. The accuracy evaluation and dynamic 3D facial expressions reconstruction verify the success of the proposed method.