Tracking of facial deformations in multi-image sequences with elimination of rigid motion of the head

Abstract

The paper deals with measurement of human facial deformations from synchronized image sequences taken with multiple calibrated cameras from different viewpoints. SIFT (Scale Invariant Feature Transform) keypoints are utilized as image feature points in the first place to determine spatial and temporal correspondences between images. If no temporal match is found for an image point by keypoint matching, then the tracking of the point is switched to least squares matching provided the point has one or more spatial corresponding points in the other views of the previous frame. For this purpose, a new method based on affine multi-image least squares matching is proposed where multiple spatial and temporal template images are simultaneously matched against each search image and part of the spatial template images also change during adjustment. A new method based on analyzing temporal changes in the image coordinates of the tracked points in multiple views is then presented for detecting the 3-D points which move only rigidly between consecutive frames. These points are used to eliminate the effect of rigid motion of the head and to obtain the changes in the 3-D points and in the corresponding image points due to pure deformation of the face. The methods are thoroughly tested with three multi-image sequences of four cameras including also quite large changes of facial deformations. The test results prove that the proposed affine multi-image least squares matching yields better results than another method using only fixed templates of the previous frame. The elimination of the effect of rigid motion works well and the points where the face is deforming can be correctly detected and the true deformation estimated. A method based on a novel adaptive threshold is also proposed for automated extraction and tracking of circular targets on a moving calibration object.