Deformable image alignment as a source of stereo correspondences on portraits

Abstract

Mesh-based deformable image alignment (MDIA) is an algorithm that warps a template image onto a target by deforming a 2D control mesh in the image plane, using an image-based nonlinear optimization strategy. MDIA has been successfully applied to various nonrigid registration problems, deformable surface tracking and stabilization of scene-to-camera motion in video. In this paper we investigate the use of image-based MDIA for computing dense correspondences for 3D reconstruction of human heads from high resolution portrait images. Human heads are topologically simple in 3D while providing textures which are challenging to match, such as hair and skin. We find that even with a simple piecewise affine deformation model MDIA delivers excellent correspondence results. We propose a robust, piecewise optimization scheme to compute MDIA on very high resolution images. We address issues of regularization and luminance correction and discuss the role of epipolar constraints. The correspondences retrieved with our approach facilitate the estimation of camera extrinsics and yield highly detailed meshes of the head.