Perspective reconstruction of non-rigid surfaces from single-view videos

Abstract

A novel algorithm to reconstruct the three-dimensional (3D) structure of non-rigid shapes based on a single view video, considering the perspective projection camera model, is presented in this paper. Even though perspective projection is considered to be the most realistic model and is ideal for a wide range of cameras in machine vision applications, but because of its complex and often non-linear equations, mostly, is approximated by some simpler camera projection models such as orthographic projection, weak perspective projection and so forth. Orthographic reconstruction of non-rigid surfaces has been done by singular value decomposition algorithm and using the orthogonal characteristics of the rotation matrix, but using this simple method, for perspective projection camera model in reconstructing non-rigid surfaces, due to the high complexity and the large number of unknowns, seems impossible. On the other hand since the orthographic projection camera model can be applied only in cases where the 3D scene is in the far distance from the camera (or in the other word, where the depth of the object is assumed to be small compared to the distance of the object to the camera), the need to apply the perspective projection camera model will arises. In this paper we have proposed a novel algorithm for perspective projection reconstruction of non-rigid surfaces from single view videos and overcome all of the complexities and all non-linearity of the perspective projection equations, for the first time. Due to the high number of unknowns, the problem has been divided into two parts of projective depth coefficients extraction and 3D shape reconstruction. The proposed method is evaluated on popular video segmentation datasets and its performance is compared to other available methods. The experiments show that the obtained results of the proposed method were acceptable when compared to the results of the previous methods, while it resolves the failures of those approaches.