NRSfPP: non-rigid structure-from-perspective projection

Abstract

A state-of-the-art algorithm for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos is proposed. It overcomes the limitations arising from the usage of orthographic camera model and also the complexity and non-linearity issues of perspective projection equation. Unlike traditional non-rigid structure-from-motion (NRSfM) methods, which have been studied only on synthetic datasets and controlled lab environments that require some prior constraints (such as manually segmented objects, limited rotations and occlusions, and full-length trajectories); the proposed method can be used in realistic video sequences. In addition, contrary to previous methods that use multiple cameras with different relative viewing angles, only a single-view video is required to reconstruct the 3D structures. By only using the 2D frames of incoming video stream, the proposed method extracts the projective depth coefficients of each point in each input frame, rotation matrix, translation vector, varying camera parameters (such as focal lengths for each input frame), and finally reconstructs the 3D deformable shape. Due to the high number of unknowns, the problem has been divided into two parts of projective depth coefficients extraction and 3D shape reconstruction. Perspective reconstruction of non-rigid surfaces has been extended to be certainly converged which leads to a significant increase in execution frequency of the iterated algorithm that has been presented for projective depth coefficients extraction. As such, it produces promising results for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos. The accuracy and robustness of the proposed method is demonstrated quantitatively on synthetic data and qualitatively on real image sequences. The experimental results show that NRSfPP provides the state-of-the-art results and resolves the failures of previous approaches by eliminating all of the predefined situations and constraints of applying perspective projection as the camera model.