Interactive optimization of 3D shape and 2D correspondence using multiple geometric constraints via POCS

Abstract

The traditional approach of handling motion tracking and structure from motion (SFM) independently in successive steps exhibits inherent limitations in terms of achievable precision and incorporation of prior geometric constraints about the scene. This paper proposes a projections onto convex sets (POCS) framework for iterative refinement of the measurement matrix in the well-known factorization method to incorporate multiple geometric constraints about the scene, thereby improving the accuracy of both 2D feature point tracking and 3D structure estimates. Regularities in the scene, such as points on line and plane and parallel lines and planes, that can be interactively identified and marked at each POCS iteration, enforce rank and parallelism constraints on appropriately defined local measurement matrices, one for each constraint. The POCS framework allows for the integration of the information in each of these local measurement matrices into a single measurement matrix that is "closest" to the initial observed measurement matrix in Frobenius norm, which is then factored in the usual manner. Experimental results demonstrate that the proposed interactive POCS framework consistently improves both 2D correspondences and 3D shape/motion estimates and similar results cannot be achieved by enforcing these constraints as either post or preprocessing.