Factorization based structure from motion with object priors

Abstract

This paper presents an efficient framework to include the information of objects position in classical multi-view geometry problems for 3D reconstruction. In particular, we present two main contributions to Structure from Motion (SfM) using factorization methods for the affine camera case. First, we introduce a method based on factorization that extends the classical 3D point cloud reconstruction based on 2D point correspondences to objects using detection correspondences. In this case, objects are approximated as quadrics in 3D (or more specifically as ellipsoids) and therefore projected as conics in 2D onto the image plane. Therefore, instead of having 2D point to point correspondences, we solve a conic to conic correspondence problem in the setting of affine factorization methods. The solution to this problem provides a 3D location/occupancy of the object together with an affine camera calibration. This is shown to be a generalisation of the standard Tomasi and Kanade factorization method with rigid objects. Secondly, we use the estimated object locations/occupancies to robustly estimate the 3D point cloud from 2D point correspondences by constructing a prior that relates 2D points locations and the positions of the object ellipsoids in 3D. This is done by recasting the problem as a probabilistic matrix factorization where the priors are not generic but truly representative of the scene structure as a composition of objects. In particular we show that by using objects to points relations, we achieve compelling results with high rate of missing data and noisy 2D data, a common occurrence when dealing with man-made textureless objects.