3D model generation using unconstrained motion of a hand-held video camera

Abstract

We have developed a shape and structure capture system which constructs accurate, realistic 3D models from video imagery taken with a single freely moving handheld camera. Using an inexpensive off the shelf acquisition system such as a hand-held video camera, we demonstrate the feasibility of fast and accurate generation of these 3D models at a very low cost. In our approach the operator freely moves the camera within some very simple constraints. Our process identifies and tracks high interest image features and computes the relative pose of the camera based on those tracks. Using a RANSAC-like approach we solve for the camera pose and 3D structure based on a homography or essential matrix. Once we have the pose for many frames in the sequence we perform correlation-based stereo to obtain dense point clouds. After these point clouds are computed we integrate them into an octree. By replacing the points in a particular cell with statistics representing the point distribution we can efficiently store the computed model. While being efficient, the integration technique also enables filtering based on occupancy counts which eliminates many stereo outliers and results in an aesthetic viewable 3D model. In this paper we describe our approach in detail as well as show reconstructed results of a synthetic room, an empty room, a lightly furnished room, and an experimental vehicle.