Fast and scalable 3D cyber-physical modeling for high-precision mobile augmented reality systems

Abstract

Mobile augmented reality is an emerging technique which allows users to use a mobile device's camera to capture real-world imagery and view real-world physical objects and their associated cyber-information overlaid on top of imagery of them. One key challenge for mobile augmented reality is the fast and precisely localization of a user in order to determine what is visible in their camera view. Recent advances in Structure-from-Motion (SfM) enable the creation of 3D point clouds of physical objects from an unordered set of photographs taken by commodity digital cameras. The generated 3D point cloud can be used to identify the location and orientation of the camera relative to the point cloud. While this SfM-based approach provides complete pixel-accurate camera pose estimation in 3D without relying on external GPS or geomagnetic sensors, the preparation of initial 3D point cloud typically takes from hours to a day, making it difficult to use in mobile augmented reality applications. Furthermore, creating 3D cyber-information and associating it with the 3D point cloud is also a challenge of using SfM-based approach for mobile augmented reality. To overcome these challenges in 3D point cloud creation and cyber-physical content authoring, the paper presents a new SfM framework that is optimized for mobile augmented reality and rapidly generates a complete 3D point cloud of a target scene up to 28 times faster than prior approaches. Key improvements in the proposed SfM framework stem from the use of (1) state-of-the-art binary feature descriptors, (2) new filtering approach for accurate 3D modeling, (3) optimized point cloud structure for augmented reality, and (4) hardware/software parallelism. The paper also provides a new image-based 3D content authoring method designed specifically for the limited user interfaces of mobile devices. The proposed content authoring method generates 3D cyber-information from a single 2D image and automatically associates it with the 3D point cloud.