Projection-based augmented reality with automated shape scanning

Abstract

We propose a new framework for interactive Augmented Reality (AR) and Mixed Reality (MR) representation using both visible and invisible projection onto physical target objects. Projection-based approach for constructing AR/MR uses physical objects such as walls, books, plaster ornaments and whatever the computer generated contents can be optically projected onto. Namely, projection makes it possible to use real objects as displays. We mainly focus on capturing and utilizing the 3D shape of the object surface, whose information allows the AR/MR system to take into account the visual consistency when merging the physical and rendered objects. 3D shape data of the object can be used to compensate the distortion caused by the difference between positions of projectors and the viewer. The other advantage is the capability to generate proper visual occlusion between physical and virtual objects so that they seem to coexist in front of the viewer. What we demonstrate in this study is to employ near-infrared pattern projection for triangulation so that scanning and updating the geometry data of the object is automatically performed in background process, thus parallel processing to provide AR/MR representation can be achieved according to dynamic physical geometry changes.