View-dependent Projection Mapping Enhanced by Real Background

Abstract

"Spatial augmented reality (SAR)" or "projection mapping" projects an image of a virtual object on the surface of a real object. A "view-dependent" display shows the virtual object with its correct appearance for an arbitrary viewer's position. If the virtual object and the real object have different shapes, the virtual object's image to project needs to be correctly distorted according to the viewer's position. Besides, the difference causes the real object's surface to have some empty areas onto which the virtual object is not projected. Such empty areas seriously degrade the viewer's feeling that the virtual object merges into the real world. We propose a method to eliminate undesired empty areas by projecting the real background behind the real object in a view-dependent way. Our method converts a real background's image captured by a fixed camera to an appropriate image for a viewer's position based on homography. This image conversion approximates the background's shape by a plane adjusted for an arbitrary viewer's position. The plane is determined by practical parameters interpolated on a 3D grid space. Consequently, the viewer does not feel the presence of the real object and feels the virtual object merging into the real world.