Real-time exploration and photorealistic reconstruction of large natural environments

Abstract

This paper presents a hybrid (geometry- and image-based) framework suitable for providing photorealistic walkthroughs of large, complex outdoor scenes, based only on a small set of real images from the scene. To this end, a novel data representation of a 3D scene is proposed, which is called morphable 3D panoramas. Motion is assumed to be taking place along a predefined path of the 3D environment and the input to the system is a sparse set of stereoscopic views at certain positions (key positions) along that path (one view per position). An approximate local 3D model is constructed from each view, capable of capturing the photometric and geometric properties of the scene only locally. Then, during the rendering process, a continuous morphing (both photometric as well as geometric) takes place between successive local 3D models, using what we call a ‘morphable 3D model’. For the estimation of the photometric morphing, a robust algorithm capable of extracting a dense field of 2D correspondences between wide-baseline images is used, whereas, for the geometric morphing, a novel method of computing 3D correspondences between local models is proposed. In this way, a physically valid morphing is always produced, which is thus kept transparent from the user. Moreover, a highly optimized rendering path is used during morphing. Thanks to the use of appropriate pixel and vertex shaders, this rendering path can be run fully in 3D graphics hardware and thus allows for high frame rates. Our system can be extended to handle multiple stereoscopic views (and therefore multiple local models) per key position of the path (related by a camera rotation). In this case, one local 3D panorama (per key position) is constructed, comprising all local 3D models therein, and so a ‘morphable 3D panorama’ is now used during the rendering process. For handling the geometric consistency of each 3D panorama, a technique which is based on solving a partial differential equation is adopted. The effectiveness of our framework is demonstrated by using it for the 3D visual reconstruction of the Samaria Gorge in Crete.