Abstract

Image-based rendering (IBR) attempts to synthesize novel views using a set of observed images. Some IBR approaches (such as light fields) have yielded impressive high-quality results on small-scale scenes with dense photo capture. However, available wide-baseline IBR methods are still restricted by the low geometric accuracy and completeness of multi-view stereo (MVS) reconstruction on low-textured and non-Lambertian surfaces. The issues become more significant in large-scale outdoor scenes due to challenging scene content, e.g., buildings, trees, and sky. To address these problems, we present a novel IBR algorithm that consists of two key components. First, we propose a novel depth refinement method that combines MVS depth maps with monocular depth maps predicted via deep learning. A lookup table remap is proposed for converting the scale of the monocular depths to be consistent with the scale of the MVS depths. Then, the rescaled monocular depth is used as the constraint in the minimum spanning tree (MST)-based nonlocal filter to refine the per-view MVS depth. Second, we present an efficient shape-preserving warping algorithm that uses superpixels to generate the warped images and blend expected novel views of scenes. The proposed method has been evaluated on public MVS and view synthesis datasets, as well as newly captured large-scale outdoor datasets. In comparison with state-of-the-art methods, the experimental results demonstrated that the proposed method can obtain more complete and reliable depth maps for the challenging large-scale outdoor scenes, thereby resulting in more promising novel view synthesis.

Highlights

  • With the increasing demand for immersive 3D content, many view synthesis methods [1]–[5] for providing realistic interactive virtual navigation have been proposed

  • To improve the quality of depth estimation and view synthesis for large-scale outdoor scenes, in this work, we propose an image-based rendering (IBR) method that is based on fusion of monocular and multi-view stereo (MVS) depth

  • Since the MVS depth and the monocular depth are from distributions that differ substantially in terms of scale, we present a novel layerwise mapping between the monocular depth and the MVS depth via a lookup table

Read more

Summary

INTRODUCTION

With the increasing demand for immersive 3D content, many view synthesis methods [1]–[5] for providing realistic interactive virtual navigation have been proposed. To improve the quality of depth estimation and view synthesis for large-scale outdoor scenes, in this work, we propose an IBR method that is based on fusion of monocular and MVS depth. Our main contributions are summarized as follows: 1) A lookup-table-based strategy that remaps the monocular depth to the scale of the MVS depth; 2) An MST-based algorithm for fusing the monocular depth and the MVS depth, which can fill in irregularities and large holes of MVS depth maps while preserving geometric details; 3) A complete pipeline for image-based outdoor scenes navigation, which includes a refinement method for depth estimation, and a superpixel-based shapepreserving warp for view synthesis.

RELATED WORK
DEPTH ESTIMATION Multi-View 3D Reconstruction
WARPING AND RENDERING
RESULTS AND COMPARISONS
EVALUATION OF THE DEPTH REFINEMENT RESULTS
CONCLUSIONS
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call