DiffSVR: Differentiable Neural Implicit Surface Rendering for Single-View Reconstruction with Highly Sparse Depth Prior

Abstract

It is well-known that 3D shape and texture reconstruction from a single-view image is a very challenging and an ill-posed problem, especially without 3D supervision/ground truth. Existing neural implicit surface reconstruction approaches do easily get trapped in the local minima and cannot produce high-fidelity geometry and high-quality textures (and rendered images) under single-view setting, even with provided highly sparse depth prior. In this paper, we propose a new self-supervised learning method DiffSVR that represents a complicated surface as a new depth-aware occupancy function (DOF) and utilizes an end-to-end differentiable surface rendering paradigm to optimize the neural DOF field relying only on single-view image with highly sparse depth information. The developed surface-aware sampling, occupancy self-labeling, and differentiable surface rendering with inverse computation techniques can enhance both the neural implicit surface reconstruction and the neural renderer. The extensive experiments and comparisons on two real-world benchmark datasets (e.g., DTU and KITTI) demonstrate that our approach not only numerically outperforms the current state-of-the-art methods by a large margin, but also produces surface mesh model with qualitatively better geometric details and more accurate textures, as well as exhibits good performance on generalizability and flexibility. The code and data are available at https://github.com/akomarichev/DiffSVR.