Voxel grid performer: efficient radiance fields generalization for human novel view synthesis

Abstract

Novel view synthesis is a long-standing problem. Despite the rapid development of neural radiance field (nerf), in terms of rendering dynamic human body, NeRF still cannot achieve a good trade-off in precision and efficiency. In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performers in an efficient way, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting these generalization approchs to human achieve reasonable rendering result. However, due to the difficulties of modeling the complex appearance of human and the dynamic sense, it is challenging to train nerf well in an efficient way. We find that the slow convergence of the human body reconstruction model is largely due to the nerf representation. In this work, we introduce a voxel grid based representation for human view synthesis, termed Voxel Grid Performer(VGP). Specifically, a sparse voxel grid is designed to represent the density and color in every space voxel, which enable better performance and less computation than conventional nerf optimization. We perform extensive experiments on both seen human performer and unseen human performer, demonstrating that our approach surpasses nerf-based methods on a wide variety of metrics. Code and data will be made available at https://github.com/fanzhongyi/vgp.