ProLiF: Progressively-connected Light Field network for efficient view synthesis

Abstract

This paper presents a simple yet practical network architecture, ProLiF (Progressively-connected Light Field network), for the efficient differentiable view synthesis of complex forward-facing scenes in both the training and inference stages. The progress of view synthesis has advanced significantly due to the recent Neural Radiance Fields (NeRF). However, when training a NeRF, hundreds of network evaluations are required to synthesize a single pixel color, which is highly consuming of device memory and time. This issue prevents the differentiable rendering of a large patch of pixels in the training stage for semantic-level supervision, which is critical for many practical applications such as robust scene fitting, style transferring, and adversarial training. On the contrary, our proposed simple architecture ProLiF, encodes a two-plane light field, which allows rendering a large batch of rays in one training step for image- or patch-level losses. To keep the multi-view 3D consistency of the neural light field, we propose a progressive training strategy with novel regularization losses. We demonstrate that ProLiF has good compatibility with LPIPS loss to achieve robustness to varying light conditions, and NNFM loss as well as CLIP loss to edit the rendering style of the scene.