Deep graph learning for spatially-varying indoor lighting prediction

Abstract

Lighting prediction from a single image is becoming increasingly important in many vision and augmented reality (AR) applications in which shading and shadow consistency between virtual and real objects should be guaranteed. However, this is a notoriously ill-posed problem, especially for indoor scenarios, because of the complexity of indoor luminaires and the limited information involved in 2D images. In this paper, we propose a graph learning-based framework for indoor lighting estimation. The core is a new lighting model (DSGLight) based on depth-augmented spherical Gaussians (SGs) and a graph convolutional network (GCN) that infers the new lighting representation from a single low dynamic range (LDR) image of limited field-of-view. Our lighting model builds 128 evenly distributed SGs over the indoor panorama, where each SG encodes the lighting and the depth around that node. The proposed GCN then learns the mapping from the input image to DSGLight. Compared with existing lighting models, our DSGLight encodes both direct lighting and indirect environmental lighting more faithfully and compactly. It also makes network training and inference more stable. The estimated depth distribution enables temporally stable shading and shadows under spatially-varying lighting. Through thorough experiments, we show that our method obviously outperforms existing methods both qualitatively and quantitatively.