NeuralPlan: Neural floorplan radiance fields for accelerated view synthesis

Abstract

We propose an approach for quickly building a visual representation of a full indoor building. Our goal is to enable intelligent systems which frequently and regularly monitor buildings to assist personnel operating remotely, a need of special importance in these days. Prior work in neural scene representations for view synthesis focuses on single objects and small scenes and does not scale to full buildings in short timeframes. We propose introducing the floorplan and learning a neural floorplan radiance field, mapping floorplan 3D points and view directions to emitted radiance, and rendering via a sinusoidal multi-layer perceptron (MLP) neural renderer. To incorporate local priors and further accelerate the overall learning, we use a hypernetwork which maps a floorplan surface normal to the parameters of the neural renderer, thus defining the scene by a space of local neural rendering functions across the building. This allows shared knowledge, reasoned in function space, of performing the neural rendering from various vantage points in the scene based on similar building structure represented in the floorplan surface normal, and facilitates meta-knowledge pre-training across multiple buildings. The meta-knowledge is used to initialize the parameters of the hypernetwork at test time for the target building. Our approach performs significantly accelerated learning of neural floorplan radiance fields in around 15 min for full buildings on a single commodity GPU, and renders in real-time at 64 Hz, allowing for immersive visual experiences.