Near-field illumination for mixed reality with delta radiance fields

Abstract

Apart from geometric registration, fusing synthetic objects with a real context requires believable interaction of real and virtual light. Where shadows provide the visual cues to locate a synthetic object in a real scene and transferred light from real light sources let it appear in harmony with its surroundings, the mutual indirect interaction of illumination is a necessary detail to convince an observer that the rendered result is not merely augmented but part of the scene. Such a mixed reality (MR) system has applications in movie production, advertisement of unfinished products or cultural heritage visualization. While there are a range of relighting tools available for offline renderers or static scenes (e.g. photos), interactive MR systems usually disregard proper lighting entirely or greatly simplify shading. A method often employed is to merge virtual light and shadows cast from synthetic objects with a real background with Differential Rendering, leaving out any other light interaction such as indirect light bounces. Attempts have been made to resolve this issue with Instant Radiosity [Knecht et al. 2010; Lensing and Broll 2012]. To suppress flickering, a large number of virtual point lights (VPL) is necessary, drastically taxing execution speed. We propose to model all light, virtual and real, as a unified radiance field instead to avoid performance issues from oversampling and to maintain temporal coherence.