Multi-camera system for depth based visual effects and compositing

Abstract

Post-production technologies like visual effects (VFX) or digital grading are essential for visual storytelling in today's movie production. Tremendous efforts are made to seamlessly integrate computer generated (CG) elements into live action plates, to refine looks and to provide various delivery formats such as Stereo 3D (S3D). Thus, additional tools to assist and improve element integration, grading and S3D techniques could ease complicated, time consuming, manual and costly processes. Geometric data like depth information is a key for these tasks but a typical main unit camera shoot does not deliver this information. Although e.g. light detection and ranging (LIDAR) scans are used to capture 3D set information, frame by frame geometric information for dynamic scenes including moving camera, actors and props are not being recorded. Stereo camera systems deliver additional data for depth map generation but accuracy is limited. This work suggests a method of capturing light field data within a regular live action shoot. We compute geometric data for post-production use cases such as relighting, depth-based compositing, 3D integration, virtual camera, digital focus as well as virtual backlots. Thus, the 2D high quality life action plate shot by the Director of Photography (DoP) comes along with image data from the light field array that can be further processed and utilized. As a proof-of-concept we produced a fictitious commercial using a main camera and a multi-camera array fitted into an S3D mirror rig. For this reference movie, the image of the center array cam has been selected for further image manipulation. The quality of the depth information is evaluated on simulated data and live action footage. Our experiments show that quality of depth maps depends on array geometry and scene setup.