Multiple HD Screen-Based Virtual Studio System with Learned Mask-Free Portrait Harmonization

Abstract

Virtual studio technology allows producers to combine live-action footage and computer-generated imagery and has promoted the development of film and television industry. However, the existing green screen-based virtual studio limits the visual effects just in postproduction. In this paper, we propose a new virtual studio system based on the recent emerging technology using multiple high-definition (HD) screens. Besides using traditional computer graphics like other similar systems, our system enables to capture panoramic video from the real world as the background and project it onto multiple HD screens, so as to recreate the scene in the studio. In addition, we propose a mask-free portrait harmonization network to make sure that the appearances of the foreground region and the background are consistent. Our portrait harmonization network does not need any auxiliary foreground mask and works in an end-to-end manner, which meets the postprocessing requirements for our virtual studio. Experimental result on real composite images shows that our network is superior to the state-of-the-art image harmonization method under the task of portrait harmonization. We also demonstrate our system with a CAVE system.