CubeGAN: Omnidirectional Image Synthesis Using Generative Adversarial Networks

Abstract

AbstractWe propose a framework to create projectively‐correct and seam‐free cube‐map images using generative adversarial learning. Deep generation of cube‐maps that contain the correct projection of the environment onto its faces is not straightforward as has been recognized in prior work. Our approach extends an existing framework, StyleGAN3, to produce cube‐maps instead of planar images. In addition to reshaping the output, we include a cube‐specific volumetric initialization component, a projective resampling component, and a modification of augmentation operations to the spherical domain. Our results demonstrate the network's generation capabilities trained on imagery from various 3D environments. Additionally, we show the power and quality of our GAN design in an inversion task, combined with navigation capabilities, to perform novel view synthesis.