Panoramic Image Generation: From 2-D Sketch to Spherical Image

Abstract

The 360-degree video/image, also called an omnidirectional video/image or panoramic video/image, is very important in some emerging areas such as virtual reality (VR). Therefore, corresponding image generation algorithms are urgently needed. However, existing image generation models mainly focus on 2-D images and do not consider the spherical structures of panoramic images. In this article, we propose a panoramic image generation method based on spherical convolution and generative adversarial networks, called spherical generative adversarial networks (SGANs). We adopt the sketch map as the input, which is a concise geometric structure representation of the panoramic image, e.g., comprising approximately 7% of the pixels for a 583 × 1163 image. Through adversarial learning, a realistic-looking, plausible and high-fidelity spherical image can be obtained from the sparse sketch map. In particular, we build a dataset of the sketch maps using a visual computation-based sketching model. Then, by optimizing SGANs with GAN loss, feature matching loss and perceptual loss, realistic textures and details are recovered gradually. On one hand, it is an improvement using the sparse sketch map as input rather than the denser input, e.g., the features of the textures and colors. On the other hand, spherical convolution helps to remedy space-varying distortions of the planar projection. We conduct extensive experiments on some public panoramic image datasets and compare them with state-of-the-art techniques to validate the superior performance of the proposed approach.