360 Panorama Synthesis from a Sparse Set of Images on a Low-Power Device

Abstract

A full 360 $^\circ$ × 180 $^\circ$ image provides an unlimited field of view (FOV) and an immersive experience for the users without any loss of information of the surrounding. In this study, a deep learning based approach is proposed to synthesize a 360 $^\circ$ image from a sparse set of images captured with a limited FOV. The proposed network consists of a cascade of the FOV estimation network and the panorama synthesis network. We propose a hierarchical generative network to synthesize high quality 360 $^\circ$ panorama images. The design of progressive multi-scale generator and multiple discriminator reduces the high frequency artifact which is commonly observed in image synthesis using generative networks. The network is further compressed to run on a low-power device such as a smartphone using our proposed size and latency optimization. Experimental result demonstrates that the proposed method produces 360 $^\circ$ panorama with satisfactory image quality of up to 512 × 1024 resolution. It is also shown that the proposed method outperforms the alternative method and can be generalized for non-panoramic scenes and real images captured by a smartphone camera.