Cross-View Panorama Image Synthesis

Abstract

In this paper, we tackle the problem of ground-view panorama image generation conditioning on top-view aerial image, which is a challenging problem due to large gap between image domains associated with different view-points. Instead of learning the underlying cross-view mapping by a feedforward network as previous methods, we propose a novel adversarial feedback GAN framework named PanoGAN consisting of two key components: an adversarial feedback module and a dual branch discrimination strategy. First, aerial image is fed into the generator to produce panorama image and segmentation map, which facilitates the use of semantic layout information for model training. Second, the discriminator's feature responses of model outputs are encoded by the adversarial feedback module and then fed back to the generator for the next round of generation. Continual improvement of generated image quality is achieved through an iterative generation process. Third, to pursue high-fidelity and semantic consistency of the generated panorama image, we propose a pixel-segmentation alignment mechanism under the dual branch discrimiantion strategy that promotes the cooperation between generator and discriminator. Extensive experimental results on two challenging cross-view image datasets show that the proposed PanoGAN generates high-quality panorama images with more convincing details than state-of-the-art methods. The source code and trained models are available at https://github.com/sswuai/PanoGAN.