Adversarial learning-based camera pose-to-image mapping network for synthesizing new view in real indoor environments

Abstract

Given a new 6DoF camera pose in an indoor environment, we investigate the challenging problem of predicting the view from that pose based on a set of reference RGBD views. Existing explicit or implicit 3D geometry construction methods are computationally expensive, while those based on learning have primarily focused on isolated views of object categories with regular geometric structures. In contrast to the traditional render-inpaint approach for new view synthesis in real indoor environments, we propose a conditional generative adversarial neural network (P2I-NET) that directly predicts the new view from the given pose. P2I-NET learns the conditional distribution of environment images to establish the correspondence between the camera pose and its view, achieved through innovative designs in its architecture and training loss function. Two auxiliary discriminator constraints are incorporated to ensure consistency between the pose of the generated image and that of the corresponding real-world image in both the latent feature space and the real-world pose space. Moreover, a deep convolutional neural network (CNN) is introduced to further reinforce this consistency in the pixel space. Extensive new view synthesis experiments and pose estimation experiments have been conducted on real indoor datasets. The results demonstrate that P2I-NET outperforms a number of strong baseline models based on NeRF. In particular, it is shown that P2I-NET is 40 to 100 times faster than these competitor techniques while producing images of similar quality. The results also reveal that the discriminator in P2I-NET achieves comparable positioning performance to other positioning methods, despite having a smaller model size. Furthermore, we contribute a new publicly available indoor environment dataset containing 22 high resolution RGBD videos where each frame also has accurate camera pose parameters.