Image generation from bounding box-represented semantic labels

Abstract

Image generation with pixel-wise semantic information is suitable for the development of adversarial learning techniques. In this study, we propose a method for synthesizing objects with class-specific textures and fine-scale details based on bounding box-represented semantic labels. To achieve this goal, we note that the traditional generative adversarial network (GAN) uses noise as an input to generate realistic images with sufficient textures and details, but it cannot be guided by specific targets and requirements. By contrast, conditional GAN (cGAN) can involve various types of guiding information but it often ignores specific textures and details, thereby leading to less realistic results and low resolution. Thus, we propose a new translator-enhancer framework by combining cGAN and GAN to achieve high quality image generation. cGAN is used as a translator to match the semantic constraints whereas GAN is employed as an enhancer to provide details and textures. We also propose a new form of semantic label map as an input, which is represented by instance-level bounding boxes rather than segmentation masks. The semantic label map represented by bounding boxes makes it easier for users to provide the inputs and it also gives greater flexibility when generating object boundaries. The results obtained from qualitative and quantitative experiments showed that our method can generate realistic images of objects with semantic labels represented by bounding boxes. Our method can be used to generate images of novel scenes to support learning tasks during training with various scenes, which are difficult to capture in the real world.