Controllable Scene Generation from Natural Language

Abstract

We propose a novel framework to generate recognizable scenes conditioned on natural language (NL) descriptions. The proposed modular approach decomposes the scene synthesis process into several manageable steps, in which it first infers a spatial layout of the desired scene from input descriptions by a spatial layout generator and generates the scene with a scene generator. Specifically, the proposed approach allows interactive tuning of the synthesized scene via NL, which helps to generate more complex and meaningful scenes, and to correct training errors or bias. We demonstrate the capability of the proposed approach on the challenging MS-COCO dataset and show that our approach can improve the quality of generated scenes, interpretability of the drawn scenes and semantic alignment to the input language descriptions.