Towards controllable image descriptions with semi-supervised VAE

Abstract

Image captioning models successfully describe the visual contents of images using natural language. To generate more natural and diverse descriptions, a model must learn style-specific patterns and requires collecting style-specific datasets, which is time-consuming. To address this issue, we propose a semi-supervised deep generative model, Semi-supervised Conditional Variational Auto-Encoder (SCVAE). Our model is capable of leveraging more labelled and unlabelled data in the generative model schema. Extensive empirical results demonstrate that compared with the start-of-art models, our proposed method is able to generate more accurate image captions with more extensive styles.