Topic-word-constrained sentence generation with variational autoencoder

Abstract

• A model that can generate sentences under topic and word constraints was proposed. • It uses multiple Gaussian distributions as the prior for the latent code in the VAE. • It has an expanded semantically meaningful and smooth latent space for sentences. • It gets rid of the KL collapse problem in the VAE. • It uses the topic and sentence latent codes to guide generation together. We propose a topic-word-constrained sentence-generation model with a variational autoencoder and convolutional neural network . It can generate sentences conditioned on a given topic distribution and a certain word. Unlike the vanilla variational autoencoder that assumes a standard Gaussian prior for the latent code, our model specifies the prior for the topic latent code as multiple Gaussian distributions, where each Gaussian distribution corresponds to a topic vector parameterized by a convolutional neural topic model. For word constraints, the decoder in the variational autoencoder generates sentences backward and forward starting from a given word. The topic latent space is arranged by the similarity of topic vectors, and the topic latent code restricts the sentence latent code through a loss term, through which expanded semantically meaningful latent spaces can be learned and provide topic guidance while generating sentences. Experimental results show that our model can generate coherent and diverse sentences related to given topics and words, while also avoiding the Kullback–Leibler divergence collapse problem. Moreover, it outperforms alternative approaches in terms of sentence reconstruction, latent space property and the quality, diversity, and topic controllability of generated sentences.