MDM: Meta diffusion model for hard-constrained text generation

Abstract

Hard-constrained text generation (Hard-CTG) task aims to generate text with given keywords, which is helpful for summarization, data augmentation, story writing, etc. Existing Hard-CTG models face two significant challenges. Firstly, hard-CTG models based on the editing method suffer from error propagation, resulting in low generation quality. Secondly, Hard-CTG models utilizing the prompt method cannot guarantee high keyword coverage. To tackle these challenges, we propose Meta Diffusion Model (MDM), a non-autoregressive diffusion model with novel meta-learning module. Specifically, we fix the embedding of keywords in the generation process, while all non-keyword tokens evolve simultaneously and contribute to each other towards the target sentence under given keywords, addressing the above issues. Moreover, existing diffusion models for the text domain have an inconsistency in the training and inference stages. To unify the training and inference processes, we present an adaptively denoising method derived from meta-learning, and further improves generation quality. Experiments on three representative datasets demonstrate that our method achieves state-of-the-art performance evaluated on empirical metrics. Especially, compared with strong baselines, MDM significantly improves the BLEU-4, CIDEr, and ROUGE by 0.48%—11.56%, 17.33%—82.87%, and 23.15%–29.78%, respectively. In terms of keyword coverage, our MDM outperforms ChatGPT by 2.93%–7.88%.