Generic Mechanism for Reducing Repetitions in Encoder-decoder Models

Abstract

Encoder-decoder models have been commonly used; they have achieved state-of-the-art results for many natural language generation tasks. However, according to the reports of previous studies, encoder-decoder models suffer from generating redundant repetitions. Thus, we herein propose a repetition reduction module (RRM) for encoder-decoder models that estimates the semantic difference of a source sentence before and after it is fed into the model to capture the consistency between the two sides. As an autoencoder, the proposed mechanism supervises the training of encoder-decoder models to reduce the number of repeatedly generated tokens. The evaluation results of the publicly available machine translation and response generation datasets demonstrate the effectiveness of our proposal.