Information-Enhanced Hierarchical Self-Attention Network for Multiturn Dialog Generation

Abstract

Transformer structure has shown promising results in multiturn dialog generation. The self-attention mechanism can learn global dependencies but ignores local information, limiting the model's ability to model context information. In this article, we propose an information-enhanced hierarchical self-attention network (IEHSA). In the word-level encoder, words in successive windows are automatically encoded as local information, and words with dependent words are automatically encoded as syntactic information, both of which are used to enhance word information and then feed into the self-attention mechanism. In the utterance-level encoder, adjacent utterance representations are automatically encoded as dialog structure information, and the self-attention mechanism is used to update the utterance representations. The context and masked response representation are then updated using the self-attention mechanism in the decoder. Finally, the correlation between context and reply is calculated and used in further decoding. We compared IEHSA with the current popular hierarchical model on several datasets, and the experiments show that the proposed method has substantial improvements in both metric-based and human evaluations.