A sequence to sequence model for dialogue generation with gated mixture of topics

Abstract

In this paper, we propose GMoT-Seq2Seq, a sequence to sequence (Seq2Seq) model with a gated mixture of topics (MoT) designed to utilize topic information to generate fluent and coherent responses. Seq2Seq model is good at capturing the local structure of word sequence which affects the fluency due to their sequential nature, but probably has difficulty to extract topic information from the utterance. In contrast, topic models are very capable of capturing global semantic information that has a direct impact on the coherence. Absorbing the advantages of both, the proposed GMoT-Seq2Seq model uses a Seq2Seq to capture the temporal dependencies, and an MoT layer to obtain the topic vector that provides global semantic dependencies in the conversation. The MoT layer can summarize the utterances into a proportion vector over several underlying topics. To balance the fluency and coherence, we utilize a topic gate to dynamically control the information from the inferred topic vector and the partially generated responses. Experiment results show that our proposed model outperforms the compared baselines, and can generate more fluent and coherent responses.