GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation

Abstract

Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in real-world scenarios, such resources are rarely available for new domains, and the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we present grt r, a hybrid generative-retrieval model based on the large-scale general-purpose language model GPT[2] fine-tuned to the multi-domain m eta lwo z dataset. In addition to robust and diverse response generation provided by the GPT[2], our model is able to estimate generation confidence, and is equipped with retrieval logic as a fallback for the cases when the estimate is low. grt r is the winning entry at the fast domain adaptation task of DSTC-8 in human evaluation ( $>$ 4% improvement over the 2nd place system). It also attains superior performance to a series of baselines on automated metrics on m eta lwo z and m ulti woz , a multi-domain dataset of goal-oriented dialogues. In this paper, we also conduct a study of grt r's performance in the setup of limited adaptation data, evaluating the model's overall response prediction performance on m eta lwo z and goal-oriented performance on m ulti woz .