Abstract
We present DREAM, the first dialogue-based multiple-choice reading comprehension data set. Collected from English as a Foreign Language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our data set contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension data sets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule-based approach. We next investigate the effects of incorporating dialogue structure and different kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehension models. Experimental results on the DREAM data set show the effectiveness of dialogue structure and general world knowledge. DREAM is available at https://dataset.org/dream/ .
Highlights
IntroductionChoice (Lai et al, 2017; Khashabi et al, 2018; Ostermann et al, 2018) and extractive (Hermann et al, 2015; Hill et al, 2016; Rajpurkar et al, 2016; Trischler et al, 2017) reading comprehension data sets (Section 2)
A significant amount of research has focused on the construction of large-scale multiple-choice (Lai et al, 2017; Khashabi et al, 2018; Ostermann et al, 2018) and extractive (Hermann et al, 2015; Hill et al, 2016; Rajpurkar et al, 2016; Trischler et al, 2017) reading comprehension data sets (Section 2)
Because the original finetuned transformer LM (FTLM) framework already leverages rich linguistic information from a large unlabeled corpus, which can be regarded as a type of tacit general world knowledge, we investigate whether additional dialogue structure can further improve this strong baseline
Summary
Choice (Lai et al, 2017; Khashabi et al, 2018; Ostermann et al, 2018) and extractive (Hermann et al, 2015; Hill et al, 2016; Rajpurkar et al, 2016; Trischler et al, 2017) reading comprehension data sets (Section 2). Source documents in these data sets have generally been drawn from formal written texts such as news, fiction, and Wikipedia articles, which are commonly considered wellwritten, accurate, and neutral. Answering 34% of DREAM questions requires unspoken commonsense knowledge, for example, unspoken scene information This might be due to the nature of dialogues: For efficient oral communication, people rarely state obvious explicit world knowledge (Forbes and Choi, 2017) such as ‘‘Christmas Day is celebrated on December 25th.’’Understanding
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Transactions of the Association for Computational Linguistics
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
