Active Reading Comprehension: A Dataset for Learning the Question-Answer Relationship Strategy

Diana Galván-Sosa

doi:10.18653/v1/p19-2014

Abstract

Reading comprehension (RC) through question answering is a useful method for evaluating if a reader understands a text. Standard accuracy metrics are used for evaluation, where high accuracy is taken as indicative of a good understanding. However, literature in quality learning suggests that task performance should also be evaluated on the undergone process to answer. The Question-Answer Relationship (QAR) is one of the strategies for evaluating a reader’s understanding based on their ability to select different sources of information depending on the question type. We propose the creation of a dataset to learn the QAR strategy with weak supervision. We expect to complement current work on reading comprehension by introducing a new setup for evaluation.

Highlights

Computer system researchers have long been trying to imitate human cognitive skills like memory (Hochreiter and Schmidhuber, 1997; Chung et al, 2014) and attention (Vaswani et al, 2017). These skills are essential for a number of Natural Language Processing (NLP) tasks including reading comprehension (RC)
Question-Answer Relationship (QAR) states that an answer and its source of information are directly related to the type of question being asked
We propose to model QAR learning as a multiclass classification task with weak supervision

Summary

Introduction

Computer system researchers have long been trying to imitate human cognitive skills like memory (Hochreiter and Schmidhuber, 1997; Chung et al, 2014) and attention (Vaswani et al, 2017). When a reading comprehension system is not able to identify the correct answer, product-based evaluation can result in the false impression of weak understanding (i.e., misunderstanding of the text, the question, or both) or the absence of required knowledge. For the question “What were the consequences of Elizabeth Choy's parents and grandparents being ‘more advanced for their times’?” the correct answer is in the text but it is located in different sentences. We propose to adopt the thesis that reading is not a passive process by which readers soak up words and information from the text, but an active process by which they predict, sample, and confirm or correct their hypotheses about the text (Weaver, 1988) One of these hypotheses is which source of information the question requires. We discuss our proposed approach to create a new dataset for learning the QAR strategy using existing reading comprehension datasets

Related work

Question-answer relationship

QAR use cases

Dataset

In the text questions

In my head questions

Summary