Do systems pass university entrance exams?

Abstract

Reading Comprehension tests are commonly used to assess the degree to which people comprehend what they read. This is why we work with the hypothesis that it is reasonable to use these tests to assess the degree to which a machine “comprehends” what it is reading. In this work, we evaluate Question Answering systems using Reading Comprehension tests from exams to enter University. This article analyses the datasets generated, the kind of inferences required, the methodology followed in three evaluation campaigns, the approaches presented by participants and current results. Besides, we study the evolution of systems and the main lessons learned in this evaluation process. We also show how current technologies are unable to pass university-entrance exams. This is because these tests require a deep understanding of texts, as well as detecting the similar meaning of phrases with different words. Future directions focused on these ideas seem more promising than including a massive amount of data for training systems, what has allowed systems to obtain outstanding results in Reading Comprehension tests with more straightforward questions. We think this study helps to increase the knowledge about how to develop better Question Answering systems.