Crowdsourcing evaluation of the quality of automatically generated questions for supporting computer-assisted language teaching

Maria Chinkina,Detmar Meurers,Simón Ruiz

doi:10.1017/s0958344019000193

Abstract

AbstractHow can state-of-the-art computational linguistic technology reduce the workload and increase the efficiency of language teachers? To address this question, we combine insights from research in second language acquisition and computational linguistics to automatically generate text-based questions to a given text. The questions are designed to draw the learner’s attention to target linguistic forms – phrasal verbs, in this particular case – by requiring them to use the forms or their paraphrases in the answer. Such questions help learners create form-meaning connections and are well suited for both practice and testing. We discuss the generation of a novel type of question combining a wh- question with a gapped sentence, and report the results of two crowdsourcing evaluation studies investigating how well automatically generated questions compare to those written by a language teacher. The first study compares our system output to gold standard human-written questions via crowdsourcing rating. An equivalence test shows that automatically generated questions are comparable to human-written ones. The second crowdsourcing study investigates two types of questions (wh- questions with and without a gapped sentence), their perceived quality, and the responses they elicit. Finally, we discuss the challenges and limitations of creating and evaluating question-generation systems for language learners.

Highlights

Questions are habitually used by teachers to test comprehension, encourage discussion, and check understanding of learning materials
Automatically generated questions are qualitatively comparable to those devised by a language teacher, and we argue that this technology can be integrated into future language instruction via computer-assisted language teaching applications
As the null and the alternative hypothesis are reversed in equivalence testing, statistically significant results indicate that the two samples are equivalent

Summary

Introduction

Questions are habitually used by teachers to test comprehension, encourage discussion, and check understanding of learning materials. We argue that in a language-learning classroom particular questions can facilitate the acquisition and practice of different linguistic forms by creating a functional need to notice and process a linguistic form (Robinson, Mackey, Gass & Schmidt, 2012). This idea is supported by a large body of research on input enhancement (Sharwood Smith, 1993; see Simard, 2018, for a recent overview) and processing instruction, research on structured input activities (see VanPatten, 2017, for a recent review). Crowdsourcing evaluation of the quality of automatically generated questions for supporting computer-assisted language teaching.

Objectives

Methods

Results

Discussion

Conclusion