An exploratory criterion validation of three meaning-recall vocabulary test item formats

Abstract

In an upcoming coverage-comprehension study, we plan to assess learners’ meaning-recall knowledge of words as they occur in the study’s reading passage. As several meaning-recall test formats exist, the purpose of this small-scale study (N = 10) was to determine which of three formats was most similar to a criterion interview regarding mean score and the consistency of correct/incorrect classifications (match rate, k = 30). In Test 1, the prompt consisted of only the target item, and a written translation of its meaning was elicited. In Test 2, the prompt was a short sentence containing the highlighted target item, and again a written translation of only the target item was requested. In Test 3, the prompt was the same sentence as in Test 2, but the target item was unhighlighted, and participants translated the entire sentence. In the criterion interview, knowledge of the target items in the same prompt sentences as in Tests 2-3 was ascertained. The results indicated that the Test 3 format produced a mean score and match rate most similar to the interview. Test 2 also performed well. The paper discusses several factors explaining differences in test performance that were explored during the interview.