Critical Factors to Consider in Evaluating Standard-Setting Studies to Map Language Test Scores to Frameworks of Language Proficiency

Abstract

In this article, we consolidate and present in one place what is known about quality indicators for setting standards so that stakeholders may be able to recognize the signs of standard-setting quality. We use the context of setting standards to associate English language test scores with language proficiency descriptions such as those presented in the Common European Framework of Reference (Council of Europe, 2001). The criteria we discuss, however, apply to any language test and language framework. Standard-setting practices are often used to associate test scores with descriptions of language proficiency, as that is one way to provide meaning to the scores. However, there are a large number of standard-setting methods and procedures, and although there are suggested guidelines on how to implement such studies, not all studies necessarily follow the guidelines or follow them to the same extent. This makes it difficult to evaluate the quality of standard-setting results. Our intention is to offer guidance to policymakers, decision makers, and researchers about important factors they may consider to evaluate the rigor and credibility of standard-setting studies conducted to associate language test scores with language frameworks.