Using Content Relevance and Representativeness Indices in Instrument Revision

Abstract

ABSTRACT Combining methods from earlier content validity and more contemporary content alignment studies may allow a more complete evaluation of the meaning of test scores than if either set of methods is used on its own. This article distinguishes item relevance indices in the content validity literature from test representativeness indices in the subsequent content alignment literature. We then demonstrate that setting minimum thresholds for both types of indices in a sequential process can affect conclusions about test content representativeness, which are integral to validity evaluation. We illustrate this outcome using test item-to-target domain objective match data from subject-matter expert panelists who reviewed sixteen US state achievement tests. Although the illustration incorporates educational achievement test data, the process has broader applicability to development of survey questionnaires, workplace tests, and other instruments.