Embedded Standard Setting: Aligning Standard‐Setting Methodology with Contemporary Assessment Design Principles

Abstract

AbstractIn this paper we assert that the practice of principled assessment design renders traditional standard‐setting methodology redundant at best and contradictory at worst. We describe the rationale for, and methodological details of, Embedded Standard Setting (ESS; previously, Engineered Cut Scores. Lewis, 2016), an approach to establish performance standards that augments and is embedded within existing principled assessment design practices. Under ESS, achievement level descriptor writing and the targeting of items to specified achievement levels are viewed as the fundamental standard‐setting activities and cut scores are estimated by optimizing the relationship between the item targets and the empirically based grouping of items. If the relationship between item targets and empirical data are sufficiently strong, then the traditional standard‐setting workshop is either unnecessary or altered in nature and significantly reduced in scope. We provide empirical evidence supporting the efficacy of ESS and describe how ESS is embedded throughout the assessment lifecycle.