Standardized Diagnostic Assessment Design and Analysis: Key Ideas from Modern Measurement Theory

Abstract

As a response to the ever-increasing demand for diagnostic assessments that can provide more informative feedback about students’ knowledge state, assessment design frameworks are needed that can help designers incorporate relevant cognitive theories into the development, implementation, and analysis process. In this chapter, we describe one prominent framework for principled diagnostic assessment design called evidence-centered design (ECD) (e.g., Mislevy et al. A brief introduction to evidence-centered design. CSE Technical Report 632. Los Angeles: The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, 2004) as well as a class of statistical models called diagnostic classification models (DCMs) (e.g., Rupp et al. Diagnostic assessment methods: theory and application. The Guilford Press, New York, 2010) that can make inferences about student profiles within this framework. With respect to DCMs we describe key terminology, concepts, and a unified estimation framework known as the log-linear cognitive diagnosis model (LCDM) (Henson et al. Psychometrika 74(2):191–210, 2009). We present three examples to illustrate how particular DCMs can be specified to address different cognitive theories concerning the process of knowledge processing. At the end of this chapter, we illustrate the utility of DCMs with a real-data set on arithmetic ability in elementary school to illustrate the type of diagnostic inferences we can make about students’ attribute profiles.