Evaluating Panelists’ Understanding of Standard Setting Data

Abstract

AbstractSince the No Child Left Behind Act (No Child Left Behind [NCLB], 2001) was enacted, the Bookmark method has been used in many state standard setting studies (Karantonis and Sireci; Zieky, Perie, and Livingston). The purpose of the current study is to evaluate the criticism that when panelists are presented with data during the Bookmark standard setting process, these data are often misunderstood. We collected survey responses from eight panels of teachers who worked on an alternate assessment standard setting workshop. We found that although many panelists understood these data, others misunderstood them. For example, when panelists reviewed panel judgment statistics, some extrapolated beyond what these data represent. Our results include themes describing the types of misconceptions we observed, and the need for training and evaluation related to understanding and use of data used in standard setting. We share some suggestions for consideration when implementing the Bookmark method.