A Qualitative Investigation of Panelists' Experiences of Standard Setting Using Two Variations of the Bookmark Method

Abstract

Only a small number of qualitative studies have investigated panelists' experiences during standard-setting activities or the thought processes associated with panelists' actions. This qualitative study involved an examination of the experiences of 11 panelists who participated in a prior, one-day standard-setting meeting in which either the traditional bookmark method or the single-passage bookmark method was used. Data were collected using in-depth focus group interviews. A whole text analysis resulted in three major categories. Many participants found the fundamental task of choosing a specific item as a bookmark to be inherently difficult. Some also deviated from the prescribed bookmark procedure and relied on alternative strategies that were highly resistant to change. Participants' views of item difficulty and correct item order also directly impacted the bookmarking process. Implications of the findings for standard-setting procedures and for further research are also addressed.