Commentary: The response process validity of a script concordance test item

Abstract

Listen

Translate article

While the Script Concordance Test (SCT) item format has generated considerable researchthat has reflected positively on its reliability (Charlin et al. 1998) and ease of construction(Fournier et al. 2008), a recent review suggests the response process validity of the SCT isweakly supported (Lubarsky et al. 2011). The Gagnon et al. (2011) paper published in thisissue of Advances in Health Sciences Education suggests that the validity of an SCT scoreas a measure of clinical reasoning hinges on the degree to which aggregate scoring conveysa concordance between experts’ and examinees’ scripts. However, the assumption that anexaminee’s solution process reflects their clinical scripts has not been confirmed. Since thevalidity of an SCT score derives from the congruence between current score interpretationsand a logical analysis of the task presented by the item, a sound validity argument mustconsider what the SCT task is actually asking the examinee to do. While it is obviouslyimportant to consider the underlying cognitive processes used to respond to the item, aneven more fundamental prerequisite is to logically decompose the SCT task into itscomponent parts. By breaking down the SCT problem into its logical steps, it will be easierto understand the cognitive demands of an SCT item.Figure 1 presents a flow diagram of the four task stimuli (S1–S4) and the three cal-culation steps (P1, P2, {P2–P1}) required by an SCT-based question. Since Gagnon et al.(2011) already present an excellent example and description of the SCT item format, hereit will be sufficient to simply note that an SCT first presents a clinical scenario (S1) to theexaminee and then suggests a hypothesis (S2). Next, the item presents a single discrete newpiece of information (S3), and then asks how that new clinical information impacts theprobability of the suggested hypothesis (S4). As Fig. 1 indicates, after both S2 and S3 theexaminee is implicitly asked to make a probabilistic calculation (P1 & P2 respectively) inorder to answer the formal question (S4). The first probability (P1) represents the likeli-hood of the hypothesis (S2) given the clinical scenario (S1). The second probability (P2) isthe likelihood of the hypothesis (S2) given both the scenario (S1) and the new piece of