Script concordance test item response process: The argument for probability versus typicality

Abstract

Kreiter’s (2011) incisive commentary cuts to the quick of an unresolved issue pertaining to script concordance testing: what are examinees thinking as they work through the test items? One possibility, advanced by Kreiter, is that SCT examinees rely on Bayesian reasoning, whereby they engage in a series of probability assignations and computations. This account has examinees estimating the probability of the hypothesis provided in the first column (P1, per Kreiter), followed by the probability of the same hypothesis given the new piece of clinical information provided in the second column (P2). They are then presumed to calculate the difference between P2 and P1, and to translate the result into an appropriate response (from -2 to ?2) on the Likert scale provided in column 3. This depiction of the thought and response processes of SCT examinees is unlikely to be accurate. Even experienced practitioners are notoriously inept at framing clinical problems in Bayesian terms. In a classic study, Eddy (1982) found that the vast majority of physicians made errors solving a problem in which probabilistic reasoning was required to determine a screened patient’s risk of developing breast cancer. If SCT demanded pure Bayesian analysis, one would not expect for SCT performance to improve with experience and expertise, as has been consistently observed in participants tested across various medical disciplines. Furthermore, carefully-constructed SCTs obviate the need for the type of probabilistic reasoning postulated by Kreiter (2011). Test-makers are deliberately instructed to avoid proposing hypotheses in column 1 that are not credible or relevant to a given vignette, i.e. ones with low a priori probability (Fournier et al. 2008). (If a hypothesis in the ‘‘If you were thinking...’’ column generates a reaction akin to ‘‘What? I would never have considered that!’’ in reasonable examinees, then that item probably should be considered unfair and discarded). Hypotheses with overly high pre-test probabilities are also meant to be eschewed, since they do not infuse SCT items with the requisite condition of