Reliability of criteria‐based content analysis of child witness statements: Cohen's kappa doesn't matter

Abstract

The recent attempt by Horowitz, Lamb, Esplin, Boychuk, Orit & Reiter‐Lavery (1997) to establish reliabilities of evaluators applying criteria‐based content analysis (CBCA) leads to misleading conclusions about the reliability of CBCA as a protocol. The history and procedure of statement validity assessment (SVA) of which CBCA is a component, has not established CBCA as the equivalent of a psychometric type instrument or method with any particular level of instrinsic reliability. There are many reasons why certainty as to whether given criteria are agreed to fit a slice of narrative may be high or low. The adequacy of that particular slice of narrative for such criterion fitting is as important as the evaluators agreeing they mean the same thing and are trying to do the same thing on one or more occasions. The outcome reliabilities statistically computed are a composite of both. Professional evaluators do not need to rely on the results of forcing themselves to decide without adequate certainty whether any or all criteria are ‘present’ or ‘absent’. Their reliability when they make a decision with adequate certainty is what matters. Horowitz et al.'s recommendations that certain criteria should be further defined ‘more clearly’ or dropped is particularly erroneous. This would lead to the loss of important if infrequently found instances of presence or absence of criteria which are telling because of what is known about how children recall sexual victimization, and not because a statistical manual sets an empirical threshold.