Interrater Agreement Measures: Comments on Kappan, Cohen's Kappa, Scott's π, and Aickin's α

Abstract

The Cohen (1960) kappa interrater agreement coefficient has been criticized for penalizing raters (e.g., diagnosticians) for their a priori agreement about the base rates of categories (e.g., base rates of disorders). A modification of kappa, called kappan (alias S coefficient, C coefficient, G index, and RE coefficient) has been proposed as an alternative to Cohen's kappa: Kappan was intended to reward rather than penalize classification agreements attributable to interrater agreement about base rates. In this article, we show that kappan has some serious limitations: It can be large when raters who randomly assign objects (e.g., patients) to categories (diagnoses) radically disagree about base rates, and it can be much larger when these raters have very different beliefs about base rates than when they are in complete agreement about base rates. Contrary to the views of recent critics of Cohen's kappa, we argue that Cohen's kappa (which does not have these serious limitations) is generally preferable to...