Boolean Analysis of Interobserver Agreement: Formal and Functional Evidence Sampling in Complex Coding Endeavors

Abstract

AbstractThis article proposes a Boolean approach to representing and analyzing interobserver agreement in dichotomous coding. Building on the notion that observations are samples of a universe of observations, it submits that coding can be viewed as a process in which observers sample pieces of evidence on constructs. It distinguishes between formal and functional interobserver agreement based on whether events coded by observers are, respectively, the same or different but equivalent samples of the same classes of events. Formal and functional agreement are defined as Boolean functions. The article reports on an investigation in which a Boolean approach was used to examine agreement in the coding of teacher–student social interactions. Agreement maps (devices that represent the relative frequencies of coding cases—different combinations of 1s and 0s in the observers’ coding) allowed detail examination of how interobserver agreement was shaped by the nature of the events coded and the characteristics of the individuals whose behaviors were coded. Results provide empirical evidence of formal and functional agreement as distinguishable forms of interobserver agreement. Results also show that Boolean coding analysis allows detection of ways in which coding procedures confound absence of evidence with evidence of absence. Practical and methodological implications are discussed.