An Empirical Investigation of Interrater Agreement for Single-Subject Data Using Graphs with and without Trend Lines

Abstract

The interrater agreement of visual judgments made from single-subject data was examined. Seventy-nine raters were given 21 single-subject graphs. Thirty-nine of the raters examined graphs containing single-subject data arrayed in the traditional format. The remaining 40 subjects reviewed AB graphs that were supplemented by a trend line. As measured by intraclass correlation coefficients, interrater agreement was higher for the trend line group than for the group relying only on visual analysis. There was a statistically significant correlation between the change in slope across the phases of the AB design and a score reflecting disagreement among raters in the visual analysis group. This relationship between change in slope and rater disagreement was not present in the trend line group. The results suggest that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements to visually inspected charts.