The Relationship between Intercoder Reliability of Data Extraction and Effect Measure Calculation in Single-Case Meta-Analysis

Abstract

Meta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from linear graphs in primary studies, prior to calculating and aggregating single-case effect measures. Existing research suggests data extraction yields reliable and valid data; however, we have an incomplete understanding of the downstream effects of relying on data extracted by two or more people. This study was undertaken to enhance that understanding in the context of SCEDs published in school psychology journals. Data for 91 unique outcomes across 67 cases in 20 SCEDs were extracted by two data extractors. Four different single-case effect measures were calculated using data extracted by each data extractor and then compared to determine the similarity of the effect measures. Overall, intercoder reliability metrics suggested a high degree of agreement, and there were minimal differences in single-case effect measures calculated from data extracted by different researchers. Intercoder reliability metrics and differences in single-case effect measures were generally negatively related, though the strength varied depending on the single-case effect measure. Hence, it is unlikely that the small differences in effect measure estimates due to the slight unreliability of the data extraction process would have a considerable impact on the interpretation of single-case effect measures. Impact Statement Two people extracted highly similar numerical data from the same linear graphs using plot digitizing software. Differences in calculations across data extracted by two people were trivial. Results suggest researchers can likely have confidence in the calculation of effect measures aggregated in meta-analyses of single-case experimental designs, provided they achieve comparable levels of agreement amongst data extractors.