Comparing Visual and Statistical Analysis in Single-Subject Studies

Abstract

Objective. There has been an ongoing scientific debate regarding the most reliable and valid method of single-subject data evaluation in the applied behavior analysis area among the advocates of the visual analysis and proponents of the interrupted time-series analysis (ITSA). To address this debate, a head-to-head comparison of both methods was performed, as well as an overview of serial dependency, effect sizes and sample sizes. Method. The comparison of both methods was conducted in two independent studies. In the first study, conclusions drawn from visual analysis of the graphs published in the Journal of Applied Behavior Analysis (2010) were compared with the findings based on the ITSA of the same data; in the second study, conclusions drawn from visual analysis of the graphs obtained from the textbook by Alan E. Kazdin (2011) were used. These comparisons were made possible by the development of software, called UnGraph® which permits the recovery of the raw data from the graphs, allowing the application of ITSA. Results. In both studies, ITSA was successfully applied to over 90% of the examined time-series data with numbers of observations ranging from 8 to 136. Over 60% of the data had moderate to high level first order autocorrelations (> .40). A large effects size (≥ .80) was found for over 70% of eligible studies. Comparison of the conclusions drawn from visual analysis and ITSA revealed an overall low level of agreement (Kappa = .14) in the first study and moderate level of agreement (Kappa = .44) in the second study. Conclusions. These findings show that ITSA can be broadly implemented in applied behavior analysis research and can facilitate evaluation of the intervention effect, particularly when specific characteristics of single-subject data limit the reliability and validity of visual analysis. Comparison of the two methods revealed low to moderate agreement between visual analysis and ITSA. Overall, the two methods should be viewed as complimentary and used concurrently.