(Effect) size matters: And so does the calculation.

Abstract

In 2001, the American Psychological Association (APA) noted in its publication manual that effect size calculations should be included in manuscripts submitted for publication. However, researchers utilizing single subject designs have not typically embraced the approach of any analyses beyond that of the traditional visual analysis (Marascuilo & Busk, 1988; Parsonson & Baer, 1977). In visual analysis of single subject data, researchers have examined data for three changes in the data: trend, variability, and level. Using trend analysis, researchers have examined the direction of the data for an increasing (i.e., upward) or decreasing (i.e., downward) trend. Researchers have also inspected for change in data variability or bounce. Finally, researchers have noted changes in level or mean performance. Recent trends in the field of education have resulted in an increased need to synthesize data sets from single subject studies. For example, the No Child Left Behind Act (NCLBA; 2001) brought considerable attention to the term evidence-based practice. As Odom and colleagues described (2005), some have claimed that only randomized experimental group designs are appropriate for demonstrating scientific evidence. This precluded single subject studies from being included in contributions of scientific evidence on effective intervention methods. However, others have noted that rigorous single subject research has much to contribute when determining scientific knowledge within the field (Horner, et al., 2005). In order to support the use of single subject research as evidence-based, a process of synthesizing single subject data is needed. Additionally, the Individuals with Disabilities Education Act (2004) mandated that teachers use strategies based on evidence based research. It would be tragic for teachers to utilize only teaching strategies proven with group design research; hence a second need to summarize data from single subject studies. Finally, researchers conducting meta-analyses or research syntheses have needed a method for interpreting and comparing intervention effectiveness of single subject studies. Researchers and practitioners in the field have tried to synthesize intervention research and effect sizes have been calculated on single subject data (e.g., Ma, 2006; Parker, Hagan-Burke, & Vannest, 2007; Wanzek, et al., 2006).Therefore, the purpose of this paper is to present the types of measures that may be used to describe intervention effects of single subject research designs. Strengths and limitations of each method will be described. Finally, a recommendation will be made to assist in determining which method should be used with which types of single subject data. Regression Approaches Allison and Gorman described the use of regression models to calculate effect sizes with single subject data (Allison & Gorman, 1993; Faith, Allison, & Gorman, 1996). In doing so, the dependent measure in the study (e.g., reading fluency or out of seat behavior) served as the dependent measure in the analysis while the intervention sessions serve as the independent variable. A separate regression equation was then obtained for the baseline and intervention data resulting in two regression equations. Finally, the intervention was subtracted from the baseline and divided by the standard deviation of baseline (Hershberger, Wallace, Green, & Marquis, 1999). It should be noted that data portrayed in single subject graphs are not independent of one another. Often in single subject research, experimenters visually analyze intervention data following each intervention session. This visual analysis might result in modifications to intervention procedures during the subsequent session resulting in data that are dependent on preceding data. For example, if a child was being taught to exchange a graphic symbol for a preferred item, the dependent variable might be rate of independent exchanges. …