Education Research: Call for Controls

Abstract

L. Deslauriers and colleagues (“Improved learning in a large-enrollment physics class,” Reports, 13 May, p. [862][1]) report a significant advantage for a nontraditional method of teaching physics compared with a traditional method. However, the design of their study appears to contain a number of scientific problems. A randomized controlled trial is the experimental design best suited to evaluating the effectiveness of an intervention ([ 1 ][2]). Randomization creates groups that are equivalent in all known and unknown variables. Because Deslauriers et al. did not use randomization, their groups may have differed in an unmeasured variable that could have been responsible for the observed differences. Furthermore, attrition differed between the groups in Deslauriers et al. 's study. In the experimental group, 23% more students attended the outcome test than in the control group. This discrepancy introduces a potential source of selection bias that could have led the authors to under- or overestimate the treatment effects. In addition, Deslauriers and colleagues did not clarify whether those ascertaining the outcome were blind to group allocation. Deslauriers et al. report a huge effect size from a brief intervention, but they provide no alternative explanations for the results (for example, selection bias, attrition bias, outcome ascertainment bias, or teacher effect). Further confounding the results, the students in the control group were exposed to a single teacher and those in the experimental group to two teachers. The authors argue that their study can be generalized, but the data provided are insufficient. This study is a group or cluster trial with one group per treatment arm, which is equivalent to running a placebo-controlled drug trial with one patient in each group. Given this design, interpretation is impossible. Investigating the effectiveness of the intervention described in the Report would require a large, rigorously designed and conducted randomized controlled trial, with randomization of individual students to a number of different teachers. 1. [↵][3]1. D. J. Torgerson, 2. C. J. Torgerson , Designing Randomised Controlled Trials in Health, Education, and the Social Sciences (Palgrave Macmillan, New York, 2008). [1]: /lookup/doi/10.1126/science.1201783 [2]: #ref-1 [3]: #xref-ref-1-1 View reference 1 in text