Abstract
Researchers compare aggregate test data across treatment and control group in randomized controlled trials to determine intervention impact on student achievement. We show that item-level data and analyses can provide information about item-level treatment effect heterogeneity; such findings may be useful for improving interventions and making decisions about outcomes in subsequent studies. We apply Differential Item Functioning techniques to examine variation in the degree to which items show treatment effects. Based on our analysis of 7,244,566 item responses (265,732 students responding to 2,119 items) from 15 RCTs, we find evidence for variation in gains across items. These analyses identify items that are highly sensitive to the interventions—in one case, a single item drives nearly 40% of the observed treatment effect. Our analyses also identifies items that are insensitive and cases wherein treatment effects are consistent across items. We also show that the variation of item-level treatment sensitivity has implications for effect estimate precision. Of the treatments with significant effect estimates, 31% have patterns of item-level treatment sensitivity that may lead to a null effect when considering this source of uncertainty.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call