Is the treatment weak or the test insensitive: Interrogating item difficulties to elucidate the nature of reading intervention effects

Abstract

Intervention research in education is sometimes criticized for the use of experimenter developed assessments, especially when these are over aligned with treatment. At the same time, intervention researchers sometimes prefer locally developed assessments because they appear to be more sensitive to treatment effects even when the test is not subject to the criticism of over alignment. This paper examines the question of test sensitivity to treatment effects for experimenter developed and standardized tests for the specific case of reading in grade 8. We examine similarities and differences between a specific experimenter developed test and widely used standardized reading assessment. Analyses show these particular tests to be quite comparable. The paper concludes with an examination of test sensitivity by simulating treatment effects of different magnitudes. These analyses highlight some potential limitations of the standardized test for detecting small to moderate effects depending on the ability range of the students participating in intervention. The implications for intervention research and identification of students under response to intervention are discussed.