Developing indicators of classroom practice to evaluate the impact of district mathematics reform initiative: A generalizability analysis

Abstract

This paper reported results from a generalizability study that examined the process of developing classroom practice indicators used to evaluate the impact of a school district's mathematics reform initiative. The study utilized classroom observational data from 32 second, fourth, eighth, and tenth grade teachers. The study addresses important measurement issues in developing classroom practice indicators used as evaluation outcome measures. This study also demonstrates that the G-theory measurement framework has important applications in practice-oriented settings. Namely, the use of G-theory provides insights into the tradeoffs between number of observations and number of raters when using observational data to conduct evaluation. The use of decision analysis provided a procedure for forecasting the optimum mix of the two for each situation.