THE EFFICIENCY IMPLICATIONS OF USING PROPORTIONAL EVALUATIONS TO SHAPE THE TEACHING WORKFORCE

Abstract

I. INTRODUCTION State and local education agencies across the United States are working to improve teacher quality through the adoption of rigorous teacher evaluation systems. (1) The teacher-performance signals that come out of these systems can be acted on in a number of ways to improve outcomes for students in K-12 schools. However, despite the rapid growth in the development of evaluation systems nationwide, there is still much controversy surrounding the specifics of how to measure teacher performance. The lack of consensus in this area is reflected in the variety of different approaches that state and local education agencies use to evaluate teachers. This article contributes to the literature by examining the efficiency effects of using different evaluation metrics to rank-order teachers with the objective of using the rankings to help shape the teaching workforce. We define efficiency in terms of student achievement--the most efficient policy is the one that results in the highest achievement in total. We compare evaluation systems that rank-order teachers based on (1) proportional estimates of teacher quality, which force comparisons to be between equally circumstanced teachers and (2) global estimates of quality that compare teachers to each other regardless of teaching circumstance. (2) The context for the comparison is a removal policy targeted at the bottom 10% of teachers. Our analysis is performed using simulated data, which we construct following the literature on test-based of teacher performance because the properties of test-based are well understood, at least relative to available alternatives (e.g., classroom observations, student evaluations). (3) However, the substance of our findings will apply to any measure of teacher performance, including those commonly used in the combined measures that are being developed by a number of state and local education agencies (Mihaly et al. 2013). To illustrate how proportional and global teacher rankings can differ consider the following example: suppose that there are two types of schools, type-A and type-B, and that teacher quality is higher in type-A schools. (4) A quality-based removal policy that depends on global rankings will identify more teachers from type-B schools to be removed. In contrast, an analogous policy based on proportional rankings, which force equally circumstanced comparisons, will ensure that an equal number of teachers from type-A and type-B schools are removed. It is straightforward to show that the proportional policy is more efficient when there is a gap in average quality between teachers who teach in different schooling contexts, which recent research suggests is likely (e.g., see Arcaira et al. 2013; Goldhaber, Walch, and Gabele 2013; Isenberget al. 2013; Sass et al. 2012). The key insight underlying the efficiency gain is that the effect of a targeted removal policy depends not only on the quality of the teachers being removed, but also on the quality of replacements. Continuing with the example from above, note that under plausible conditions the gap in quality between teachers in type-A and type-B schools will persist for replacement teachers at these schools as well. After taking direct account of the link between observed teacher quality and replacement-teacher quality for schools in different contexts, we show that the proportional policy is the most efficient in terms of raising total student achievement. Although there is a strong efficiency rationale for proportionality, we find that in real-world applications the efficiency gain from imposing proportionality will be small. One reason is that teachers are evaluated in practice using imprecise measures, which attenuates the efficiency gain. Another reason is that the efficiency gain can be offset by gaps in the variance of teacher quality across different types of schools. However, it is important to recognize that as long as the proportional policy does not meaningfully underperform the global policy in our study, it may be preferable for several reasons. …