Value-added Models Research Articles

Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates

Are teachers' impacts on students' test scores (value-added) a good measure of their quality? One reason this question has sparked debate is disagreement about whether value-added (VA) measures provide unbiased estimates of teachers' causal impacts on student achievement. We test for bias in VA using previously unobserved parent characteristics and a quasi-experimental design based on changes in teaching staff. Using school district and tax records for more than one million children, we find that VA models which control for a student's prior test scores provide unbiased forecasts of teachers' impacts on student achievement. (JEL H75, I21, J24, J45)

Teacher Evaluation and Collective Bargaining

This case study analyzes controversial teacher evaluation policies in the context of collective bargaining. Dr. Jill Abrams, a new superintendent in a struggling school district, is at the center of the case. Her school board demands a form of teacher evaluation she finds problematic because it includes value-added modeling. Moreover, the board insists on excluding the union in the process of developing the new evaluation system. This case explores the potential consequences of implementing specific teacher evaluation policies and competing demands of relevant stakeholders in the process. The case weaves rich policy issues that frequently present to school leaders at the local level. Other policy and ethical dilemmas emerge as the case develops.

The Predictive Validity of Measures of Teacher Candidate Programs and Performance

Calls for evidence-based reform of teacher preparation programs (TPPs) suggest the question: Do the current indicators of progress and performance used by TPPs predict effectiveness of their graduates when they become teachers? In this study, the indicators of progress and performance used by one program are examined for their ability to predict value-added scores of program graduates. The study finds that rating instruments, including disposition surveys, clinical practice observation ratings, and portfolio assessments, each measure a single underlying dimension rather than the multiple constructs they were designed to measure. Neither these instruments nor teacher candidates’ scores on standardized exams predict their later effectiveness in the classroom based on value-added models of student achievement. Candidates’ grade point averages during their preparation program and number of math courses were positively associated with their students’ math score gains. These findings suggest a need for better instruments to measure prospective teachers’ progress toward proficiency.

Value Added and Its Uses: Where You Stand Depends on Where You Sit

In this policy brief we argue that there is little debate about the statistical properties of value-added model (VAM) estimates of teacher performance, yet, despite this, there is little consensus about what the evidence about VAMs implies for their practical utility as part of high-stakes performance evaluation systems. A review of the evidence base that underlies the debate over VAM measures, followed by our subjective opinions about the value of using VAMs, illustrates how different policy conclusions can easily arise even given a high-level general agreement about an existing body of evidence. We conclude the brief by offering a few thoughts about the limits of our knowledge and what that means for those who do wish to integrate VAMs into their own teacher-evaluation strategy.

Can Small Class Policy Close the Gap? An Empirical Analysis of Class Size Effects in Japan

Can smaller classes lead to better educational outcomes and greater equality in achievement? We estimate the causal effects of class size on achievement tests by using discontinuous changes in class size under the Japanese public compulsory education system. We employ a value-added model that uses achievement tests conducted at two different times during the same school year. Our results show that a reduction in class size has significantly positive effects on Japanese language test scores in the sixth grade, especially at schools in wealthy areas. However, we find no evidence that a universal small class policy closes the achievement gap among schools.

Rethinking Teacher Evaluation: A Conversation about Statistical Inferences and Value-Added Models

In this article, the authors provide a methodological critique of the current standard of value-added modeling forwarded in educational policy contexts as a means of measuring teacher effectiveness. Conventional value-added estimates of teacher quality are attempts to determine to what degree a teacher would theoretically contribute, on average, to the test score gains of any student in the accountability population (i.e., district or state). Everson, Feinauer, and Sudweeks suggest an alternative statistical methodology, propensity score matching, which allows estimation of how well a teacher performs relative to teachers assigned comparable classes of students. This approach more closely fits the appropriate role of an accountability system: to estimate how well employees perform in the job to which they are actually assigned. It also has the benefit of requiring fewer statistical assumptions—assumptions that are frequently violated in value-added modeling. The authors conclude that this alternative method allows for more appropriate and policy-relevant inferences about the performance of teachers.

Patterns in Student Assignment to Elementary School Classrooms

In an effort to better understand aggregate patterns in the way elementary school students are assigned to classes, we conduct a careful analysis of observed classroom assignment outcomes in the 5th grade in North Carolina elementary schools. First, we model the probability that a pair of students are classmates as a function of the characteristics of that pair of students. This novel methodological technique enables us to directly observe the degree to which actual assignment patterns differ from what might be expected under random assignment for a wide variety of student characteristics. Second, we analyze patterns in classroom assignment and discuss the implications of these patterns. We show that classroom assignments tend to deviate from random assignment in a way that tends to group similar students and that these deviations tend to be greatly magnified in Magnet schools. Importantly, we find evidence that administrators sort students based on attributes not normally observable by researchers. These findings have important implications for researchers using value-added modeling (VAM) techniques. Finally, we find that classroom assignment patterns are generally stable across the racial, income, and geographic characteristics of schools.

Teacher Value-Added at the High-School Level

This article reports on findings based on analyses of a unique dataset collected by ACT that includes information on student achievement in a variety of subjects at the high-school level. The authors examine the relationship between teacher effect estimates derived from value-added model (VAM) specifications employing different student learning assumptions. Specifically, they compare teacher effectiveness estimates derived from a traditional lagged score VAM using pretests and posttests in a single subject area to those derived from VAM specifications employing a cross-subject student fixed-effects approach. The latter approach offers advantages for teacher evaluation systems due to significantly reduced data requirements; however, there is evidence that both the estimated effect size of teacher quality as well as estimates of individual teacher performance vary depending on the VAM methodology. In particular, teacher effects identified based on within-student cross-subject variation results in significantly smaller effect size estimates than do those generated from a more traditional lagged score model. The correlation across model specification ranges from .25 to .96 depending on the subject area.

Investigating the “black box” of effective teaching: the relationship between teachers’ perception and student achievement in a large urban district

Given the international need to improve student learning, there is nothing more important than classroom teachers. Obtaining a deeper understanding of effective classrooms is a priority if educational reform efforts are to succeed in any educational system around the world. In the last decade, educational researchers have expanded the knowledge base about the impact of teachers, but far less is known about what effective teachers do that make such a positive difference. This investigation focuses on the relationship between attributes of effective teaching, as perceived by both more and less effective teachers, and fourth grade reading achievement results of their students. This investigation used value-added modeling methods as the first step for identifying the more and less effective fourth grade teachers (N = 261) and their students (N = 6,962). Survey research methods were used to compare the perceptions about characteristics of effective teaching for teachers previously identified—in the value-added model—as (a) more effective and (b) less effective. Findings indicated that the more effective teachers value classroom management and organization as the number one characteristic of effective teaching; that, in turn, enables the more effective teachers to focus classroom time on student learning. Implications for theory and practice are discussed.

The use of a school value-added model for educational improvement: a case study from the Portuguese primary education system

This article focuses on the use of a value-added approach for promoting school improvement. It presents yearly value-added estimates, analyses their stability over time, and discusses the contribution of this methodological approach for promoting school improvement programmes in the Portuguese system of evaluation. The value-added model is applied to longitudinal data gathered from a representative sample of students enrolled in primary school, using mathematics scores as outcome variable. The estimates are used to determine a systematic positive or negative school value-added effect. Results indicate that at least 41% of schools remain at the same level over 3 years, 15% present an instable pattern, and approximately 40% show a positive or negative trend. It is argued that these results, especially those that reveal systematically underperforming schools, could have relevant implications for educational policy geared towards school improvement.

What Are Error Rates for Classifying Teacher and School Performance Using Value-Added Models?

This article addresses likely error rates for measuring teacher and school performance in the upper elementary grades using value-added models applied to student test score gain data. Using a realistic performance measurement system scheme based on hypothesis testing, the authors develop error rate formulas based on ordinary least squares and Empirical Bayes estimators. Empirical results suggest that value-added estimates are likely to be noisy using the amount of data that are typically used in practice. Type I and II error rates for comparing a teacher’s performance to the average are likely to be about 25% with 3 years of data and 35% with 1 year of data. Corresponding error rates for overall false positive and negative errors are 10% and 20%, respectively. Lower error rates can be achieved if schools are the performance unit. The results suggest that policymakers must carefully consider likely system error rates when using value-added estimates to make high-stakes decisions regarding educators.

From Perceptions to Practice: Examining How Administrators Make Use of Value-Added Data in Schools

Accountability mandates increasingly lead schools and districts to make use of student performance metrics. This study begins to unpack the factors behind uptake and use of value-added models by administrators in schools within a district and state that strongly promote data use as a tool to improve student outcomes, measure performance, and ascertain accountability. This study examines the components that influence how and to what extent school administrators make use of value-added performance metrics to shed light on how the availability of data affects schools beyond the external accountability factors commonly presented. The usage in schools in this study has two key components that drive the data cultures of the individual schools: (1) administrators’ perceptions of the usefulness and alignment of data and (2) the external factors that influence data uptake and usage.

Comparing Least-Squares Value-Added Analysis and Student Growth Percentile Analysis for Evaluating Student Progress and Estimating School Effects

This paper compares two functionally different approaches to analyzing standardized test data: least-squares based value-added analysis, geared principally to supporting teacher and school accountability; and Betebenner’s (2009) student growth percentiles, which focuses primarily on tracking individual student progress in a normative context and projecting probable trajectories of future performance. Applying the two methods to Australian standardized numeracy and reading test scores (NAPLAN) in grades 3 to 5 and 7 to 9, we find that although they are used differently, the two methods share key structural elements, and produce similar quantitative indicators of both individual student progress and estimated school effects.

Value-added modeling and the power of magical thinking

This article explains the impetus in the United States behind the drive to extend test-based accountability to teachers and the growing interest in employing value-added models to generate the indicators to be used in teacher evaluation. The empirical literature finds that teaching quality is the most important school-related determinant of student. Yet, in the main, teacher evaluations are done poorly - if at all - and compensation has been largely determined by seniority and credentials. Policy makers see strengthening teacher accountability as a priority. In particular, they are looking to increase the role of outputs in comparison to inputs. However, given the technical problems associated with status-based indicators of teacher and school effectiveness, the focus has turned to indicators based on some measure of the progress students have made during the academic year. Value-added analysis relies on sophisticated statistical models to generate estimates of the relative effectiveness of teachers, based on a measure related to student progress. This article provides a brief introduction to value-added models and summarizes key research findings. Although value-added estimates have some desirable properties, they do not represent a simple, neat solution to a complex evaluation problem. In this spirit, the article concludes by describing some of the many concerns regarding the use of value-added scores for high-stakes decisions and suggests some ways to enhance the likelihood that teacher accountability will contribute constructively to the improvement of teaching.

Effectiveness measures for cross-sectional studies: a comparison of value-added models and contextualised attainment models

Educational effectiveness research often appeals to value-added models (VAM) to gauge the impact of schooling on student learning net of the effect of student background variables. A huge amount of cross-sectional studies do not, however, meet VAM's requirement for longitudinal data. Contextualised attainment models (CAM) measure the influence of schools on student outcomes controlling for family background characteristics in cross-sectional studies. It is argued that the latter are adequate substitutes for student prior attainment. Drawing on data from a 3-point longitudinal study in the city of Berlin, Germany (n = 3,074), reading and mathematics achievement of primary students are investigated to assess effectiveness measures of schools. Estimates are compared for a 3-level growth curve analysis (VAM), a hierarchical linear model controlling for background characteristics (CAM), and one additionally controlling for prior achievement scores (prior attainment model). The article contributes to the enhancement of a feedback culture for cross-sectional study results.

Value-added models of teacher and school effectiveness in Ireland: wise or otherwise?

The government of Ireland, like many European countries, is currently under severe pressure from external forces to grow the economy. One possible way to maintain and grow its economy is through the production of a highly educated and globally competitive workforce. In an effort to develop such a workforce, the government, through the Department of Education, is considering ways to increase accountability in its schools. This paper examines value-added accountability systems used in the USA and raises critical questions about their perceived value for Irish schools.

Legal Issues in the Use of Student Test Scores and Value-added Models (VAM) to Determine Educational Quality

A growing number of states and local schools across the country have adopted educator evaluation and accountability programs based on the use of student test scores and value-added models (VAM). A wide array of potential legal issues could arise from the implementation of these programs. This article uses legal analysis and social science evidence to discuss potential legal challenges by educators to the use of VAM that should be considered by public policy makers. It also discusses potential ways VAM might be used as evidence in support of legal claims by students concerning access to educational opportunity.

Value-Added Model (VAM) Research for Educational Policy: Framing the Issue.

In this manuscript, the guest editors of the EPAA Special Issue on “Value-Added Model (VAM) Research for Educational Policy” (1) introduce the background and policy context surrounding the increased use of VAMs for teacher evaluation and accountability purposes across the United States; (2) summarize the five research papers and one research-based commentary that were peer-reviewed and selected for inclusion in this special issue; and (3) discuss the relevance of the papers both individually and collectively. Their importance is discussed in terms of each paper’s contribution to the general research on this topic and each paper’s potential to inform educational policy. In addition, the papers reflect our shared thinking about VAMs, VAM output, and the inference-based decisions for which VAMs are increasingly being used.

Does the Market Value Value-Added? Evidence from Housing Prices after a Public Release of School and Teacher Value-Added

Value-added data are an increasingly common evaluation tool for schools and teachers. Many school districts have adopted these methods and released the results publicly. In this paper, we study the release of value-added data in Los Angeles by the Los Angeles Times newspaper to identify how measured value-added is capitalized into housing prices. This analysis is the first in the school valuation literature to examine property value responses to a value-added information shock, which is of interest as this measure is less correlated with demographics than typical school quality measures. Unique to this setting as well is the release of both school and teacher-level value-added data, which allows us to examine how property values respond to both types of information. Using a difference-indifferences methodology surrounding the release, we find that neither school nor teacher value-added scores are capitalized into home prices. Our results suggest that, despite the contentiousness following these data releases, homeowners do not consider value-added models as currently constructed to be a relevant school quality measure on the margin.

Development of a battery electric vehicle sector in North-West Europe: challenges and strategies

Electric propulsion is a technological alternative to the internal combustion engine. However, the transition to electric propulsion represents a considerable challenge. The European Network on Electric Vehicles and Transferring Expertise (ENEVATE) has undertaken an analysis of the supply chain for the Battery Electric Vehicle (BEV) in North-West Europe to identify the challenges and develop strategies for a strong BEV sector. A production structure analysis, make or buy analysis, value-add analysis, white spot analysis and competitor analysis were conducted in that context. It was found that North-West Europe has a good potential for the development of a healthy BEV sector owing to its innovative automotive industry, strong financial background, flexible production lines and skilled workers. However, without improvements in the level of BEV production, Research and Development (R&D) investments, infrastructure and market share, these advantages are not necessarily sufficient for the region to be a leader in the global BEV industry.