School Accountability Systems Research Articles

Mixed-Effects Location Scale Models for Joint Modeling School Value-Added Effects on the Mean and Variance of Student Achievement

School value-added models are widely applied to study, monitor, and hold schools to account for school differences in student learning. The traditional model is a mixed-effects linear regression of student current achievement on student prior achievement, background characteristics, and a school random intercept effect. The latter is referred to as the school value-added score and measures the mean student covariate-adjusted achievement in each school. In this article, we argue that further insights may be gained by additionally studying the variance in this quantity in each school. These include the ability to identify both individual schools and school types that exhibit unusually high or low variability in student achievement, even after accounting for differences in student intakes. We explore and illustrate how this can be done via fitting mixed-effects location scale versions of the traditional school value-added model. We discuss the implications of our work for research and school accountability systems.

Beyond the threshold: The implications for pupil achievement of reforming school performance metrics

We study the effects of a major change to the school accountability system in England. In 2015, the leading published school performance metric was switched from a threshold measure (essentially the fraction of students above a test score level) to an average score measure. Using 7 years of data on all secondary schools in England, we show that this intervention relatively reduced the test scores of students near the threshold, in favour of groups above the threshold (marginally) and below (substantially). We check the sensitivity of our results to different decisions, and present findings on heterogeneous treatments.

A comparison of value-added models for school accountability

ABSTRACT School accountability systems increasingly hold schools to account for their performances using value-added models purporting to measure the effects of schools on student learning. The most common approach is to fit a linear regression of student current achievement on student prior achievement, where the school effects are the school means of the predicted residuals. In the literature, further adjustments are usually made for student sociodemographics and sometimes school composition and “non-malleable” characteristics. However, accountability systems typically make fewer adjustments: for transparency to end users, because data are unavailable or of insufficient quality, or for ideological reasons. There is therefore considerable interest in understanding the extent to which simpler models give similar school effects to more theoretically justified but complex models. We explore these issues via a case study and empirical analysis of England’s “Progress 8” secondary school accountability system.

The role of parents in the school accountability system: Insights from a Hong Kong case

Parents, as a group, are important stakeholders that can influence education policy, as evidenced by the 2015−16 public opposition to a test-based school accountability measure in Hong Kong, which forced the government to remove the high stakes attached to test results. Parents and their role in education accountability systems, however, remain under-researched. This study analysed interviews with 42 parents to examine how they interpreted the Hong Kong Territory-wide System Assessment (TSA) and its functions. One particular area of theoretical interest was understanding what triggered parents to protest against a policy that seemingly protects their interests, and determining whether the alliance between market and bureaucracy, or between neoliberalism and neo-conservatism, has been weakened in some way. The research, however, found the power alliance remained strong. Three distinctive patterns of parental attitudes were identified, indicating a rather loose coalition among parents, the majority of whom endorsed the policy’s intentions and only a small portion was critical of its consequences and ethics. Findings are discussed in terms of parents’ role in the theoretical accountability triangle and the importance of educators’ professional accountability for transforming the policy rhetoric of assessment for learning into classroom reality. Implications are discussed for Hong Kong and beyond.

A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses and challenges for 'Progress 8' raised by COVID-19.

School performance measures are published annually in England to hold schools to account and to support parental school choice. This article reviews and evaluates the ‘Progress 8’ secondary school accountability system for state‐funded schools. We assess the statistical strengths and weaknesses of Progress 8 relating to: choice of pupil outcome attainment measure; potential adjustments for pupil input attainment and background characteristics; decisions around which schools and pupils are excluded from the measure; presentation of Progress 8 to users, choice of statistical model, and calculation of statistical uncertainty; and issues related to the volatility of school performance over time, including scope for reporting multi‐year averages. We then discuss challenges for Progress 8 raised by the COVID‐19 pandemic. Six simple recommendations follow to improve Progress 8 and school accountability in England. Context and implicationRationale for this studyThis study fulfils a need for a comprehensive statistical evaluation of the Progress 8 school accountability system in England, which has been in place since 2016.Why the new findings matterA review of Progress 8 is an important contribution as decisions on its calculation and presentation have important consequences for schools and pupils.Implications for educational researchers and policy makersOur findings on the statistical strengths and weaknesses of Progress 8 and the resulting recommendations made for improvements to the measure and school accountability have clear implications for informing policy makers in the educational system. This includes wider lessons extending beyond Progress 8 to other performance metrics and to other school systems internationally where similar issues apply. Additionally, the review highlights key concerns that the public and school practitioners may like to consider when drawing upon Progress 8 to make decisions around school choice and in assessing how well a school is performing.

Status, Growth, and Perceptions of School Quality

States and districts are increasingly incorporating measures of achievement growth into their school accountability systems, but there is little research on how these changes affect the public’s perceptions of school quality. We conduct a nationally representative online survey experiment to identify the effects of providing participants with information about their local public schools’ average achievement status and/or average achievement growth. Prior to receiving any information, participants already possess a modest understanding of how their local schools perform in terms of status, but they are largely unaware of how these schools perform in terms of growth. Participants who live in higher status districts tend to grade their local schools more favorably. The provision of status information does not fundamentally change this relationship. The provision of growth information, however, alters Americans’ views about local educational performance. Once informed, participants’ evaluations of their local schools better reflect the variation in district growth.

Test-Based Accountability and the Effectiveness of School Finance Reforms

A recent literature provides new evidence that school resources are important for student outcomes. This paper examines whether school accountability systems that incentivize performance (such as No Child Left Behind) raise the efficiency with which additional resources get spent. We leverage the timing of school finance reforms to compare funding impacts on student test scores between states that had accountability in place at the time of the reform and states that did not. The results show that finance-reform-induced increases in student performance are driven by those states where the reform was accompanied by the presence of test-based accountability.

The moral life of econometric equations: Factoring class inequality into school quality valuations in Chile

AbstractRecent sociological scholarship on market design is ill-equipped to understand the normative and political aspects of experts’ practices in connection to political conflicts over the commodification of social rights. I develop an original approach to the politicized use of market devices to address collective concerns in a noneconomic policy field: education. When designing a high-stakes school accountability system, policymakers in Chile confronted a moral conundrum: should schools be valued according to their students’ absolute proficiency, or according to the school’s relative effectiveness? Progressive and conservative experts in charge of settling this dilemma pushed for using the statistical model (OLS vs. HLM) that yielded rankings that fit their moral preferences. Through qualitative analyses of experts’ real-world application of quantitative methods, as well as experts’ interpretations of these methods’ performative consequences, I mobilize the much-debated concept of “moral background” to unravel the conditions for subsuming ideological dissent into consensual forms of decision-making.

How are the ‘losers’ of the school accountability system constructed in Chile, the USA and England?

ABSTRACT Performance-based accountability systems that rank schools based on their effectiveness produce ‘winners’ and ‘losers’. Substantial evidence has pointed to the (side)effects of these classifications, particularly in the most disadvantaged communities. Whilst previous studies have compared schools under different effectiveness categories within and between countries, this qualitative study takes a cross-case comparison approach to analyse education policies, grey literature and previous research evidence to explore the mechanisms that construct ‘failing’ schools in three notable high-stakes accountability systems worldwide: Chile, the USA and England. After describing (1) the identification and classification of ‘failing’ schools; (2) the instruments used to justify these classifications; and (3) who make these judgements, we conclude that the construction of ‘failing’ schools serves the competition and differentiation required for maintaining neoliberal hierarchical and unequal market-oriented regimes. Instead of disciplining only ‘failing’ schools, these labels also provide a caution for the whole school system.

Have Income-Based Achievement Gaps Widened or Narrowed?

Since 1990, U.S. policymakers have worked to close gaps in academic achievement by income and race (e.g. with school finance reform and school accountability systems) even as rising income inequality and income-based residential segregation have threatened to widen them. Using estimates of the mean and variance in household income for sampled schools, we reconstruct the student-level relationship between achievement and household income in the National Assessment of Educational Progress (NAEP) from 1990 to 2015. We find that achievement at all levels of parental income rose substantially in 4th and 8th grade. In contrast to Reardon (2011), we find that achievement gaps narrowed substantially in 4th grade reading and math and in 8th grade math, while the gaps remained stable in 8th grade reading. As a robustness check, we used the March Current Population survey to impute income for dependent children by race, mother’s education, urbanicity and state and then calculated mean achievement for those same groups in the NAEP. Again, we found gaps in achievement narrowing between groups with high and low predicted mean household incomes. Our results challenge the prevailing understanding that income-based achievement gaps have widened in the United States over the last 30 years. Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

Sensitivity of self-reported noncognitive skills to survey administration conditions

Noncognitive skills (e.g., persistence and self-control) are typically measured using self-reported questionnaires in which respondents rate their own skills. In many applications-including program evaluation and school accountability systems-such reports are assumed to measure only the skill of interest. However, self-reports might also capture other dimensions aside from the skill, such as aspects of a respondent's situation, which could include incentives and the conditions in which they complete the questionnaire. To explore this possibility, this study conducted 2 experiments to estimate the extent to which survey administration conditions can affect student responses on noncognitive skill questionnaires. The first experiment tested whether providing information about the importance of noncognitive skills to students directly affects their responses, and the second experiment tested whether incentives tied to performance on another task indirectly affect responses. Both experiments suggest that self-reports of noncognitive skills are sensitive to survey conditions. The effects of the conditions are relatively large compared with those found in the program evaluation literature, ranging from 0.05 to 0.11 SDs. These findings suggest that the effects of interventions or other social policies on self-reported noncognitive skills should be interpreted with caution.

An Argument in Search of Evidence: A Response to Camara et al. (2019)

AbstractThis article offers a critique of the validity argument put forward by Camara, Mattern, Croft, and Vispoel (2019) regarding the use of college‐admissions tests in high school assessment systems. We challenge their argument in two main ways. First, we illustrate why their argument fails to address broader issues related to consequences of using admissions tests as an achievement indicator in high school accountability systems. Second, we find the evidence put forth by Camara et al. argument incomplete and/or overstated, even if evaluated against a more limited set of claims. We offer several additional comments regarding the construction of their validity argument, the lack of transparency for evaluating many of the claims, and the potential unintended negative consequences associated with the use of the ACT or SAT as the primary achievement indicator in a state accountability system.

Conflicts and Issues on Parental Participation in School Management

교육감 직선제가 시행된 이후, 민주성과 개방성 원칙에 따라 교육정책의 다양한 분야에서 협치 장치들이 제도화되고 운영되어 왔다. 단위학교 책임경영제의 운영 범위가 예산결정과 평가로까지 확산됨에 따라 학부모의 역할이 부각되고 있다. 특히 교육수요자로서 예산과 평가 등 단위학교 의사결정과정에 참여하는 학부모의 역할과 그 영향에 대해 기대와 우려가 교차하고 있다. 본 연구는 학교 자율화와 이에 따른 단위학교 책임경영을 기조로 하는 교육행정 체제 변화로 인해 확대된 학부모의 의사결정 참여 영역과 수준의 변화를 고찰하고, 교육주체로서 학부모의 역할의 지향점을 모색하였다. 이를 위해 학부모의 학교 운영 참여에 대한 상반된 이론적 흐름과 이를 구체화한 상반되는 제도적 현황을 제시하였다. 이렇게 확인된 교육주체로서의 학부모 역할을 도출하고, 이를 토대로 현행 단위학교책임경영 제도 내 학부모 의사결정 수렴 구조의 쟁점을 도출하였다. 본 연구를 통해 단위학교를 중심으로 한 교육공동체 내 교육주체로서 학부모의 역할을 규정하는 데 있어서 기존의 당위적 정당성 부여와 전문성 부재에 따른 배제적 관점의 이분법적 논의를 극복하려고 하였다. 또한 자율과 자치로서 학교평가와 학교운영이 효과적으로 작동하기 위해 교육주체를 참여시킴에 있어 요구되는 원칙과 체계를 모색하고자 하였다.Recently, in accordance with the principle of democracy and openness, the parental participation in school management have been institutionalized and operated in various fields of educational policy. The purpose of this study is to examine changes in the level decision making and management participation of parents along with the changes of educational administrative system based on school based management (SBM) and school accountability system. On this point, we present the contradictory theoretical flow of the parental participation in school management and the conceptual framework different status among institutions in terms of introducing level and range of parental participation. Based on theories and practices, we draw out the issues of the structure of the parental school management participation structure in the current school based management system. Through this study, we try to overcome the dichotomy of the exclusionary viewpoint in terms of defining the role of the parents as the educational subject in the educational community centering in the era of school based management. Through this paper, we expect that comprehensive understanding on parental participation in school management can be continued.

How Much Should We Rely on Student Test Achievement as a Measure of Success?

The use of test scores as a performance measure in high-stakes educational accountability has become increasingly popular since the enactment of the No Child Left Behind Act of 2001 (NCLB), which imposed sanctions such as the threat of losing federal funds unless a state implemented a school accountability system that measures student progress continuously. Since then, many in the education community have questioned whether differences in student test scores reflect actual discrepancies in the long-term well-being of individuals. In this review, we try to address this question in the light of the extant literature that examines the relationship between test scores and later life outcomes. We show that while there are certainly studies that contradict the causality of this relationship, there is also abundant evidence suggesting a causal link between test scores and later life outcomes. We conclude that any debate about the use of test scores in educational accountability (1) should be framed by use of all relevant empirical evidence, (2) should also consider the predictive validity of nontest measures of student success, and (3) should keep in mind that the predictive validity of test scores could be stronger in some contexts than others.

Examining the State-Trait Composition of Social-Emotional Learning Constructs: Implications for Practice, Policy, and Evaluation

Social-emotional learning (SEL) is gaining prominence in education practice and policy. Research shows that SEL can be improved by short-term, targeted interventions and longer-term strategies to improve school contextual factors. However, little is known about how much of the variance in SEL constructs is stable over time versus specific to a given time/context. The present study examines the stability of academic achievement relative to four SEL domains (growth mindset, self-efficacy, social awareness, and self-management). While the rank ordering of math and reading skills are highly stable over time, the four SEL domains seem to be more strongly influenced by contextual factors. Implications for school accountability systems and intervention approaches are discussed.

Recruiting for school improvement: the relationship between teacher-centric school quality factors and school improvement designations in Kentucky

In 2018, Kentucky began the implementation of a new school accountability system. Like other states across the nation, Kentucky’s new system was designed to bring state policy into alignment with federal changes brought about by the Every Student Succeeds Act (ESSA) of 2015. In addition to the creation of new school improvement labels, Kentucky’s system also reports on a variety of teacher-centric school quality factors. A statistical analysis was conducted to determine if a relationship exists between these school quality factors and the school’s improvement designation. The study found that schools identified as needing the greatest improvement (Comprehensive Support and Improvement [CSI]) are statistically more likely to have higher teacher turnover rates, higher percentages of new teachers, and employ teachers with lower rates of advanced education in the year prior to identification than their counterparts with either the Targeted Support and Improvement (TSI) or Other designations. These findings should inform future policy making and elevate teacher recruitment and retention as a school improvement priority.

Value-added indicators for a fairer Chilean school accountability system: a pending subject

ABSTRACT Although schools’ relative contribution to pupils’ progress is increasingly used in accountability systems around the world, momentum for value-added models (VAM) has not been reached in Chile. This small-scale study explores qualitatively the policy context in which this omission takes place, by analyzing policy documents and interviewing local policymakers and researchers about their views on VAM. These agents and official documents not only point to political, methodological and pragmatic reasons for and against value-added indicators, but also highlighted ethical and legal reasons pro (but not against) VAM. Overall, the most prominent reason for including VAM into the new accountability system was the ethical consideration. The notion that a fairer indicator could make justice, especially for those schools working in disadvantaged contexts, was a recurrent idea. In contrast, the most recurrent reasons against VAM were methodological. Whilst research on VAM for the Chilean school system has been conducted over the last decade, it has impacted very little the policy arena. Given that the creation of frameworks for assessing value-added indicators takes time, needs policy and research support and needs to consider potential unintended consequences, the road towards including them into the Chilean school accountability system is just starting.

The importance of adjusting for pupil background in school value‐added models: A study of Progress 8 and school accountability in England

In the UK, USA and elsewhere, school accountability systems increasingly compare schools using value‐added measures of school performance derived from pupil scores in high‐stakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that value‐added measures unadjusted for pupil background are biased in favour of schools with more ‘educationally advantaged’ intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low‐performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the ‘Progress 8’ secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high‐stakes testing and school value‐added models, raise serious doubts about their use in school accountability systems.

School accountability in England: A critique

No school accountability system is perfect, but will mooted changes to inspection in England tackle some of the unintended consequences that school leaders face? Hilary Grayson draws some lessons from international practice

Alternative Accountability Models

The passage of ESSA in 2015 created a novel mandate for States to develop school accountability systems that use alternative measures of school quality beyond test scores. This created an opportunity for States to be innovative and make new forms of data available that could potentially lead to new strategies for improving schools. This study explored early experiments in alternative accountability measures in two urban districts. Drawing on interviews, documents, and observations gained through participant research, this study highlights the variable types of data the models made available to stakeholders and how those differences suggest different theories of change for improving schools. The findings highlight how different ways of specifying school quality point to different ideas about what schools should be responsible for improving.