Test-based Accountability Systems Research Articles

Test-based accountability and educational equity: Breaking through local district politics?

A central premise of the U.S. No Child Left Behind Act (NCLB) was that test-based accountability systems would break through local politics and lead districts to prioritize the needs of disadvantaged students. Yet, no research examines the equity-related effects of NCLB’s district accountability system. Focusing on a state with rich data, we find that negative ratings (which put districts at higher risk of state intervention) led to the disenrollment of economically disadvantaged students (ostensibly leading to a decline in income-based school segregation) and a corresponding (suspicious) jump in test scores. Negative ratings also led to changes in collectively bargained personnel policies and the replacement of teachers in disadvantaged students’ schools. There was no redistribution of funds toward disadvantaged students’ schools, however, and likely minimal impact on district elections. The results put into question whether a shift in district governance helps explain NCLB’s well-documented positive impacts on low-income students’ academic outcomes.

Consequential research of accountability testing: the case of the CET

This article examines the changes taken place over the decades after a test-based accountability system was in place and proposes an agenda for consequential research of the testing program. The article starts with a review of theories of test-based accountability (Supovitz, Journal of Educational Change 10:211–227, 2009) and a socially grounded, integrated approach to test validation (Chalhoub-Deville, Language Testing 33:453–472, 2016; Chalhoub-Deville and O’Sullivan, Validity: Theoretical development and integrated arguments, 2020). Following Supovitz’s frameworks of accountability testing, the case of the College English Test (CET), a national English language testing program in China, is analyzed to explain the nature of the consequences of accountability testing and show the motivational effects, alignment effects, the informational role and the symbolic values of the accountability testing program. The case analysis highlights the importance of investigating consequences of an accountability test at the aggregate and the system levels. Guided by Chalhoub-Deville and O’Sullivan’s (Validity: Theoretical development and integrated arguments, 2020) integrated validity framework, the article delineates the claims for the theory of action argument and the communication engagement argument for future investigation of the consequences of the CET.

A validity framework for accountability: educational measurement and language testing

High stakes test-based accountability systems primarily rely on aggregates and derivatives of scores from tests that were originally developed to measure individual student proficiency in subject areas such as math, reading/language arts, and now English language proficiency. Current validity models do not explicitly address this use of aggregate scores in accountability. Historically, language testing and educational measurement have been related, yet parallel disciplines. Accountability policies have increasingly forced these disciplines under one umbrella with a common system of rewards and sanctions based on results achieved. Therefore, a validity framework, as suggested in the present paper, is relevant to both.

TEST-BASED ACCOUNTABILITY: CAN TESTING CHILDREN TELL US ANYTHING SIGNIFICANT ABOUT THE PERFORMANCE OF SCHOOLS?

This article would argue that test-based accountability has its rationality and is necessary for policy makers to control and improve public education. It could demonstrate to the public that public funds and resources are used reasonably and effectively. However, it indeed led to unintended issues among system-level, school-level and affected individuals in many ways. The main focus is on criticizing the effect and consequences of test-based accountability system by analyzing an act incorporating the system — “No Child Left Behind”. Besides, some background information as well as main legislations would be included in this article.

Literacies and social practice: sociological perspectives on reading research

ABSTRACT This article considers the distinctive contribution that sociological perspectives have made to understanding reading as a profoundly social and cultural activity, differentiated by the specifics of time and place. Literacy researchers, using a social lens, have sought to explore, explain and redress inequalities in access to and uses of literacy in its different forms. This has often meant championing pedagogies premised on dialogue and connection. Yet this may conflict with an emphasis in current policy on children mastering a relatively narrow skills-based definition of literacy as they begin to learn to read. The article explores how the tension points in the literacy curriculum, intensified in test-based accountability systems, can be navigated in this light.

The game of accountability: perspectives of urban school leaders on standardized testing in Ontario and British Columbia, Canada

ABSTRACT This paper investigates how test-based, standardized accountability is impacting on urban school leaders’ everyday work in Ontario and British Columbia, Canada. This critical policy analysis draws on theories of new public management and governmentality to document the experiences of school leaders as they enact and navigate test-based accountability policies. The data for this empirical study was collected from semi-structured interviews with 18 school leaders in 4 high schools from two cities: Toronto and Vancouver. As a result of different education policy contexts in Ontario and British Columbia, findings reveal that there were both similarities and differences between the experiences of school leaders therein. In Ontario, school leaders face tremendous pressure to provide evidence of improved test scores as part of an accountability regime that is high-stakes. In both provincial contexts, the mediatization of test scores places new demands on school leaders who act as entrepreneurs, marketing and selling their schools. Ultimately, the research highlights the unintended consequences of test-based accountability systems as contributing to the marketization of education in Canada with deleterious consequences for marginalized communities and the practice of social justice leadership in schools. There is a need to identify democratic and holistic measures of accountability.

Negotiating the Political and Pedagogical Tensions of Writing Rubrics: Using Conceptualization to Work toward Sociocultural Writing Instruction

An increased emphasis on writing standards has led many U.S. states to incorporate on-demand writing assessments into their test-based accountability system. We argue this creates political and pedagogical tensions for teachers to navigate. We discuss how rubric conceptualization (1) is a process wherein a teacher iteratively (co-)constructs meaning from a rubric’s design via classroom instruction; (2) is informed by implicit theories of learning; and (3) often requires a teacher to negotiate the competing pedagogical and political meanings of a rubric. While test-based accountability frameworks promote rubric use that equates learning with student achievement, rubric conceptualization is a process where teachers have some agency to resist behaviorist approaches to instruction.

Assessment Reform and the Case for Learning-Focused Accountability

Calls for greater accountability within schools have led to an expansion, and in some cases, retraction of test-based accountability systems within many English-speaking Western nations. Countries such as England, United States, and Canada have all developed educational standards that are primarily measured in relation to standardized achievement test results. This paper discusses the documented impact and social consequences of external testing measures on students, teachers, and school systems within the Western culture of student performance. The discussion considers alternative assessment systems that are currently evolving into a framework of learning-focused accountability. The article concludes by discussing the implications of these progressive alternatives for policy-makers, school personnel, and institutions concerned with the training of teachers and administrators.

Subjective Performance Evaluation in the Public Sector

This paper investigates the effects of being evaluated under a novel subjective assessment system where independent inspectors visit schools at short notice, disclose their findings, and sanction schools rated fail. I demonstrate that a fail inspection rating leads to test score gains for primary school students. I find no evidence to suggest that fail schools are able to inflate test score performance by gaming the system. Relative to purely test-based accountability systems, this finding is striking and suggests that oversight by evaluators who are charged with investigating what goes on inside the classroom may play an important role in mitigating such strategic behavior. There appear to be no effects on test scores following an inspection for schools rated highly by the inspectors. This suggests that any effects from the process of evaluation and feedback are negligible for nonfailing schools, at least in the short term.

The GED and the Rise of Contextless Accountability

Background/ContextPolicy discussions in the U.S. and abroad have become increasingly studded with reference to the results of international tests like PISA. Unlike most assessments, PISA is not designed to measure whether students have mastered a particular school curriculum but rather provide a measure of students’ ability to meet future challenges irrespective of where in the world they live. Though growing in influence, the concept of a “contextless” form of accountability has an important antecedent in the history of American education: the Tests of General Educational Development (GED), which were developed in the 1940s to assist the transition of American World War II servicemen and women.PurposeThe purpose of the study is to use the history of the development and subsequent spread of the GED examine the general challenge posed by contextless accountability measures.Research DesignThis study draws on a wide range of primary and secondary sources to present a historical analysis of the development and diffusion of the Tests of General Educational Development.Conclusions/RecommendationsNoting the strong parallels between the history of the GED and the current popularity of international measures like PISA, this paper examines the history and development of the GED in order investigate the allure, promise, and pitfalls of contextless assessment and accountability. In so doing, this paper illustrates the importance of quantification as a means of creating useful abstractions but also the inherent danger of the perceived certainty of these kinds of metrics. In the decades following the 1940s, the GED retained its reputation as an objective, readily available, measure of high school achievement that could be used in any context and with any population—a task it was never intended or designed to fulfill. Thus, this paper argues, the American experience with the GED offers important lessons and insights in a world where PISA continues the reign of contextless, test-based accountability systems. Namely, that the level of abstraction required to develop these measures makes them ill-suited to inform the kind of specific policy discussion in which they are frequently invoked.

Responses of schools to accountability systems using multiple measures: the case of New York City elementary schools

Many studies point to potential unintended consequences of accountability systems such as when schools narrow their teaching to fixate on tested subjects. As a result, some states and districts in the USA have complemented the federal test-based accountability system with additional measures of educational practices to hold schools accountable on multiple measures. To explore the consequences of such systems, this study focuses on the responses of nine elementary schools to a multiple-measure accountability system in New York City, including high-stakes tests and quality reviews. While some schools showed broader improvement efforts, results suggest the state test remains the dominant measure in driving responses of schools, and in some cases, the quality review further reinforces the schools' focus on the test.

Pedagogy of the Impossible: Neoliberalism and the Ideology of Accountability

This article analyzes the ideology of accountability in contemporary education within the context of neoliberalism and its reconstruction of social relationships on the basis of the market, competition and efficiency. Drawing on contemporary critical philosophical accounts, it argues that the scholarship on education and accountability has not fully registered the way that ideology in neoliberalism works through modes of fantasy and enclosure, inhering not only in perspectives and understandings, but also in procedures, rituals and structures of subjectivity. Analyzing the political logic of test-based accountability, and taking up several specific examples of its reorganization of curriculum and assessment, the article challenges the tendency within educational theory to understand ideology in education according to the Gramscian model of hegemony and proposes a reconceptualization of accountability's ideological effects. In particular, it shows that the tradition of understanding ideology in schooling in terms of the production of ‘common sense’ overlooks the ideological force in contemporary education of the procedures themselves of standardized assessment and scripted curriculum. The article concludes with a consideration of the implications of this analysis for critical teaching in the context of the constraints created by test-based accountability systems.

Evidence, Methodology, Test-Based Accountability, and Educational Policy

This exchange represents a follow-up to an article on the effects of state high school exit examinations that previously appeared in this journal ( Warren, Grodsky, & Kalogrides 2009 ). That 2009 was article featured prominently in a report by the National Research Council (NRC) that evaluated the efficacy of test-based accountability systems. Hanushek (2012 ) was highly critical of the NRC’s interpretation of the existing evidence, including Warren, Grodsky, & Kalogrides’ 2009 piece. Here, Warren and Grodsky explain why they believe that Hanushek incorrectly evaluated their original article on state high school exit examinations. Hanushek then responds.

Meeting the Curricular Challenges of Inclusive Assessment: The role of alignment, opportunity to learn, and student engagement

Inclusive test-based accountability systems are intended to focus attention on the teaching and learning of important knowledge and skills for students with disabilities. Test score inferences to this end are often based on unverified assumptions of curricular alignment, access, and engagement. Focusing on students with disabilities, we discuss key concepts and practices relevant for measuring and facilitating alignment, opportunity to learn (OTL), and student engagement. In conclusion, three recommendations are provided calling for the collection and evaluation of alignment, OTL, and engagement data to support the validity of test score inferences and to inform the design of interventions that improve student OTL and engagement.

Rules and Discretion in the Evaluation of Students and Schools: The Case of the New York Regents Examinations

The challenge of designing effective performance measurement and incentives is a general one in economic settings where behavior and outcomes are not easily observable. These issues are particularly prominent in education where, over the last two decades, test-based accountability systems for schools and students have proliferated. In this study, we present evidence that the design and decentralized, school-based scoring of New York’s high-stakes Regents Examinations have led to pervasive manipulation of student test scores that are just below performance thresholds. Specifically, we document statistically significant discontinuities in the distributions of subject-specific Regent scores that align with the cut scores used to determine both student eligibility to graduate and school accountability. Our results suggest that roughly 3 to 5 percent of the exam scores that qualified for a high school diploma actually had performance below the state requirements. Using multiple sources of data, we present evidence that score manipulation is driven by local teachers’ desire to help their students avoid sanctions associated with failure to meet exam standards, not the recent creation of school accountability systems. We also provide some evidence that variation in the extent of manipulation across schools tends to favor traditionally disadvantaged student groups.

Avoidable losses: High-stakes accountability and the dropout crisis.

In the state of Texas, whose standardized, high-stakes test-based accountability system became the model for the nation's most comprehensive federal education policy, more than 135,000 youth are lost from the state's high schools every year. Dropout rates are highest for African American and Latino youth, more than 60% for the students we followed. Findings from this study, which included analysis of the accountability policy in operation in high-poverty high schools in a major urban district, analysis of student-level data for more than 271,000 students in that district over a seven-year period under this policy, and extensive ethnographic analysis of life in schools under the policy, show that the state's high-stakes accountability system has a direct impact on the severity of the dropout problem. The study carries great significance for national education policy because its findings show that disaggregation of student scores by race does not lead to greater equity, but in fact puts our most vulnerable youth, the poor, the English language learners, and African American and Latino children, at risk of being pushed out of their schools so the school ratings can show "measurable improvement." High-stakes, test-based accountability leads not to equitable educational possibilities for youth, but to avoidable losses of these students from our schools.

Validity of Inferences from Test-based Educational Accountability Systems

Many states have adopted test-based accountability systems before the No Child Left Behind (NCLB) Act of 2001 became law. NCLB has added a new dimension to test-based educational accountability systems. Features of state and NCLB accountability are discussed with an emphasis on questions of the validity of inferences that are made about school quality. It is concluded that none of the current approaches toe test-based accountability support causal inferences about school quality. It is also shown that tracking progress toward the important goal of closing gaps in achievement requires more than just monitoring changes in the percentages of students who are proficient.

Alternative Approaches to High-Stakes Testing

UNDER No Child Left Behind (NCLB), the federal government requires state governors, superintendents, and school principals, through a regime of annual testing, to demonstrate to the taxpaying public that education dollars are being used effectively to improve student achievement. Developing, administering, and scoring the required assessments call for highly specialized skills and experience that states often lack, leading them to hire outside testing companies and consultants. Student performance on these tests--given at the end of the year to all students in grades 3 through 8--determines rewards and sanctions for schools, teachers, and students. But these standardized tests were not designed for accountability purposes, and experience in isolating and measuring the effects that schools have on student learning is rare. This test-based accountability system is being controlled by people who may know how to develop standardized achievement tests but know very little about the institutional realities of accountability--and even less about how to improve instruction in schools. For the most part, the assessments currently in use are not capable of accounting for the many non school factors that influence test scores, including student background, home environment, poverty level, and English-language proficiency. Such standardized tests are not always reliable measures of what is learned in the classroom, according to assessment expert James Popham. (1) And now NCLB has established these single, end-of-year tests as the dominant measures of school success or failure. HIGH-STAKES TESTING Tests used to make high-stakes decisions, especially in light of the fact that they are given only once or twice a year, necessarily have to meet a set of strict requirements. High-stakes tests must be sensitive--that is, they must be capable of determining changes in achievement related to instructional improvements. The validity of an accountability system depends on designing the right tests. These tests should meet the following criteria: (2) 1. Assessments used to measure student mastery of specific content must include clear descriptions--brief, jargon-free, and teacher-friendly--of what is going to be assessed. Classroom teachers need these descriptions to understand in detail what is expected of their students. 2. Effective assessments focus on a modest number of significant curricular aims, drawn from content standards. The selected content standards clearly must be of major importance. 3. An instructionally sensitive test to be used for accountability purposes must report student performance in a way that enables teachers to know what aspects of their instruction need to be improved and what aspects are working well. The tests currently being used to satisfy NCLB have been judged by the majority of teachers to be instructionally insensitive--incapable of measuring the effects of instruction on student performance. This can lead to tragic results because high-stakes tests can distort instruction and may encourage teachers to teach to the Teaching to a bad test and spending months on drill and skill may boost scores but surely ends up turning off students. The NCLB mandates for AYP (adequate yearly progress) and public reporting of results put enormous pressure on students, teachers, principals, and superintendents to raise test scores. This pressure can lead, in extreme cases, to cheating. The more general response is for teachers to practice on past (and even future) versions of the tests and to restrict instruction to just those subjects that will be tested; this is known as item teaching. These outcomes of high-stakes testing distort the traditional ideal of the teacher as one who makes every effort to achieve the goals of the curricula without regard to any particular test. The real blame for inappropriate forms of teaching in response to testing lies not with teachers but with state and national policy makers who create accountability systems centered on ever-higher test scores (AYP) with little regard for how these scores relate to better learning. …

Accountability, California Style: Counting or Accounting?

Across the nation and at nearly all levels of our educational system, efforts to hold schools accountable for student learning dominate strategies for improving the quality of education. At both the national and state level, student testing stands at the center of educational accountability programs, such that schools are effectively held accountability for increases in student test scores. This working definition of accountability contrasts sharply with the formal definition of accountability in which those discharged with duties are expected to provide an account—that is, a description and an explanation—of their duties and conduct and to assist in determining whether said conduct was responsible. This article focuses specifically on the accountability system introduced in California in 1999 and identifies several shortcomings of test-based accountability systems. These shortcomings fall into two broad categories: unrealistic expectations established by the system and failure to identify why schools succeed or fail to meet test-based goals. Based on shortcomings identified in California, as well as limitations of test-based accountability itself, several recommendations for an improved accountability system are offered. Chief among these recommendations is that notions of accountability be expanded such that accountability becomes a two-way process in which all levels from the classroom up to the state should be asked to account for their practices and the impact those practices have on students and their learning.

Accountability, California Style: Counting or Accounting?

Across the nation and at nearly all levels of our educational system, efforts to hold schools accountable for student learning dominate strategies for improving the quality of education. At both the national and state level, student testing stands at the center of educational accountability programs, such that schools are effectively held accountability for increases in student test scores. This working definition of accountability contrasts sharply with the formal definition of accountability in which those discharged with duties are expected to provide an account—that is, a description and an explanation—of their duties and conduct and to assist in determining whether said conduct was responsible. This article focuses specifically on the accountability system introduced in California in 1999 and identifies several shortcomings of test-based accountability systems. These shortcomings fall into two broad categories: unrealistic expectations established by the system and failure to identify why schools succeed or fail to meet test-based goals. Based on shortcomings identified in California, as well as limitations of test-based accountability itself, several recommendations for an improved accountability system are offered. Chief among these recommendations is that notions of accountability be expanded such that accountability becomes a two-way process in which all levels from the classroom up to the state should be asked to account for their practices and the impact those practices have on students and their learning.