Educational Accountability Systems Research Articles

The Role of Politics and Governance in Educational Accountability Systems

This brief utilizes case study methodology to illustrate the role of governance in educational accountability systems. Most research on the effectiveness of such systems has focused on technical components, such as standards-setting, assessments, rewards and sanctions, and data collection and reporting. This brief seeks to demonstrate that this focus may miss the importance of the institutional set-up. We argue that effective accountability systems are largely dependent on associated government structures, rules, and procedures, and the individuals responsible for implementing them. We use an illustrative case from the state of Oklahoma, where a lack of independent oversight, few checks and balances, and little in-state technical capacity combine to call into question the effectiveness of this state's accountability system. We urge researchers and policy makers to focus more attention on the “messy” governance and politics of educational accountability, and conclude the brief with specific policy proposals to strengthen state education accountability systems.

A Proposed Framework for Evaluating Alignment Studies

Evaluating the multiple characteristics of alignment has taken a prominent role in educational assessment and accountability systems given its attention in the No Child Left Behind legislation (NCLB). Leading to this rise in popularity, alignment methodologies that examined relationships among curriculum, academic content standards, instruction, and assessments were proposed as strategies to evaluate evidence of the intended uses and interpretations of test scores. In this article, we propose a framework for evaluating alignment studies based on similar concepts that have been recommended for standard setting (Kane). This framework provides guidance to practitioners about how to identify sources of validity evidence for an alignment study and make judgments about the strength of the evidence that may impact the interpretation of the results.

Challenges in Interpreting Accountability Results for Schools’ Facilitation of Parent Involvement Under IDEA

Parents’ involvement in their child’s education has been an important tenet of the Individuals with Disabilities Education Act. Under the special education accountability system that went into effect in 2005, states have been required to report annually to the federal government on the percentage of parents with a child receiving special education services who report that schools facilitated parent involvement. This study investigated the variability of states’ reported results owing to states’ different choices with regard to items, metrics, and standards used to calculate their percentage on the indicator. Findings indicated that the application of different methodologies could result in reported percentages that vary by as much as 40 percentage points. It is argued that when reported results do not mean the same thing across states, the accountability system may fail in its mission to provide stakeholders with clear information to evaluate program performance and guide improvement efforts.

The Persistent Problems and Confounding Challenges of Educator Incentives

While education accountability systems emphasize teacher quality as a prerequisite for student learning, education administrators have struggled to staff low-performing schools with effective teachers. Fueled in part by the federal Teacher Incentive Fund, compensation reforms have gained center stage status among strategies aimed at improving human capital in schools. This article presents findings from a formative evaluation of the initial year of implementation of the Financial Incentive Rewards for Supervisors and Teachers (FIRST) program in the Prince George’s County Public School System. Our review of the relevant theoretical and empirical literature on financial incentive systems in public school systems identified four sets of implementation challenges: securing and maintaining stakeholder support; developing and implementing accurate and credible measures of educator performance; developing the district and site capacity required to implement and sustain the initiative; and aligning the reform with human resource goals, school improvement initiatives, and features of the work environment. Our field research on the formulation, implementation, and initial impact of the FIRST initiative demonstrates that even when an educator incentive initiative reflects “lessons learned” from the empirical and theoretical literature, challenges that undermine the initiative’s potential to realize its major goals persist. We argue that this case extends our understanding of the key challenges identified in the literature and we highlight implications for policy and research.

Including students with disabilities in national academic assessments in Korea

In Korea, the effort to include students with disabilities in the educational accountability system has just begun. This paper reviews how Korean students with disabilities have been tested using the National Assessment of Educational Achievement (NAEA) and what issues have emerged as a result of the testing. Analysis of the 2009 and 2010 NAEA data reveals that only a small proportion of students with disabilities participated in these tests and that those who did participate showed low academic achievement compared with general education students. In addition, even though alternate assessments are conducted by special education teachers to assess the academic performance of students with disabilities, national academic assessments in Korea do not include a system for alternate assessment. Based on findings of the analysis, several suggestions are made for improvement in Korea’s educational accountability system for students with disabilities.

An Examination of Algebra for All through Historic Context and Statewide Assessment Data

Since 2003, California has enacted a policy through its education accountability system that encourages schools and districts to place all 8th grade students into algebra courses and therefore, be tested in algebra in the statewide assessment program. Ten years later, there are a great many more 8th graders taking algebra now. However, there are also many students repeating algebra, instead of going on taking higher level mathematics tests. This article aims to provide the historic context of this policy, previous and recent studies on 8th grade algebra, and our study based on the California Standardized Testing and Reporting (STAR) data. We analyzed 8th grade algebra test-taking and the following years� higher level mathematics test-taking to examine the college preparation course taking pipeline. Our longitudinal study compared two groups of students� performance on 9th grade algebra between those who previously scored below proficient on algebra at 8th grade and those who scored proficient or above on general mathematics at 8th grade. Further, another longitudinal study linked 7th grade mathematics sub-scores to 8th grade algebra achievement. The results show that �algebra for all� policy increased the number of students taking algebra at 8th grade and subsequently, taking higher level mathematics tests. However, the pipeline of the college preparation course taking has a significant leak because the number of students taking higher level mathematics decreased dramatically after algebra. Longitudinal study shows that students who pass the general mathematics test at 8th grade have a 69% greater chance to pass the algebra test at 9th grade compared to their peers who failed the algebra test at 8th grade. We also find that the sub-score rational numbers is a strong predictor of 8th grade algebra achievement. Alternatives to help all students achieve in mathematics learning are also discussed in addition to recommendations for future research.

Shining a Light or Fumbling in the Dark? The Effects of NCLB’s Subgroup-Specific Accountability on Student Achievement

The theory of action behind the No Child Left Behind Act of 2001 is that “shining a light” on subgroup performance will increase reading and math test scores for minority and disadvantaged students. Using a panel of all students in Grades 3 through 8 in North Carolina from 2000 to 2008 ( N = 1.7 million students in 1,800 schools), the authors estimate double- and triple-differenced models with school fixed effects to examine whether subgroup-specific accountability threats increase high-stakes test scores. These sanctions are found to have positive effects for minority and disadvantaged students. Larger positive effects emerge for the lowest achieving schools rather than schools near the margin of passing. Some evidence of adverse effects is also found for low and high achievers in math, but not in reading, a finding attributed to the combination of increases in the rigor of state standards in math and responses to an accountability metric based on test score status rather than growth. The implications of the findings for the design of educational accountability systems are discussed.

Early Childhood Special Education and Early Intervention Personnel Preparation Standards of the Division for Early Childhood

Results of the field validation survey of the revised initial and new advanced Council for Exceptional Children (CEC) Division for Early Childhood (DEC) early childhood special education (ECSE)/early intervention (EI) personnel standards are presented. Personnel standards are used as part of educational accountability systems and in teacher personnel preparation program accreditation to train highly qualified teachers. DEC and the National Association for the Education of Young Children (NAEYC) members were partners in the validation survey. The method and survey procedures are described along with the knowledge and skill standards selected as essential and retained in the final DEC ECSE/EI approved sets of standards. Purposeful sampling was used to select DEC respondents for their knowledge and expertise in personnel preparation and operation of ECSE/EI and early childhood education (ECE) programs. Results from the DEC and NAEYC respondents are compared and contrasted. Challenges and limitations are discussed.

Conceptual and empirical differences among various value-added models for accountability

Accountability systems in education generally include indicators of student performance. However, these indicators often differ considerably among the various systems. More and more countries try to include value-added measures, mainly because they do not want to hold schools accountable for differences in their initial intake of students. This study presents a conceptual framework of these value-added measures, resulting in an overview of 5 different types. Using data from Dutch secondary schools, we empirically provide estimates of these different measures. Our analyses show that the correlation between the different types of school effects estimated is rather high, but that the different models implicate different results for individual schools. Based on theoretical considerations, arguments are given to use the following indicators in the value-added accountability models: prior achievement, student-level background characteristics, and compositional characteristics of the student population.

The Effect of Ignoring Classroom-Level Variance in Estimating the Generalizability of School Mean Scores

Contemporary educational accountability systems, including state-level systems prescribed under No Child Left Behind as well as those envisioned under the “Race to the Top” comprehensive assessment competition, rely on school-level summaries of student test scores. The precision of these score summaries is almost always evaluated using models that ignore the classroom-level clustering of students within schools. This paper reports balanced and unbalanced generalizability analyses investigating the consequences of ignoring variation at the level of classrooms within schools when analyzing the reliability of such school-level accountability measures. Results show that the reliability of school means cannot be determined accurately when classroom-level effects are ignored. Failure to take between-classroom variance into account biases generalizability (G) coefficient estimates downward and standard errors (SEs) upward if classroom-level effects are regarded as fixed, and biases G-coefficient estimates upward and SEs downward if they are regarded as random. These biases become more severe as the difference between the school-level intraclass correlation (ICC) and the class-level ICC increases. School-accountability systems should be designed so that classroom (or teacher) level variation can be taken into consideration when quantifying the precision of school rankings, and statistical models for school mean score reliability should incorporate this information.

Are Education Cost Functions Ready for Prime Time? An Examination of Their Validity and Reliability

This article makes the case that cost functions are the best available methodology for ensuring consistency between a state's educational accountability system and its education finance system. Because they are based on historical data and well-known statistical methods, cost functions are a particularly flexible and low-cost way to forecast what each school district must spend to meet the standards in a state's accountability system. However, the application of cost functions to education must confront several challenges in both data collection and estimation methodology. This article describes the strengths and weaknesses of various ways to address these challenges and illustrates how the reliability and forecasting accuracy of cost functions can be tested using data for Missouri school districts.

Validate High Stakes Inferences by Designing Good Experiments, Not Audit Items: A Comment on “Self-Monitoring Assessments Educational Accountability Systems”

Validate High Stakes Inferences by Designing Good Experiments, Not Audit Items: A Comment on “Self-Monitoring Assessments Educational Accountability Systems”

High stakes testing, bilingual education and language endangerment: a Yup'ik example

A growing body of research documents how educational policies and accountability systems can open or close ‘ideological and implementational spaces’ for bilingual education, shaping the language planning efforts of Indigenous communities. Using collaborative research, Indigenous and non-Indigenous researchers investigated the intersection of policy, schooling, and language maintenance/shift within a school district serving 22 Yup'ik villages in Alaska. This article demonstrates how, as multiple communities are witnessing emerging signs of a language shift to English, high stakes testing practices accompanying No Child Left Behind (NCLB) legislation are simultaneously weakening support for bilingual programs in village schools. Yet the article also illuminates the ways in which Yup'ik educators are acting as local language planners, negotiating language maintenance/shift/revitalization, and testing regimes in contested school spaces. Authors discuss the urgent need for, and the promise within, spaces for locally directed language investigation and language planning in national contexts of educational standardization and high stakes assessments.

Educational Accountability System Based on Student Academic Achievement Information: Case of Texas State Accountability System

Educational Accountability System Based on Student Academic Achievement Information: Case of Texas State Accountability System

Self-Monitoring Assessments for Educational Accountability Systems

Test-based accountability is now the cornerstone of U.S. education policy, and it is becoming more important in many other nations as well. Educators sometimes respond to test-based accountability in ways that produce score inflation. In the past, score inflation has usually been evaluated by comparing trends in scores on a high-stakes test to trends on a lower-stakes audit test. However, separate audit tests are often unavailable, and their use has several important drawbacks, such as potential bias from motivational differences. As an alternative, we propose self-monitoring assessments (SMAs) that incorporate audit components into operational high-stakes assessments. This paper provides a framework for designing SMAs. It describes 5 specific SMA designs that could be incorporated into the non-equivalent groups anchor test linking approaches used by most large-scale assessments and discusses analytical issues that would arise in their use.

The Effects of Accountability on Higher Education

This paper analyzes the effects of a higher education accountability system in Brazil. For each discipline, colleges were assigned a grade that depended on the scores of their students on the ENC, an annual mandatory exam. These grades were then disclosed to the public and colleges were rewarded or penalized based on them. I find that the ENC had a positive effect on the education and the proportion of full-time faculty and that it increased the number of vacancies offered, applicants and enrollments. Colleges were affected differently depending on grades, ownership and academic organization, changing the distribution of students among them.

The effects of accountability on higher education

This paper analyzes the effects of a higher education accountability system in Brazil. For each discipline, colleges were assigned a grade that depended on the scores of their students on the ENC, an annual mandatory exam. These grades were then disclosed to the public and colleges were rewarded or penalized based on them. I find that the ENC had a positive effect on the education and the proportion of full-time faculty and that it increased the number of vacancies offered, applicants and enrollments. Colleges were affected differently depending on grades, ownership and academic organization, changing the distribution of students among them.

Trick or treat: new ecology of education accountability system in the USA

This study tracks American states’ policy choices under the No Child Left Behind Act and explores their consequences for student achievement. Using the path analysis of relationships among state‐level policy input, context, and outcome variables, the study portrays a Halloween‐like ‘trick‐or‐treating’ game between the federal and state governments in the new ecology of the test‐driven education accountability system. States that chose the ‘trick’ path with a calculative policy negotiation and manipulation strategy made significant gains on their own state assessments but not on the national assessment. In contrast, states that followed the ‘treat’ path with a faithful policy implementation for funding strategy have not yet brought about significant gains on either the national or state assessments. The first‐generation accountability states with a prior history of high‐stakes testing tended to employ both strategies at the same time. However, neither effective illusion nor ineffective implementation serves the goal of long‐term, sustainable academic improvement. Implications for research and policy are discussed.

Dual standards of school performance and funding? Empirical searches of school funding adequacy in Kentucky and Maine

This study examines potential consequences of the discrepancies between national and state performance standards for school funding in Kentucky and Maine. Applying the successful schools observation method and cost function analysis method to integrated data‐sets that match schools’ eight‐grade mathematics test performance measures to district funding, this study conducts empirical searches for adequate instructional expenditures per pupil to meet desired proficiency targets on national versus state assessments. While Kentucky (high‐stakes testing state) had a lower performance standard than Maine (low‐stakes testing state), this study reveals a relatively lower level of school funding adequacy and a weaker relationship between school expenditures and performance for Kentucky than for Maine. The study suggests that state educational accountability systems and policies may influence the level of state performance standards and the proficiency gaps between national and state assessments, which in turn lead to potential gaps in school funding. Implications for policy and research are discussed to address problems with dual standards of school performance and to improve school funding adequacy and efficiency.

A Study on the Test-Based Educational Accountability System

A Study on the Test-Based Educational Accountability System