State Accountability Systems Research Articles

Refining and expanding accountability policy for English learners to support Lau implementation: A critical policy analysis

ABSTRACT Title I accountability requirements under the Every Student Succeeds Act (ESSA) represent the present-day instantiation of Lau, which requires schools to provide a program for English learners (ELs) that supports their meaningful engagement and to provide transparent information about EL program quality. This study uses critical policy analysis to explore whether ESSA accomplishes what it intends. Employing a parallel mixed methods design, we first analyze longitudinal data from two states to examine whether their approaches to assessing EL students’ academic performance and progress support valid claims about EL achievement. Then, we analyze focus group and interview data from educators, advocates, and families in states to understand how they assess the quality of a school’s EL program. Taken together, our findings highlight the need for the incorporation of multiple perspectives into the design of state accountability systems and for the development of robust data systems that facilitate the sharing of accurate and nuanced information related to EL student outcomes and opportunities.

Digital Module 35: Through‐Year Assessment

Module AbstractThrough‐year assessments are assessments that are administered in multiple parts and at different times over the course of a school year that also produce summative scores that can be used with state accountability systems (Lorié et al., 2021; Dadey & Gong, 2023). These assessments are alternatively known as instructionally embedded, through‐course, or periodic assessments. There are a number of possible through‐year assessment models, and they have recently been the subject of much policy interest as they have the potential to inform subsequent instruction, be more closely aligned with and responsive to curricula and instruction, provide more proximal measures of learning, and be a more sensitive measure of student progress or growth than typical year‐end summative assessments (Clark & Karvonen, 2021; Gong, 2021; NWEA, 2021; Wise, 2011). More research is needed, however, to substantiate these potential uses.

Performance Information and Personnel Decisions in the Public Sector

Firms and other organizations establish the criteria under which employees will be judged and the performance measures made available to supervisors, the board of directors and other stakeholders, and these structures almost certainly influence behavior and organization outcomes. Any divergence of the chosen performance metric from an ideal measurement of productivity may lead to suboptimal outcomes, particularly in the public sector where outside interest groups may rely more heavily on easily accessible ratings than better-informed insiders. In the case of public education, federal and state accountability systems provide considerable information about student outcomes and rate schools on that basis. However, the No Child Left Behind accountability legislation’s focus on pass rates rather than learning and achievement growth introduces the possibility that inadequate information and a flawed structure each compromise public school quality. This study of school principal labor market outcomes investigates the relationship between principal labor market success and a set of performance measures that differ on the basis of accessibility to stakeholders and link with true principal productivity. The results from the empirical analysis provide evidence that information and design deficiencies introduce a lack of alignment between incentives and principal productivity and adversely affect the quality of education in Texas public schools.

Adding “Student Voice” to the Mix: Perception Surveys and State Accountability Systems

For the past two decades, student perception surveys have become standard tools in data collection efforts. At the state level, however, “student voice” is still used sparingly. In this study, we examine the ways in which including student survey results might alter state accountability determinations. Reconstructing the accountability system in Massachusetts, we draw on a unique set of student survey data, which we add to the state’s formula at a maximally feasible dosage in order to determine new school ratings. As we find, student survey data shift school accountability ratings in small but meaningful ways and appear to enhance functional validity. Student survey results introduce information about school quality that is not captured by typical accountability metrics, correlate moderately with test score growth, and are not predicted by student demographic variables.

Teacher and Student Reciprocal Agency in Odds-Beating Schools

Background/Context In the field of education, the lens of agency has provided a valuable conceptual alternative to deterministic portrayals of schools as oppressive institutions where teachers and students have little power over the conditions in which they teach and learn. A number of studies have investigated teacher and student agency, but few have explored the relationships between the two, particularly in regard to how teacher and student agency relate in high-need and high-diversity contexts with exemplary student graduation outcomes. Purpose/Focus of the Study In an effort to address the paucity of research investigating the relationships of teacher and student agency, this analysis draws on seven qualitative case studies of secondary schools achieving a trend of above-predicted (i.e., odds-beating) graduation outcomes. We pursued the overarching research question: What are the relationships between teacher and student agency in odds-beating schools? Setting The study took place in a purposeful sample of odds-beating secondary schools identified through multiple regression analyses. The sample included schools in rural, suburban, and more urban communities and were distributed across various geographic regions in the state of New York. All schools shared a pattern of above-predicted graduation outcomes, taking into account student demographic factors (percentage of students economically disadvantaged, African American/Black, Hispanic/Latinx, and English language learners). All schools met the criteria of being within the normal range for wealth ratio as well. Research Design This qualitative multiple case study focused on data generated from interviews and focus groups with 302 participants, including teachers, student support specialists, and school and district leaders, as well as field notes gathered during guided school tours, and documents. These data were analyzed using qualitative comparative analysis methods in multiple phases of deductive and inductive coding. Conclusions Study findings indicate that when teachers are offered opportunities to act as agents, they tend to offer opportunities to their students similarly. This research also suggests that affordances for teachers to assert agency can mitigate the constraining effects of state accountability system compliance-oriented practices. We conclude that teacher and student reciprocal agency merits further study and offers theoretical insights of particular import in high-needs and high-diversity school contexts.

Understanding congressional coalitions: A discourse network analysis of congressional hearings for the Every Student Succeeds Act

The purpose of this study is to investigate policy coalitions of the Every Student Succeeds Act (ESSA) at U.S. congressional hearings. This study is grounded in the advocacy coalition framework, which argues that advocacy coalitions are forged by policy actors who have similar policy preferences. To identify the coalitions, according to the policy claims articulated by policy actors, discourse network analysis was performed to examine 30 testimonies in the congressional hearings on ESSA since its passage in 2015. The policy actors fall into eight categories: (1) federal administrative and executive offices, (2) state administrative and executive offices, (3) teachers unions, (4) interest groups, (5) superintendents, (6) teacher, (7) education professor, and (8) Congress members. The results of discourse network analysis suggest four coalitions based on the actors’ policy claims on (1) equity, (2) assessment and accountability, (3) states have changed/passed legislation to align the state accountability systems with ESSA goals, and (4) the U.S. Department of Education’s state plan approval was inconsistent with the ESSA statutory provisions. The findings provide timely insights into the ongoing process of ESSA implementation at the federal, state, and local levels.

School Board Sensemaking of Federal and State Accountability Policies

This qualitative case study examines how board members make sense of federal accountability policies and how their sensemaking shapes their use of assessment data as a policy instrument. Deviating from previous work on practitioner sensemaking, the participants’ interpretations of assessments did not align with their ensuing use of the data. Furthermore, board members’ use of assessment data diverged from both federal and state messaging, illustrating board members’ synthesis and adaptation of external messaging into a locally driven narrative. As the nation has shifted to state accountability systems under Every Student Succeeds Act (ESSA), the findings provide insights to policymakers and practitioners to support local implementation.

National Council on Measurement in Education Position Statement on the Use of College Admissions Test Scores as Academic Indicators in State Accountability Systems

National Council on Measurement in Education Position Statement on the Use of College Admissions Test Scores as Academic Indicators in State Accountability Systems

An Argument in Search of Evidence: A Response to Camara et al. (2019)

AbstractThis article offers a critique of the validity argument put forward by Camara, Mattern, Croft, and Vispoel (2019) regarding the use of college‐admissions tests in high school assessment systems. We challenge their argument in two main ways. First, we illustrate why their argument fails to address broader issues related to consequences of using admissions tests as an achievement indicator in high school accountability systems. Second, we find the evidence put forth by Camara et al. argument incomplete and/or overstated, even if evaluated against a more limited set of claims. We offer several additional comments regarding the construction of their validity argument, the lack of transparency for evaluating many of the claims, and the potential unintended negative consequences associated with the use of the ACT or SAT as the primary achievement indicator in a state accountability system.

Student growth measures: What we’ve been missing

The move toward assessments that measure student growth, rather than just proficiency, has been perceived as an improvement in state accountability systems. However, Michael Watson explains that, for many students, these measures present an incomplete picture. Because they are based on grade-level assessments, any growth achieved by students who are above or below grade level is lost. He recommends the use of adaptive assessments that incorporate material from multiple grade levels.

Designing Accountability Assessments for Teaching.

A fundamental assumption of assessment and accountability programs is that teachers will attempt to adjust the content of their curricula to mirror the tests their students will take. In order to effect positive instructional change, these tests are supposed to conform, at least in part, to the desired curriculum’s learning targets. It is argued here that mandated tests, such as those administered through state accountability systems, can best meet the goal of curricular reform by making their domains of learning targets transparent to users. Test maps are proposed as an effective device to accomplish that goal. Following these suggestions would have little impact on how tests are developed or scored, but we believe they would have a large impact on how they are described and used, and ultimately on their effectiveness as agents of curriculum reform. Finally, we describe some suggestions about assessment program design and product development in order to create an information-rich classroom environment that capitalizes on the new domain descriptions.

Redefining or reinforcing accountability? An examination of meeting routines in schools

Accountability has been a major feature of educational policy making across the globe, including in the US where there is a persistent focus on student achievement results. This paper examines how accountability influences organizational routines in US schools, paying particular attention to meeting routines. We draw upon in-depth qualitative data gathered in four urban middle schools in which approximately 40 math teachers were engaged in collaboration and data use in order to improve instruction. Over a period of 3 years, we conducted extensive observations of teacher team meetings and interviews with teachers and administrators. An analysis of data reveals that many meetings were dominated by organizational routines that reflected state accountability systems, regardless of whether the topic was assessment, curricular pacing, or planning. At times, teachers and administrators made attempts to shift meeting routines from test-based accountability to a focus on instruction and professional accountability; however, existing routines endured. The findings from this study have important implications for thinking through whether and how the field may move towards more intelligent forms of accountability in the presence of external demands and deeply embedded patterns.

4-H at Work: Career Lessons Provide New Dimension to School Classrooms

Georgia 4-H has been partnering with public schools for over 100 years to provide youth development programming for elementary, middle, and high schools. In 2013, a new state accountability system required the addition of career exploration lessons in elementary schools. Georgia 4-H staff and faculty developed a career curriculum taught by county Extension agents in public school classrooms. This curriculum allowed Georgia 4-H to continue its historic partnership with Georgia public schools and assisted elementary schools in meeting new educational requirements. Evaluation data show that youth participants increased knowledge in respective career areas. State-standards-based 4-H career education delivered in classrooms and elsewhere can meet varied needs and yield valuable outcomes.

The Texas special education cap: Exploration into the statewide delay and denial of support to students with disabilities

State accountability systems have been a primary school reform initiative in the U.S. for the past twenty years, but often produce unintended negative consequences. In 2004, the Texas Education Agency (TEA) implemented the Performance Based Monitoring and Analysis System (PBMAS) which included an accountability indicator focused on the percentage of students found eligible for special education under the Individuals with Disabilities Education Act (IDEA), the nation’s special education law. From 2004 through 2016, the percentage of students found eligible for special education in Texas declined significantly, while the national rate held constant. Eventually, the U.S. Department of Education (ED) investigated TEA and the statewide implementation of IDEA. The purpose of this study is two-fold: (a) to evaluate the potential impact of the the PBMAS indicator on manipulation of special education identification practices; and (b) to describe how the indicator may have influenced school and district personnel. We highlight several concerning trends in state and district data and, through an analysis of publicly available reports from the ED, show how district and school personnel knowingly and unknowingly acted in ways that delayed and denied special education to potentially eligible students. We conclude with recommendations for TEA and implications for future research and policy.

Opportunity to Learn What Is on the Test and Performance on the Test

State accountability systems assume standards based instruction and test content are highly aligned and opportunities to learn the content exist equally for all students. This alignment between content taught and tested is important to understand achievement, yet it is rarely examined. Teachers from Grades 3 to 8 participated along with students without disabilities ( n = 116) and students with a disability (n =104) who received all mathematics instruction in their general education. Teachers recorded over 155 days of instructional information for mathematics and administered an interim mathematics test at the end of the year. We found an average of 44% of mathematics content standards were taught and tested, while 22% were not taught but tested. The results indicated students without disabilities did significantly better than students with disabilities on content taught and tested, but not so on content not taught but tested. Limitations and research needed conclude the article.

Trends in Performance and Growth by Students With and Without Disabilities on Five State Summative Assessments

ABSTRACT Indicators of student academic growth are desired in state accountability systems in order to approximate student learning over time and attribute observed growth to schooling inputs. Through an extant analysis of five states’ assessment data, this study offers evidence about whether longitudinal match rates and measures of growth differ at the state level for students with disabilities, relative to students without disabilities. There were three main findings: 1) In states in which a modified assessment was offered, students with disabilities were more likely to have missing prior year scores, and consequently missing growth scores; 2) Low scoring students, many of whom had a disability, were more likely to have missing prior scores on the state general assessment, and consequently missing growth scores; 3) Students with and without disabilities showed similar growth using transition and gain score definitions of growth, but students with disabilities had lower growth when estimated via a regression-based model. Measurement and policy considerations are discussed.

Principals’ Working Conditions, Job Stress, and Turnover Behaviors Under NCLB Accountability Pressure

Purpose: Many studies have investigated the impact of the No Child Left Behind (NCLB) Act and its sanctions on students and teachers. Little research, however, has systematically examined the relationship between NCLB and its sanctions and school principals. Given the importance of school leadership and the accountability systems implemented at different levels of government, there has been a need for research that examines how school leaders behave under the accountability pressure. Research Method: Using the Schools and Staffing Survey data, the Principal Follow-up Survey data, and detailed school-level adequate yearly progress (AYP) and assessment data systematically collected from 45 states, I investigate the association between NCLB sanctions and principals’ working conditions, job stress level, and turnover behaviors. I use a distance variable as a key matching variable in propensity score matching and estimate the association. Findings: I find strong evidence that NCLB sanctions are associated with a higher level of principals’ job stress and a higher turnover rate. These findings hold even after controlling for the influence of state accountability systems. I do not find evidence that these associations are moderated by principal or school characteristics. I also find little evidence that job stress is a mediator of the relationship between NCLB sanctions and principal turnover. Implications: Sanction-based accountability pressure appears to make the principalship more stressful and lead to turnover. Policy makers currently redesigning state and local accountability systems under the Every Student Succeeds Act may include support programs for school leaders to reduce their job stress level and retain them for a longer period of time.

Class Placements and Competing Priorities

Justine Finley, newly appointed principal of Northside Elementary, had been tasked with increasing her school’s grade under the state accountability system and keeping neighborhood families from choosing to enroll in the charter school down the street. The superintendent made these priorities very clear. After spending the first 3 months observing and analyzing data, Justine realized that a new model for placing students in classes could be a lever for increasing achievement scores. What Justine failed to realize was the conflict that the new model would generate among parents and teachers. This case highlights the challenges a new principal encountered when attempting to implement a change in student and teacher placement procedures amid competing priorities.

Evidence-Based Staffing in High Schools: Using Student Achievement Data in Teacher Hiring, Evaluation, and Assignment

ABSTRACTRecent research has demonstrated that elementary school leaders, under pressure to meet benchmarks set by state and federal governments, have begun “staffing to the test,” moving to tested grades and subjects teachers whose previous students made substantive learning gains. This is particularly true in lower-performing schools or schools that recently experienced a drop in performance, as measured by the grade they received from the state accountability system. What is not known, however, is how widespread this phenomenon has become. Moreover, little is known about whether and how performance data is used by administrators of high schools, where scheduling complexity is amplified by graduation requirements, larger enrollments, and credentialing requirements that mandate teachers to be certified in all the subjects that they teach. By surveying high school principals in Florida and Texas, this article attempts to fill this gap.

Writing Evaluation: Rater and Task Effects on the Reliability of Writing Scores for Children in Grades 3 and 4.

We examined how raters and tasks influence measurement error in writing evaluation and how many raters and tasks are needed to reach a desirable level of .90 and .80 reliabilities for children in Grades 3 and 4. A total of 211 children (102 boys) were administered three tasks in narrative and expository genres, respectively, and their written compositions were evaluated in widely used evaluation methods for developing writers: holistic scoring, productivity, and curriculum-based writing scores. Results showed that 54% and 52% of variance in narrative and expository compositions were attributable to true individual differences in writing. Students' scores varied largely by tasks (30.44% and 28.61% of variance), but not by raters. To reach the reliability of .90, multiple tasks and raters were needed, and for the reliability of .80, a single rater and multiple tasks were needed. These findings offer important implications about reliably evaluating children's writing skills, given that writing is typically evaluated by a single task and a single rater in classrooms and even in state accountability systems.