Educational Accountability Systems Research Articles

Assessing Learning Outcomes in Higher Education: From Practice to Systematization

Accountability systems in higher education ensure that academic programs meet learning outcomes and address student cognitive and affective development. The learning outcome assessment is one of the most important elements to enhance and assure the quality of programs. In this paper, we present a “tour” of how an assessment cycle is performed in a national/public university in Qatar. The study draws on “good” practices and how a higher education institution systemizes the process of assessing learning. The approach demonstrates how learning is connected to program objectives and the curriculum through a rational, logical, conceptual, consensual, and interpretable process. Employing specification tables known as curriculum and assessment plan matrices, with tools being assessments and rubrics, learning outcomes measures can be reached. The approach as well as system calls for an institutional structure and organization which place responsibilities on all stakeholders involved in academic programs delivery. The futuristic approach and practice posit a reconceptualization of how students will be assessed, more likely a paradigm shift necessitating a change in practice and trend in which students’ success (achievement) can be based on the learning outcomes attained.

Evaluation of a Muslim Pre-School in the United States Using Elements of the Ofsted Model

This article reviewed the Ofsted inspection model in England and reported on its implementation in a Muslim pre-school located in an American southeastern state. The inspection methods included classroom observations, interviews with staff, parent survey, and scrutiny of artifacts. After the pre-school was inspected and the feedback was given to the administration orally and in a detailed written report, an improvement plan was developed by the author to address the inspection recommendations. The results of the inspection showed that implementing elements of the Ofsted inspection model in the pre-school contributed to overall educational improvement. A follow-up communication with the administration showed that curriculum changes were made, teachers were provided with more professional development opportunities, and their teaching practices had improved. Considering these findings, this study demonstrated that there was an alignment between the mission of the pre-school and the parents’ aspirations; and that an educational accountability system such as the Ofsted inspection model have had a positive impact on the educational provision even though not all elements of the inspection model were implemented, many of which require a team of inspectors and a quality assurance mechanism.

Can a school climate survey accurately and equitably measure school quality? Examining the multilevel structure and invariance of the Georgia School Climate Scale

School climate measures are increasingly utilized as one indicator of school quality within educational accountability systems. However, concerns have been raised about the accuracy of these indicators given that school climate surveys are often not validated using multilevel methods. Further, cross-school comparisons in climate may not be trustworthy because the school-level invariance of climate surveys has not been investigated. There is a need to examine the validity of school-level climate constructs and to determine if surveys measure climate equitably for schools that serve underrepresented populations. The aim of the current study was to examine the multilevel factor structure of a statewide school climate survey to determine whether it measured climate equitably for students of different races/ethnicities and across schools with varying racial/ethnic and socioeconomic compositions. Participants included 259,778 students from 427 middle schools throughout a southeastern U.S. state. Cross-level invariance analyses revealed that the climate constructs were measured differently across levels of analysis, and school-level climate could not be interpreted as merely the aggregate of individual-level climate. Student- and school-level factorial invariance was tested using multilevel modeling procedures. Results revealed item bias with respect to student and school characteristics, and the relationships between school climate and student and school demographics changed after accounting for identified bias. As more educational agencies consider including school climate surveys in their accountability systems, these findings suggest that multilevel validation procedures and school-level invariance analyses are necessary to ensure accurate and equitable measurement.

Student mobility: Extent, impacts and predictors of a range of movement types for secondary school students in England

AbstractMoving school is a major event for students, with potential impacts on both student and school performance. Students can experience a diversity of move types, including variation in timing, origin and destination, though this complexity is not always acknowledged in studies or educational policies, which tend towards binary distinctions of movers versus non‐movers. However, it is crucial to understand the vulnerabilities and outcomes associated with different mobility types, including to inform more effective and efficient targeting of support. We therefore study the extent, impacts and predictors of a diverse range of school moves during secondary schooling for a cohort of students completing their end‐of‐school examinations in 2018/2019 in England. We highlight a negative association between mobility and achievement for all movement types, however, we find substantial heterogeneity in the strength of their association with student performance and highlight the likelihood of making different types of school moves vary by student characteristics. Therefore, we recommend the wider consideration and examination of diverse mobility features by research and in educational accountability systems.

The role of parents in the school accountability system: Insights from a Hong Kong case

Parents, as a group, are important stakeholders that can influence education policy, as evidenced by the 2015−16 public opposition to a test-based school accountability measure in Hong Kong, which forced the government to remove the high stakes attached to test results. Parents and their role in education accountability systems, however, remain under-researched. This study analysed interviews with 42 parents to examine how they interpreted the Hong Kong Territory-wide System Assessment (TSA) and its functions. One particular area of theoretical interest was understanding what triggered parents to protest against a policy that seemingly protects their interests, and determining whether the alliance between market and bureaucracy, or between neoliberalism and neo-conservatism, has been weakened in some way. The research, however, found the power alliance remained strong. Three distinctive patterns of parental attitudes were identified, indicating a rather loose coalition among parents, the majority of whom endorsed the policy’s intentions and only a small portion was critical of its consequences and ethics. Findings are discussed in terms of parents’ role in the theoretical accountability triangle and the importance of educators’ professional accountability for transforming the policy rhetoric of assessment for learning into classroom reality. Implications are discussed for Hong Kong and beyond.

In Praise of Ordinary Measures: The Present Limits and Future Possibilities of Educational Accountability

AbstractState and federal policymakers “see” school performance via formal measures — data collected with attendance sheets and standardized tests. Such an approach, though not without its merits, is extremely limited and inherently exposed to the threat of systematic misperception and unintended consequences, especially as policymakers try to use data to leverage on‐the‐ground change. In this essay, Jack Schneider and Derek Gottlieb discuss the limitations of present accountability systems and advocate for the inclusion of what they call “ordinary measures.” Long positioned as inferior to their formal counterparts, such measures offer much to clarify the picture of schools that good governance depends on. According to Schneider and Gottlieb, using ordinary measures, paired with deliberative evaluation processes, will improve the validity and utility of educational accountability systems.

The Effect of Results-Based Intergovernmental Transfers in the Educational Accountability System: An Examination of the Race to the Top Program

Results-based intergovernmental transfers have recently attracted attention from both scholars and practitioners. In contrast to other types of grants, the results-based approach can lead to improved accountability and responsive performances on the part of subnational governments while still enabling broad local flexibility. In education, the principles of results-based transfers fit in with the mechanism of the educational accountability system. A case in point is the Race to the Top (RttT) program, which aimed to support large-scale educational reform. This study examines whether the RttT program achieved its goal of improving student achievement in Florida. A difference-in-differences comparison shows some positive effects of the RttT program when considering spillover effects or comparing Florida with a state that did not receive this fund. The overall estimation results, however, do not support the view that the program has had a significant positive impact on educational achievements. This finding may result from the lack of a clear and singular focus on the specified outputs or a lack of citizen-based accountability.

Cross-Classified Random Effects Modeling for Moderated Item Calibration

In the research reported here, we propose a new method for scale alignment and test scoring in the context of supporting students with disabilities. In educational assessment, students from these special populations take modified tests because of a demonstrated disability that requires more assistance than standard testing accommodation. Updated federal education legislation and guidance require that these students be assessed and included in state education accountability systems, and their achievement reported with respect to the same rigorous content and achievement standards that the state adopted. Routine item calibration and linking methods are not feasible because the size of these special populations tends to be small. We develop a unified cross-classified random effects model that utilizes item response data from the general population as well as judge-provided data from subject matter experts in order to obtain revised item parameter estimates for use in scoring modified tests. We extend the Metropolis–Hastings Robbins–Monro algorithm to estimate the parameters of this model. The proposed method is applied to Braille test forms in a large operational multistate English language proficiency assessment program. Our work not only allows a broader range of modifications that is routinely considered in large-scale educational assessments but also directly incorporates the input from subject matter experts who work directly with the students needing support. Their structured and informed feedback deserves more attention from the psychometric community.

Toward a Framework for Public Accountability in Education Reform

Public accountability through information disclosure is a pillar of modern education reform efforts. Despite the ubiquity of this approach, we argue that public accountability in education is undertheorized and often predictably unlikely to achieve its intended policy goals. Drawing on examples from an equity-oriented court case in California and the literatures on democratic engagement and parent use of school performance data, we propose a framework for thinking about the design of public accountability systems in education. The framework could provide guidance for policymakers considering new efforts at improving schools through the production and dissemination of educational data.

Recommendations for Developing and Implementing State-level Physical Education Accountability Systems in Student Learning

With the introduction of the Every Student Succeeds Act (ESSA) in 2015, physical education is now considered a part of a “well-rounded education.” One of the critical features of ESSA is the growing expectation for states to establish accountability systems that monitor student learning. In 2018, a group of researchers conducted a study to examine the status of state-level accountability systems in physical education. Based on the findings, critical elements for robust accountability systems of student learning were identified. The purpose of this article is to discuss recommendations for policymakers and school leaders to develop and implement robust state-level accountability systems to inform policy, standards, curriculum, instruction, and assessment in physical education. To better advocate for the societal value of physical education, robust accountability systems must become the norm, and student learning should be evident.

Higher Education Accountability, Robert Kelchen, Johns Hopkins University Press, Baltimore, 2018. 272 pp. ISBN: 978‐1‐42142‐473‐6. $39.95 (hardcover). eISBN: 978‐1‐42142‐474‐3

In his book Higher Education Accountability, Robert Kelchen comprehensively discusses the concept of accountability in nine chapters in a pleasant and convincing way. He aims to cover the landscape of key accountability systems in US Higher Education by discussing the theoretical underpinnings of accountability, their evolution, and the perspectives from and interactions of different stakeholders at different levels in the Higher Education system. The last two chapters are reflective in nature by presenting ten lessons on accountability and by speculating about what the future might bring.

Common standards, different stakes: A comparative and multi-scalar analysis of accountability reforms in the Spanish education context

An increasing number of countries are adopting accountability systems in education that rely on the external evaluation of students’ learning outcomes through standardized assessments. The international dissemination of this form of accountability, often known as test-based accountability, does not imply that exactly the same policy is adopted everywhere. Accountability reforms, as any other globalizing policy model, are context-specific. The concrete form that accountability reforms adopt is contingent on a range of political, historical and institutional conditions, and to policy-making dynamics and logics that operate at multiple scales. This paper analyzes the trajectory of accountability reforms in two Spanish regions, Madrid and Catalonia, from a comparative and multi-scalar perspective. Based on document analysis of media and official sources, and exploratory interviews with key informants, the paper shows that, although these two regions have pioneered the adoption of test-based accountability reforms in the Spanish context, their accountability systems have evolved quite differently. While accountability reforms in Madrid have been oriented toward the promotion of school choice and competition, Catalonia has adopted an uneven lower-stakes accountability approach with multiple ramifications. In this paper, we explain how and why such diverging trends have been possible within the context of a common general regulatory framework.

Cultural Capital Efficacy in Parental and Student Aligned Expectations for Postsecondary Matriculation

Cultural Capital Efficacy in Parental and Student Aligned Expectations for Postsecondary Matriculation Dr. Kennetra Bryant; Dr. Ronnie Davis; Dr. Teryana Lamb; Mr. Orlando Milton, Jr; Dr. Joyce Montgomery; Dr. Dominic Thomas Abstract The purpose of this study was to investigate the predictability and relationship of selected student-related and institutional factors on the graduation rates of community colleges in multiple regions in the largest state in the Southern Region of the United States. The selected total population for this study included public twoyear community colleges within six community college regions in the largest state in the Southern Region of the United States. The target state’s Higher Education Accountability System (Community College Success, Participation, Institutional Efficiency and Effectiveness reports were used to collect the data. The data revealed that the student-related and institutional factors of postgraduate employment, receipt of financial assistance, number of full-time faculty, number of part-time faculty and size of institution collectively accounted for 77% of the variance of graduation rates of community colleges. Full Text: PDF DOI: 10.15640/jehd.v5n2a4

Measuring what really matters

Since the passage of No Child Left Behind, large-scale assessments have come to play a central role in federal and state education accountability systems. Teachers and parents have expressed a number of concerns about their state testing programs, such as too much time devoted to testing and the high-stakes use of testing for teacher evaluation. We don't dispute that federal and state testing and accountability policies have been problematic or that there are ethical issues with high-stakes uses of a single measure. But rather than doing away with state tests, we should improve them so they do measure what matters, and so that what matters gets taught in our public schools. In short, we need a better system of assessments.

A feel for numbers: affect, data and education policy

This article explores the relationship between commensuration and affect in various contexts of education policy. Commensuration is the process through which disparate qualities are transformed into a common metric and is central to the production of performance data. The rise of governance through numbers in education has resulted in a proliferation of performance data, comparisons and rankings that influence political debate and policymaking. The efficacy of data as a governance mechanism depends on their usage to shift perceptions of performance, and this involves both conscious interpretation and affective sense-making of data and their representation in multiple forms. For example, performance data used within accountability systems in education are linked to sanctions and rewards, and their effects are partially due to the feelings that are provoked. The relationship between affect and data is also important in the mobility of policy ideas, which spread via meetings that enable affective proximity between participants. This article draws on a philosophical concept of affect, defined as the feeling of transition in bodily states, and topological concepts that are being taken up in social theory, to consider new perspectives on education policy research that these thinking tools may afford.

All that is global is not world culture: accountability systems and educational apparatuses

This article explores why we see educational accountability systems circulating transnationally. It argues that researchers in the field of comparative and international education need to use the concepts of diffusion and translation to think about the formation, coordination and extension of networks and discursive formations through which heterogeneous, disparate objects are brought into relation. Approaching accountability in education as an ‘apparatus’ helps us engage with the research challenges presented by globalisation. This article proposes a way of seeing accountability as constitutive of the global and not as an after-effect. This approach helps us avoid the distracting and ultimately irrelevant fixation on a so-called ‘global/local nexus’ that is characteristic of much work in the field of comparative and international education. It also aims to improve on world culture theory explanations for why we are presently witnessing a global trend towards the increased ‘monitoring of monitoring’, i.e., increased self-organising reflexivity in the self-description and self-observation that school systems are called to engage in.

Medical Adaptation to Academic Pressure

Despite the rise of medical interventions to address behavioral issues in childhood, the social determinants of their use remain poorly understood. By analyzing a dataset that includes the majority of prescriptions written for stimulants in the United States, we find a substantial effect of schooling on stimulant use. In middle and high school, adolescents are roughly 30 percent more likely to have a stimulant prescription filled during the school year than during the summer. Socioeconomically advantaged children are more likely than their less advantaged peers to selectively use stimulants only during the academic year. These differences persist when we compare higher and lower socioeconomic status children seeing the same doctors. We link these responses to academic pressure by exploiting variation between states in educational accountability system stringency. We find the largest differences in school year versus summer stimulant use in states with more accountability pressure. School-based selective stimulant use is most common among economically advantaged children living in states with strict accountability policies. Our study uncovers a new pathway through which medical interventions may act as a resource for higher socioeconomic status families to transmit educational advantages to their children, either intentionally or unwittingly.

Educational accountability system in local government: Current conditions and impact

Educational accountability system in local government: Current conditions and impact

Let's Measure What No One Teaches: PISA, NCLB, and the Shrinking Aims of Education

Background/Context PISA has come up with an ingenious solution to the problem of how to measure student achievement across national school systems with different curricula. Instead of measuring how well students learn what they are taught in each system, it measures a set of economically useful skills that no one teaches. Purpose The aim is to figure out how this odd situation came about in the current global policy context. Research Design This paper explores PISA as one type of educational accountability system, based on how well students demonstrate mastery of particular cognitive skills, and compares it with the current state-level accountability systems in the U.S. (NCLB), which are based on how well students demonstrate mastery of the formal curriculum. Conclusions/Recommendations Both PISA and NCLB, I argue, are cases of how we are shrinking the aims of education. One approach focuses on mastery of skills that are relevant but not taught and the other on mastery of content that is taught but not relevant. Neither seems a sensible basis for understanding the quality of schooling or for making educational policy.

Evaluating Growth for ELL Students: Implications for Accountability Policies

In recent years, many U.S. states have introduced growth models as part of their educational accountability systems. Although the validity of growth‐based accountability models has been evaluated for the general population, the impact of those models for English language learner (ELL) students, a growing segment of the student population, has not received sufficient attention. We evaluated three commonly used growth models: value tables or transition matrices, projection models, and student growth percentiles (SGP). The value table model identified more ELL students as on track to proficiency, but with lower accuracy for ELL students. The projection and SGP models were more accurate overall, but classified the fewest ELL students as on track and were less likely to identify ELL students who would later be proficient. We found that each model had significant trade‐offs in terms of the decisions made for ELL students. These findings should be replicated in additional state contexts and considered in the development of future growth‐based accountability policies.