National Large-scale Assessment Research Articles

Confronting the disadvantage gap: the challenges to transformative leadership in a high-stakes assessment system

ABSTRACT Students with low socio-economic status (SES) often have lower levels of academic achievement. In England, various UK governments have sought to address this disadvantage gap through reforms to national large-scale assessments, school accountability mechanisms, and educational governance structures. While scholarly research highlights the important mediatory role of educational leaders in policy implementation and student performance, typologies of successful leadership tend to emphasise ‘what works’ at the organisational level. Conversely, theories of transformative leadership acknowledge that efforts to provide an equitable education are related to, and should address how to transform, wider social power structures. With an abductive methodology, this qualitative study explores the experiences and perspectives of three academy headteachers in their work with the most disadvantaged students in their school communities. Semi-structured interview data were analysed thematically to evaluate whether and how their leadership practices might be considered transformative. The findings suggest that these headteachers are mostly transformative at the individual level which aligns with the current UK government social mobility agenda. Moreover, despite their courage to promote equity of opportunity, transformative leadership is restricted by government cuts to public expenditure, a high-stakes accountability system focused on students’ academic performance, and the limited political capital of students and headteachers.

Life Course in the Making

Abstract: TREE (Transitions from Education to Employment) is a prospective interdisciplinary multi-cohort panel survey following up on the (post-compulsory) education and employment trajectories of two large samples of Swiss compulsory school leavers. The first TREE cohort (TREE1) was launched in 2000, drawing on the sample tested on the occasion of Switzerland’s first-time participation in PISA ( Nt0 = 6,343, Nt10 in 2020 ≈ 3,900). Since then, the sample has been followed up by means of 10-panel waves, the most recent one conducted in 2019/20. Further panel waves are planned at 5 years intervals. To date, TREE1 respondents have reached an average age approaching 40 and have been surveyed for a period of over 20 years, spanning from early adolescence up to early middle age. Under a replication design allowing for cohort comparison, the second TREE cohort (TREE2) covers a comparable population of school leavers who left compulsory education in 2016. As its baseline survey, it draws on a national large-scale assessment of mathematics skills. Since then, the TREE2 sample ( Nt0 = 8,429, Nt6 in 2022 ≈ 4,500) has been re-surveyed six times at yearly intervals, up to the average age of 21. Further panel waves at 2–5 years intervals are planned. The present contribution includes a detailed description of TREE’s study and survey design as well as a synoptic summary of salient results from some of the several hundred publications that draw on the TREE data.

Teachers’ perceptions of national large-scale assessment: the pedagogical dimension

ABSTRACT Large-scale assessments have become a basic national policy for educational improvement encouraging standards, decentralisation and school accountability. The current study focuses on the pedagogical dimension of large-scale assessments, examining its uses as a policy instrument for effecting pedagogical change. The paper presents and discusses the case of the Israeli NLSA (national large-scale assessment) regime – the Meitzav (Hebrew acronym for: Growth and Effectiveness Measures for Schools) tests. Although it aimed to design a low-stakes testing regime, its implementation was a top-down procedure which, in practice, restricts principals’ and teachers’ autonomy. Using qualitative and quantitative methods, the findings showed that the Meitzav test results are barely used as a means leading to pedagogical change to improve learning. Teachers considered these tests an unreliable assessment tool that in the main, does not reflect the school curriculum or student learning, while producing a high level of pressure on the teaching routine. In consequence of the Meitzav test results, the most common pedagogical change in practice chosen by teachers was: “teaching to the test”. Other pedagogical changes following the Meitzav were implemented to a minor extent. Policy implications are discussed.

Taming the time zone: National large-scale assessments as instruments of time in the Russian Federation

Nationwide large-scale assessments (NLSA)—an example of cross-border policy mobility—manifest a proliferating means of governing formal schooling. In the Russian context, NLSA takes the form of a compulsory graduation examination called the Unified State Examinations (USE). In this article, we explore how a mobile policy instrument of the NLSA participates in the relational processes of time- and space-making in a particular federated context of Russia, and how this process intertwines with and is shaped by the presence of multiple time zones. We argue that NLSA is an instrument of time that attempts to achieve centralization of the complex federated structure of the Russian Federation. Yet, the work of the NLSA is not a smooth process in a country characterized by territorial vastness, a complex federated structure, and the existence of multiple time zones. Guided by the theory of logistical power and sociological perspectives on time, as well as empirical insights, we show how the time zones need to be tamed in order for the NLSA to exercise its centralizing role. Discursively, the time zone is introduced and publicly discussed to symbolically characterize Russia and justify political actions or their outcomes. Bureaucratically, the desire for simultaneity and synchronicity takes the form of a meticulous ordering of a sequence of actions through prescriptive documentation that regulates the NLSA. Technologically, synchronicity, simultaneity, and instantaneousness rely on and engender an expanding national infrastructure that mediates social relations and the processes of conducting the NLSA, cutting across the time zones and federal units. Based on this analysis, we propose that scholarship on policy mobility and education policy sociology at large could benefit from examining the relationship between time and education policy and governance in four intertwined ways: the time of policy, context as time, policy instruments as instruments of time, and time in policy instruments.

Italian teachers’ conceptions of large-scale assessment: what are the implications for assessment literacy?

ABSTRACT Over recent decades, a great amount of attention has been globally directed to school achievement evidenced by large-scale assessment programmes. Additionally, there has been an increase in expectations regarding teachers’ use of data. Against this backdrop, there arose a clear demand for assessment-literate teachers, mapping out the changes that teachers need in practice in order to ensure sound instructional decision-making and enable better learning for students. Despite the current educational policy and trends in research, teachers in Italy tend to avoid using large-scale assessment results and continue to perceive this assessment as inappropriate and ineffective. To shed light on the reasons for teachers’ negative conceptions of the national large-scale assessment programme, a qualitative study was conducted drawing data from semi-structured interviews with 70 teachers. The discussion on limitations of and issues encountered in large-scale assessments addresses policy implications for teachers’ assessment literacy, as well as for their professional development.

Teachers' pedagogical implementation of the National Large-Scale Assessment (NLSA): improving and advancing teaching-learning processes

PurposeLarge-scale assessment has been used in many education systems as an instrument to evaluate educational performance nationally. This practice is based on the concept of epistemic governance which encourages school accountability. This study aims to explore teachers' perspectives regarding the value and uses of national large-scale assessment (NLSA), highlighting its relevance across contexts.Design/methodology/approachUsing qualitative data, this paper presents the case of the Israeli NLSA tests – the Meitzav, while examining the perceptions and actions in which teachers engage to follow-up on the test results, and the extent to which they implement pedagogical change in light of the test results.FindingsThe findings showed that teachers tend to use the NLSA test results as a pedagogical tool to improve learning processes to a limited extent. They concede that most activity involving the tests at the school and class levels is dedicated to preparation and not to pedagogical change. Some explanations are suggested.Originality/valueThis paper discusses the theoretical and practical implications of the NLSA testing regime for the school, curriculum and pedagogy.

Governing education in times of crisis: State interventions and school accountabilities during the COVID-19 pandemic

Strategic government interventions in public education have shifted and blurred the boundaries between state, market and civil society modes of governance. Within this matrix of interdependent relations, schools operate under increasingly hybrid accountability arrangements in which public accountability can both complement and compete with market and social regimes and their associated institutional logics, goals, values and mechanisms. During the first wave of the COVID-19 pandemic, however, national governments implemented a wide range of emergency measures which had consequences for the mixes and layers of school accountabilities. This article examines the principal policy changes in Denmark, England and Italy. Drawing on state theories and the concept of ‘hybrid accountability’, semi-structured interviews with national and local policymakers and school practitioners were analysed thematically. While cultural nuances exist between the cases, our findings reveal that state interventions reinforce a public–professional accountability hybrid and hierarchies of control and command within and outside networks. Concomitantly, state non-interventions and the distinct underlying institutional logics associated with national large-scale assessments suggest policy inertia with implications for professional accountability and institutionalised change. Future research might investigate whether educators’ experiences influence the direction of national and local accountability policy reforms in a post-pandemic era.

Enacting performance-based accountability in a Southern European school system: between administrative and market logics

Performance-based accountability (PBA) policies are increasingly adopted in a wide range of education systems in order to reform school governance and to improve students’ results and schools’ performance. Countries around the world have been implementing national large-scale assessments to make school actors more accountable and responsible for students’ results. This policy model has been generalized in countries with different administrative traditions, including those with a short tradition in New Public Management. This is the case in Spain, where PBA has been adopted unevenly in different regions, with Madrid being one of the earliest adopters. In recent decades, Madrid has developed a model that combines administrative test-based accountability with a system of broad parental school choice, which also facilitates the activation of market forms of accountability. However, the combination and interaction between market and administrative forms of accountability is understudied. This paper adopts a policy enactment perspective to analyze, through a case study approach, the interaction of administrative and market forms of accountability and its enactment at the school level. The case study is based on a set of 41 semi-structured interviews with teachers, principals, and school inspectors in a sample of eight schools in Madrid, combined with document analysis of school educational projects and improvement plans. The evidence suggests that administrative and market forms of accountability tend to generate dynamics of interdependence, resulting in increasing external pressures which schools tend to address with superficial responses, including teaching to the test, or second-order competition between schools.

The Role of Policy Legacies in the Alternative Trajectories of Test-Based Accountability

National large-scale assessments (NLSA) have spread rapidly since the 1990s, but contrary to arguments of diffusion theories, not every country with large-scale assessments develop test-based accou...

When didactics meet data science: process data analysis in large-scale mathematics assessment in France

During this digital era, France, like many other countries, is undergoing a transition from paper-based assessments to digital assessments in education. There is a rising interest in technology-enhanced items which offer innovative ways to assess traditional competencies, as well as addressing problem solving skills, specifically in mathematics. The rich log data captured by these items allows insight into how students approach the problem and their process strategies. Educational data mining is an emerging discipline developing methods suited for exploring the unique and increasingly large-scale data that come from such settings. Data-driven methods can be helpful when trying to make sense of process data. However, studies have shown that didactically meaningful findings are most likely generated when data mining techniques are guided by theoretical principles on subjects’ skills. In this study, theoretical didactical grounding has been essential for developing and describing interactive mathematical tasks as well as defining and identifying strategic behaviors from the log data. Interactive instruments from France’s national large-scale assessment in mathematics have been pilot tested in May 2017. Feature engineering and classical machine learning analysis were then applied to the process data of one specific technology-enhanced item. Supervised learning was implemented to determine the model’s predictive power of students’ achievement and estimate the weight of the variables in the prediction. Unsupervised learning aimed at clustering the samples. The obtained clusters are interpreted by the mean values of the important features. Both the analytical model and the clusters enable us to identify among students two conceptual approaches that can be interpreted in theoretically meaningful ways. If there are limitations to relying on log data analysis in order to determine learning profiles, one of them is the fact that this information remains partial when it comes to describing the complete cognitive activity at play, the potential of technology-enriched problem solving situations in large-scale assessments is nevertheless obvious. The type of findings this study produced is actionable from teachers’ perspective in order to address students’ specific needs.

Measuring mathematics competence in international and national large scale assessments: Linking PISA and the national educational panel study in Germany

This study examines a linkage between the international mathematics scale of the Programme for International Student Assessment (PISA 2012) and the mathematics assessment taken from the German National Educational Panel Study (NEPS). The linking was realized by a separate linking study that uses a single group design. The sample consists of n = 1,270 9th graders from 78 German secondary schools. The equipercentile linking leads to close descriptive scale characteristics (means, standard deviation, and skewness) between the original PISA mathematics scale and the PISA score equivalents. The linking was stable over the four subgroups (gender, migration background, books at home, and school type). Altogether, the results indicate that assigning students to PISA proficiency levels given their NEPS mathematics test score, the PISA score equivalents produce a similar distribution of students reaching the PISA proficiency levels at a group level.

Keyword: Reading literacy. Reading competencies in Germany and underlying cognitive skills

There is a general consensus that reading competencies, being a core element of basic education, are an essential prerequisite for the acquisition, organization, and application of knowledge in various subjects and domains. Thus, reading competencies represent a key factor for educational and professional success, lifelong learning, and social participation. In the first part of this article, we summarize the main findings of recent international and national large-scale assessment studies suggesting that a significant proportion of children, adolescents, and adults in Germany do not attain a sufficient level of reading competence to understand and learn from texts. Because there is evidence that a share of these poor readers may have limited basic reading abilities that underlie text comprehension, in the second part of this article we review meta-analyses on the relations between cognitive skills and reading competence. Finally, we discuss challenges for school and adult education, and mention possible perspectives for future research.

Reforming governance through policy instruments: how and to what extent standards, tests and accountability in education spread worldwide

ABSTRACTIn the last decades, most countries have adopted data-intensive policy instruments aimed at modernizing the governance of education systems, and strengthening their competitiveness. Instruments such as national large-scale assessments and test-based accountabilities have disseminated widely, to the point that they are being enacted in countries with very different administrative traditions and levels of economic development. Nonetheless, comparative research on the trajectories that governance instruments follow in different institutional and socio-economic contexts is still scarce. On the basis of a systematic literature review (n = 158), this paper enquires into the scope and modalities of educational governance change that national large-scale assessments and test-based accountability instruments have triggered in a broad range of institutional settings. The paper shows that, internationally, educational governance reforms advance through path-dependent and contingent processes of policy instrumentation that are markedly conditioned by prevailing politico-administrative regimes. The paper also reflects on the additive and evolving nature of educational governance reforms.

The growth and spread of large-scale assessments and test-based accountabilities: a political sociology of global education reforms

ABSTRACTThe Global Education Reform Movement (GERM) is expanding internationally and reaching countries that seemed to be immune to this education reform approach until quite recently. Accordingly, more and more educational systems in the world are articulated around three main policy principles: accountability, standards and decentralisation. National large-scale assessments (NLSAs) are a core component of the GERM; these assessments are increasingly used for accountability purposes as well as to ensure that schools achieve and promote centrally defined and evaluable learning standards. In this paper, we explore these trends on the basis of a new and original database on NLSAs, as well as on data coming from the Programme for International Student Assessment (PISA) questionnaires. In the paper we also discuss how different theories on policy dissemination/globalisation explain the international spread of NLSAs and test-based accountability worldwide, and reflect on the potential of a political sociology approach to analyse this globalising phenomenon.

Iranian university students lifestyle and health status survey: study profile

BackgroundThe physical health assessment of university students in Iran is a national large scale assessment examining health behaviors among tertiary education students. Understanding risky health behaviors which are the major sources of global mortality and morbidity in adulthood is the key objective of this assessment.MethodsIn academic year of 2012–2013, newly admitted students (N = 151,671) at 74 governmental eligible universities that had health center from 28 provinces were invited to participate in the health assessment program. The physical health behaviors of the students were evaluated by using questionnaire. The test-retest reliability method was applied to estimate the reliability of physical health questionnaire. After filling out the questionnaires, students were led to the examination room for the measurement of height, weight and blood pressure.ResultsFrom the total study population, 84,298 student’s ages between 18 and 29 years old, were participated in the health assessment. The mean response proportion was 63%. The mean age of students was 21.5 ± 4.01, with 49.20% percent being <20 years old. 32.31% were between 20 and 24 years, 13.44% between 25 and 29 years, 69% of the participants were undergraduate 34.9% were master’s students, and 2.9% were Ph.D. students. The mean BMI for total students was 22.5 ± 4.0 and regarding to gender, the mean BMI for male and female were 23.0 ± 4.1 and 22.2 ± 3.8 respectively.ConclusionAnalysis of student’s findings will generate multiple studies which report different aspects of physical health of Iranian university students who constitute a large proportion of young adult aged 18–29 years in the country. This assessment also provides opportunity to compare Iranian student’s behavioral patterns with the behavioral pattern of students worldwide.

Diagnosing Competency Mastery in Science: An Application of GDM to TIMSS 2011 Data

ABSTRACTNumerous diagnostic studies have been conducted on large-scale assessments to illustrate the students’ mastery profile in the areas of math and reading; however, for science a limited number of investigations are reported. This study investigated Iranian eighth graders’ competency mastery of science and examined the utility of the General Diagnostic Model (GDM) to produce diagnostic information using TIMSS 2011 data. Eight diagnostic attributes were extracted, using documentary analysis of the major large-scale assessment frameworks, including basic science knowledge, using models, reasoning, using science, representing data, explaining of phenomena, predicting, and scientific inquiry. These attributes were then assigned to each item in order to construct the Q matrix, through focus group discussions, survey of head science teachers, think-aloud verbal protocol, and analysis of written answers. Results show the utility of GDM to generate rich diagnostic information for a national large-scale assessment. Moreover, the findings indicated that students performed relatively well in using science, but performed weakly in reasoning, explaining of phenomena, and scientific inquiry, which all required complex skills and higher-order thinking abilities.

Determinants of Academic Achievement of Middle Schoolers in Turkey

(ProQuest: ... denotes formulae omitted.)While education provides individuals an opportunity for cognitive, social and emotional development, maintains and creates social stratification (Montt, 2011). Socioeconomic factors, along with school- and community-related factors, cause inequalities in education. For instance, gender (being female) and wealth (being poor) are major obstacles to school enrollment and achievement (Filmer, 2005; Nguyen, 2006). In some countries, females still do not have the same opportunities as males. Females do not have equal access to education and have lower academic achievement than their male peers (Nguyen, 2006). The literature has pointed to socioeconomic-related educational inequalities in underdeveloped (Grimm, 2011) and developing countries (Martins & Veiga, 2010). Students who grow up in families of low status (SES), specifically, those whose parents have low levels of education, low incomes, or low-prestige occupations, generally show slower cognitive development than students whose parents have high SES (Gamboa & Waltenberg, 2012; Hertzman, 1994; Hertzman & Weins, 1996). This can be explained with Bourdieu's cultural capital reproduction theory (1986), which holds that socioeconomic inequalities in education persist because highly educated parents give their children a better understanding of the dominant culture and an ability to act within it (p. 1017, as cited in Martins & Vegia, 2010). Identifying the sources of inequalities in educational attainment and achievement and, reducing their effects are major concerns of educational researchers and policymakers worldwide.School variables such as school culture, resources (e.g., books, teacher-student ratio), and the composition of a school can create learning situation that exacerbate imbalances in student achievement across schools (Baker, Goesling, & Letendre, 2002; Thrupp, Lauder, & Rabinson, 2002). The Equality of Educational Opportunity Study (Coleman et al., 1966), also known as the Coleman Study, surprised many researchers and policymakers, not only in the United States but throughout the world; showed that, rather than the school itself, was the and ethnic background of families that constituted the source of variation in achievement. Since then, much research has been conducted to verify whether these findings hold true in other countries as well. Some studies conclude that the impact of schools on student achievement is small in wealthy countries but relatively strong in poorer countries (Buchmann, 2002; Fuller & Clarke, 1994; Heyneman & Loxley, 1983). Other findings, however, are consistent with those of the Coleman Study (Baker et al., 2002).Beyond the acknowledged effect of family factors on school participation, the influence of school- and community-related factors on student achievement remains essentially unexplored. Binder (1999) and numerous other researchers have drawn attention to this gap in the literature. Few studies have been conducted in Turkey to examine school and family effects on school outcomes such as achievement on national large-scale assessment and the Program for International Student Assessment (PISA) (Alacaci & Erbac, 2010; Dincer & Uysal, 2010; Guncer & Kose, 1993; Tomul & Savacci, 2010). Existing Turkish studies have their own methodological limitations. For instance, they used data collected by the Program for International Student Assessment (PISA), which raises concerns about the validity of test results, sample coverage and representativeness (Ferreira & Gignoux, 2011). Pisa Sampling coverage rate is below 50% and no information about non-participant students is available (Carvalho, Gamboa, & Waltenberg, 2012).The purpose of this study is to explore student and school factors that contribute to inequalities in seventh- and eighth-grade student achievement in the Turkish context by using national achievement test scores. …

Principals and teachers: Different perceptions of large-scale assessment

This study examined whether differences exist between principals’ and teachers’ perceptions of the goals of a large-scale assessment, and whether they view it as formative or summative. Interviews were held with 30 principals and 92 teachers. A difference was found between their perceptions of the goals and the national large-scale assessment goals. The main goal is perceived to be student achievements, and not a tool for improving the schools’ work as declared by the Ministry of Education. Principals perceive the assessment mainly as formative, whereas teachers perceive it mainly as summative. Teachers who hold administrative roles perceive some of the goals as formative assessment, as do principals, whereas other teachers perceive it as summative.

Evolving data use policy in Trinidad and Tobago: the search for actionable knowledge on educational improvement in a small island developing state

This paper presents a single-country case study of the use of large scale assessment (LSA) data to generate actionable knowledge at school and system levels. Actionable knowledge is data-informed insight into school and system processes that can be used to direct corrective action. The analysis is framed from the perspective of the country’s evolving national policy on data use for educational improvement between 1990 and 2013. Trinidad and Tobago first participated in international large scale assessments (ILSAs) in 1991 but also developed a centralized system of national large scale assessments (NLSAs) in 2004. Analyses of both datasets consistently pointed to low quality and high inequality as the main actionable issues in the education system. NLSA data also hinted at notable variation in performance across schools and education districts. Analyses for and of policy point to the need for multiple school performance measures to better inform site-based, formative action. Over the period, actionable knowledge appears to have had greater impact at school level, with evidence being used by some low-performing schools to improve. However, at the system level, the frequent non-use and misuse of actionable knowledge suggest the need to promote and strengthen structures and processes related to evidence-informed policy-making.

The Annual Status of Education Report (ASER)

The Annual Status of Education Report (ASER) is a national citizen-led rapid assessment of children's ability to read simple text and do basic arithmetic. ASER is designed and facilitated by the Indian nongovernment organisation Pratham, and has been conducted every year since 2005 by partner organisations in every rural district of India, reaching more than 600,000 children annually. The assessment differs from most other international and national large-scale assessments in several key respects, such as the use of household rather than school-based sampling and the focus on simple tools and indicators that are easy to administer and understand. All ASER metrics, measures and processes are intended to engage ordinary citizens in thinking about and acting to improve basic learning outcomes in India. By conducting a massive national survey each year, ASER has demonstrated that it is possible to use simple, reliable and scientific methods of sampling and assessment on a large scale for high impact at a very low cost. Key to this aspect of ASER has been its ability to mobilise over 25,000 volunteers each year. ASER has been responsible to a large extent for putting the issue of learning on the agenda in India. More recently, the model has been adapted for use in several African and Asian countries. Taken together, these initiatives reached more than a million children in 2012.