International Large-scale Assessments Research Articles

Learning from elsewhere: A hundred years of policy learning revisited

Abstract This article discusses four significant changes in lesson drawing, policy learning, or policy borrowing. Thematically, the issue of educational expansion has preoccupied policy-makers worldwide for the past hundred years. They have been eager to learn from experiences elsewhere, initially how others decentralized the finance and management of schooling to enable universal access and later how they addressed the fallout of decentralization reforms, notably inequity and quality erosion. Regarding the modalities of policy learning, the ‘travelling observer’ has been replaced by myriad digital platforms that propagate best practices and ‘actionable’ policy recommendations. As a consequence of the decentralization of governance and finance and, in some countries, structural adjustment policies, the drivers of policy borrowing have shifted from ministries of education to a wide range of stakeholders, including ministries of finance, offices of presidents and prime ministers, and private foundations and businesses. Finally, changes in the objects of emulation or reference societies reflect a spatial reconfiguration of a special sort. Along with countries in the same geopolitical or cultural space, the education systems of league leaders in international large-scale assessments, such as Finland, Korea, and Singapore, have become objects of policy attraction for both OECD and non-OECD countries.

Home Environment Matters: Predicting TIMSS Math Success in Bosnia and Herzegovina

The Trends in International Mathematics and Science Study (TIMSS) is a large-scale international assessment that measures students’ achievement in mathematics and science at the fourth and eighth-grade levels. Coordinated by the International Association for the Evaluation of Educational Achievement (IEA), TIMSS is conducted every four years. The 2023 assessment represented the eighth cycle of TIMSS, collecting data from 64 participating countries. This exploratory study examines the influence of several key factors—gender, home learning resources, experiences of bullying, disruptive classroom behavior, and students’ sense of school belonging—on fourth-grade mathematics achievement in Bosnia and Herzegovina. The research sample included 2,712 students (mean age: 10.2 years; SD = 0.4; 1,388 girls and 1,324 boys). Results showed that home learning resources were the most important predictor of mathematics achievement, emphasizing the essential role that home educational conditions play in student academic outcomes. The observed difference between boys and girls in mathematics scores was minimal. Both bullying and disruptive behavior were found to have a statistically significant negative impact on mathematics performance, whereas the effect of students’ sense of school belonging did not reach statistical significance. Home learning resources proved to be the most decisive variable in forecasting mathematics achievement among fourth-grade students in Bosnia and Herzegovina, reinforcing the value of home-based educational support in fostering academic progress. These results suggest the necessity for educational policies and interventions that ensure fair access to learning resources within the home environment. The paper concludes with a discussion of the implications of these findings and outlines potential avenues for future research and educational practice.

Testing for the money: an analysis on the interdependence of participation in international large-scale assessments and development aid networks

IntroductionInternational Large-Scale Assessments (ILSAs) are widely utilized to benchmark to evaluate the quality of education systems globally. Since 2000, the number of participating countries in these assessments has increased dramatically, revealing the dominance of a few education systems while exposing the deficiencies of many others. Especially for the latter group the question, as to why they participate in ILSAs, is puzzling. Existing literature attributes this participation to domestic factors or the influence of International Organizations. However, we argue that the global spread of ILSAs can also be explained by direct country-to-country dependencies through development aid.MethodsThis study takes an interdependency-inspired approach, focusing on how bilateral development aid networks influence ILSA participation. Using advanced network diffusion and hurdle models, we analyse data from a global set of countries over the period from 1990 to 2012. The models examine the role of development aid distribution in shaping ILSA participation, with particular attention to how participation in ILSAs affects the likelihood of attracting additional donors and aid.ResultsOur analysis reveals that the distribution of development aid has a significant impact on countries’ decisions to participate in ILSAs. Furthermore, we find that countries already involved in ILSAs are more likely to attract additional donors and aid within established donor-recipient dyads. These results demonstrate a clear connection between development aid distribution and the global spread of ILSAs.DiscussionThis study highlights the role of monetary incentives, facilitated by the distribution of development aid, in promoting participation in ILSAs. Our findings suggest that the rapid spread of ILSAs is not solely driven by domestic factors or international organizations but also by the interdependent relationships formed through aid distribution networks. These insights contribute to a broader understanding of the factors driving global educational assessments and provide implications for policy-making in education and development aid.

Evaluating uncertainty: the impact of the sampling and assessment design on statistical inference in the context of ILSA

This paper informs users of data collected in international large-scale assessments (ILSA), by presenting argumentsunderlining the importance of considering two design features employed in these studies. We examine a commonmisconception stating that the uncertainty arising from the assessment design is negligible compared with that arisingfrom the sampling design. This misconception can lead to the erroneous conclusion that there is always a relatively lowrisk of ignoring the uncertainty arising from the assessment design when reporting estimates of population parameters. We use the design effect framework to assess the impact that the sampling and the assessment design have on theestimation. We first evaluate the loss in efficiency in the estimation of a population parameter attributable to each ofthe designs. We then examine whether knowledge about the effect of one design feature can justify any belief about theeffect of the other design feature. We repeat this examination across different parameters characterizing theachievement distribution in a population. We provide empirical results using data collected for PIRLS 2016. Our empirical results can be summarized in two general findings. First, when estimating mean achievement, the effectof the sampling design is often substantially larger than that of the assessment design. This finding might explain themisconception we try to address. However, we show that this is not true in all instances, and the magnitude of thedifference between both design effects is context dependent and hence not generalizable. Second, differences in designeffects become less predictable when estimating other parameters, e.g. the proportion of students reaching a certainthreshold in the achievement scale (i.e., benchmarks), or an association estimated using linear regression. This contribution underlines that accounting for all sources of uncertainty in the estimation is of paramount importanceto obtain credible inferences. We conclude that it is difficult to justify a priori the belief that the effect of the samplingdesign in the estimation is always greater than that of the assessment design.

Alignment Optimization in International Large-Scale Assessments: A Scoping Review and Future Directions

Alignment Optimization in International Large-Scale Assessments: A Scoping Review and Future Directions

Stacking Models of Growth: A Methodology for Predicting the Pace of Progress to the Education Sustainable Development Targets Using International Large-Scale Assessments.

To assess country-level progress toward these educational goals it is important to monitor trends in educational outcomes over time. The purpose of this article is to demonstrate how optimally predictive growth models can be constructed to monitor the pace of progress at which countries are moving toward (or way from) the education sustainable development goals as specified by the United Nations. A number of growth curve models can be specified to estimate the pace of progress, however, choosing one model and using it for predictive purposes assumes that the chosen model is the one that generated the data, and this choice runs the risk of "over-confident inferences and decisions that are more risky than one thinks they are" (Hoeting et al., 1999). To mitigate this problem, we adapt and apply Bayesian stacking to form mixtures of predictive distributions from an ensemble of individual models specified to predict country-level pace of progress. We demonstrate Bayesian stacking using country-level data from the Program on International Student Assessment. Our results show that Bayesian stacking yields better predictive accuracy than any single model as measured by the Kullback-Leibler divergence. Issues of Bayesian model identification and estimation for growth models are also discussed.

An Improved Satterthwaite (1941, 1946) Effective df Approximation

This study introduces a correction to the approximation of effective df as proposed by Satterthwaite, specifically addressing scenarios where component df are small. The correction is grounded in analytical results concerning the moments of standard normal random variables. This modification is applicable to complex variance estimates that involve both small and large df, offering an enhanced approximation of the higher moments required by Satterthwaite’s framework. Additionally, this correction extends and partially validates the empirically derived adjustment by Johnson and Rust, as it is based on theoretical foundations rather than simulations used to derive empirical transformation constants. Finally, the proposed adjustment also provides a correction to the estimate of the total variance in cases missing data have been replaced by multiple imputations such as in the case of plausible values in national and international large scale assessments.

Proposal for modifying procedures for declaring significant trends in TIMSS

International large-scale assessments (ILSA) are an important source of information for education policymakers across the globe. Despite sponsors’ warnings, when results are published, media attention focuses on country rankings and changes in scores. Score changes are evaluated using a two-sided z-statistic, with statistical significance declared if the statistic exceeds 1.96 in absolute value. Findings of significance occasion much commentary and may also have policy implications. Particularly problematic, however, are cases in which a significant trend in one direction is followed in the next cycle by a significant trend in the opposite direction. Such reporting reversals are often difficult to explain on substantive grounds and, consequently, can undermine the credibility of the ILSA. This article proposes and evaluates a new approach to determining the significance of observed score changes. It employs a two one-sided test (TOST) procedure. A key feature is establishing an equivalence zone around zero. A change is declared only if the test statistic does not fall in the equivalence zone. Thus, the procedure combines consideration of statistical significance and substantive importance. We augment the TOST procedure with the Benjamini–Hochberg procedure (BH-TOST) to control the False Discovery Rate , to address the problem of multiplicity. Using data from TIMSS (2011, 2015, 2019), we explore different parameter choices for BH-TOST and evaluate the operating characteristics of the selected version with respect to three different criteria. Our results indicate that BH-TOST is generally superior to the current procedure, concluding that it merits serious consideration as a basis for reporting TIMSS results.

Open Classroom Climate in International Large-Scale Assessments: Individual Construct or Climate Construct?

This study examines the conceptualization and operationalization of the open classroom climate construct in existing empirical works. Open classroom climate is a classroom environment where adolescents can freely discuss political issues and express their opinions. The study reviewed 45 secondary data analysis articles that examined the open classroom climate construct from the Civic Education Study and the International Civic and Citizenship Education Study , the major international large-scale assessments. This review identified inconsistencies in how these studies define and analyze the open classroom climate construct. These inconsistencies arise from the different interpretations of the construct whether as an individual construct that reflects students’ distinctive perceptions or as a climate construct that represents a collective classroom experience. Due to varied conceptualizations, researchers employed different operationalization approaches, each incorporating distinct statistical methods. It led to varied results and impeded the development of a unified understanding of the concept. Our study emphasizes the necessity for a clear and consistent conceptualization of the open classroom climate construct for future works. It implores researchers to deliberate on the construct’s nature, endorsing a proper operationalization to understand open classroom climate aptly.

Teacher Satisfaction And Grade 4 Reading Literacy Achievement: An Austrian Perspective Using International Large-Scale Assessment Data

The complexity of the teaching environment requires now more than ever a satisfied and fulfilled teacher population. This study uses the Austrian Progress in International Reading Literacy Study (PIRLS) 2021 data to uncover those factors specifically associated with the school environment when controlling for school socio economic status as predictors of teacher dissatisfaction. Results show that the current model explains 22% of the variance, yet school emphasis on academic success, teaching being limited by a number of lacking elements and school safety were found to be of no statistical significance. The study provides two important points to consider, namely the role of short cycle teachers in producing reading literacy scores well above those of their degreed counterparts, and the significant role of socio-economic status and its effect on achievement. The study concludes that a systems perspective that considers the shared beliefs and values, relationships and social interactions, leadership, and physical environment may shed more light on the underlying dynamics that can be associated with those less than satisfied teachers.

Embracing complexity: abductive reasoning as a versatile tool for analyzing international large-scale assessments

AbstractThis paper explores the potential of abductive reasoning to enhance the analysis of international large-scale assessments which have traditionally relied on deductive and inductive reasoning. While these conventional methods have provided valuable insights into global student achievement, they often fail to capture the complexity of educational systems and the nuanced interplay of variables influencing student outcomes. By proposing abductive reasoning as a complementary analytical approach, this study argues for its ability to account for surprising data points and embrace the complexity inherent in educational systems. Abductive reasoning, characterized by the formulation of the most plausible explanations for observed phenomena, offers a promising avenue for generating new hypotheses and uncovering unexpected relationships. Through a detailed examination of the strengths and weaknesses of inductive and deductive reasoning in the context of ILSA data and the introduction of abductive reasoning’s unique characteristics, this paper highlights the benefits and practical applications of abduction in educational research. It also addresses potential limitations and challenges, advocating for a structured application to enhance rigor and mitigate biases. Ultimately, abductive reasoning is presented as a versatile and creative tool that, when used thoughtfully alongside traditional methods, can significantly advance our understanding of global education systems and inform more nuanced and impactful educational policies.

A Comparative Analysis of PISA 2015 Türkiye Studies: Introducing A Variable Selection Model to International Large-Scale Assessments

International large-scale assessments have a key role in improving educational, economical, and political systems. By using the data of these assessments, countries can draw conclusions about the status of educational systems. Studies and reports generally tend to choose variables available in data set to model the relationships among the variables. In this study, we aimed to introduce a variable selection method to analyze large-scale assessments to be able to decide which variables might be included in modelling country data. We used the entire data set of Türkiye PISA 2015 through elastic net regression to decide which variables should be selected for further analysis. We also provided a summary of the available studies based on Türkiye PISA 2015 data and compared the results. Based on the series of analyses, this study revealed that test anxiety, environmental awareness, interest in broad topics in science, playing video games after school, mathematics literacy, reading literacy, and collaborative problem-solving skills were the explanatory variables contributed most to the degree of scientific literacy of students. This study has a potential to provide an example of shrinkage methods applied in educational context and offer another standpoint for providing a rationale to select which variables can be included.

Should mathematicians worry with PISA and TIMSS math results?

Since the end of last century, international large-scale assessment studies have been providing systematic information about countries’ education results. The news they convey regarding European education are not at all reassuring and have worsened with school closures and the disruption due to the recent Covid-19 pandemic. The size of the education losses and the distance of most European countries to other areas of the world, namely South East Asia, are often overlooked. A simple statistical analysis shows most European students are about one to two school years behind their peers in most advanced countries and regions, and the gap is increasing. These facts will negatively impact European workforce, economic competitiveness, and development. They will most likely also negatively impact mathematics higher education and research in our continent. To recognise the problem is the first step to cope with it. [1]

Integrating the Key Competencies of International Large-Scale Assessment (ILSA) into the Teacher-Education Curriculum in the Philippines

This study proposes a framework to integrate competencies from various international large-scale assessments (ILSAs) into the program design of a teacher educational institution (TEI) in the Philippines. Using a descriptive-developmental research design, the study examined how ILSA key competencies can be incorporated into the curriculum, along with practices of outcomes-based educational programs at the TEI. Inductive thematic analysis of focus group interviews revealed one overarching program-level theme: using ILSA as a benchmark for curriculum development. At the micro level, four core themes emerged: (1) strengthened content teaching, (2) increased pedagogical content knowledge (PCK) courses, (3) explicit targeting and teaching of ILSA competencies through course-intended learning outcomes (CILOs), and (4) ILSA-like assessments. The study recommends that the Philippine Department of Education (DepEd) and TEIs collaborate to align pre-service teacher training with global literacy standards, ensuring that pre-service teachers are equipped to teach ILSA-recommended skills.

"Re-thinking equity: the need for a multidimensional approach in evaluating educational equity through TIMSS data"

BackgroundIn recent years, data derived from international large-scale assessments have significantly influenced the discourse surrounding educational equity. However, the use of such data has often neglected the full spectrum of dimensions that equity encompasses, while being limited to exploring the relationship between achievement variations and student background. This approach, however, constrains our understanding of the rich notion.MethodsThis paper aims to contribute to the advancement of current research by advocating for an alternative approach that encompasses all relevant dimensions of equity and examines their impact on both achievement and motivational outcomes. A cluster analysis manifests itself as the designated method to employ, as the concept of equity remains theoretically challenged and the detection of data patterns can serve as a strong step in its multifaceted inquiry. Moreover, this method enables us to incorporate the multidimensional concept in a country-level context.ResultsEmploying data from the Trends in International Mathematics and Science Study at grade four, this paper presents five distinct clusters of educational systems that illustrate diverse equity configurations. Notably, none of the identified clusters consistently scores high or low on all equity dimensions. This suggests that all of the clusters perform inconsistently to different degrees across the dimensions of equity. Furthermore, no single cluster emerges as superior to the others across both outcome measures.ConclusionsThese findings shed light on the intricate interplay between equity, achievement, and motivation within the examined educational systems. The unique configuration of the equity dimensions within each cluster underscores the importance of adopting a nuanced evaluation of equity, which can enhance our understanding of equity patterns.

Detecting latent subpopulations in international large-scale assessments by fitting MixIRT models using NUTS

The focus of this study is to use the mixture item response theory (MixIRT) model while implementing the no-U-turn sampler as a technique for investigating the presence of latent classes (i.e., subpopulations) among eighth-grade students who were administered TIMSS 2019 mathematics subtest in paper format from the gulf cooperation council (GCC) countries. One-, two-, and constrained three-parameter logistic MixIRT models with one to four classes were used to fit to the data, where the model data fit was assessed using Bayesian fit indices. The results indicate that multiple latent classes or subpopulations can better reflect the mathematical proficiency of eighth graders from the four GCC countries, and specifically the two-class constrained three-parameter MixIRT model provides a relatively better fit to the data. The results also indicate that when a mixture of several latent classes present, the conventional unidimensional IRT model is limited in providing information for multiple latent classes and shall be avoided. In addition to adding to the existing literature on MixIRT models for international large-scale assessments such as TIMSS on its heterogenous subpopulations from a fully Bayesian approach, this study sheds light on the limitation of conventional unidimensional IRT models and subsequently directs attention to the use of the more complex MixIRT model for such assessments.

Deciphering socioeconomic inequality in science literacy: Evidence from PISA 2015 in Türkiye

Deciphering socioeconomic inequality in science literacy: Evidence from PISA 2015 in Türkiye

Writing PISA-Like Mathematics Items: The Case of Tertiary Mathematics Instructors from a State University in the Philippines

<p style="text-align:justify">Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) are nested in contexts defined in their assessment framework (e.g., the Personal, Occupational, Societal, and Scientific contexts in PISA). This study followed the item-writing activities of four tertiary mathematics instructors in the Philippines as they constructed context-based mathematics items. They were tasked to write four items each, following a set of specifications for PISA content and context categories. The data consisted of transcripts from the focus-group discussion which was conducted days after the task. The transcripts were then analyzed using thematic analysis. The results of this study showed that the phenomenon of item-writing in the context of writing PISA-like mathematics items had two themes: the phases of item-writing and the dimensions of item-writing. Findings showed that the respondents struggled to find realistic contexts and that they engaged in a problem-solving task likened to solving a puzzle as they attempted to satisfy the content, context, and process categories in the table of specifications (TOS). This study contributes to filling in the research gap on item-writing activities, particularly those of mathematics teachers in the Philippines- a country whose recent mathematical performance in the PISA 2018, TIMSS 2019, and PISA 2022 was nothing short of dismal.</p>

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

AbstractIn international large-scale assessments, student performance comparisons across educational systems are frequently done to assess the state and development in different domains. These results often have a large impact on educational policy and on the perceptions of an educational system’s performance. Early assessments, such as the First and Second International Science Studies (FISS and SISS), have been used alongside recent studies to create unique scales for investigating changes in constructs. The implicit assumptions in system comparisons are that the measures are valid, reliable, and comparable. However, these assumptions have not always been investigated thoroughly. This study aims to investigate the validity and cross-system comparability of scores from the FISS and SISS, conducted by the International Association for the Evaluation of Educational Achievement in 1970–1971 and 1983–1984. Findings based on item response theory (IRT) modeling indicate that scores in most educational systems can be viewed as reliable measures of a single science construct, supporting the validity of test score interpretations in these educational systems individually. In a robust assessment of measurement invariance using standard IRT methods, an alignment-based method, and the root mean square difference (RMSD) fit statistic, we demonstrate that measurement invariance is violated across systems. The alignment-based method identified a well-fitting model with complex restrictions but no items exhibited invariance across all systems, a result supported by the RMSD statistics. These results question the appropriateness of score comparisons across systems in FISS and SISS. We discuss the implications of these results and outline consequences for score comparisons across time.

Does between-school tracking increase school segregation among migrant students? A difference-in-differences and multiverse analysis of international large-scale assessment data

ABSTRACT This study examines the role of educational tracking in shaping school segregation among first and second-generation migrants, a pressing issue for societal integration. Drawing on the pooled data from international student assessments (PISA, PIRLS, & TIMSS) from 1995 to 2019, we examine a total of 75 countries and assess the extent to which early tracking exacerbates segregation compared to more integrated school systems. Our analysis, based on a Difference-in-Differences approach, reveals no significant differences in migrant school segregation between tracked and integrated systems. Robustness checks, employing multiverse analyses, confirm the stability of our findings across alternative model specifications. However, considerable variation in migrant school segregation exists within both types of education systems. This underscores the importance of investigating additional factors beyond early tracking contributing to variation in migrant school segregation. Our study highlights the complex interplay between educational policies and migrant integration in school settings.