International Large-scale Assessment Data Research Articles

Does between-school tracking increase school segregation among migrant students? A difference-in-differences and multiverse analysis of international large-scale assessment data

ABSTRACT This study examines the role of educational tracking in shaping school segregation among first and second-generation migrants, a pressing issue for societal integration. Drawing on the pooled data from international student assessments (PISA, PIRLS, & TIMSS) from 1995 to 2019, we examine a total of 75 countries and assess the extent to which early tracking exacerbates segregation compared to more integrated school systems. Our analysis, based on a Difference-in-Differences approach, reveals no significant differences in migrant school segregation between tracked and integrated systems. Robustness checks, employing multiverse analyses, confirm the stability of our findings across alternative model specifications. However, considerable variation in migrant school segregation exists within both types of education systems. This underscores the importance of investigating additional factors beyond early tracking contributing to variation in migrant school segregation. Our study highlights the complex interplay between educational policies and migrant integration in school settings.

The limits of inference: reassessing causality in international assessments

AbstractThis paper scrutinizes the increasing trend of using international large-scale assessment (ILSA) data for causal inferences in educational research, arguing that such inferences are often tenuous. We explore the complexities of causality within ILSAs, highlighting the methodological constraints that challenge the validity of causal claims derived from these datasets. The analysis begins with an overview of causality in relation to ILSAs, followed by an examination of randomized control trials and quasi-experimental designs. We juxtapose two quasi-experimental studies demonstrating potential against three studies using ILSA data, revealing significant limitations in causal inference. The discussion addresses the ethical and epistemological challenges in applying quasi-experimental designs to ILSAs, emphasizing the difficulty in achieving robust causal inference. The paper concludes by suggesting a framework for critically evaluating quasi-experimental designs using ILSAs, advocating for a cautious approach in employing these data for causal inferences. We call for a reevaluation of methodologies and conceptual frameworks in comparative education, underscoring the need for a multifaceted approach that combines statistical rigor with an understanding of educational contexts and theoretical foundations.

A Bayesian workflow for the analysis and reporting of international large-scale assessments: a case study using the OECD teaching and learning international survey

This paper aims to showcase the value of implementing a Bayesian framework to analyze and report results from international large-scale assessments and provide guidance to users who want to analyse ILSA data using this approach. The motivation for this paper stems from the recognition that Bayesian statistical inference is fast becoming a popular methodological framework for the analysis of educational data generally, and large-scale assessments more specifically. The paper argues that Bayesian statistical methods can provide a more nuanced analysis of results of policy relevance compared to standard frequentist approaches commonly found in large-scale assessment reports. The data utilized for this paper comes from the Teaching and Learning International Survey (TALIS). The paper provides steps in implementing a Bayesian analysis and proposes a workflow that can be applied not only to TALIS but to large-scale assessments in general. The paper closes with a discussion of other Bayesian approaches to international large-scale assessment data, in particularly for predictive modeling.

Exclusion rates from international large-scale assessments: an analysis of 20 years of IEA data

Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.

Investigating the measurement of academic resilience in Aotearoa New Zealand using international large-scale assessment data

Academic resilience captures academic success despite adversity and thus is an important concept for promoting equity within education. However, our understanding of how and why rates of academic resilience differ between contexts is currently limited by variation in the ways that the construct has been operationalised in quantitative research. Similarly, comparing the strength of protective factors that promote academic resilience is hindered by differing approaches to the measurement of academic resilience. This methodological variation has complicated attempts to reconcile disparate findings about academic resilience. The current study applied six commonly used operationalisations of academic resilience that combined different thresholds of high risk and high achievement, to three international large-scale assessments, to explore how these different operationalisations impacted the findings produced. The context of Aotearoa New Zealand was chosen as a case study to further academic resilience research within this context and investigate how academic resilience manifests in an education system with relatively high levels of average achievement alongside low levels of educational equity. Within international large-scale assessment datasets, prevalence rates differed markedly across subject areas, grade levels, and collection cycles, as a function of the measure of academic resilience employed, while the strength of protective factors was more consistent. Thresholds that were norm-referenced produced more consistent findings across the different datasets compared to thresholds that were criterion-referenced. High levels of missing data prevented the analysis of some datasets, and differences in the way that key constructs were measured undermined the comparability of findings across international large-scale assessments. The findings emphasise the strengths and limitations of utilising international large-scale assessment data for the study of academic resilience, particularly within the Aotearoa New Zealand context. Furthermore, the study highlights that researchers’ methodological decisions have important impacts on the conclusions drawn about academic resilience.

High Achievement in Mathematics and Science among Students in Ireland: An In-Depth Analysis of International Large-Scale Assessment Data Since 2000

ABSTRACT National policy initiatives in Ireland, such as Project Maths and the Science, Technology, Engineering, and Mathematics (STEM) Education Policy Statement have sought to increase the engagement and performance of students in mathematics and science. The current study investigated the performance of students in Ireland in these areas and in reading in international large-scale assessments (Programme for International Student Assessment [PISA], Trends in International Mathematics and Science Study [TIMSS], and Progress in International Reading Literacy Study [PIRLS]), with a view to better understand the performance of high-achieving students, relative to their counterparts in other countries in general, and in countries with similar average performance. Lower than expected proportions of high achievers were noted and a pattern of relative underachievement was observed among high achievers – those performing at the highest levels of proficiency, and those performing at key benchmarks, including the 75th and 90th percentiles – in mathematics and science to a greater extent than in reading. These issues were found to be consistent over time, and more prevalent at post-primary level, compared with primary level. The findings of this study are discussed with reference to individual and societal costs, and specific issues that need to be investigated further are identified.

The influence of school climate and achievement on bullying: Comparative evidence from international large-scale assessment data

ABSTRACT Background Bullying is widely acknowledged as one of the most harmful events in a child’s life, leading to negative life experiences and outcomes. However, ‘school effects of bullying’ are rarely studied from an international perspective, especially with international large-scale data. Purpose In this study, we aim to look at bullying through an international comparative approach, focusing on the contribution of the school, the education system and culture, and pupil level factors such as socio-economic status (SES) and gender. Our key question is whether school factors can affect bullying prevalence. Methods We used data from six countries (USA, Finland, England, Romania, Korea and Italy) from the Trends in International Mathematics and Science Study (TIMSS), a large-scale international study. We used multilevel modelling to analyse the dataset. Findings We find little evidence of a relation between country policies and levels of bullying, though there are differences in the extent to which school and pupil factors are related to bullying. The most consistent relationship was between gender and bullying, with prevalence higher among boys, while school factors were not significant. Conclusion The findings indicate that ‘one size fits all’ school policies might not be the best course of action, and individual support might be a more fruitful avenue.

Importance of sampling weights in multilevel modeling of international large-scale assessment data

ABSTRACTMultilevel modeling is an important tool for analyzing large-scale assessment data. However, the standard multilevel modeling will typically give biased results for such complex survey data. This bias can be eliminated by introducing design weights which must be used carefully as they can affect the results. The aim of this paper is to examine different approaches and to give recommendations concerning handling design weights in multilevel models when analyzing large-scale assessments such as TIMSS (The Trends in International Mathematics and Science Study). To achieve the goal of the paper, we examined real data from two countries and included a simulation study. The analyses in the empirical study showed that using no weights or only level 1 weights sometimes could lead to misleading conclusions. The simulation study only showed small differences in estimation of the weighted and unweighted models when informative design weights were used. The use of unscaled or not rescaled weights however caused significant differences in some parameter estimates.

The effects of motivational constructs and engagements on mathematics achievements: a comparative study using TIMSS 2011 data of Chinese Taipei, Singapore, and the USA

Motivational constructs and students’ engagements have great impacts on students’ mathematics achievements, yet they have not been theoretically investigated using international large-scale assessment data. This study utilized the mathematics data of the Trends in International Mathematics and Science Study 2011 to conduct a comparative and empirical study on exploring: (1) the changes of motivational constructs from 4th grade to 8th grade; and (2) the effects of motivational constructs from the expectancy-value model and students’ engagements on mathematics achievements. The countries investigated include Chinese Taipei, Singapore, and the USA. The results showed that: (1) students’ motivations deteriorate over school years; (2) self-confidence in mathematics (SCM) has the strongest relationship with mathematics achievements. Furthermore, it is evident that Singapore has the most effective schools in students’ mathematics education. More findings of this comparative study are subsequently discussed.

Causal inferences with large scale assessment data: using a validity framework

To answer the calls for stronger evidence by the policy community, educational researchers and their associated organizations increasingly demand more studies that can yield causal inferences. International large scale assessments (ILSAs) have been targeted as a rich data sources for causal research. It is in this context that we take up a discussion around causal inferences and ILSAs. Although these rich, carefully developed studies have much to offer in terms of understanding educational systems, we argue that the conditions for making strong causal inferences are rarely met. To develop our argument we first discuss, in general, the nature of causal inferences and then suggest and apply a validity framework to evaluate the tenability of claims made in two well-cited studies. The cited studies exemplify interesting design features and advances in methods of data analysis and certainly contribute to the knowledge base in educational research; however, methodological shortcomings, some of which are unavoidable even in the best of circumstances, urge a more cautious interpretation than that of strict “cause and effect.” We then discuss how findings from causal-focused research may not provide answers to the often broad questions posed by the policy community. We conclude with examples of the importance of the validity framework for the ILSA research community and a suggestion of what should be included in studies that wish to employ quasi-experimental methods with ILSA data.

An analysis of the relationships between Taiwanese eighth and fourth graders’ motivational beliefs and science achievement in TIMSS 2011

The purpose of this study is to examine the issues of students’ motivational beliefs in science learning and their relations with science achievement. Data of Taiwanese fourth and eighth graders from the Trends in International Mathematics and Science Study 2011 were examined. Students’ self-concept and intrinsic interest as motivational beliefs, and science scores in general and in different cognitive domains were used as measures in this study. A series of quantitative analyses along with statistical techniques accounting for the properties of international large-scale assessment data were utilized to investigate these issues. The results indicate that students’ motivational beliefs decrease over grades. The relations between the eighth graders’ motivational beliefs and science scores are positively stronger than those of the fourth graders. Meanwhile, students’ self-concept has a greater predictive power than intrinsic interest for science scores in both grades. Beyond documenting students’ motivational beliefs over grades, the conclusion also sheds light on the role of motivational beliefs, self-concept in particular, in explaining students’ science achievement.

Competitive comparison and PISA bragging rights: sub-national uses of the OECD's PISA in Canada and the USA

Participation in the Organization for Economic Co-operation and Development's (OECD) Program for International Student Assessment (PISA) has continuously expanded: from 43 systems in 2000 to 65 systems in the 2012 cycle, with 71 signed up for PISA 2015. There also has been a growth in sub-national participation, expanding PISA's reach beyond the nation-state. This paper explores sub-national PISA participation in Canada and the USA, asking how PISA is being used within sub-national education policy spaces. We draw on analysis of documents and data from interviews with officials at sub-national, national, and international levels. Findings illustrate some of the diverse motivations and uses of PISA, providing insights into the effects of PISA at the sub-national scale. As such, we argue that competitive comparison in education has deepened through the enhanced granularity of international large-scale assessment data to new scales beyond the nation-state.

EVALUATING MEASUREMENT PROPERTIES OF ATTITUDINAL ITEMS RELATED TO LEARNING SCIENCE IN TAIWAN FROM TIMSS 2007

In the recent empirical studies utilizing existing items and derived variables of international large-scale assessment (ILSA) data, the three major methodological deficiencies, including the use of a single item to define a construct, the statistical properties of ordinal data, and the fitness of the measurement structure for different scenarios, are examined. To overcome these issues, this study proposes an integrated approach to evaluating items and constructing derived variables in a given situation. Exploratory factor analysis, confirmatory factor analysis, and the item response model are utilized to evaluate student attitudinal items and derived variables from the Trends in International Mathematics and Science Study (TIMSS) 2007 Taiwanese data. The results suggest that the three-factor model composed of 12 items is optimal for the data, not the default factor structure in the database. The implications of evaluating items and creating derived variables from ILSA data for the education research community are also discussed. Key words: attitudinal items, factor analysis, Trends in International Mathematics and Science Study.

VET workers’ problem-solving skills in technology-rich environments: European approach

The European workplace is challenging VET adults’ problem-solving skills in technology-rich environments (TREs). So far, no international large-scale assessment data has been available for VET. The PIAAC data comprise the most comprehensive source of information on adults’ skills to date. The present study (N=50 369) focuses on gaining insight into the problem-solving skills in TREs of adults with a VET background. When examining the similarities and differences in VET adults’ problem-solving skills in TREs across 11 European countries, two main trends can be observed. First, our results show that only a minority of VET adults perform at a high level. Second, there seems to be substantial variation between countries with respect to the proportion of VET adults that can be identified as “at-risk” or “weak” performers. For the future, our findings indicate the variations that can be used as a starting point to identify beneficial VET approaches.

STATISTICAL TECHNIQUES UTILIZED IN ANALYZING PISA AND TIMSS DATA IN SCIENCE EDUCATION FROM 1996 TO 2013: A METHODOLOGICAL REVIEW

We conducted a methodological review of articles using the Programme for International Student Assessment (PISA) or Trends in International Mathematics and Science Study (TIMSS) data published by the SSCI-indexed science education journals, such as the International Journal of Science and Mathematics Education, the International Journal of Science Education, the Journal of Research in Science Teaching, and Science Education, from 1996 to 2013. A total of 51 empirical articles from 8 journals were identified as the sample for this study. These articles were analyzed in terms of the 2 essential statistical techniques, sampling weights and design effects, used to analyze the international large-scale assessment (ILSA) data. The study also summarized the most commonly used quantitative methods for analyzing PISA and TIMSS data in these articles. The results indicate that the weights and design effects, essential adjustments for analyzing large-scale data, were reported in less than half of the studies. Suggestions regarding the use of appropriate techniques and reporting as well as data analysis methods are made for science education researchers who use ILSA data in their research.

Understanding validity issues in international large scale assessments

Purpose – This policy brief, the third in the AERI-NEPC eBrief series “Understanding validity issues around the world”, discusses validity issues surrounding International Large Scale Assessment (ILSA) programs. ILSA programs, such as the well-known Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), are rapidly expanding around the world today. In this eBrief, the authors examine what “validity” means when applied to published results and reports of programs like the PISA. Design/methodology/approach – This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users. Findings – ILSA programs and tests, while offering valuable information, should be read and used cautiously and in context. All parties need to be on the same page to maximize valid use of ILSA results, to obtain the greatest educational and social benefits, and to minimize negative consequences. The authors propose several recommendations for test makers and ILSA program leaders, and ILSA users. To ILSA leaders and researchers: provide more cautionary information about how to correctly interpret the ILSA results, particularly country rankings, given contextual differences among nations. Provide continuing psychometric or research resources so as to address or reduce various sources of error in reports. Encourage policy makers in different nations to share the responsibility for ensuring more contextualized (and valid) interpretations of ILSA reports and subsequent policy development. Raise awareness among policy makers to look beyond simple rankings and pay more attention to inter-country differences. For consumers of ILSA results and reports: read the fine print, not just the country rankings, to interpret ILSA results correctly in particular regions/nations. When looking to high-ranking countries as role models, be sure to consider the “whole picture”. Use ILSA data as complements to other national- and state-level educational assessments to better gauge the status of the country's education system and subsequent policy directions. Originality/value – By translating complex information on validity issues with all concerned ILSA stakeholders in mind, this policy brief will improve uses and applications of ILSA information in national and regional policy contexts.