International Large-scale Assessments Research Articles

Welfare state policy and educational inequality: a cross-national multicohort study

Abstract Proponents of welfare policy have argued that publicly funded early childhood education and care (ECEC), paid parental leave, and family benefits spending can weaken the influence of social background on educational outcomes by providing a supplementary source of early investment that particularly benefits disadvantaged families. We analyze whether the welfare state context in which children spend their early childhood (ages 0–5) moderates the association between parental educational attainment and the child’s educational achievement at age 10. We combine data from two large-scale international student assessments with data about welfare state policies. Results from multilevel models show that countries with higher public ECEC spending and higher family benefits spending exhibited a weaker association between parental education and student math achievement. Countries with longer parental leave exhibited a stronger association between parental education and student math, science, and reading achievement. Findings provide evidence of the mixed role of welfare state policies for social inequality in student achievement.

A Bayesian workflow for the analysis and reporting of international large-scale assessments: a case study using the OECD teaching and learning international survey

This paper aims to showcase the value of implementing a Bayesian framework to analyze and report results from international large-scale assessments and provide guidance to users who want to analyse ILSA data using this approach. The motivation for this paper stems from the recognition that Bayesian statistical inference is fast becoming a popular methodological framework for the analysis of educational data generally, and large-scale assessments more specifically. The paper argues that Bayesian statistical methods can provide a more nuanced analysis of results of policy relevance compared to standard frequentist approaches commonly found in large-scale assessment reports. The data utilized for this paper comes from the Teaching and Learning International Survey (TALIS). The paper provides steps in implementing a Bayesian analysis and proposes a workflow that can be applied not only to TALIS but to large-scale assessments in general. The paper closes with a discussion of other Bayesian approaches to international large-scale assessment data, in particularly for predictive modeling.

Cross-National Measurement of Mathematics Intrinsic Motivation: An Investigate of Measurement Invariance with MG-CFA and Aligment Method Across Fourteen Countries

One of the main objectives of international large-scale assessments is to make comparisons between different countries, education policies, education systems, or subgroups. One of the main criteria for making comparisons between different groups is to ensure measurement invariance. The purpose of this study was to test the measurement invariance of the mathematics intrinsic motivation scale across 14 countries. For this purpose, the "students like learning mathematics" scale, which measures intrinsic motivation for mathematics, was included in the TIMSS 2019 cycle. The study sample consisted of a total of 152992 students, 70192 4th grade and 82800 8th grade students from 14 different countries participating in the TIMSS 2019 cycle. Measurement invariance was tested with Multi-Group Confirmatory Factor Analysis (MG-CFA) and Alignment Method. The mathematics intrinsic motivation scale provides only configural invariance according to MG-CFA at the 4th grade level, whereas the scale provides approximate invariance according to the alignment method. At the 8th grade level, the scale provides configural and metric invariance according to MG-CFA, whereas the scale provides approximate invariance according to the alignment method. The results indicate that the mathematics intrinsic motivation scale provides approximate measurement invariance at both grade levels and that comparisons can be made between the scores of the identified countries.

A network ethnography of international large-scale assessment contracting: scientific knowledge as messy, provisional, complex, and subjective

ABSTRACT While methodologies are often presented as standardised procedures which, under specified conditions, should lead to the same conclusions, this case study presents the complex and deeply personal process of research (Addey and Piattoeva 2022). We analyse our application of network ethnography – an approach presented by Ball (2016) – to the study of contractors developing international large-scale assessments, exploring how we, as scholars, become with our methodology and navigate the ‘messiness’ of research (Law 2004). Drawing on Science and Technology Studies (STS) to understand the constitutive role of methodology and performativity of knowledge-making (Law and Singleton 2013, Rimpiläinen 2015), we show how methodological decisions construct what is studied and ourselves. Finally, we discuss challenges of visual representation, applying Galloway’s (2011) ‘conversion rules’ to examine what was unrepresented – or unrepresentable. This paper shows the complex, subjective, and provisional nature of knowledge, theorising ‘heterogeneity and variation’ (Law 2004) as an inherent part of methodological application.

Comparison of Different Computerized Adaptive Testing Approaches with Shadow Test Under Different Test Length and Ability Estimation Method Conditions

Adaptive testing approaches have been used and adopted in many international large-scale assessments (PISA, TIMSS, PIRLS, etc.). The shadow test approach, on the other hand, is an innovative testing approach that both meets all test specifications and constraints and aims to provide maximum information at the test taker's true ability level. The aim of this study is to investigate the effectiveness of four different adaptive testing approaches created with shadow test (CAT, 2-Stage O-MST, 3-Stage O-MST and LOFT) according to the test length and ability estimation method. With the Monte Carlo (MC) study in R software, 200 item parameters and 2000 test takers were generated under the 3PL model and the results were calculated over 50 replications. The results show that CAT, 2-Stage O-MST and 3-Stage O-MST are quite similar in effectiveness, while LOFT is less effective than these techniques. As the test length increases, the measurement precision increases in all different types of adaptive tests. Although EAP method generally presents better measurement precision than MLE method, at the extremes of the ability scale MLE has been found to present good measurement precision. In the research, it is discussed that large-scale assessments can benefit from adaptive testing created with shadow test approach.

Collaborative problem-solving design in large-scale assessments: Shedding lights in sequential conversation-based measurement

Collaborative problem solving (CPS) is inherently an interactive, conjoint, dual-strand process that considers how a student reasons about a problem as well as how s/he interacts with others to regulate social processes and exchange information (OECD, 2013). Measuring CPS skills presents a challenge for obtaining consistent, accurate, and reliable scale across individuals and user populations. The Programme for International Student Assessment (PISA)’s 2015 cycle first introduced an assessment of CPS in international large-scale assessments in which computer-based conversational agents were adapted to represent team members with a range of skills and abilities. This study draws on measures of the CPS domain in PISA 2015 to address the challenges and solutions related to CPS item design and shed lights on sequential conversation-based measurement. Specifically, we present the process of CPS item design, the development of scoring rules through CPS conversation paths, and discuss the possible approaches to better estimate CPS beyond item response models.

How Did Spain Perform In PISA 2018? New Estimates Of Children’s PISA Reading Scores*

ABSTRACT International large-scale assessments have gained much attention since the beginning of the twenty-first century, influencing education legislation in many countries. This includes Spain, where they have been used by successive governments to justify education policy change. Unfortunately, there was a problem with the PISA 2018 reading scores for this country, meaning the OECD refused to initially release the results. Therefore, in this paper we attempt to estimate the likely PISA 2018 reading scores for Spain, and for each region within. The figure finally published by the OECD for Spain – in terms of reading scores – was 476.5 points, which is between the lower and upper bound of the interval we find (475 to 483 test points in 2018). Additionally, we report some robustness checks for the OCED countries participating in PISA 2018, which show that the difference between the actual scores and the ones we found with the imputation methods are quite close.

Reading to Young Children: Higher Home Frequency Associated with Higher Educational Achievement in PIRLS and PISA

The benefits of reading aloud to young children for their reading development are well documented, and international large-scale assessments (ILSAs) offer an opportunity to explore its unique contribution to literacy achievement at both the primary and secondary levels. Using Portuguese data from ILSAs, this study shows the relationship between reading to young children in the home context and their later reading performance. Specifically, we use the Program for International Reading Literacy Study (PIRLS) 2011, which tests fourth-grade students, and the Program for International Student Assessment (PISA) 2018, which is used for the assessment of 15-year-olds. Data sources from these surveys include the mean reading performance of similar cohorts of students and home/parental questionnaires that include questions about the frequency of home book-reading, as well as other background variables. Linear regression analyses show a positive and significant relationship, both at the fourth-grade level and in secondary school, between students’ performance and having been read to at home during early childhood. These findings indicate that the advantages associated with book reading in the early years are maintained throughout students’ schooling. In addition, the analysis shows that, in both surveys, girls score higher than boys in reading, and that there is a positive association between parental education and reading achievement. Implications about how children’s early literacy development sets the foundation for future educational achievement are discussed, namely in the context of country-specific reading initiatives and reading practices.

Variable selection in latent variable models via knockoffs: an application to international large-scale assessment in education

Abstract International large-scale assessments (ILSAs) play an important role in educational research and policy making. They collect valuable data on education quality and performance development across many education systems, giving countries the opportunity to share techniques, organisational structures, and policies that have proven efficient and successful. To gain insights from ILSA data, we identify non-cognitive variables associated with students’ academic performance. This problem has three analytical challenges: (a) academic performance is measured by cognitive items under a matrix sampling design; (b) there are many missing values in the non-cognitive variables; and (c) multiple comparisons due to a large number of non-cognitive variables. We consider an application to the Programme for International Student Assessment, aiming to identify non-cognitive variables associated with students’ performance in science. We formulate it as a variable selection problem under a general latent variable model framework and further propose a knockoff method that conducts variable selection with a controlled error rate for false selections.

Challenges for the Design of International Assessments: Sampling, Measurement, and Causality

This paper examines critical design factors that influence data quality in educational research, using international large-scale educational assessments as an example. We will focus on statistical challenges related to sampling, measurement, and causality. While international assessments employ rigorous random sampling techniques, deviations such as exclusions and non-participation can introduce bias and affect representativeness. In terms of measurement, although these assessments excel in core domains, there is a growing call for broader assessment areas, such as environmental literacy and civic education. Additionally, concerns are emerging about the quality of context surveys. Causality remains a central concern, and despite the challenges posed by the cross-sectional design, combining data and applying sophisticated analytical methods can help address causal questions. Recognising the interconnectedness of sampling, measurement, and causality is essential for conducting robust research and informing evidence-based policies and practices.

Data Friction and Precarious Knowledge: IEA and the Movement of Data in the 1960s and 1970s

The creation of international large-scale assessments in the 1960s and 1970s was inevitably a product of the movement of data. Large amounts of information about school performance circled the globe: first collected in individual schools, then aggregated at the national level, then sent to an international centre, and finally returned for further national analysis and dissemination. While the movement of data was a prerequisite for international testing, there was an inherent risk in how the work was organised. The concepts of data friction and precarious knowledge are used to describe how the pace of information transfer was slowed or how information was in danger of being lost as it circled the globe. The movement of data brought with it the double possibility of creating and destroying knowledge.

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

Process data are becoming more and more popular in education research. In the field of computer-based assessments of collaborative problem solving (ColPS), process data have been used to identify students’ test-taking strategies while working on the assessment, and such data can be used to complement data collected on accuracy and overall performance. Such information can be used to understand, for example, whether students are able to use a range of styles and strategies to solve different problems, given evidence that such cognitive flexibility may be important in labor markets and societies. In addition, process information might help researchers better identify the determinants of poor performance and interventions that can help students succeed. However, this line of research, particularly research that uses these data to profile students, is still in its infancy and has mostly been centered on small- to medium-scale collaboration settings between people (i.e., the human-to-human approach). There are only a few studies involving large-scale assessments of ColPS between a respondent and computer agents (i.e., the human-to-agent approach), where problem spaces are more standardized and fewer biases and confounds exist. In this study, we investigated students’ ColPS behavioral patterns using latent profile analyses (LPA) based on two types of process data (i.e., response times and the number of actions) collected from the Program for International Student Assessment (PISA) 2015 ColPS assessment, a large-scale international assessment of the human-to-agent approach. Analyses were conducted on test-takers who: (a) were administered the assessment in English and (b) were assigned the Xandar unit at the beginning of the test. The total sample size was N = 2,520. Analyses revealed two profiles (i.e., Profile 1 [95%] vs. Profile 2 [5%]) showing different behavioral characteristics across the four parts of the assessment unit. Significant differences were also found in overall performance between the profiles.

Socially Jetlagged and Late for School: Chronotypes, Achievement and Truancy among 15-Year-Old Students in PISA 2018

This paper uses data from PISA 2018 in Germany to explore how the students‘chronotype (known as larks and owls) and their academic achievement is related to truancy. We used an adapted version of the Munich Chronotype Questionnaire (MCTQ) to measure the students’ chronotype. The results demonstrate a correlation between the students’ social jet lag, student achievement, and truancy. Finally, we discuss the potential inclusion of chronotypes as a measure in large-scale international student assessments.

Exclusion rates from international large-scale assessments: an analysis of 20 years of IEA data

Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.

Cultural variation in the SES-gender interaction in student achievement.

Is the socioeconomic gap in academic achievement larger among boys than girls? Several scholars have proposed such an interaction between socioeconomic status (SES) and gender. Prior empirical studies have yielded mixed evidence, but they have been conducted almost exclusively in Western countries. Here we propose the hypothesis that the SES-gender interaction is stronger in less gender-equal societies. We estimated the SES-gender interaction in 36 countries using data from two international large-scale assessments (PIRLS and TIMSS). The degree of gender equality was measured by the Global Gender Gap Index. Consistent with the hypothesis, the SES-gender interaction was stronger in societies with less gender equality. Our findings suggest that cultural factors determine how the socioeconomic achievement gap differs between boys and girls.

Measuring education: Do we need a plethora of assessment studies or just a single score?

The major international large-scale assessments in education prioritize the three core domains such as mathematics, science, and reading, neglecting other important domains such as civic education or computer and information literacy. The present study aims at determining the added value of different international large-scale assessments to evaluating educational systems across the globe. Using country-level data from all recent international assessments by the IEA and the OECD, we conducted a series of correlational analyses and multidimensional scaling to determine the similarities and differences among educational assessments. The findings of this study show that the various measures of achievement correlate to different degrees at country level. While mathematics, science, and reading are the most closely correlated, the correlations with civic education and computer and information literacy are considerably lower. We conclude that subject domains as civic education and computer and information literacy add a significant value to the assessment of education.

Will the gap ever be bridged? A cross-national comparison of non-native students’ educational achievements

A crucial element in evaluating the success of immigrant integration policies is to compare the school performances of immigrant students with that of natives. According to large-scale international assessments, immigrant students tend to underperform their native peers even after controlling for socioeconomic conditions, with relevant differences depending on countries’ migration histories. This article analyses the evolution in the skills gap between native students and students with an immigrant background by comparing traditional (France, Germany, and the United Kingdom) and new immigration countries (Italy and Spain) using data collected from the Program for International Student Assessment survey from 2009 to 2018. We model students’ learning outcomes in mathematics and reading in a multilevel setting by clustering students at the high school level. This approach allows us to compare countries’ relative performances concerning the immigrant integration process by accounting for the role played by high schools and students’ backgrounds. Results show that the differences between native and immigrant students in both sets of countries are narrowing but still present.

Teacher quality and cross-country differences in learning in francophone Sub-Saharan Africa

We study the effects of two dimensions of teacher quality, subject knowledge and didactic skills, on student learning in francophone Sub-Saharan Africa. We use data from an international large-scale assessment in 14 countries that include individual-level information on student achievement and country-level averages of teacher subject knowledge and teacher didactic skills in reading and math. Exploiting variation between subjects in a student fixed-effects model, we find that teacher subject knowledge has a large positive effect on student achievement, whereas the effect of teacher didactic skills is comparatively small but imprecisely estimated. Differences in teacher subject knowledge account for 37 percent of the variation in average student achievement across countries.

Predicting cross-national sex differences in large-scale assessments of students' reading literacy, mathematics, and science achievement: Evidence from PIRLS and TIMSS

Causes of sex differences in educational achievement have been controversially discussed in the extant literature. It has been speculated that differing prosperity and equality of opportunities may be linked to these differences, but conclusive empirical evidence for such effects is unavailable. Here, we present evidence for sex differences in international large-scale assessments of reading literacy, mathematics, and science across 16 cohorts from 1995 to 2019. Our analyses of PIRLS and TIMSS reading literacy, mathematics, and science achievement data (N = 3,999,062; 90 countries) showed consistent advantages for girls in reading literacy (d range: −0.02 to 0.66). For mathematics and science this pattern was less unambiguous, yielding non-trivial effects in both directions (d ranges: −0.44 to 0.36 and − 0.50 to 0.46, respectively). Sex differences in all three domains were more pronounced in more egalitarian countries (β range 0.16 to 0.20). Higher national prosperity and educational investment predicted larger sex differences favoring fourth grade boys in mathematics and science (β range: 0.07 to 0.39) and became less meaningful with increasing student ages (β range for eighth graders: 0.17 to 0.21). In all, our findings suggest that influences of economic macro-indicators on sex differences in educational achievement are differentiated according to subject, indicating larger sex differences in mathematics and science in more egalitarian and prosperous countries.

Comparing different trend estimation approaches in country means and standard deviations in international large-scale assessment studies

One major aim of international large-scale assessments (ILSA) like PISA is to monitor changes in student performance over time. To accomplish this task, a set of common items (i.e., link items) is repeatedly administered in each assessment. Linking methods based on item response theory (IRT) models are used to align the results from the different assessments on a common scale. This work employs the one-parameter logistic (1PL) and the two-parameter logistic (2PL) IRT models as scaling models for dichotomous item response data. The present article discusses different types of trend estimates in country means and standard deviations for countries in ILSA. These types differ in three aspects. First, the trend can be assessed by an indirect or direct linking approach for linking a country’s performance at an international metric. Second, the linking for the trend estimation can rely on either all items or only the link items. Third, item parameters can be assumed to be invariant or noninvariant across countries. It is shown that the most often employed trend estimation methods of original trends and marginal trends can be conceived as particular cases of indirect and direct linking approaches, respectively. Through a simulation study and analytical derivations, it is demonstrated that trend estimates using a direct linking approach and those that rely on only link items outperformed alternatives for the 1PL model with uniform country differential item functioning (DIF) and the 2PL model with uniform and nonuniform country DIF. We also illustrated the performance of the different scaling models for assessing the PISA trend from PISA 2006 to PISA 2009 in the cognitive domains of reading, mathematics, and science. In this empirical application, linking errors based on jackknifing testlets were utilized that adequately quantify DIF effects in the uncertainty of trend estimates.