International Large-scale Assessments Research Articles

Should mathematicians worry with PISA and TIMSS math results?

Since the end of last century, international large-scale assessment studies have been providing systematic information about countries’ education results. The news they convey regarding European education are not at all reassuring and have worsened with school closures and the disruption due to the recent Covid-19 pandemic. The size of the education losses and the distance of most European countries to other areas of the world, namely South East Asia, are often overlooked. A simple statistical analysis shows most European students are about one to two school years behind their peers in most advanced countries and regions, and the gap is increasing. These facts will negatively impact European workforce, economic competitiveness, and development. They will most likely also negatively impact mathematics higher education and research in our continent. To recognise the problem is the first step to cope with it. [1]

Integrating the Key Competencies of International Large-Scale Assessment (ILSA) into the Teacher-Education Curriculum in the Philippines

This study proposes a framework to integrate competencies from various international large-scale assessments (ILSAs) into the program design of a teacher educational institution (TEI) in the Philippines. Using a descriptive-developmental research design, the study examined how ILSA key competencies can be incorporated into the curriculum, along with practices of outcomes-based educational programs at the TEI. Inductive thematic analysis of focus group interviews revealed one overarching program-level theme: using ILSA as a benchmark for curriculum development. At the micro level, four core themes emerged: (1) strengthened content teaching, (2) increased pedagogical content knowledge (PCK) courses, (3) explicit targeting and teaching of ILSA competencies through course-intended learning outcomes (CILOs), and (4) ILSA-like assessments. The study recommends that the Philippine Department of Education (DepEd) and TEIs collaborate to align pre-service teacher training with global literacy standards, ensuring that pre-service teachers are equipped to teach ILSA-recommended skills.

"Re-thinking equity: the need for a multidimensional approach in evaluating educational equity through TIMSS data"

BackgroundIn recent years, data derived from international large-scale assessments have significantly influenced the discourse surrounding educational equity. However, the use of such data has often neglected the full spectrum of dimensions that equity encompasses, while being limited to exploring the relationship between achievement variations and student background. This approach, however, constrains our understanding of the rich notion.MethodsThis paper aims to contribute to the advancement of current research by advocating for an alternative approach that encompasses all relevant dimensions of equity and examines their impact on both achievement and motivational outcomes. A cluster analysis manifests itself as the designated method to employ, as the concept of equity remains theoretically challenged and the detection of data patterns can serve as a strong step in its multifaceted inquiry. Moreover, this method enables us to incorporate the multidimensional concept in a country-level context.ResultsEmploying data from the Trends in International Mathematics and Science Study at grade four, this paper presents five distinct clusters of educational systems that illustrate diverse equity configurations. Notably, none of the identified clusters consistently scores high or low on all equity dimensions. This suggests that all of the clusters perform inconsistently to different degrees across the dimensions of equity. Furthermore, no single cluster emerges as superior to the others across both outcome measures.ConclusionsThese findings shed light on the intricate interplay between equity, achievement, and motivation within the examined educational systems. The unique configuration of the equity dimensions within each cluster underscores the importance of adopting a nuanced evaluation of equity, which can enhance our understanding of equity patterns.

Detecting latent subpopulations in international large-scale assessments by fitting MixIRT models using NUTS

The focus of this study is to use the mixture item response theory (MixIRT) model while implementing the no-U-turn sampler as a technique for investigating the presence of latent classes (i.e., subpopulations) among eighth-grade students who were administered TIMSS 2019 mathematics subtest in paper format from the gulf cooperation council (GCC) countries. One-, two-, and constrained three-parameter logistic MixIRT models with one to four classes were used to fit to the data, where the model data fit was assessed using Bayesian fit indices. The results indicate that multiple latent classes or subpopulations can better reflect the mathematical proficiency of eighth graders from the four GCC countries, and specifically the two-class constrained three-parameter MixIRT model provides a relatively better fit to the data. The results also indicate that when a mixture of several latent classes present, the conventional unidimensional IRT model is limited in providing information for multiple latent classes and shall be avoided. In addition to adding to the existing literature on MixIRT models for international large-scale assessments such as TIMSS on its heterogenous subpopulations from a fully Bayesian approach, this study sheds light on the limitation of conventional unidimensional IRT models and subsequently directs attention to the use of the more complex MixIRT model for such assessments.

Writing PISA-Like Mathematics Items: The Case of Tertiary Mathematics Instructors from a State University in the Philippines

<p style="text-align:justify">Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) are nested in contexts defined in their assessment framework (e.g., the Personal, Occupational, Societal, and Scientific contexts in PISA). This study followed the item-writing activities of four tertiary mathematics instructors in the Philippines as they constructed context-based mathematics items. They were tasked to write four items each, following a set of specifications for PISA content and context categories. The data consisted of transcripts from the focus-group discussion which was conducted days after the task. The transcripts were then analyzed using thematic analysis. The results of this study showed that the phenomenon of item-writing in the context of writing PISA-like mathematics items had two themes: the phases of item-writing and the dimensions of item-writing. Findings showed that the respondents struggled to find realistic contexts and that they engaged in a problem-solving task likened to solving a puzzle as they attempted to satisfy the content, context, and process categories in the table of specifications (TOS). This study contributes to filling in the research gap on item-writing activities, particularly those of mathematics teachers in the Philippines- a country whose recent mathematical performance in the PISA 2018, TIMSS 2019, and PISA 2022 was nothing short of dismal.</p>

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

AbstractIn international large-scale assessments, student performance comparisons across educational systems are frequently done to assess the state and development in different domains. These results often have a large impact on educational policy and on the perceptions of an educational system’s performance. Early assessments, such as the First and Second International Science Studies (FISS and SISS), have been used alongside recent studies to create unique scales for investigating changes in constructs. The implicit assumptions in system comparisons are that the measures are valid, reliable, and comparable. However, these assumptions have not always been investigated thoroughly. This study aims to investigate the validity and cross-system comparability of scores from the FISS and SISS, conducted by the International Association for the Evaluation of Educational Achievement in 1970–1971 and 1983–1984. Findings based on item response theory (IRT) modeling indicate that scores in most educational systems can be viewed as reliable measures of a single science construct, supporting the validity of test score interpretations in these educational systems individually. In a robust assessment of measurement invariance using standard IRT methods, an alignment-based method, and the root mean square difference (RMSD) fit statistic, we demonstrate that measurement invariance is violated across systems. The alignment-based method identified a well-fitting model with complex restrictions but no items exhibited invariance across all systems, a result supported by the RMSD statistics. These results question the appropriateness of score comparisons across systems in FISS and SISS. We discuss the implications of these results and outline consequences for score comparisons across time.

Does between-school tracking increase school segregation among migrant students? A difference-in-differences and multiverse analysis of international large-scale assessment data

ABSTRACT This study examines the role of educational tracking in shaping school segregation among first and second-generation migrants, a pressing issue for societal integration. Drawing on the pooled data from international student assessments (PISA, PIRLS, & TIMSS) from 1995 to 2019, we examine a total of 75 countries and assess the extent to which early tracking exacerbates segregation compared to more integrated school systems. Our analysis, based on a Difference-in-Differences approach, reveals no significant differences in migrant school segregation between tracked and integrated systems. Robustness checks, employing multiverse analyses, confirm the stability of our findings across alternative model specifications. However, considerable variation in migrant school segregation exists within both types of education systems. This underscores the importance of investigating additional factors beyond early tracking contributing to variation in migrant school segregation. Our study highlights the complex interplay between educational policies and migrant integration in school settings.

Uncovering student profiles. An explainable cluster analysis approach to PISA 2022

Educational data mining (EDM) applied to the wealth of data generated from international large-scale assessments (ILSAs) shows potential for identifying successful educational initiatives. Despite limited research on clustering methods in ILSAs, leveraging these methods to uncover student profiles can help decision-making in designing tailored programs. This study aims to identify and characterize 15-year-old student profiles using PISA 2022 data and reveal insights into the relationship between these profiles and factors such as ICT availability and use, gender, academic performance, and educational expectations. We analyzed PISA 2022 Spanish student data (n = 30,800) with a selection of 74 contextual variables, applying an end-to-end explainable cluster analysis methodology that integrates different machine learning (ML) and explainable artificial intelligence (XAI) techniques. This methodology covered data pre-processing, dimensionality reduction, clustering, and classification to ensure data quality and result explainability. We obtained 16 derived variables, 7 student clusters, and an optimal XGBoost classifier with a global accuracy of 0.8643. Using local and global SHAP values, we interpreted clusters, finding that socio-economic status and ICT availability and use at home are the most important factors differentiating student profiles. Our findings suggest the need to emphasize (i) proper ICT accessibility and use, as well as student support networks to improve academic performance, (ii) gender-specific well-being programs, and (iii) the encouragement of educational expectations tailored to students’ gender and their exposure to higher education. These results pave the way for personalized academic policies and programs through ML-based tools for uncovering student profiles.

The International Large-Scale Assessment market: Understanding its emergence and rationales

How did a network of passionate academics with limited funding but big, comparative, education research ideas lead to an international assessment market? This paper explores the interests, capitals, and power dynamics embedded in the International Large-Scale Assessment (ILSA) market through a network ethnography to understand how the market emerged and what drives actors despite the lack of direct profit. This paper constructs an interdisciplinary framework combining sociological and business studies to analyse qualitative data on OECD and IEA ILSAs. Highlighting the role of social-cultural-economic capitals and business expanding processes, the paper describes an ILSA market that was and remains a non-competitive, desirable, and exclusive market that is impenetrable to outside actors. The paper concludes by asking how the ILSA market contributes to education as a common good.

Contradictory rationales behind national education export: tracing the policy formation processes of EDU-Port Japan

ABSTRACT National education export strategies are on the rise. Many countries of the Global North brand and export their educational models, with Finland and Singapore doing so on the back of their success in international large-scale education assessments. This article investigates the policy formation processes of EDU-Port Japan, a nationally coordinated education export strategy launched in 2016. Touted as multi-ministerial and public–private partnerships, EDU-Port aims to globally promote and export ‘Japanese-style education’. The article elucidates how EDU-port emerged as a tentative assemblage of contingent political, economic and geopolitical conditions faced by the Japanese state at a specific historical juncture. By illuminating tensions and contradictions within EDU-Port, reflective of the conflicting interests among the diverse policy actors, the article challenges the assumed linearity and coherence of national education export strategies, uncovering complexities often overlooked in current scholarship on national education export.

Circumventing construct-irrelevant variance in international assessments using cognitive diagnostic modeling: A curriculum-sensitive measure

International large-scale assessments such as TIMSS administer achievement tests that are based on an analysis of national curricula to compare student achievement across countries. The organizations that coordinate these studies use Rasch or more generalized item response theory (IRT) models in which all test items are assumed to measure a single latent ability. The test responses are then used to estimate this ability, and the ability scores are used to compare countries.A central but yet-to-be-contested assumption of this approach is that the achievement tests measure an unobserved unidimensional continuous variable that is comparable across countries. One threat to this assumption is the fact that countries and even regions or school tracks within countries have different curricula. When seeking to fairly compare countries, it seems legitimate to account for the fact that applicable curricula differ and that some students may not have been taught the full test content yet. When seeking to fairly compare countries, it seems imperative to account for the fact that national curricula differ and that some countries may not have taught the full test content yet. Nevertheless, existing IRT-based rankings ignore such differences.The present study proposes a direct method to deal with differing curricula and create a fair ranking of educational quality between countries. The new method compares countries solely on test content that has already been taught; it uses information on whether students have mastered skills taught in class or not and does not consider contents that have not been taught yet. Mastery is assessed via the deterministic-input, noisy, “and” gate (DINA) model, an interpretable and tractable cognitive diagnostic model. To illustrate the new method, we use data from TIMSS 1995 and compare it to the IRT-based scores published in the international study report. We find a mismatch between the TIMSS test contents and national curricula in all countries. At the same time, we observe a high correlation between the scores based on the new method and the conventional IRT scores. This finding underscores the robustness of the performance measures reported in TIMSS despite existing differences across national curricula.

Using international large-scale assessment for learning: Analyzing U.S. students’ geometry performance in TIMSS

The data from international large-scale assessments, such as The Trends in International Mathematics and Science Study (TIMSS), is often designed and used as an assessment of learning rather than an assessment for learning. This research employed TIMSS 2011 data, focusing on the geometric performance of fourth-grade students in the United States, to demonstrate how large-scale assessments can be utilized qualitatively to identify students' learning challenges, providing valuable insights to inform geometry teaching practices. Using van Hiele’s levels of geometric thinking as a guiding framework, the 24 released geometry items were analyzed to identify students’ struggle points in geometry nationwide. Our analysis revealed that students performed well on tasks that were visual, contextually grounded, and had clearly defined manipulative expectations.. However, students needed further support to solve items requiring a deep understanding of geometrical concepts, such as finding the perimeter of a square. Additionally, students excelled at solving tasks involving the identification of relative positions of objects on a map and recognizing lines of symmetry in regular shapes. However, they struggled with measurement-related tasks. Our analysis also identified a range of item features that might cause difficulties that impacted the way students responded to these items. Instructional implications for elementary mathematics education are discussed.

The promises and expectations of ILSAs regarding policymaking: lessons from Latin America

ABSTRACT Standardised International Large-Scale Assessments (ILSAs) have gained prominence in global and national educational discussions. ILSAs claim to offer valuable insights for improving education systems, but their impact on educational policy varies and has become a contested arena. This article analyses how these assessments fed educational policymaking in six Latin American countries based on a review of policy documents; the article advances three theses on how ILSAs are used by policymakers: First, there is a tokenistic usage of ILSAs; second, ILSAs must be considered more as political devices bolstering national reputation rather than studies in the academic sense; third, ILSAs can serve as leverage tools that can be mobilised for broader political ends. The study shows that the promises, designs and reporting of ILSAs are not necessarily aligned, that participation in ILSAs has become a symbolic gesture, and that ILSAs’ data are often cherry-picked to support pre-existing diagnoses and policy agendas.

Designing an International Large-Scale Assessment of Professional Competencies and Employability Skills: Emerging Avenues and Challenges of OECD’s PISA-VET

Globally, vocational education and training (VET) is considered important for ensuring the supply of skilled labour to the economy and economic competitiveness but also for helping the next generation with the transition to working life and integration into society. However, despite this importance, there are no international comparative studies on the effectiveness of the very different VET systems. In March 2024, the Organisation for Economic Co-operation and Development (OECD) published the ‘Analytical and Assessment Framework’ for PISA-VET, an international study on professional competencies and employability skills in VET. In this paper, some of the lead experts that contributed to the framework provide an outline of the aims of the initiative, the target groups, the assessment approaches as well as strength and weaknesses to stimulate discussion in the scientific community. VISA-VET aims to deliver comprehensive data, inform decision making, facilitate peer learning between countries, and promote the image of VET, in general. Target populations are learners toward the end of their VET programmes in the occupational areas of automotive technicians, electricians, business and administration, health care, or tourism and hospitality. Assessment approaches to domain-specific professional skills are simulation-based questions, digital simulations, and live or recorded demonstrations. The professional skills assessments are expanded by the assessment of employability skills and comprehensive data collections on national contextual and system-level factors. This paper discusses the selection and breakdown of occupational areas, the various assessment approaches and possible supplementary studies. Its overall aim is to initiate a broader discussion in the scientific community about the design of and expected insights from PISA-VET.

Using convolutional neural networks to automatically score eight TIMSS 2019 graphical response items

International large-scale assessments (ILSAs) have used graphical response-based items to measure student ability for decades, but they have yet to implement automated scoring of these responses and instead rely on human scoring alone. To investigate how scores provided by machine algorithms compare to those provided by human raters, we applied convolutional neural networks (CNNs) to classify image-based responses from eight Timss 2019 items. Our results show that the most accurate CNN models classified over 99% of the image responses into the appropriate scoring category for dichotomous items and almost 98% for one trichotomous item. Additionally, during the modeling process, the CNNs correctly classified numerous image responses that human raters had scored incorrectly. For most items, the number of incorrectly human-scored responses exceeded the average number of responses misclassified by the most accurate models. These results suggest that automated scoring using CNNs is comparable to, and in many cases more accurate, than human raters, even across a wide variety of graphing tasks. This paper argues that the machine learning procedure explored could be implemented in ILSAs as a verification method to improve the accuracy and consistency of graphical response item scores. In lieu of additional human raters, ILSAs could implement CNN-based automated scoring to provide a second set of scores, thus reducing the workload and costs associated with human scoring.

Treason and revenge: the emergence and continuation of ILSA contracting

ABSTRACT OECD and IEA International Large-Scale Assessments (ILSAs) contractors have remained largely invisible despite playing a key role in the making of education assessment data. With the help of Bourdieu’s thinking, this paper analyses data generated through document analysis of ILSA Reports and in-depth interviews with staff working on ILSAs at the IEA, the OECD and ILSA contractors. The paper traces education assessment developments in the USA and at the IEA and the OECD since the 1980s, and points to the critical role of Education Testing Services (ETS). Although the selection of ILSA contractors ostensibly occurs through a globally competitive tendering process, this paper reveals the pressures and struggles that emerged when economic capital became available at the IEA and the OECD. The struggles reveal PISA was created as a competing project to TIMSS. The paper demonstrates how the social and cultural capitals that individuals acquired, and in particular trust, shaped who can and who cannot produce ILSA data today. Overall, the paper makes visible the ILSA data chefs and their recipes: ILSAs are not raw, they are cooked under pressure with personal and organizational tensions, struggles, conflicts, and emotional bonds.

Cognitive skills and economic growth in the twenty-first century: Evidence from PISA and cognitive ability studies

In recent years, some educational researchers have questioned the theory of a robust relationship between test scores in international large-scale assessments such as PISA and lagged economic growth. However, both the proponents and opponents of such a human capital approach have mainly focused on earlier growth patterns during the twentieth century. Hence, the current article analysed the relationship between PISA 2000 and economic growth in 2010–2019 for 31 countries, and composite cognitive skills for 80 countries and their relationships with economic growth in 2010–2019. The results indicate that PISA scores or other cognitive skill measures could to some extent predict economic growth between 2010 and 2019, even when controlling for a significant number of variables. In a consecutive analysis, the relations were substantially associated with Middle Eastern countries because many countries from this region experienced political instability and decreasing oil revenues throughout this period. Hence, it is possible that regional and contextual relationships affect these patterns.

The potential of following-up international large-scale assessment studies: using PISA 2018 to develop a comprehensive model of effective teaching

ABSTRACT The paper investigates the potential of using international large-scale assessment studies for conducting follow-up studies testing models of educational effectiveness. The impact of teacher factors coming from the dynamic model of educational effectiveness and the dialogic education theory on student literacy achievement is examined. Modern Greek teachers from secondary schools that participated in PISA 2018 (Accessed on https://www.oecd.org/pisa/data/pisa2018technicalreport/) in Cyprus were recruited. Two Grade 11 classes per teacher were selected, giving 392 students in total. Observation data from Greek lessons and student literacy achievement data were collected. Two multilevel regression analyses were conducted: one with the whole student sample and one with only students from whom PISA prior achievement measure was identified. When prior achievement (i.e., PISA measure) was considered, additional and larger effects of factors on student final achievement were detected. Both analyses reveal that when factors from both frameworks were considered, larger variance in student achievement was explained.

Combining machine translation and automated scoring in international large-scale assessments

BackgroundArtificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies.MethodsThis study proposes combining state-of-the-art machine translations (i.e., Google Translate & ChatGPT) and artificial neural networks (ANNs) to mitigate two key concerns of human scoring: inconsistency and high expense. We applied AI-based automated scoring to multilingual student responses from eight countries and six different languages, using six constructed response items from TIMSS 2019.ResultsAutomated scoring displayed comparable performance to human scoring, especially when the ANNs were trained and tested on ChatGPT-translated responses. Furthermore, psychometric characteristics derived from machine scores generally exhibited similarity to those obtained from human scores. These results can be considered as supportive evidence for the validity of automated scoring for survey assessments.ConclusionsThis study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs.

Assessment of Academic Resilience and Its Associated Factors Among Pharmacy Students in Twelve Countries

ObjectiveAcademic resilience, a critical determinant of academic achievement, is affected by various factors. There is a paucity of large-scale international assessments of academic resilience among pharmacy students. Therefore, this study aimed to assess academic resilience among pharmacy students in 12 countries and to evaluate factors associated with their academic resilience levels. MethodsA cross-sectional online survey-based study was conducted among randomly selected pharmacy students in 12 countries: Egypt, Türkiye, Indonesia, Pakistan, Bangladesh, Iraq, Jordan, Nigeria, Malaysia, Saudi Arabia, Sudan, and the United Arab Emirates. After pilot testing, the validated 30-item academic resilience scale (ARS) was used for the assessment. The data were collected between November 1, 2022 and April 15, 2023. Descriptive and inferential statistics were performed, as appropriate. ResultsA total of 3950 were received from the 12 participating countries. The mean age was 21.68 ± 2.62 years. About two-thirds of the responses were from female participants and those studying for Bachelor of Pharmacy degrees. Overall, the findings show moderate academic resilience, which varied across countries. The median (IQR) of the total ARS-30 was 114 (103−124). Females exhibited lower negative affective and emotional response subscale levels than males. There were significant cross-country variations in the ARS-30 and all subscales. The highest overall levels were reported for Sudan, Pakistan, and Nigeria and the lowest were reported for Indonesia and Türkiye. Students in private universities tended to have higher overall ARS levels than public university students. Higher academic performance was significantly associated with ARS levels, whereas those with excellent performance exhibited the highest ARS levels. Students with exercise routines had higher ARS levels than those without exercise routines. Finally, students who were engaged in extracurricular activities had higher ARS levels than those who did not participate in these activities. ConclusionThe study offers insights into the factors affecting academic resilience in pharmacy students across several countries. The findings could guide interventions and support activities to improve resilience and academic outcomes.