Programme For International Student Assessment Assessments Research Articles

Understanding the disparities in Mathematics performance: An interpretability-based examination

Problem:Educational disparities in Mathematics performance are a persistent challenge. This study aims to unravel the complex factors contributing to these disparities among students internationally, with a focus on the interpretability of the contributing factors. Methodology:Utilizing data from the Programme for International Student Assessment (PISA), we conducted rigorous preprocessing and variable selection to prepare for applying binary classification interpretability models. These models were trained using the Stratified K-Fold technique to ensure balanced representation and assessed using six key metrics. Solution:By applying interpretability models such as Shapley Additive Explanations (SHAP) analysis, we identified critical factors impacting student performance, including reading accessibility, critical thinking skills, gender, and geographical location. Results:Our findings reveal significant disparities linked to resource availability, with students from lower socioeconomic backgrounds possessing fewer books and demonstrating lower performance in Mathematics. The geographical analysis highlighted regional educational disparities, with certain areas consistently underperforming in PISA assessments. Gender also emerged as a determinant, with females contributing differently to performance levels across the spectrum. Conclusion:The study provides insights into the multifaceted determinants of student Mathematics performance and suggests potential avenues for future research to explore global interpretability models and further investigate the socioeconomic, cultural, and educational factors at play.

Is the SES and academic achievement relationship mediated by cognitive ability? Evidence from PISA 2018 using data from 77 countries.

Earlier research has suggested that that the international large-scale assessment, PISA (Programme for International Student Assessment), may be looked upon as a form of school test that is mostly explained by participating students' socioeconomic status, non-cognitive factors, and various school factors, whereas another strand of research focuses on the similarities between PISA and cognitive ability assessments such as IQ tests. The latter position does also highlight the strong relationships between PISA scores and IQ test scores, typically aggregated to the country level. The current article adds to this scholarly debate by examining the latest PISA survey from 2018. Correlation, regression, moderator, and mediation analyses were run for aggregated country-level data (N = 77) from PISA 2018. The results indicate that PISA scores indeed are strongly associated with both cognitive ability test scores and socioeconomic status indicators such as average annual national income. A more nuanced position suggests that PISA should be labelled a test that measures cognitive school skills rather than a proxy of national IQ, as the link between country-level abilities and school age abilities is neither causal, nor theoretically and empirically palpable, yet partly robust.

Twenty years on – what can PISA tell us about educational inequality in France and England?

ABSTRACT This article examines French and English average performances and educational inequality in the Programme for International Student Assessment (PISA) between 2000 and 2018. It asks why English average scores in 2018 are much higher than those of France when they were fairly similar in previous PISA assessments. It questions why the impact of socio-economic background on educational performance in England decreased while the reverse has been the case for France. This is all the more damning given the French republican espousal of égalité. It examines policies in both countries and analyses to what extent PISA has impacted on these. It argues that in England, policies of autonomy, accountability and competition, in line with the OECD and its own tradition of liberalism, have been implemented rigorously. In France, because of left-right oppositional dynamics and the role of embedded republican principles, the status quo has been maintained.

Linking via Pseudo‐Equivalent Group Design: Methodological Considerations and an Application to the PISA and PIAAC Assessments

AbstractThis article presents the pseudo‐equivalent group approach and discusses how it can enhance the quality of linking in the presence of nonequivalent groups. The pseudo‐equivalent group approach allows to achieve pseudo‐equivalence using propensity score reweighting techniques. We use it to perform linking to establish scale concordance between two assessments. The article presents Monte‐Carlo simulations and a real data application based on data from the Survey of Adult Skills (PIAAC) and the Programme for International Student Assessment (PISA). Monte‐Carlo simulations suggest that the pseudo‐equivalent group design is particularly useful whenever there is a large overlap across the two groups with respect to balancing variables and when the correlation between such variables and ability is medium or high. The example based on PISA and PIAAC data indicates that the approach can provide reasonable accurate linking that can be used for group‐level comparisons.

An Exploratory Study on Key Issues and Expected Benefits from the Introduction of Financial Literacy Assessment in PISA in Economic Education

본 연구에서는 대표적인 국제역량평가인 국제학업성취도평가(Programme for International Student Assessment: 이하 PISA)의 금융소양 영역에 대해 소개함으로써 경제교육 관련자들의 관심을 촉구하고, 이에 대한 우리나라와 주변국의 적극적인 참여를 요청하고 있다. PISA 금융소양 영역은 학생들이 오늘날 사회에서 직면하는 금융 분야의 문제 상황에 대해 자신이 가지고 있는 지식을 어떻게 활용할 수 있는가를 평가하는 것에 초점이 맞추어져있다. 우리나라는 예산의 부담, 시행 상의 어려움, 주요 비교대상국의 부재 등의 문제로 PISA 금융소양 평가에 참여하지 않고 있다. 하지만 PISA 금융소양 평가에 참여함으로써 얻을 수 있는 이득은 적지 않다. PISA 결과에 나타난 국가 내 심층 분석 자료는 교육 정책 개선의 방향을 제시하며, 국가 간 비교 자료는 정책을 구체화시킬 수 있는 아이디어를 제공한다. 또한 경제교육과 정이 지식뿐만 아니라 실제 경제생활과 관련된 문제 해결 능력이나 지식 적용 능력을 길러줄 수 있는 방향으로 개선될 수 있도록 하는 데 도움을 줄 수 있다. 마지막으로 금융교육에 대해 미온적인 일반의 인식을 개선하고 학교 및 학교 밖 금융 및 경제교육을 활성화할 수 있는 계기로 삼을 수 있다는 점에서 의미가 있다.This study introduced the financial literacy assessment in PISA(Programme for International Student Assessment), one of the best international competence assessment, and called for the concerns and interests of the authorities concerned with financial education as well as the participation of Korea and neighboring countries in PISA. The financial literacy assessment in PISA focuses on how to assess the students utilizing their knowledge to address the financial problems they face daily. Korea currently does not participate in this PISA assessment due to several reasons: budget constraints for the payment of participant dividends, class absence, loss of focus due to additional assessment hour, and the absence of comparative countries. Nonetheless, this study believe that the following benefits by introducing and participating in the financial literacy assessment in PISA. The national data analysis of PISA test results can suggest the ways to improve financial education policies, while inter-national analysis will provide us of the more specific ideas to realize those policies. It also can help to reform the national curriculum on financial and economic education to cultivate not only economic knowledge but competencies for economic problem-solving and knowledge application in an actual economic life. Lastly, it can be an opportunity for us to invigorate social as well as school financial education, and to modify the public indifferences toward financial education.

Poverty and educational achievement in the US: A less-biased estimate using PISA 2012 data

ABSTRACTIn the current paper, we employ the most recent Programme for International Student Assessment (PISA) data to calculate a less-biased estimate of poverty on US achievement. The PISA was specifically chosen as it is an assessment removed from a specific curriculum and instead focuses on concepts that students should know in order to participate in a global economy. Using a propensity score matching approach, our findings suggest that US students in poverty have notable educational attainment deficiencies compared to a matched group of students who are not in poverty. In other words, when we select two students who have a great deal in common but for the fact that one comes from a poverty background, the student in poverty is expected to perform nearly 28 points, or about a quarter of a standard deviation lower, on the PISA assessment. In real terms, this puts math achievement for children not in poverty on-par with the Organisation for Economic Co-operation and Development (OECD) average, while chi...

Chinese responses to Shanghai’s performance in PISA

ABSTRACTThis article analyses the public responses in China to Shanghai’s performance in the 2012 Programme for International Student Assessment (PISA). Based on data obtained from media accounts and other materials published between 2013 and 2016, the research findings show that the responses in China are generally reflective, measured and self-critical. Drawing upon Gadamer’s notions of ‘tradition’, ‘horizon’ and ‘prejudice’, this paper contends that the responses reflect the prevailing worldviews in China that perceive Shanghai’s education system to be academically rigorous but too exam-oriented and burdensome. It is further argued that Confucian knowledge traditions and structures in China shape the Chinese interpretations of the PISA assessment format, leading them to downplay Shanghai’s success. This study introduces the metaphor of ‘triadic eyes’ and highlights the mediating effects of the ‘local eyes’ of policy actors in an era of global educational governance.

Does more exposure to the language of instruction lead to higher academic achievement? A cross-national examination

Aims and objectives: As some Programme for International Student Assessment (PISA) studies claimed that native speaking (NS) students outperform language minority (LMi) students, far-reaching inferences have been drawn by policymakers. However, previous PISA assessments were not appropriate because they only included a dichotomous home language variable. The main objective of this study is to gain a better understanding of how students’ language background and use are related to academic achievement. Design: The PISA data from 2012 provides a unique opportunity to fill this research lacuna as it includes a more elaborated questionnaire on language background and use. Data and analysis: Multivariate three-level analyses are conducted on PISA 2012 data from 18 countries, covering about 5,000 schools and 120,000 students. Findings: The results show that there is an achievement gap between LMi and NS students for both reading and math. After controlling for students and school characteristics, the LMi–NS achievement gap narrows, but remains significant. This holds true for most countries. However, language use per se is not the cause of underachievement: LMi students who more often speak a minority language with their parents do not achieve less. In some countries, speaking a minority language more often with parents is actually positively related to math and reading achievement. Nevertheless, speaking the instruction language in the school context is positively associated with math and reading achievement. Originality and significance: This study revealed that the relation between language use and academic achievement is more complex than it was conceptualized in most previous PISA studies. Scholars need to go beyond the dichotomous approach to achieve a better understanding of language use. Our results show that linguistic diversity could function as an asset for academic performance, at least if a good balance between focus on minority languages at home and instruction language at school can be found.

School curriculum in Japan

ABSTRACTThis article examines Japanese education system especially relevant to the school curriculum, which might support Japanese high performance in the OECD's Programme for International Student Assessment (PISA), mainly through Japanese policy documents. The Japanese education systems have been constructed by the local context of society and politics, and it is hard to analyse the reasons why Japan has seen better results in the previous PISA assessments than other OECD countries. However, the 2008–2009 revision of the Course of Study (the national curriculum in Japan) shows that the Japanese government was really concerned about the decline of Japan's ranking in the previous PISA assessments and revised the Course of Study in response to children's skills which the PISA aimed to measure (key competencies). The Japanese government has placed importance on the cultivation of human resources which are necessary to compete within the international community, and the PISA has had a large effect on the direction of Japanese education.

PISA, TIMSS and Finnish mathematics teaching: an enigma in search of an explanation

Finnish students’ success on all three content domains of each of the four cycles of the OECD’s Programme for International Student Assessment (PISA) has created much international interest. It has also prompted Finnish academics to offer systemic explanations typically linked to the structural qualities of Finnish schooling and teacher education. Less well-known has been the modest mathematics performance of Finnish grade 8 students on the two Trends in International Mathematics and Science Study (TIMSS) in which Finland has participated, which, when compared with its PISA successes, has created something of an enigma. In this paper, we attempt to shed light on this enigma through analyses of Finnish mathematics classroom practice that draw on two extant data sets—interviews with Finnish teacher educators and video-recordings of sequences of lessons taught on standard topics. Due to the international interest in Finnish PISA success, the analyses focus primarily on the resonance between classroom practice and the mathematical literacy component of the PISA assessment framework. The analyses indicate that Finnish mathematics didactics are more likely to explain the modest TIMSS achievements than PISA successes and allude to several factors thought to be unique to the Finns, which, unrelated to mathematics teaching practices, may be contributory to the repeated Finnish PISA successes. Some implications for policy-borrowing are discussed.

Special educational needs, disability and school accountability: an international perspective

Standards-based accountability approaches in education are increasingly being adopted in many industrialised countries. This tends to involve the scrutiny of student performance in national academic tests and then holding schools accountable for subsequent progress. International assessments such as Programme for International Student Assessment (PISA) are not accountability measures in themselves, but are often utilised in a similar way and national policies are judged against performance in these tests. Drawing upon the examples of PISA and national assessments in the USA, this paper explores how students with special educational needs and disability (SEND) are included in, and served by, these assessment policies. The analysis suggests that while these approaches are inclusive in their intent (most obviously through efforts to include students with SEND in testing) they may poorly represent the progress of these students. It is argued that more equitable inclusion of students with SEND must find a balance between accountability measures which measure performance in traditional attainment-focused tests and other relevant assessments which are meaningful to the given student and/or SEND sub-group.

Sex differences in mathematics and reading achievement are inversely related: within- and across-nation assessment of 10 years of PISA data.

We analyzed one decade of data collected by the Programme for International Student Assessment (PISA), including the mathematics and reading performance of nearly 1.5 million 15 year olds in 75 countries. Across nations, boys scored higher than girls in mathematics, but lower than girls in reading. The sex difference in reading was three times as large as in mathematics. There was considerable variation in the extent of the sex differences between nations. There are countries without a sex difference in mathematics performance, and in some countries girls scored higher than boys. Boys scored lower in reading in all nations in all four PISA assessments (2000, 2003, 2006, 2009). Contrary to several previous studies, we found no evidence that the sex differences were related to nations’ gender equality indicators. Further, paradoxically, sex differences in mathematics were consistently and strongly inversely correlated with sex differences in reading: Countries with a smaller sex difference in mathematics had a larger sex difference in reading and vice versa. We demonstrate that this was not merely a between-nation, but also a within-nation effect. This effect is related to relative changes in these sex differences across the performance continuum: We did not find a sex difference in mathematics among the lowest performing students, but this is where the sex difference in reading was largest. In contrast, the sex difference in mathematics was largest among the higher performing students, and this is where the sex difference in reading was smallest. The implication is that if policy makers decide that changes in these sex differences are desired, different approaches will be needed to achieve this for reading and mathematics. Interventions that focus on high-achieving girls in mathematics and on low achieving boys in reading are likely to yield the strongest educational benefits.

Hypothetical Use of Multidimensional Adaptive Testing for the Assessment of Student Achievement in the Programme for International Student Assessment

The usefulness of multidimensional adaptive testing (MAT) for the assessment of student literacy in the Programme for International Student Assessment (PISA) was examined within a real data simulation study. The responses of N = 14,624 students who participated in the PISA assessments of the years 2000, 2003, and 2006 in Germany were used to simulate MAT with different restrictions (unrestricted, treatment of link items, treatment of open items, content balance, unitwise item selection, all restrictions). Compared with conventional testing based on the booklet design of PISA 2006, unrestricted MAT increases measurement efficiency by 74% and reduces the average number of presented items from 55 to 26 without a loss in measurement precision. The incorporation of restrictions reduces the advantages of MAT. MAT is recommended for the assessment of newly introduced constructs but not for the assessment of the literacy domains in PISA.

What enhances mathematical ability? A cross-country analysis based on test scores of 15-year-olds

This article examines the mathematical abilities of 15-year-olds in a range of countries which participated in the 2003 cycle of the Organization for Economic Cooperation and Development (OCED)'s Programme for International Student Assessment (PISA). Utilizing information on the scores obtained by individual students in the mathematical part of the PISA assessment, we use a range of indicators from the literature on inequality and poverty to evaluate the ‘mathematical performance’ of participating countries. Since data from PISA contained a wealth of information on the circumstances of the students in terms of their home and school environment, we identify and examine the relative influence of factors which serve to enhance the mathematical performance of students in the PISA assessment.

Test motivation in the assessment of student skills: The effects of incentives on motivation and performance

There is widespread concern that assessments which have no direct consequences for students, teachers or schools underestimate student ability, and that the extent of this underestimation increases as the students become ever more familiar with such tests. This issue is particularly relevant for international comparative studies such as the IEA’s Third International Mathematics and Science Study (TIMSS) and the OECD’s Programme for International Student Assessment (PISA). In the present experimental study, a short form of the PISA mathematical literacy test is used to explore whether the levels of test motivation and test performance observed in the context of the standard PISA assessment situation can be improved by raising the stakes of testing. The impact of (1) informational feedback, (2) grading, and (3) performance-contingent financial rewards on the personal value of performing well, perceived utility of participating in the test, intended and invested effort, task-irrelevant cognitions, and test performance are investigated. The central finding of the study is that the different treatment conditions make the various value components of test motivation equally salient. Consequently, no differences were found either with respect to intended and invested effort or to test performance.