Programme For International Student Assessment Research Articles

Illuminating the shadows: the role of private supplementary tutoring on student math performance in PISA 2022

Recent decades have witnessed a global expansion of private supplementary tutoring, known as shadow education—additional support in academic subjects that takes place outside of regular school hours. Using the data of 55 countries from the Programme for International Student Assessment (PISA) 2022, this study examines the association between students’ participation in shadow education and their academic achievement in mathematics and how these relationships change for developed, developing, and East Asian regions. This study also sheds light on the moderating effect that socio-economic status has on the relationship between engagement in shadow education and math performance. Multilevel regression analyses revealed that, at the between-school level, only one of the five forms of shadow education, namely the degree to which students received asynchronous video-recorded instruction, was associated with improved math performance. Similarly, at the within-school level, only the degree to which students received video-recorded instruction was statistically significantly associated with improved mathematical outcomes. Incidentally, we find that institutional-related school-level factors such as the overall quality of student–teacher relationships, provision of math-related extra-curricular activities, and support in terms of peer-to-peer tutoring appear to be associated with improved school math performance, while the quality of student–teacher relationships and level of math teacher support also feature positively for students within-schools. Regarding differential effects by region, at the between-school level, only the degree to which students received large-group study or practice was associated with math performance in East Asian countries. With regards to the moderating effect of SES on the positive relationship between asynchronous shadow education and math performance, we find the effects to be negative, suggesting that asynchronous engagement might be more beneficial for students from lower SES backgrounds. Finally, while the overall negative relationships between synchronous forms of shadow education and math performance appear counter-intuitive, we speculate that this may reflect the use of such tutoring for remedial purposes. We suggest that further research into these “remedial reverse effects” in post-COVID AI-bot-capable educational contexts may provide a more definitive understanding of the role that student engagement in shadow education has on their academic performance.

The Influence of Leadership, Cooperation, Welfare, and Work Ethic on Madrasah Educator Performance: The Role of Motivation and Competence

Education is the key to progress and a challenge for every country, especially for developing countries like Indonesia. The problem of education quality in Indonesia can be seen from the low achievement of madrasahs and the results of the 2022 Program for International Student Assessment (PISA) evaluation. This research aims to explore the variables that influence the performance of educators in Bandar Lampung, with a focus on the leadership of madrasah heads, cooperation, welfare, motivation, personality competence, and work ethic of educators. This research uses a qualitative approach with exploratory and causal methods to identify and analyze the relationship between these variables. Data was collected through surveys and analyzed using multiple linear regression. The results of the research show that partially and simultaneously, the leadership of the madrasa head, cooperation, welfare, motivation, personality competence and work ethic have a significant effect on the performance of educators. Educator competency is the most dominant variable influencing educator performance. This research underlines the importance of good management of human resources in education to improve the quality of education. Recommendations are given to improve the performance of educators through improving leadership management, collaboration between educators, increasing welfare, motivation, as well as developing the competence and work ethic of educators.

Expecting less in hard times: How the state of the economy influences students’ educational expectations

Students’ realistic aspirations about their educational attainment (expectations) are predictive of their efforts, actions, and future outcomes. Limited evidence suggests these expectations are affected by the macroeconomy; however, the direction is ambiguous. We combine seven waves of Programme for International Student Assessment (PISA) data to examine this relationship in 38 OECD countries. Using within-country fixed-effect regressions, we find students have significantly lower educational expectations when GDP growth is low or negative. Assessing the heterogeneity of these relationships, we find that the expectations of students with below-average reading skills or who attend non-metropolitan schools are most strongly affected by the state of the economy. The results also suggest that when GDP growth is weak, students are significantly more likely to complete zero homework and arrive late to school, are less likely to participate in extracurricular academic programs, and expect lower labour market returns to completed education.

Enhancing science achievement through inquiry activities: the critical role of perceived teacher support

ABSTRACT In science education, scientific inquiry is crucial for developing inquiry skills and constructing scientific knowledge. However, the impact of scientific inquiry activities on student achievement continues to be debated. A number of counterintuitive findings highlight the need to explore the underlying factors influencing the effectiveness of inquiry-based learning. This study employed hierarchical linear models (HLM) to investigate the role of students’ perceived school teacher support in moderating the relationship between inquiry-based science activities and science achievement. The sample used in this study was drawn from Beijing, Shanghai, Jiangsu, and Guangdong in mainland China (B-S-J-G-China). These four provinces participated in the 2015 Programme for International Student Assessment (PISA) as a collective group. The dataset comprised 9,592 students (4,543 females and 5,049 males) from 261 schools. Results revealed a negative correlation between inquiry-based activities and science achievement, indicating that a higher frequency of inquiry-based activities was associated with lower science achievement. However, this association was positively and significantly moderated by the extent of school teacher support. These findings highlight the pivotal role of students’ perceived school teacher support in enhancing the effectiveness of inquiry-based practices and shed light on optimising inquiry-based educational strategies. This suggests that the effective implementation of inquiry-based activities requires the backing of supportive teachers, with broader implications for curriculum design and teacher development in science education.

Decompositions of Information Criteria in Joint Model Under Probit Link for Response

ABSTRACTThe item response and response time are naturally encountered in variety of computerized testing, which are jointly analysed to achieve efficient model fitting. Several novel Bayesian assessment criteria, which quantify the enhancement in the fit of one dataset given the other, are proposed by utilizing decompositions of the deviance information criterion (DIC) and the logarithm of the pseudo‐marginal likelihood (LPML) under a logistic link. However, the calculation of these criteria under a probit link, a commonly employed alternative, remains unexplored. Moreover, decomposing DIC and LPML under probit link presents greater challenges than in the logistic model due to the intricate additional dimension integral. This paper addresses this gap by demonstrating the application of Bayesian model assessment criteria with a normal ogive model in the context of item response data within a joint modelling framework. Additionally, we also illustrate how to implement Bayesian sampling via a powerful R package ‘nimble’. Extensive simulation studies are conducted to evaluate the empirical performance of those criteria. Further, a detailed analysis of the Programme for International Student Assessment (PISA) data is carried out to illustrate the methodology.

Leveraging professional learning communities in linking digital professional development and instructional integration: evidence from 16,072 STEM teachers

BackgroundIntegration of digital tools and resources in STEM instruction has garnered significant attention due to its high potential. Digital professional development is identified as a pivotal factor for equipping teachers with necessary digital skills to effectively orchestrate digital resources. Notably, the role of professional learning communities is considered critical. However, the intricate relationships among digital professional development, professional learning communities, and digital instructional integration among STEM teachers remain underexplored. Utilizing partial least-squares–structural equation models (PLS–SEM), the present study examined links in digital professional development, professional learning communities, and digital instructional integration among STEM teachers (N = 16,072) who participated in the Programme for International Student Assessment (PISA) 2022.ResultsFindings from the PLS–SEM analysis indicate that digital professional development exhibits a direct positive relationship with professional learning communities and digital instructional integration. Relatedly, professional learning communities is positively correlated with digital instructional integration. In terms of indirect effect, findings show that professional learning communities play a significant positive mediating role in linking digital professional development and digital instructional integration.ConclusionsThis study reports new evidence on the influence of digital professional development on digital instructional integration through professional learning communities among 16,072 STEM teachers and concludes that, when STEM teachers regularly immerse themselves in professional learning communities, they are more likely to benefit from their digital professional development by integrating digital technologies in classroom instruction. Policymakers and educational leaders should consider promoting digital professional development and professional learning communities among STEM teachers, along with efforts to encourage digital instructional integration.

Realistic mathematics education and mathematical literacy: a meta-analysis conducted on studies in Indonesia

A considerable body of literature exists pertaining to realistic mathematics education (RME) and its correlation with mathematical literacy, with numerous studies demonstrating incongruent results. The principal objective of this meta-analysis is to systematically investigate the overarching influence of RME on mathematical literacy within the context of Indonesia. The collection of documents comprises a total of seventeen publications that were released between the years 2014 and 2023. The estimation methodologies utilized in this study were grounded on a random-effects model, and statistical computations were conducted utilizing the comprehensive meta-analysis (CMA) software in academic writing. The equation proffered by Hedges was employed for the quantification of effect magnitude. The outcomes of the investigation reveal that the implementation of RME learning yielded a noteworthy and advantageous impact (effect size = 1.031; p 0.05) on the adeptness of students in the domain of mathematical literacy. Moreover, many moderating factors, including class capacity, educational level, geographical location, content of the Programme for International Student Assessment (PISA), and the combination of learning, did not significantly impact students' diverse mathematical literacy proficiency. This study proposes that mathematics educators should consider utilizing the RME as a means of improving students' proficiency in mathematical literacy.

Comparison Despite Uniqueness: A Thematic Analysis of the PISA 2022 Results for the United Arab Emirates

The United Arab Emirates (UAE) participates in the Programme for International Student Assessment (PISA) with the aims of educational improvement and future economic development. The PISA results are significantly comparison oriented with every country listed for each part of the test, index or relationship described in the results reports. This research was conducted by analyzing each mention of the UAE in the three main results documents for the PISA 2022. Through critical thematic analysis (Braun & Clarke, 2012), the uniqueness of UAE schools, students and the PISA results emerged. These three main themes are discussed along with my experience at the UAE PISA 2022 results announcement in December 2023. The article concludes with a discussion of the use of comparison in spite of the uniqueness of the UAE, the implications for the usefulness of the PISA results considering the differences of the UAE, the implications of using PISA results for education policy, and how UAE education policy may be constrained by a focus on PISA results.

Beyond Benchmarking: A Diagnostic Inquiry into the Underlying Determinants of Low Performance in Philippine PISA Science

The Philippines’ consistently low performance in the Program for International Student Assessment (PISA) Science Assessment has sparked concerns about the country’s educational quality and competitiveness. The science scores are due to socioeconomic status, teacher quality, resource deficiencies, and systemic issues. A multi-pronged approach with evidence-based interventions is recommended to improve scientific literacy. This paper conducts a diagnostic inquiry into the underlying determinants contributing to Filipino students’ underachievement in PISA science. Drawing from an extensive literature review, the analysis identifies several interrelated factors spanning multiple levels of the educational system. Socioeconomic status, motivation, self-efficacy beliefs, and language proficiency emerged as significant predictors at the student level. School-level determinants include resource deficiencies, lack of qualified science teachers, and inadequate instructional practices. Furthermore, systemic issues like fragmented policy implementation, curriculum misalignment, and insufficient investment in education undermine performance. The paper argues for a holistic, multi-pronged approach targeting these diverse determinants through evidence-based interventions and policy reforms to enhance scientific literacy and elevate educational outcomes in the Philippines.

PISA in the mirror

This paper offers a conceptual analysis of the OECD’s Programme of International Student Assessment (PISA) through the lens of the organizational sociology. Building upon a narrative review of the literature, it explores drivers behind the current prominence of PISA, using concepts derived from rational systems, social efficiency, and new institutionalist approaches. From the analysis of PISA’s rationales, logics of operation and impacts, the key finding is that PISA’s power and limitations function as a mirror, reflecting coinciding features of the program’s design and OECD’s characteristics mobilized by critics and supporters in opposite directions. Given PISA’s appeal for policymakers, the paper calls for further research to uncover its cross-cultural learning potential, rather than reinforce it as a tool of global accountability and standardization.

Introduction to Critical Thinking in Educational Model for Visual Communication Design Methods: Study Case Design Methods Class for Visual Communication Design Program at Binus University, Indonesia.

Based on the Programme for International Students Assessment (PISA), where problem-solving skills and critical thinking were measured the Indonesian students ranked below standard. In a smaller scope, there were other phenomenon in the learning process of a Design Methods class, in the Visual Communication Design programme at Binus University where students struggled to perform design analysis specifically on the part where the process of classifying which data belong to each of the three analytical aspects of the Triadic, despite definitions had been given in the prior session. In The Ethics, Aristotle thought about classifying as one of the thinking processes that would help in the process of analyzing. According to the Reboot Foundation, an institution dedicated to critical thinking research in the United States of America, critical thinking is needed in the decision-making process. The study was conducted to find the influence of the addition critical thinking method in C-F-C Triadic, one of the design thinking methods, in its learning process. The method in this investigation is qualitative primary data collection based on the observation of two different classes, class A and class B. In class A, the learning process was given following the existing learning process, while in class B, an additional short critical thinking demo was given in the first session of the class. The result was elevation in the analysis performance of the students in class B.

The Comparison of PISA 2015-2018 Mathematics Trend Items Based on Item Response Times

This study aims to explore the intricate relationship between students' response times, item characteristics, and the effort invested during the Programme for International Student Assessment (PISA) 2015 and 2018 cycles. Through the analysis of data obtained from 69 mathematics trend items administered in a computer-based format across both cycles, this research investigates the dynamics of students' response times and their implications on effort and item characteristics. Findings reveal a significant increase in students' mean response times in the 2018 cycle compared to 2015, indicating potentially heightened effort and solution behavior. Notably, item formats exerted a substantial influence on response times, with open-ended items consistently eliciting lengthier response times compared to multiple-choice items. Additionally, a correlation between response times and item difficulty emerged, suggesting that more challenging items tend to consume more time, possibly due to the complexity of involved cognitive processes. Item based effort, assessed through Response Time Fidelity (RTF) indices, highlighted that a majority of students exhibited solution behavior across both cycles to the items.. Moreover, a decrease in the proportion of students displaying rapid-guessing behavior was observed in the 2018 cycle, potentially reflecting increased engagement with the assessment. While providing insights into the interplay of response times, item characteristics, and effort, this study emphasizes the need for further exploration into the multifaceted nature of effort in educational assessments. Overall, this research contributes valuable perspectives on nuances surrounding test performance and effort evaluation within PISA mathematics assessments.

The influence of the social environment on the development of critical thinking skills in young people: A study in the city of Durres, Albania

Critical thinking is an essential skill for individuals, especially young people, enabling them to analyze and evaluate information to make informed decisions. Young people who are able to make informed decisions have the opportunity to improve their future and contribute to a better society. In Albania, the level of development of young people's critical thinking skills, according to the Program for International Student Assessment (PISA), is very low. In this context, this work aims to explore the social environments that influence the development of critical thinking in young people. The study employed quantitative and qualitative approaches, as well as an analysis of secondary data extracted from similar studies. The quantitative method consists of the analysis of a questionnaire that was completed by 260 young people from 15–30 years of age. The study data show that schools, society, and social media play a very important role in the development of young people's critical thinking skills. The data confirmed that the socioeconomic status of families influences critical thinking abilities. This highlights the importance of supporting young people from all walks of life to ensure that they all have the opportunity to develop critical thinking, regardless of social background. The findings of this study are important for understanding and fostering the development of critical thinking skills in young people. This research supports teachers, parents, policymakers, researchers, and others in their efforts.

“Phones off while school's on”: Evaluating problematic phone use and the social, wellbeing, and academic effects of banning phones in schools

AbstractBackground and aimsMobile phone bans in secondary schools are claimed to reduce student distraction and promote learning and face-to-face socializing. Currently, the evidence on phone bans is limited. The aim of this preregistered study was to evaluate the South Australian mobile phone ban's effects on students' problematic phone use, academic engagement, school belonging, and bullying. The study also sought to identify student variables that predict phone ban compliance.MethodsAs the ban was phased in over 2023, a 2 (phone ban: yes/no) × 2 (time: baseline, 1-month follow-up) repeated-measures design was employed. Students (n = 1,282 at baseline; n = 1,256 at follow-up) in Grades 7 to 12 were recruited from five public secondary schools. Surveys included measures drawn from the Longitudinal Study of Australian Children (LSAC) and the Programme for International Student Assessment (PISA).ResultsProblematic phone use was reported by 2.6% of the sample. Being older and a more frequent user of social media predicted lower phone ban compliance. Linear mixed models indicated that ban and no ban school groups did not differ significantly in terms of problematic phone use, academic engagement, and school belonging. There was slightly higher bullying in the ban group but bullying decreased significantly in both groups.DiscussionImposing access restrictions may not affect the underlying psychological mechanisms that drive problematic phone use. Although these results indicate limited to no short-term benefits of the ban, further evaluation with more sensitive methodologies is recommended.ConclusionsStudent-technology interactions in learning institutions should be continually monitored to determine the optimal balance to support student etiquette, learning, and wellbeing.

The politics of tinkering: developing the PISA assessment of global competence

ABSTRACT The popularity of OECD’s PISA tests as benchmarks for shaping global and national education policies is premised on its reputation for developing technically robust surveys by engaging leading experts who are untouched by politics. Drawing on the Science and Technology Studies (STS) subfield of the sociology of measurement, we illustrate how the view of these surveys as a-political and purely technical achievements is a myth. Using detailed documentary analysis supported by interviews with PISA insiders, we describe the interplay between technical expertise and political expertise in the development of the 2018 surveys of global competence. This paper highlights the challenge for the PISA insider group who must negotiate how to maintain scientific authority whilst gaining political backing and trust. We refer to the key strategy of ‘tinkering’ that is required to keep various actors aligned to the project through stages of development and moments of uncertainty.

Writing PISA-Like Mathematics Items: The Case of Tertiary Mathematics Instructors from a State University in the Philippines

<p style="text-align:justify">Mathematics test items in International Large-Scale Assessments (ILSAs) such as the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) are nested in contexts defined in their assessment framework (e.g., the Personal, Occupational, Societal, and Scientific contexts in PISA). This study followed the item-writing activities of four tertiary mathematics instructors in the Philippines as they constructed context-based mathematics items. They were tasked to write four items each, following a set of specifications for PISA content and context categories. The data consisted of transcripts from the focus-group discussion which was conducted days after the task. The transcripts were then analyzed using thematic analysis. The results of this study showed that the phenomenon of item-writing in the context of writing PISA-like mathematics items had two themes: the phases of item-writing and the dimensions of item-writing. Findings showed that the respondents struggled to find realistic contexts and that they engaged in a problem-solving task likened to solving a puzzle as they attempted to satisfy the content, context, and process categories in the table of specifications (TOS). This study contributes to filling in the research gap on item-writing activities, particularly those of mathematics teachers in the Philippines- a country whose recent mathematical performance in the PISA 2018, TIMSS 2019, and PISA 2022 was nothing short of dismal.</p>

Parental emotional support, self-efficacy, and mental health problems among adolescents in Hong Kong: a moderated mediation approach.

Early signs of mental health issues may develop into severe mental illnesses over time. The influence of parental emotional support on adolescent mental health problems is well acknowledged. However, prior research has predominantly focused on psychological symptoms, often neglecting the somatic symptoms associated with mental health. Additionally, there has been insufficient exploration of the mediating role of self-efficacy and the relationship between socioeconomic status (SES) and adolescent mental health, particularly within the context of Hong Kong. Using data from 3,613 adolescents aged 15 years from 109 schools in Hong Kong who participated in the Programme for International Student Assessment 2018, this study explored the relationship between parental emotional support and two dimensions of adolescent mental health symptoms (psychological and somatic symptoms), and the mediating effect of self-efficacy and the moderating role of SES. We found that parental emotional support significantly reduced psychological and somatic symptoms, primarily by enhancing self-efficacy. MoreovSES moderated these relationships, with the impact of parental emotional support on psychological symptoms being more pronounced among adolescents from lower SES backgrounds. This study deepens the understanding of the mechanisms underlying adolescent mental health in Hong Kong. By highlighting the importance of parental emotional support and self-efficacy, as well as the moderating effect of SES, the findings offer valuable insights for effective interventions aimed at improving adolescent mental health.

STEM career expectations across four diverse countries: motivation to learn mathematics mediates the effects of gender and math classroom environments

BackgroundWe tested the broad generality of a model for predicting 9th–10th grade students’ STEM career expectations by age 30, focusing on hard science, mathematics and engineering professions only, known for driving innovation, research and development. The model’s predictors included motivation to learn mathematics, gender, and math classroom environments (disciplinary climate, teacher support and instructional strategies fostering conceptual understanding).MethodsWe used data from the Programme for International Student Assessment (PISA) 2022. Four countries were selected based on the percentage of students expecting STEM careers, representing high vs. low groups (Qatar and Morocco vs. Czech Republic and Lithuania, respectively). Analysis began with computing correlations between the variables, followed by path analyses for each country to determine both direct and indirect effects of the predictors on students’ STEM career expectations.ResultsWe found that motivation to learn mathematics not only directly predicted STEM career expectations but also mediated the influence of the remaining variables: gender (boys show higher motivation to learn math), and math classroom environments (students in well-disciplined math classes with supportive teachers who employ instructional strategies fostering math reasoning also demonstrate higher motivation to learn math). Remarkably, our model consistently demonstrated robustness across all four countries, despite their significant economic, ethnic, and religious diversity.ConclusionsTheoretically, the model reveals that 9th–10th grade students’ transitory long-term STEM career expectations are shaped by their interest in mathematics, their perceived importance of the subject, confidence in their self-efficacy to succeed in math tasks, perceptions of classroom disciplinary climate, teacher support, and their exposure to instructional strategies aimed at enhancing math reasoning. Practically, it suggests widespread potential for informing interventions aimed at increasing student motivation to pursue STEM careers through improved mathematics education practices.

A unified EM framework for estimation and inference of normal ogive item response models.

Normal ogive (NO) models have contributed substantially to the advancement of item response theory (IRT) and have become popular educational and psychological measurement models. However, estimating NO models remains computationally challenging. The purpose of this paper is to propose an efficient and reliable computational method for fitting NO models. Specifically, we introduce a novel and unified expectation-maximization (EM) algorithm for estimating NO models, including two-parameter, three-parameter, and four-parameter NO models. A key improvement in our EM algorithm lies in augmenting the NO model to be a complete data model within the exponential family, thereby substantially streamlining the implementation of the EM iteration and avoiding the numerical optimization computation in the M-step. Additionally, we propose a two-step expectation procedure for implementing the E-step, which reduces the dimensionality of the integration and effectively enables numerical integration. Moreover, we develop a computing procedure for estimating the standard errors (SEs) of the estimated parameters. Simulation results demonstrate the superior performance of our algorithm in terms of its recovery accuracy, robustness, and computational efficiency. To further validate our methods, we apply them to real data from the Programme for International Student Assessment (PISA). The results affirm the reliability of the parameter estimates obtained using our method.

Cultural capital, black holes, and the hidden curriculum: an analysis of grade repetition in Spain

ABSTRACT In this article I explain the contradiction between the averagely high and equitable scores that Spanish students obtained in the last three editions (2015, 2018, and 2022) of the Program for International Student Assessment, and Spain’s extremely high and unequitable rate of grade repetition in schools. After ruling out alternative explanations for this discrepancy, I explore the hypothesis that grade repetition in Spain is an effect of the hidden curriculum. Methodologically speaking, my explanation establishes analogies with Einstein’s theory of relativity and draws on its spatial topoi to conceptualize the difference between the official and the hidden curriculum and the undemocratic conflation between upper- and middle-class cultural capital and academic skills. This conflation, I argue, lies behind said contradiction between Spanish students’ scores in PISA and the rates of grade repetition in Spain.