Cross-national Assessments Research Articles

The politics of international testing

How does quantifying and ranking national performance influence state behavior? Cross-national assessments in education, such as the Programme for International Student Assessment (PISA) promoted by the Organisation for Economic Co-operation and Development (OECD), have become increasingly prominent in recent years. However, cross-national assessments are politically contentious, and their impact remains underexplored. We argue that assessment participation has a meaningful, positive impact on education outcomes and evaluate three hypotheses related to elite, domestic, and transnational mechanisms. Our mixed-method approach draws on a panel dataset covering all cross-national assessments and all countries as well as an original survey of education officials directly responsible for planning and implementation in 46 countries. We find that assessment participation increases net secondary enrollment rates even after accounting for potential self-selection. The magnitude of this increase is large: on a global basis, it is equivalent to improved access to higher education for 27–32 million students annually. The empirical evidence suggests elite-level mechanisms are primarily responsible for these findings.

Monitoring the Monitor? Selective Responses to Human Rights Transgressions

AbstractSanctions are among the most frequently used foreign policy tools to address human rights violations, but they can be highly politicized. Since the early 2000s, human rights sanctions have been increasingly triggered by standardized rankings of states’ performances. While research on economic statecraft suggests that coercive measures based on cross-national assessments may be less influenced by strategic considerations, scholarship on rankings highlights how standardized performance indicators can also be political. This paper investigates whether sanctions based on standardized human rights assessments are also influenced by senders’ strategic political and economic interests. Empirically, we examine the case of United States human trafficking sanctions that combines universal rankings in the first stage and country-specific sanctions waivers in the second. The analysis leverages novel data on all Trafficking in Persons (TIP) rankings by the US State Department and presidential sanctions waivers from 2003 to 2018. Despite the TIP report's reputation as a reliable indicator, we find that both stages in the process of imposing human trafficking sanctions are driven by strategic attempts to minimize the economic and political costs of sanctions for the US. These findings have broader implications for the reputation and effectiveness of other human rights rankings by the US.

Differences in the measurement of cognition for the assessment of dementia across geographic contexts: Recommendations for cross-national research.

Most cognitive assessments have been developed in high-income countries but are used in diverse contexts. Differences in culture and context may affect the performance of cognitive items. We used the Harmonized Cognitive Assessment Protocol (HCAP) surveys in the United States, Mexico, India, England, and South Africa (combined N=11,364) to quantify associations across countries between cognitive items and cognitive impairment status using age- and sex-adjusted logistic regression. Associations were stronger in the United States (median odds ratio [OR] across items=0.17) and England (median OR=0.19), compared to South Africa (median OR=0.23), India (median OR=0.29), and Mexico (median OR=0.28). Items assessing memory (e.g., delayed recall tasks) had the most consistent associations of the largest magnitudes across contexts. Transporting cognitive items among countries and cultures warrants caution. Our results can guide the design of future instruments by identifying items that performed well either in individual contexts or across the range of contexts considered. Little quantitative evidence exists to guide the design of cognitive assessments in cross-national studies. The performance of cognitive items for the measurement of dementia varied across countries. Items with lower variation across countries (e.g., delayed word recall) should be used in future cross-national assessments. Across countries, there was variability in the performance of language assessments, with the exception of the animal naming task. Results can be used to design future cross-national or location-specific cognitive assessments.

Nexus between participating in cross-national learning assessments and educational policy reforms: case study of Kenya, Tanzania and South Africa

Utilizing the theory of change, this study analyzes the extent to which the participation in cross-national learning assessments by Kenya, Tanzania and South Africa has contributed to educational policy and curriculum reforms. Data collection involved in-depth interviews with key informants and summative content analysis of policy/strategic documents. The findings reflect that overall, at least 11 policy/strategic documents formulated between the years 2000 and 2015 in the three countries respond well to recommendations or findings emanating from the cross-national learning assessments. Two curriculum reviews/reforms attributable to cross-national learning assessments are also recorded. The inferences are drawn based on the testament by the research participants and or whether the documents contain strong references to cross-national learning assessments.

How institutions structure judicial behaviour: An analysis of Alarie and Green’s Commitment and Cooperation on High Courts: A Cross-Country Examination of Institutional Constraints on Judges

No theory of judicial behaviour ignores institutions, but, all too often, their role in structuring judges’ choices goes assumed rather than directly evaluated. For this reason alone, we should applaud Benjamin Alarie and Andrew J. Green. Not only do they take institutions seriously; they attempt cross-national assessments of their effect on judging. This is their book’s overarching contribution, but there are many others along the way – so many that Commitment and Cooperation on High Courts is bound to take its place among the classics in the ever-growing field of judicial behaviour. My aim is to bring Alarie and Green’s contributions into relief by highlighting their key arguments and empirical results. Along the way, I integrate some of the existing literature if only to show where and how the authors advance our understanding of judging. All of this amounts to Parts I and II. But a simple summary of Commitment and Cooperation will not suffice because Alarie and Green invite the reader to think about extensions. In that spirit, Part III offers some suggestions for forward movement.

The Politics of International Testing

In recent years, an increasing number of countries have participated in cross-national assessments in education (CNAs), but their impact remains underexplored. We argue that CNA participation increases the capacity and motivation of policymakers to implement improvements in education through mechanisms at the elite, domestic, and transnational levels. We find evidence consistent with our propositions using a mixed-method approach, utilizing: 1. a panel dataset covering all CNAs and all countries in the international system; 2. an original survey of 77 education officials directly responsible for the planning and implementation of CNAs in 46 countries; 3. personal interviews with 48 officials in target states, assessment agencies, and donor agencies. Even after accounting for potential self-selection, CNA participation is associated with increases in net secondary enrollment, particularly for girls, and education aid inflows. Qualitative and quantitative evidence also suggests that CNA participation accelerates education reforms. The empirical results consistently support our prediction that CNA participation has a meaningful impact on education policy and outcomes.

Pedagogy in citizenship education research: A comparative perspective

Abstract The author of this article argues for using a comparative and international perspective to understand the role of pedagogy in civic education. Drawing on her own research in England, Germany and Denmark, the author demonstrates that pedagogical cultures vary. Next, she discusses what international researchers have learned about civic education pedagogy from the large cross-national assessments and what individual researchers have learned from small case studies within diverse countries, such as Canada, the United Kingdom, Australia, the Philippines and Singapore. She then discusses recent research programmes in the United States that have yielded important information related to pedagogy, including the use of discussion, simulations and digital/social media. The author concludes by proposing a list of needed comparative and international research on civic education pedagogy.

Recursive Partitioning to Identify Potential Causes of Differential Item Functioning in Cross-National Data

Differential item functioning (DIF) assessment is key in score validation. When DIF is present scores may not accurately reflect the construct of interest for some groups of examinees, leading to incorrect conclusions from the scores. Given rising immigration, and the increased reliance of educational policymakers on cross-national assessments such as Programme for International Student Assessment, Trends in International Mathematics and Science Study, and Progress in International Reading Literacy Study (PIRLS), DIF with regard to native language is of particular interest in this context. However, given differences in language and cultures, assuming similar cross-national DIF may lead to mistaken assumptions about the impact of immigration status, and native language on test performance. The purpose of this study was to use model-based recursive partitioning (MBRP) to investigate uniform DIF in PIRLS items across European nations. Results demonstrated that DIF based on mother's language was present for several items on a PIRLS assessment, but that the patterns of DIF were not the same across all nations.

How effective are educational systems? A value-added approach to measure trends in PIRLS

From an educational effectiveness perspective, research based on international large scale assessments has been limited as it neglects to take contextual conditions of educational systems into account. Further, methodological challenges of cross-sectional studies have yet prevented investigations from a longitudinal effectiveness perspective. The paper investigates how effectively educational systems grow, i.e. change, in their performance by applying a methodological approach known from school effectiveness research that captures changes at the country level within repeated cross-sectional data designs. Data from the Progress in International Reading Literacy Study (PIRLS) 2001 to 2006 trend systems is analyzed with hierarchical linear modeling. Effectiveness measures of achievement status in 2006 and of change from 2001 to 2006 are investigated and compared. Results suggest that there are systems which exceed their expected outcomes (status and change) as well as systems which stay below what could have been expected, changing the picture of “high” and “low” performing systems, when contextual conditions and prior performances are taken into account. The study contributes to methodological developments of educational effectiveness research in cross-national assessments. Its results provide complementary information for policymakers to further look at policies, practices, and structures that have favored effectiveness.

Criterion-related validity of the short International Physical Activity Questionnaire against exercise capacity in young adults

Self-reported physical activity (PA) is well associated with cardiorespiratory fitness and exercise capacity. The short International Physical Activity Questionnaire (IPAQ-short) is a frequently used instrument for cross-national assessments of PA in adults. The purpose of this study was to validate IPAQ-short against exercise capacity in Greek young adults. One hundred and thirteen men and 105 women, aged 20-29 years, were randomly selected from a larger population of young health-science students. A Greek version of IPAQ-short (IPAQ-Gr) was administered to all participants before their exercise capacity evaluation with a maximal Bruce treadmill test. Multiple regression and correlation analyses were used to examine the associations between all IPAQ-Gr outcomes with exercise capacity based on maximal treadmill time. Spearman's correlations for total and vigorous PA against maximal treadmill time were significant in all groups examined, ranging from 0.35 to 0.43. Moderate and walking PA correlations were poor and nonsignificant, ranging from near-zero values to 0.19. In multiple linear regression analysis, only sex, smoking, and vigorous PA from all personal and log-transformed IPAQ-Gr data were significantly associated with maximal treadmill time. Partial correlation analysis for the overall population, adjusted for sex and smoking, showed that total PA (r=0.37) and vigorous PA (r=0.47) were significantly associated with exercise capacity. IPAQ-Gr was tested against exercise capacity and showed acceptable validity properties in Greek young adults. Total and vigorous weekly PA expenditure were well associated with exercise capacity, presenting significant validity correlations against maximal treadmill time.

Globalization and the Growth of International Educational Testing and National Assessment

Education has long been characterized as a central requirement for national economic development and political democratization in the contemporary world. Moreover, international benchmarking has been identified as the “basis for improvement. . . . It is only through such benchmarking that countries can understand relative strengths and weaknesses of their education systems and identify best practices and ways forward” (OECD 2006, 18). Statements such as this example signal an international consensus that has emerged— at least among “developed” countries—about the legitimacy and, even more so, the necessity of international testing and national assessment. As David P. Baker and Gerald K. LeTendre (2005) observe, both international testing and national assessment are linked to efforts to reform educational systems and are often themselves stimuli for further cycles of reform. The results of international testing, they note, will fuel further interest in national assessment. Here we develop an argument about the global forces that have led to the explosive growth of national educational assessment and international testing. In particular, we argue that the international acceptance of testing comes from key ideological forces in the world polity that are associated with the accelerating globalization of national and international cultural, economic, and political structures. As we develop and warrant this argument, we also qualify it by pointing out that national adaptations to this larger world culture may vary depending on the presence and capacities of international organizations and regional associations that act to mediate and adapt these changes to conditions in individual countries. In addition, we consider the effects of subnational movements in introducing pressures for change that may favor more national assessment.

Issues of Global and Local Norm Use in Assessments of Motivation

With the growth of globalization, organizations are facing an increased challenge to manage talent, improve employee engagement, develop effective teams, create succession pipelines, and increase organizational effectiveness in multinational settings. Given the need to evaluate and compare individuals from more than one country using standardized assessment instruments, there is a consequent need to understand the issues and implications of norm selection decisions. The current investigation examines the impact of global vs. local norm selection in the assessment of motivation in work settings. A sample of 21,248 participants from organizations in Australia, Canada, South Africa, the United Kingdom, and the United States completed the Individual Directions Inventory™ as part of ongoing organizational development activities. The choice of norm appeared to have minimal impact on most scales among the Anglo cultures studied. Results illustrate the issues researchers and organizations face when creating cross-national assessments and provide recommendations regarding norm selection and the communication of results to end users.

Cross-national cognitive assessment in schizophrenia clinical trials: a feasibility study

Clinical trials for the treatment of schizophrenia now often include cognitive assessments in addition to clinical ratings of symptoms. Recently, these trials have included cross-national assessments. It is not clear if translated psychological tests produce consistent results across different languages. This paper presents the results of a study of the comparability of the results of cognitive assessments in different English-speaking countries and a number of countries where tests were translated into other languages. Performance on tests of executive functioning, verbal and visuo-spatial learning and memory, language skills, psychomotor speed, and vigilance was compared across the first episode patients with schizophrenia ( n=301) assessed in six different languages (English, French, Finnish, German, Hebrew, and Afrikaans), including two different countries where patients were assessed in English and other languages: Canada (French) and South Africa (Afrikaans). The variance in performance across the sites tested in English was as large as the variance between English and non-English speakers when all tests were considered. Performance differences across English and other languages were found only for executive functions, vigilance, and psychomotor speed, with executive functioning differences nonsignificant when education was considered. No differences were found between English and non-English speakers in Canada. These results suggest that the translation of tests of memory and verbal skills can lead to consistent results across translated versions of the tests. Differences between countries were greater than differences between languages, suggesting the need to consider representativeness of patient samples in terms of local educational attainment. In general, these data support the validity of cross-national neuropsychological assessments.

The Political Structuration of Assessment: Negotiating State Power and Legitimacy

The Political Structuration of Assessment: Negotiating State Power and Legitimacy

Transnational Advocacy, Global Civil Society? Emerging Evidence from the Field of Education

L'objectif des auteurs de cet article est double. Il s'agit d'une part de presenter une etude exploratoire sur les tendances actuelles temoignant de l'emergence d'un nouvel activisme non gouvernemental dans le champ de l'education et d'autre part de dresser un cadre analytique afin d'evaluer l'importance de cet activisme, en elaborant des concepts bases sur les recherches ayant porte sur les nouveaux mouvements sociaux et les debats theoriques actuels a propos de la societe civile mondiale.

Transnational Advocacy, Global Civil Society? Emerging Evidence from the Field of Education

Transnational Advocacy, Global Civil Society? Emerging Evidence from the Field of Education