Cross-national Comparability Research Articles

Anchoring Vignettes: A Useful Tool to Measure and Correct for Cultural Bias in Parent Reports on Their Child's Mental Health?

Parent report measures developed in the Western world are commonly used to assess children's mental health, but their cross-cultural comparability is questionable. The present study examines the use of anchoring vignettes to assess and adjust for bias in five countries: the United States, Mexico, Germany, China, and Russia. Parents (N = 500) rated their child's mental health and vignettes depicting internalizing and externalizing problem behaviors in an online survey. Vignette ratings were used to assess bias and for rescaling. Cross-national comparisons of vignette scores revealed differences in the use of the scale range and overall level of vignette scores. Measurement invariance across countries improved after rescaling, resulting in weak invariance for internalizing and strong invariance for externalizing problem behavior. Rescaled scores revealed cross-national differences that were masked using the raw score. Results confirm the lacking cross-national comparability in parent reports of child mental health, and anchoring vignettes appear to be a useful tool for reducing bias.

A critical evaluation of the validity of socioeconomic measures used in PISA

PurposeGiven the scientific and practical difficulties inherent in measuring and comparing socioeconomic deprivation (SED), and the further complexity added in cross national measurements, the main aim of this paper was to check the validity of SED measures used in PISA 2018 dataset. The SED measure used in PISA 2018 was the PISA index of economic, social and cultural status abbreviated as the ESCS index. This assessment was important as PISA analysis is based on variables derived from this instrument and the ESCS index and these reports influence and reflect international and comparative education policies and practice.Design/methodology/approachThis study critically evaluates the socioeconomic status measures in the PISA 2018 dataset, focusing on their convergent validity and cross-national comparability. Using responses from over 600,000 students in 73 countries, it examines the validity of SES indicators and their comparability across countries. The study employs principal component analysis to construct local SES measures and compares them with the existing Economic, Social, and Cultural Status (ESCS) index. It explores the relationship between these SES measures and academic achievement in reading, science, and mathematics, aiming to understand their predictive validity in diverse educational settings. Statistical analyses were conducted using the IEA’s IDB Analyser and SPSS, ensuring robustness and generalisability across the diverse participant countries.FindingsOur research findings challenge the assumed superiority of local measures over broader constructs like the Economic, Social, and Cultural Status (ESCS). It suggests that standardised measures like ESCS may provide more reliable predictions of academic achievement across various educational contexts, underscoring the complex relationship between SES measures and academic performance.Originality/valueOur novel analysis shows that local and cross-national SED measures are poorly correlated. Our findings raise questions about the measures' validity while acknowledging the methodological challenges. We provide empirical evidence to support ongoing debates on the topic.

Measurement invariance of subjective social status: The issue of single-item questions in social stratification research

Measures of objective and subjective social status are commonly used in social stratification research. While objective measures have been extensively examined for cross-national comparability, subjective indicators have received less attention. This study aims to address this research gap by investigating the measurement invariance of the three most commonly used single-item measures of subjective social status across many countries. Using a Bayesian approximation approach, we analysed data from three waves of the International Social Survey Programme conducted between 1999 and 2019. The analyses showed that our composite measure is a relatively reliable and stable construct when compared internationally. However, some single-item measures were not invariant across countries or survey rounds, suggesting that the average of a given single measure of subjective status or the relationship between it and other variables should not be compared across countries. Finally, the study showed that a subjective status item with a 10-step numerical ladder seems to be more appropriate for cross-country comparisons, showing low variation across countries. To improve the validity of future research, we recommend that at least three questions measuring subjective social status be included in international questionnaires and that subjective status be used as a latent construct whenever possible and appropriate.

The Psychological Inventory of Criminal Thinking Styles: A Scoping Review of its Reliability, Factor Structure, Concurrent Correlates, Predictive Correlates, Change Scores, and Cross-National Comparability in Non-U.S. Samples

The Psychological Inventory of Criminal Thinking Styles: A Scoping Review of its Reliability, Factor Structure, Concurrent Correlates, Predictive Correlates, Change Scores, and Cross-National Comparability in Non-U.S. Samples

Considerations for the use of the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) in cross-country comparisons of cognitive aging and dementia.

Informant reports are a critical component of dementia diagnoses, but the comparability of informant reports across countries is not well understood. We compared the performance of the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) using population-representative surveys in the United States (N=3183), England (N=1050), and India (N=4047). Analyses of regression splines and comparisons of model fit showed strong associations between IQCODE and objective cognition at low cognitive functioning in the United States and England; in India, the association was weaker but consistent over the range of cognition. Associations between IQCODE score and informant generation (analysis of variance [ANOVA] p=0.001), caregiver status (p<0.001), and years known by the informant (p=0.015) were different across countries after adjusting for objective cognition. In India, the IQCODE was less sensitive to impairments at the lowest levels of cognitive functioning. Country-specific adjustments to IQCODE scoring based on informant characteristics may improve cross-national comparisons. Associations between IQCODE and cognitive testing were similar in the United States and England but differed in India. In India, the IQCODE may be less sensitive to impairments among those with low cognition and no education. Informant characteristics may differentially impact informant reports of decline across countries. Adjustments or culturally sensitive adaptations may improve cross-national comparability.

Political trust among European youth: Evaluating multi-dimensionality and cross-national measurement comparability

Educational research has focused increasingly on comparing student civic outcomes across various educational systems. However, the study of non-cognitive, attitudinal measures has assumed a subsidiary role to the study of cognitive measures, despite its potential to shape policy and practice. This article investigates the validity and cross-country measurement equivalence or invariance (ME/I) of political trust measures among adolescents. Drawing on student data from ICCS 2016 covering 15 European educational systems (N = 52,788), the standard test for exact ME/I (i.e., Multi-Group Confirmatory Factor Analysis) is juxtaposed with a novel method for approximate ME/I (i.e., Alignment Optimization). The results underscore a two-dimensional structure of political trust, separating trust in order institutions from trust in representative institutions. While MGCFA results suggest configural invariance, rendering correlational analyses and country mean comparisons meaningless, Alignment Optimization yields valid latent means with an acceptable degree of non-invariance (i.e., 16%). Implications for comparative research on political trust among adolescents are discussed.

Anchoring Vignettes as a Diagnostic Tool for Cross-National (in)Comparability of Survey Measures: The Case of Voters’ Left-Right Self-Placement

AbstractThere are potentially multiple sources that make it difficult to compare the typical survey measure of the left-right self-placement cross-nationally. We focus on differential item functioning (DIF) due to the different use of response scales when the left-right is framed as an aggregate dimension of policies. We also examine whether and to what extent ordinary citizens’ use of the scale is cross-nationally comparable. Our goal is twofold. First, we assess the cross-national comparability of the left-right self-placement scale using the anchoring vignette method used in nine European countries. Second, we propose a measure that quantifies the extent of DIF at the country level. Our original survey and other benchmark studies suggest that the size of cross-national DIF (CN-DIF) in citizens' use of a left-right scale is relatively small when the left-right concept is considered in policy terms and when a comparison is made between Western European countries.

Feasibility and validity of using healthcare databases to conduct cross-national comparative studies of opioid use, its determinants and consequences.

A cross-national comparative (CNC) study about opioid utilization would allow the identification of strategies to improve pain management and mitigate risk. However, little is known about the accessibility and validity of information in healthcare databases internationally. This study aimed to identify the feasibility of using healthcare databases to conduct a CNC study of opioid utilization and its associated consequences. A cross-sectional survey was launched in March 2018, including experts interested in CNC studies comparing opioid utilization by purposeful sampling. An electronic survey was used to collect database characteristics, medicine information, and linkage information of each aggregate-level dataset (AD) and individual patient-level dataset (IPD). Overall, participants from 21 geographical regions reported 18 ADs and 19 IPDs. Information on dispensed medications is available from 17 ADs and 17 IPDs. Of the 16 ADs that include primary care settings, only 9 ADs can obtain information from secondary care settings. Fourteen IPDs included patients' characteristics or could be retrieved from linkage databases. Although most ADs are publicly accessible (n = 13), only five IPDs can be accessed without extra cost. Most ADs could be used to report opioid utilization in a primary care setting. IPDs with linkage databases should be applied to identify potential determinants, clinical outcomes, and policy impact. Data access restrictions and governance policies across jurisdictions can be challenging for timely analysis and require further collaboration.

The impulsive behavior short scale-8 (I-8): A comprehensive validation of the English-language adaptation.

The Impulsive Behavior Short Scale–8 (I-8) measures the psychological construct of impulsivity with four subscales comprising two items each (completion time < 1 min). The aim of the present study was threefold: (1) to assess the psychometric properties (objectivity, reliability, and validity) of the English-language I-8; (2) to compare these psychometric properties with those of the original German-language source version of the scale; and (3) to test the cross-national comparability of the scale via measurement invariance tests. For this purpose, we used heterogeneous quota samples from the UK and Germany. Our results indicate that I-8 is a reliable and valid short scale with highly comparable psychometric properties across both language versions. In addition, I-8 showed a highly similar correlational pattern with various extraneous variables across the two nations. Furthermore, partial scalar invariance and full invariance of residual variances held, allowing the comparison of latent means and observed (co)variances across nations. I-8 lends itself as a measure of impulsive behavior especially in surveys in which assessment time is limited, such as in large-scale cross-national surveys.

Road traffic noise modelling and population exposure estimation using CNOSSOS-EU: Insights from Ireland

In 2009 the European Commission sanctioned the development of a new noise calculation method - CNOSSOS-EU - as part of commitments outlined in the Environmental Noise Directive. CNOSSOS-EU aims to develop consistency and cross-national comparability of noise modelling and population exposure results across the EU. This paper assesses the implementation of the CNOSSOS-EU road traffic model for Ireland with a view to determining its effect on future population exposure statistics relative to previous estimations. In order to do this, CNOSSOS-EU and CRTN-TRL models were compared by way of a roadside measurement experiment and population exposure estimation analysis. Results from the roadside measurement experiment indicated that the CNOSSOS-EU model predicts the real noise situation more accurately compared to CRTN-TRL. This suggests that estimates of population exposure statistics calculated under the CNOSSOS-EU method may more accurately reflect the real noise exposure situation than previously reported estimates of population exposure statistics calculated using the CRTN-TRL method. Furthermore, the results found higher levels of population exposure to road traffic noise levels > 55 dB(A) Lden and > 50 dB(A) Lnight using CNOSSOS-EU relative to CRTN-TRL. This has implications for future population exposure estimations and the associated provision of noise action planning within agglomerations.

Establishing a Baseline: Bringing Innovation to the Evaluation of Cross-National Probability-Based Online Panels

A number of countries in Europe and beyond have established online or mixed mode panels with a web component based upon probability samples. This paper aims to assess the first ever attempt to do this cross-nationally using an input harmonised approach representing a major innovation in cross-national survey methodology. The European Social Survey established a panel using the face-to-face interview as the basis for recruitment to the panel. This experiment was conducted in Estonia, Slovenia and Great Britain using an input harmonised approach to the design throughout something never done before across multiple countries simultaneously. The paper outlines how the experiment was conducted. It then moves on to compare the web panel respondents to the ESS achieved face-to-face sample in each country, as well as comparing the web panel achieved sample to external benchmarks. Most importantly, since the literature is very scarce, the differences in attitudinal and behavioural characteristics are also assessed. By comparing the answers of the total achieved sample in the ESS to the subset who also answered the CRONOS web panel we assess changes in representativeness and substantive answers without confounding the findings with other changes such as mode effects. This approach is only possible where ‘piggybacking’ recruitment has been used. This in itself is rare at the national level but this is the first time survey methodologists have employed this cross-nationally allowing such an analytical approach. Our findings suggest that the CRONOS sample is not too divergent from the target population and to the ESS with the exception of the oldest age groups. However, there are cross-national differences suggesting cross-national comparability might be different when compared to estimates from a face-to-face survey.

Analyzing the cross-national comparability of party positions on the socio-cultural and EU dimensions in Europe

AbstractUsing survey vignettes and scaling techniques, we estimate common socio-cultural and European integration dimensions for political parties across the member states of the European Union. Previous research shows that party placements on the economic left-right dimension are cross-nationally comparable across the EU; however, the socio-cultural dimension is more complex, with different issues forming the core of the dimension in different countries. The 2014 wave of the Chapel Hill Expert Survey included anchoring vignettes which we use as “bridge votes” to place parties from different countries on a common liberal/authoritarian dimension and a separate common scale for European integration. We estimate the dimensions using the Bayesian Aldrich–McKelvey technique. The resulting scales offer cross-nationally comparable, interval-level measures of a party's socio-cultural and EU ideological positions.

Gender Role Attitudes in the International Social Survey Programme: Cross-National Comparability and Relationships to Cultural Values

Differences in societal views on the roles of men and women have been addressed in many large-scale comparative studies by employing indicators of gender roles attitudes from cross-sectional surveys. Assuming that cross-country differences in gender role attitudes are linked to the prevailing cultural value orientations in each society, this study aims at investigating the association between societal views on gender roles, as measured by the International Social Survey Programme (ISSP), and the prevailing cultural values, as defined by Schwartz’s theory. However, to carry out meaningful comparisons, we first assessed the prerequisite of measurement equivalence between countries. The comparability of gender role attitudes is limited when using traditional methods based on the concept of exact equivalence (multiple-group confirmatory factor analysis). However, the recently established alignment optimization procedure reveals approximate measurement equivalence and suggests that the mean comparison is trustworthy. Based on these results, we correlate the national mean levels of gender role attitudes with the cultural values of embeddedness, hierarchy and egalitarianism, showing that traditional gender roles are displayed in societies emphasizing hierarchy and embeddedness while progressive views are more expressed in egalitarian societies.

Ensuring the comparability of cross-national survey data on intimate partner violence against women: a cross-sectional, population-based study in the European Union

ObjectivesTo ensure the cross-national comparability of the set of questions addressing physical and sexual intimate partner violence against women (IPVAW) included in the European Union (EU) Agency for Fundamental Rights...

Investigating the Cross-National Comparability of Testing Using Response Times

The cross-national comparability of testing results is one of the main concerns associated with international assessment. One way to approach this concern is the theoretical framework of cross-national comparability. One such framework, proposed by Van de Vijver and Tanzer [1997], describes the bias in item meaning across countries and the bias in terms of test administration procedures and respondent behavior across countries. To study how items function in terms of respondent behavior and in terms of their psychological meaning, we introduce the concept of time-related differential item functioning (DIF). This concept is similar to traditional DIF but describes the incomparable time parameters of items across countries rather than item difficulty parameters. We discuss time-related DIF, referencing dual processing cognitive theory and illustrate how it limits the interpretability of a certain type of time-related DIF. We also demonstrate the analysis with real data, and discuss the difference introduced by response time information in parameter interpretation and modelling results.

Cross-National Comparability of Assessment in Higher Education

The last three decades have seen an increase in researchers’ interest in international comparative assessments of educational outcomes, particularly at the level of secondary schools. Achieving cross-national comparability is the main methodological challenge in the design of such studies. Cross-national comparability of test scores implies that the measure operates similarly across all the participating countries, regardless of their linguistic and cultural differences.The process of achieving cross-national comparability in higher education is more complicated due to specific features of higher education. This article explores the modern understanding of cross-national comparability of student assessment results and the possible ways of achieving it. It analyzes the specific aspects of higher education that complicate standardized measurement of educational outcomes and trivial achievement of cross-national comparability.The process of designing and conducting the Study of Undergraduate Performance — an international comparative research project aimed to assess and compare higher engineering education across nations — is described as an example of overcoming those challenges.

Psychological intimate partner violence against women in the European Union: a cross-national invariance study

BackgroundIntimate partner violence against women (IPVAW) is a worldwide public health problem. One of the most frequent forms of this type of violence in western societies is psychological IPVAW. According to the European Union (EU) Fundamental Rights Association (FRA) the prevalence of psychological IPVAW in the EU is 43%. However, the measurement invariance of the measure addressing psychological IPVAW in this survey has not yet been assessed.MethodsThe aim of this study is to ensure the cross-national comparability of this measure, by evaluating its measurement invariance across the 28 EU countries in a sample of 37,724 women, and to examine how the levels of this type of violence are distributed across the EU.ResultsOur results showed that the psychological IPVAW measure presented adequate psychometric properties (reliability and validity) in all countries. A latent structure of one factor was supported and scalar invariance was established in all countries. The average levels of psychological IPVAW were higher in countries like Finland, Latvia, Lithuania, and Sweden compared to the rest of the EU countries. In many of the other countries the levels of this type of violence overlapped.ConclusionOur findings underlined the importance of using appropriate statistical methods to make valid cross-national comparisons in large population surveys.

How to Obtain Comparable Measures for Cross-National Comparisons

Comparisons of means or associations between theoretical constructs of interest in cross-national comparative research assume measurement invariance, that is, that the same constructs are measured in the same way across the various nations under study. While it is intuitive, this assumption needs to be statistically tested. An increasing number of sociological and social psychological studies have been published in the last decade in which the cross-national comparability of various scales such as human values, national identity, attitudes toward democracy, or religiosity, to name but a few, were tested. Many of these studies did not manage to fully achieve measurement invariance. In this study we review, in a nontechnical manner, the methodological literature on measurement invariance testing. We explain what it is, how to test for it, and what to do when measurement invariance across countries is not given in the data. Several approaches have been recently proposed in the literature on how to deal with measurement noninvariance. We illustrate one of these approaches with a large dataset of seven rounds from the European Social Survey (2002–2015) by estimating the most trustworthy means of human values, even when strict measurement invariance is not given in the data. We conclude with a summary and some critical remarks.

The impact of definitions in classifying the employed, unemployed and inactive when comparing measurements from different sources

In all large-scale sample surveys and the census, gender, age, marital status, educational and occupational variables are included as background variables to provide information necessary for defining subpopulations and “contexts in which respondents’ opinions, attitudes, and behavior are socio-economically embedded”. Because these background variables play such a central role in social research, establishing their measurement’s cross-national and overtime comparability is essential and consequently, international classifications have been developed. However, in the case of the employment status, i.e. one of the occupational variables, all large-scale sample surveys and the census use a perception question for its measurement whereas the European Union Labour Force Survey (EU-LFS) is using a synthesized economic construct according to the International Labour Organization (ILO) conventional definitions of the employed, unemployed and inactive. In this paper, we investigate the classification issues arising from these two different measurements. The analysis is based on the 2008-2014 EU-LFS annual datasets and the 2001 and 2011 Integrated Public Use Microdata Series (IPUMS)-International census datasets for Eastern (Hungary, Poland, Romania) and Southern (Greece, Portugal and Spain) Europe. The results are reported for the age group 15–74 so as to allow for comparability with the ILO conventional definition of unemployment.

Perfectionism in Argentina and the United States: Measurement Structure, Invariance, and Implications for Depression

ABSTRACTThe Revised Almost Perfect Scale (APS–R; Slaney, Mobley, Trippi, Ashby, & Johnson, 1996) was designed to measure perfectionistic strivings (standards), perfectionistic concerns (discrepancy), and preferences for neatness and organization (order). In this study, English and Spanish (Arana, Keegan, & Rutsztein, 2009) versions of the APS–R were compared in samples of 283 college students in Argentina and 311 U.S. students. The results offered support for cross-national scalar invariance for discrepancy and order items but standards items, with a few exceptions, raised concerns about their cross-national comparability. Tests of latent means revealed no differences between the countries for discrepancy, but the order factor mean was higher in the U.S. sample. Age and gender had negligible effects on measurement models. Discrepancy was strongly related to depressive symptoms for both samples. Unlike in the U.S. sample, there was a significant association for the Argentina sample between standards and discrepancy. Overall, results indicated that discrepancy items performed well, order items were adequate with some measurement adjustments, and standards items should be revisited for their cross-cultural utility. Translation issues and general values in Argentina and the United States (e.g., collectivism–individualism) are discussed that might affect how perfectionism is understood and measured in different cultural contexts.