Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience

Dorothée Behr,Lars Kaczmirek,Michael Braun,Wolfgang Bandilla

doi:10.1007/s11135-012-9754-8

Dorothée Behr, Lars Kaczmirek + Show 2 more

Open Access

https://doi.org/10.1007/s11135-012-9754-8

Copy DOI

Abstract

This article focuses on assessing item comparability in cross-national surveys by asking probing questions in Web surveys. The “civil disobedience” item from the “rights in a democracy” scale of the International Social Survey Program (ISSP) serves as a substantive case study. Identical Web surveys were fielded in Canada (English-speaking), Denmark, Germany, Hungary, Spain, and the U.S. A category-selection and a comprehension probe, respectively, were incorporated into the Web surveys after the closed-ended “civil disobedience” item. Responses to the category selection-probe reveal that notably in Germany, Hungary, and Spain the detachment of politicians from the people and their lack of responsiveness is deplored. Responses to the comprehension probe show that mainly in the U.S. and Canada violence and/or destruction are associated with civil disobedience. These results suggest reasons for the peculiar statistical results found for the “civil disobedience” item in the ISSP study. On the whole, Web probing proves to be a valuable tool for identifying interpretation differences and potential bias in cross-national survey research.

Highlights

1.1 Comparability needs in cross-national survey researchLarge-scale cross-national survey projects, such as the World Value Surveys (WVS), the European Values Study (EVS), the International Social Survey Program (ISSP), the European Social Survey (ESS), and the Eurobarometer have established long time series, partly beginning in the 1970s and 1980s
This paper extends this research to the international arena and addresses the question to what extent probing techniques implemented in cross-national Web surveys allow to unravel different interpretation patterns across countries
While the means for civil disobedience is lower than the index in all countries, it is low for Canada, Denmark, and the U.S

Summary

Introduction

1.1 Comparability needs in cross-national survey researchLarge-scale cross-national survey projects, such as the World Value Surveys (WVS), the European Values Study (EVS), the International Social Survey Program (ISSP), the European Social Survey (ESS), and the Eurobarometer have established long time series, partly beginning in the 1970s and 1980s. 1.1 Comparability needs in cross-national survey research. A major goal of these surveys is the analysis of social change in a comparative perspective, which requires the continued invariant measurement of. Countries can differ in the way the majority of the population interprets a question, even if the question seems well-translated. Social change over time may lead to changes in item interpretation in a country. These aspects are a threat to the long-term validity of cross-national surveys because differences across countries and over time could be methodological artifacts rather than “real” differences between countries and time periods

Methods

Results

Conclusion