How Comparative Is Comparative Research?

Abstract

Social scientists tend to accept lower standards of rigor in cross-national surveys than in national surveys, leading to heroic conclusions about differences between nations on the basis of obviously faulty data. Arising perhaps from a commendable wish to respect different cultural norms, even some of the most conscientious cross-national studies make the mistake of permitting considerable variations by country in the type and quality of the methods they deploy. Meanwhile, analysts of cross-national data frequently abandon offering explanations and interpretations in favor of league tables of distributions showing merely “gee whiz” national differences. This article acknowledges the formidable obstacles in the way of achieving rigor in large-scale comparative studies and offers 10 possible rules to mitigate the difficulties. It suggests that bigger is usually worse and recommends routine cross-national collaboration in analysis and interpretation—not just in design, development, and execution—among scholars in each of the nations under the microscope.