Evaluating Test and Survey Items for Bias Across Languages and Cultures

Abstract

Introduction The world is growing smaller at a rapid rate in this 21st century, and it is little wonder that interest and activity in cross-cultural research are at their peak. Examples of cross-cultural research activities include international comparisons of educational achievement, the exploration of personality constructs across cultures, and investigations of employees’ opinions, attitudes, and skills by multinational companies. In many, if not all, of these instances, the research involves measuring psychological attributes across people who have very different cultural backgrounds and often function using different languages. This cultural and linguistic diversity poses significant challenges for researchers who strive for standardization of measures across research participants. In fact, the backbone of scientific research in psychology – standardization of measures – may lead to significant biases in the interpretation of results if the measuring instruments do not take linguistic and cultural differences into account. The International Test Commission (ITC) has long pointed out problems in measuring educational and psychological constructs across languages and cultures. Such problems are also well documented in the Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association, & National Council on Measurement in Education, 1999). For example, the Guidelines for Adapting Educational and Psychological Tests (Hambleton, 2005; ITC, 2001) provide numerous guidelines for checking the quality of measurement instruments when they are adapted for use across languages. These guidelines include careful evaluation of the translation process and statistical analysis of test and item response data to evaluate test and item comparability. Many of these guidelines are echoed by the aforementioned Standards . Table 8.1 presents some brief excerpts from the Guidelines and Standards that pertain to maximizing measurement equivalence across languages and cultures while ruling out issues of measurement bias. As can be seen from Table 8.1, both qualitative and quantitative procedures are recommended to comprehensively evaluate test comparability across languages. The qualitative procedures involve use of careful translation and adaptation designs and comprehensive evaluation of the different language versions of a test. Quantitative procedures include the use of dimensionality analyses to evaluate construct equivalence, differential predictive validity to evaluate the consistency of test-criterion relationships across test versions, and differential item functioning procedures to evaluate potential item bias.