The Reliability of Same-Race and Cross-Race Skin Tone Judgments

Abstract

The purpose of this study is to examine the reliability of the skin tone measures in the widely used American National Election Studies data collection (ANES 2016 Time Series). Low reliability in skin tone measurement can lead to false conclusions regarding theoretically important relationships. Consistent with previous reliability analyses based on data from the General Social Survey, we find that different interviewers agree on Black and Latinx respondent skin tone less than 20% of the time and inter-rater reliability coefficients are very low (< .3). We also exploit unique features of the ANES data that allow us to (1) assess intra-rater reliability using Krippendorff’s alpha and (2) compare observer skin tone judgments to respondent self-appraisals. We find that even for cases where the same interviewer judges the same Black or Latinx respondent 2 months later, interviewers agree with their earlier assessment less than 35% of the time—only modestly exceeding expectations based on chance alone. Furthermore, we find weak correlations between how interviewers remember Black and Latinx respondent skin tone and how respondents self-describe. Importantly, our analyses indicate that these data patterns persist regardless of whether or not interviewer race/ethnicity matches that of the respondent. Thus, our results provide little support for the claim that measurement reliability can be significantly improved through a policy of matching respondents to interviewers of the same race and ethnicity. We discuss the implications for future research on skin tone’s relationship with social attitudes and outcomes.