Inferential selection bias in a study of racial bias: Revisiting ‘Working twice as hard to get half as far’

Abstract

This study revisits an important recent article about racial bias and finds that many of its inferences are weakened when we analyze the data more completely. DeSante in 2013 reported evidence from a survey experiment indicating that Americans reward Whites more than Blacks for hard work but penalize Blacks more than Whites for laziness. However, the present study demonstrates that these inferences were based on an unrepresentative selection of possible analyses: the original article does not include all possible equivalent or relevant analyses, and when results from these additional analyses are combined with the results reported in the original article, the strength of inferences is weakened. Moreover, newly-reported evidence reveals heterogeneity in racial bias: respondents given a direct choice between equivalent targets of different races favored the Black target over the White target. These results illustrate how the presence of researcher degrees of freedom can foster production of inferences that are not representative of all inferences that a set of data could produce. This study thus highlights the value of preregistering research design protocols and required public posting of data.