Improving publication quality by reducing bias with double-blind reviewing and author response

Abstract

Because scientific progress depends on peer-reviewing, it behooves researchers to ensure that evaluations are as error free as possible and of the highest possible quality. However, scientists are human and humans have emotions and biases. The reviewing process acknowledges our humanity. In particular, single-blind reviewing never reveals the reviewers’ identity to the authors, in order to protect reviewers from author retribution. All the conference, journal, and grant processes of which I am aware use at least single-blind reviewing. Some also use double-blind reviewing. In addition to not revealing reviewer identities, the authors’ identities are not known to the reviewers, for most of the double-blind reviewing process. The purpose of double-blind reviewing is to focus the evaluation process on the quality of the submission by reducing human biases with respect to the authors’ reputation, gender, and institution, by not revealing those details. Compared to single-blind reviewing, every study so far shows double-blind reviewing improves the outcome of the process. Many ACM conferences sponsored by SIGPLAN, SIGARCH, SIGMETRICS, SIGMICRO, and SIGMOD [1, 6], some computer science journals, such as TODS [4], and many journals in other disciplines, such as The Journal of Finance [2], successfully use double-blind reviewing. The SIGPLAN PLDI community significantly prefers doubleblind reviewing. In the 2007 PLDI attendee survey, we had 148 responses from 334 attendees (a 44% response rate). Respondents indicated double-blind reviewing was: very useful (40), useful (50), neutral (30), not useful (14), or harmful (4). Only 19% were opposed to it and 60% support it. I recommend that SIGPLAN require all its conferences and journals to use double-blind reviewing for evaluating research submissions, and furthermore that SIGPLAN advocates for an ACM wide policy that requires double-blind reviewing. In the remainder of this editorial, I point to some of the literature and scientific studies on reviewing, discuss the types of biases that double-blind reviewing helps minimize, and suggest implementation strategies. These strategies include (1) author response, (2) an external review committee (instead of ad hoc external reviews and in addition to the program committee), and (3) to minimize errors, revealing authors before making final decisions, but of course after