Peer-Review: Let's Imitate the Lawyers!

Abstract

As noted by Della Sala and Grafman in their Editorial, the traditional peerreview process has many drawbacks. In this comment, we would like to emphasize one particular problem of peer review, namely how reviewers handle findings that are incompatible with their own viewpoints. Speculating as to why it took several years before his ideas on the architecture of human memory were accepted by cognitive psychologists, Tulving (1999) noted that “(...) we scientists love to hate new ideas (...) and we routinely resist facts that do not fit into whatever comfortable framework we have managed to adopt (p. 12).” If this is correct, it would mean that journal referees exhibit a “confirmation bias”, such that they tend to favour manuscripts describing results in line with their own theoretical position. Is there evidence for confirmation bias in scientists? Chinn and Brewer (1993) showed that when science students are confronted with findings contradicting their beliefs, they tend to discount such findings in various ways. Sometimes “anomalous” results are simply ignored or placed in abeyance, other times the results are rejected, excluded from the domain of the theory at stake, or reinterpreted in such a way that the original theory is retained. Occasionally, when repeatedly exposed to data that contradict their ideas, students may change their theoretical positions. It appears that confirmation bias is not limited to students. Chinn and Brewer offer many convincing examples from the history of science demonstrating that scientists also have a propensity to discount data in order to protect their viewpoints. Interestingly, some authors have warned against the deleterious effects of confirmation bias among scientists. Greenwald et al. (1986), for instance, argued that the tendency of researchers to ignore or reject data incompatible with their own viewpoints may obstruct scientific progress. These authors describe how confirmation bias among social psychologists hindered revision of the theory on the so-called sleeper effect. The sleeper effect pertains to the delayed persuasive effect of a communication accompanied by a discounting cue (i.e., information indicating that the communication is untrustworthy). This phenomenon was discovered in the late 1940s, and while research in the early 1970s showed that the sleeper effect only took place under certain circumstances, it was not until the mid 1980s that the original theory was modified. Greenwald et al. provide evidence that researchers in the 1970s did not revise their theoretical ideas on the sleeper effect because their confirmation bias led them to ignore failed replications. Of course, it would be naive to think that confirmation bias only occurs in social psychology. We have reason to believe that reviewers also suffer from confirmation bias.