Reviewer bias against the unconventional? A randomized double-blind study of peer review

Abstract

To test the hypothesis that there is a reviewer bias against publication of a test of an unconventional drug. Randomized, controlled, double-blind study of peer review. Convenience sample of 291 medical doctors from a wide variety of specialties drawn from a list of conference participants. Reviewers were randomly assigned to receive one of two versions of a manuscript. Version M related to an in-vitro experiment on a mainstream drug (Metoprolol). The otherwise identical version V used a highly unconventional drug (beef spleen cell extract) for the same experiment. Reviewers were asked to complete a standardised evaluation sheet including visual analogue scales (VASs) on a set of predefined quality criteria. All participants were debriefed after completion of the study. The response rate was 61%. There were no significant differences in VAS ratings between the two versions of the manuscript. Ratings covered the entire range of the VASs. In the present setting, there was no evidence for a reviewer-bias against testing an unconventional drug. The low inter-rater reliability, however, suggested inadequate validity of peer review.