The Anonymous American Idol Manuscript Reviewer

Abstract

In his recent Correspondence, Michele Pagano likens NIH grant reviewers to judges on the TV talent show American Idol (Pagano, 2006xPagano, M. Cell. 2006; 126: 637–638Abstract | Full Text | Full Text PDF | PubMed | Scopus (9)See all ReferencesPagano, 2006). The appealing aspect of American Idol is the face-to-face confrontation of the judges (a.k.a. “reviewers”) and contestants (a.k.a. “scientists”). The judges dole out their evaluations, while the contestants attempt to formulate a come-back (a.k.a. “resubmission”). NIH grant review is clearly not a game of shielded anonymous criticism because the reviewers must justify their criticisms to fellow reviewers. In contrast, with the exception of journal editors, reviewers of research manuscripts are entirely anonymous. So, while both review processes strive to achieve the same ends, the means of obtaining a fair and unbiased review are entirely different. I agree with Pagano's assertion that there is still trouble in paradise at NIH's Center for Scientific Review. However, I disagree with his suggestion that moving NIH grant review closer to an anonymous manuscript review-style system by decreasing face-to-face reviewer discussions will impart a fairer grant review process. Indeed, I think it would be disastrous.As any author who has submitted 3 years of their life's blood in the form of a research manuscript only to see it rejected by an out-of-hand and out-of-control “anonymous” reviewer can attest, the current manuscript review process has failings. In my view, the problem boils down to a curtain of anonymity in the review process. While anonymity can offer both a fair and unbiased manuscript assessment, it also leaves open the potential for exaggerated critiques and hidden agendas. Unlike NIH grant review where a face-to-face discourse puts the reviewer's reputation on the line and often leads to a toned down, focused criticism, the anonymous manuscript reviewer can easily kill a manuscript's chances by raising the acceptance threshold to an unattainable level and plying the editor with a laundry list of “critical” experiments.I think we should apply the lessons learnt from the NIH grant review process to increase the quality of research manuscript review. In a similar vein to NIH study sections, one obvious approach is to hold a monthly video conference meeting with the editors and reviewers of all manuscripts reviewed that month. This could narrow the error bars of the written reviews, especially the extreme comments, and hence the decisions made. By knowing the fixed review meeting date in advance, the reviewers could call in from any location in the world, and given the limited number and size of manuscripts compared to 25-page NIH grants, the meeting could be relatively short. Although this may add several weeks to the manuscript review process, the reward of less biased reviews may easily outweigh this potential downside.The clear benefit of a conference call manuscript review approach is that it would expose the anonymous “out-of-control” reviewer. Similar to grant reviews where the reviewer's reputation is on the line, this miniscule exposure from behind the curtain of anonymity could temper reviewers to make sure that what they are proposing in the form of additional experimentation is, in fact, not 2 years worth of work merely designed to kill the manuscript. Furthermore, just like NIH grant reviews, journals should publish the names of all reviewers for a given period of time that includes when your manuscript was reviewed.As for the NIH grant review process, opting for a version of our current anonymous manuscript review system by eliminating face-to-face contact of reviewers would only serve to widen the error bars among reviews. Moreover, eliminating the face-to-face discussions of study sections would have no effect on decreasing the submission-to-score time as reviews are due ∼5 days before the study section meets. This is clearly not the bottleneck. So the next time you are hiding behind the cloak of manuscript reviewer anonymity, ask yourself if you would make these same comments if you were required to summarize them at a meeting. Better yet, would you make them about your own manuscript?