Peer Review: The Best of the Blemished?

Abstract

One of Winston Churchill’s most memorable quotations comes from a speech to the House of Commons in which he remarked that, “Democracy is the worst form of government except for all the other forms that have been tried.” In my opinion, the same sentiment applies to the peer review process employed by biomedical academic journals. As is the case for Churchill’s democracy, there can be little doubt that peer review, as currently practiced, has flaws. Its dual purpose is to assess the quality of manuscripts and to help authors improve those manuscripts. Most editors of medical journals believe that employing a wide array of specialty reviewers is of considerable value in appraising submissions. In addition, many authors—but not all—are grateful for constructive criticisms that result in improvements to their published work.In recent years, both editors and scientists have begun to examine the peer review process to elucidate bias and error in the system.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar, 3Jefferson T. Alderson P. Wager E. Davidoff F. Effects of editorial peer review: a systematic review.JAMA. 2002; 287: 2784-2786Crossref PubMed Scopus (237) Google Scholar, 4Kearney M.H. Freda M.C. Nurse editors’ views on the peer review process.Res Nurs Health. 2005; 28: 444-452Crossref PubMed Scopus (37) Google Scholar, 5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google Scholar, 6Burnham J.C. The evolution of editorial peer review.JAMA. 1990; 263: 1323-1329Crossref PubMed Scopus (197) Google Scholar, 7Olson C.M. Peer review of the biomedical literature.Am J Emerg Med. 1990; 8: 356-358Abstract Full Text PDF PubMed Scopus (15) Google Scholar As might be expected, 2 reviewers examining the same manuscript might express diametrically opposed views. Indeed, it is not uncommon for me to see one reviewer suggest acceptance while another counsels against it. Studies of the reproducibility of peer review based on critiques of the same manuscript by different reviewers have demonstrated considerable disparities between reviewers’ responses.5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google ScholarAnother failing, aside from inconsistency, is delayed decision-making. Although computerization of the review process has shortened the time between author submission and an editorial decision, significant slow-downs continue to exist. Other commonly cited defects in the peer review system include reviewer bias, conflict of interest, expense, and even occasional instances of reviewer dishonesty or abuse of the author.Bias exists in all human thinking, and a number of studies have documented bias in the peer review process. Such bias may be based on the gender or national origin of the author, or it may relate to previously held prejudices concerning the type of work described or the methodology employed in the study. Today, potential conflicts of interest are generally an issue in biomedical research as well. Conflicts may be financial or may relate to other factors, such as concerns about career advancement or jealousy.8Alpert J.S. Furman S. Smaha L. Conflicts of interest: science, money, and health.Arch Intern Med. 2002; 162: 635-637Crossref PubMed Scopus (21) Google ScholarScientific misconduct also has been documented in the peer review process. There have been a number of well-publicized examples of plagiarism of ideas and of components of a peer-reviewed manuscript. Finally, the peer review system often fails to uncover examples of scientific fraud on the part of a paper’s authors until 1 or more manuscripts already have been published and subsequently shown to be false. The recent Korean stem-cell scandal received front-page coverage in almost every major newspaper in the world. Some have argued that the falsified publications of Dr. Hwang Woo-Suk in Science during 2004 and 2005 demonstrated the weakness of the present peer review mechanism. However, the opposite argument also has been advanced. In the end, peer review triumphed when Dr. Hwang Woo-Suk’s work was shown to contain falsified data.Can the process be improved? Given the many potential problems and abuses attributed to peer review, it is surprising that the system, though deemed to be faulty, has generally functioned well.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar It remains the “standard of care” for most biomedical journals published today.A number of editors and academicians have suggested reforms aimed at improving the identified shortcomings of peer review. Double-blinding of the review process; open reviews published on the Internet alongside the author’s response; and focused reviewer training have been suggested and evaluated in a handful of small trials.3Jefferson T. Alderson P. Wager E. Davidoff F. Effects of editorial peer review: a systematic review.JAMA. 2002; 287: 2784-2786Crossref PubMed Scopus (237) Google Scholar, 9Godlee F. Making reviewers visible: openness, accountability, and credit.JAMA. 2002; 287: 2762-2765Crossref PubMed Scopus (122) Google Scholar Few existing studies have assessed the efficacy and accuracy of the peer review process.5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google Scholar Moreover, the results of these small trials often have produced conflicting conclusions. No suggested improvement has emerged as the answer to previously enumerated deficiencies.So what is an editor to do? In the final analysis, I have come to believe that like the democratic system, peer review is an imperfect arrangement, but it is better than anything else that has been previously tried.Reviewers are rarely remunerated for their work, and to my knowledge, there is no current movement to change the situation. If they were paid, the annual expense would be enormous, and the readership would likely be required to contribute. At The American Journal of Medicine, we remain indebted to colleagues who serve as reviewers. We also owe a debt of gratitude to readers who share their comments on manuscripts that we have published—this is another form of peer review. As I noted in an earlier editorial, the editors of the Green Journal strive to bring you a quality product. Their decisions are based on multiple factors, including, but not limited to, reviewers’ comments. We attempt to anticipate your interest in specific topics, and we also try to publish material that will be of broad interest to the internal medicine community.Overall, the peer review process warrants our confidence; we should trust the reviewers’ and editors’ efforts at objectivity.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar, 10Alpert J.S. Dear editor.Am J Med. 2006; 119: 193Abstract Full Text Full Text PDF Scopus (1) Google Scholar, 11Freshwater D. Editors and publishing: integrity, trust, and faith.J Psychiatr Ment Health Nurs. 2006; 13: 1-2Crossref PubMed Scopus (7) Google Scholar Those of us who participate in the peer review process pledge our honesty. If we fail, it stains our reputations. Similarly, editors and their editorial boards are committed to the highest ideals of academic honesty and scholarship. Undoubtedly, The American Journal of Medicine will continue to rely on the peer-review system when making decisions until a different or modified technique is conclusively proven to be better. One of Winston Churchill’s most memorable quotations comes from a speech to the House of Commons in which he remarked that, “Democracy is the worst form of government except for all the other forms that have been tried.” In my opinion, the same sentiment applies to the peer review process employed by biomedical academic journals. As is the case for Churchill’s democracy, there can be little doubt that peer review, as currently practiced, has flaws. Its dual purpose is to assess the quality of manuscripts and to help authors improve those manuscripts. Most editors of medical journals believe that employing a wide array of specialty reviewers is of considerable value in appraising submissions. In addition, many authors—but not all—are grateful for constructive criticisms that result in improvements to their published work. In recent years, both editors and scientists have begun to examine the peer review process to elucidate bias and error in the system.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar, 3Jefferson T. Alderson P. Wager E. Davidoff F. Effects of editorial peer review: a systematic review.JAMA. 2002; 287: 2784-2786Crossref PubMed Scopus (237) Google Scholar, 4Kearney M.H. Freda M.C. Nurse editors’ views on the peer review process.Res Nurs Health. 2005; 28: 444-452Crossref PubMed Scopus (37) Google Scholar, 5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google Scholar, 6Burnham J.C. The evolution of editorial peer review.JAMA. 1990; 263: 1323-1329Crossref PubMed Scopus (197) Google Scholar, 7Olson C.M. Peer review of the biomedical literature.Am J Emerg Med. 1990; 8: 356-358Abstract Full Text PDF PubMed Scopus (15) Google Scholar As might be expected, 2 reviewers examining the same manuscript might express diametrically opposed views. Indeed, it is not uncommon for me to see one reviewer suggest acceptance while another counsels against it. Studies of the reproducibility of peer review based on critiques of the same manuscript by different reviewers have demonstrated considerable disparities between reviewers’ responses.5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google Scholar Another failing, aside from inconsistency, is delayed decision-making. Although computerization of the review process has shortened the time between author submission and an editorial decision, significant slow-downs continue to exist. Other commonly cited defects in the peer review system include reviewer bias, conflict of interest, expense, and even occasional instances of reviewer dishonesty or abuse of the author. Bias exists in all human thinking, and a number of studies have documented bias in the peer review process. Such bias may be based on the gender or national origin of the author, or it may relate to previously held prejudices concerning the type of work described or the methodology employed in the study. Today, potential conflicts of interest are generally an issue in biomedical research as well. Conflicts may be financial or may relate to other factors, such as concerns about career advancement or jealousy.8Alpert J.S. Furman S. Smaha L. Conflicts of interest: science, money, and health.Arch Intern Med. 2002; 162: 635-637Crossref PubMed Scopus (21) Google Scholar Scientific misconduct also has been documented in the peer review process. There have been a number of well-publicized examples of plagiarism of ideas and of components of a peer-reviewed manuscript. Finally, the peer review system often fails to uncover examples of scientific fraud on the part of a paper’s authors until 1 or more manuscripts already have been published and subsequently shown to be false. The recent Korean stem-cell scandal received front-page coverage in almost every major newspaper in the world. Some have argued that the falsified publications of Dr. Hwang Woo-Suk in Science during 2004 and 2005 demonstrated the weakness of the present peer review mechanism. However, the opposite argument also has been advanced. In the end, peer review triumphed when Dr. Hwang Woo-Suk’s work was shown to contain falsified data. Can the process be improved? Given the many potential problems and abuses attributed to peer review, it is surprising that the system, though deemed to be faulty, has generally functioned well.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar It remains the “standard of care” for most biomedical journals published today. A number of editors and academicians have suggested reforms aimed at improving the identified shortcomings of peer review. Double-blinding of the review process; open reviews published on the Internet alongside the author’s response; and focused reviewer training have been suggested and evaluated in a handful of small trials.3Jefferson T. Alderson P. Wager E. Davidoff F. Effects of editorial peer review: a systematic review.JAMA. 2002; 287: 2784-2786Crossref PubMed Scopus (237) Google Scholar, 9Godlee F. Making reviewers visible: openness, accountability, and credit.JAMA. 2002; 287: 2762-2765Crossref PubMed Scopus (122) Google Scholar Few existing studies have assessed the efficacy and accuracy of the peer review process.5Kumar P.D. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper.J Assoc Physicians India. 1999; 47: 198-200PubMed Google Scholar Moreover, the results of these small trials often have produced conflicting conclusions. No suggested improvement has emerged as the answer to previously enumerated deficiencies. So what is an editor to do? In the final analysis, I have come to believe that like the democratic system, peer review is an imperfect arrangement, but it is better than anything else that has been previously tried. Reviewers are rarely remunerated for their work, and to my knowledge, there is no current movement to change the situation. If they were paid, the annual expense would be enormous, and the readership would likely be required to contribute. At The American Journal of Medicine, we remain indebted to colleagues who serve as reviewers. We also owe a debt of gratitude to readers who share their comments on manuscripts that we have published—this is another form of peer review. As I noted in an earlier editorial, the editors of the Green Journal strive to bring you a quality product. Their decisions are based on multiple factors, including, but not limited to, reviewers’ comments. We attempt to anticipate your interest in specific topics, and we also try to publish material that will be of broad interest to the internal medicine community. Overall, the peer review process warrants our confidence; we should trust the reviewers’ and editors’ efforts at objectivity.1Smith R. Peer review: a flawed process at the heart of science and journals.J R Soc Med. 2006; 99: 178-182Crossref PubMed Scopus (552) Google Scholar, 2Hojat M. Gonnella J.S. Caelleigh A.S. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process.Adv Health Sci Educ Theory Pract. 2003; 8: 75-96Crossref PubMed Scopus (145) Google Scholar, 10Alpert J.S. Dear editor.Am J Med. 2006; 119: 193Abstract Full Text Full Text PDF Scopus (1) Google Scholar, 11Freshwater D. Editors and publishing: integrity, trust, and faith.J Psychiatr Ment Health Nurs. 2006; 13: 1-2Crossref PubMed Scopus (7) Google Scholar Those of us who participate in the peer review process pledge our honesty. If we fail, it stains our reputations. Similarly, editors and their editorial boards are committed to the highest ideals of academic honesty and scholarship. Undoubtedly, The American Journal of Medicine will continue to rely on the peer-review system when making decisions until a different or modified technique is conclusively proven to be better.