Anatomy of Constructive Peer Review.

Abstract

The value of peer review continues to be debated in the scientific community. In recent years, a number of prominent researchers have questioned the utility of peer review, claiming that it is inefficient, slow, expensive,1 and sometimes fails to meet its objectives: namely, to elevate worthy works of scholarship to the status of “the medical literature” and to keep less-worthy works from seeing the light of day. The critics are not without evidence that supports these claims, including studies exposing biases in the peer review process;2,3 publication in numerous journals of reports consisting of computer-generated gibberish or obviously fake science;4 the failure of reviewers to spot obvious and important errors (even after training);5 and the fact that even some Nobel-prize-winning works of scholarship were initially rejected by peer reviewers.6 A Cochrane Review from over a decade ago concluded that little empirical evidence supported the use of peer review as a way to assure the quality of published research.7 Despite this criticism, we hold peer review in the same vein as Winston Churchill did democracy: “No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.” So why should the scientific community engage in the arduous process of peer review? From the perspective of the editorial staff of scientific journals, the most obvious answer is that we want to publish valid (and reliable) findings that will contribute to the scientific understanding of a topic. At a more nuanced level, the intent is to deepen scientific knowledge as well as provide some guidance as to a profitable direction to pursue for extending that body of knowledge. A less-recognized, or at least less-acknowledged, purpose of peer review is to develop the next generation of scientists by providing constructive criticism of their work—that is, guidance as to how to improve the product. Of course, both the reviewed scientist and the reviewer potentially benefit: the former by obtaining a holy grail of academe: publications; and the latter by providing “service” to the scientific community that tenure and promotion committees want to see. A meaningful review is one that achieves these purposes. This begs the question as to what constitutes a meaningful review? Scholars have been trying to answer that question for decades, and now, many peer-reviewed journal publishers provide explicit guidance to peer reviewers.8 Guidelines9 suggest that meaningful or constructive reviews share the following characteristics: they are written in a positive, impartial, and polite tone; provide constructive guidance on a paper’s utility, accuracy, structure, and salience to the field; and, of course, they are returned as quickly as possible. Reviewers also have a responsibility to identify inaccurate or misleading work and to disclose any real or apparent conflicts of interest.10 Here at Medical Care, we recently had our own experience with bias in peer review—in this case, bias against people with disabilities.3,11 As a follow-up to existing guidelines for constructive review, we thought it would be useful to summarize some of the elements that Medical Care editors use when evaluating reviews. COMMENTS TO THE EDITOR ARE ESSENTIAL TO ADVANCING KNOWLEDGE In any review, there is always a section for “confidential comments to the editor.” This section is often left blank by the reviewer. Having done numerous reviews ourselves, we understand that reviewers often see this request as a burdensome summary of their “comments to the authors.” After all, we imagine the reasoning goes, the editors are supposedly competent scientists: why can’t they figure it out from my comments to the authors? The purpose of the section, though, is not to summarize the comments to the authors. The intent is for the reviewer—as an expert in a topic area—to inform the editor(s) as to whether the manuscript describes findings that contribute to the specific body of knowledge and what the magnitude of the contribution is. Medical Care receives hundreds of manuscripts each year on many different topics. While we try to assign each manuscript to the editor with the most appropriate background, no one is totally knowledgeable on every topic even within a rather narrow scientific area, especially in an era of voluminous information being revealed at a rapid rate. Assuming the science is solid, the editor needs to know if the manuscript presents duplicative findings, or it adds to the knowledge base, and if the contribution is modest, moderate, or huge. In situations where the knowledge base is poorly developed, modest findings may be a leap forward. The goal is to get novel, important information out to the scientific community. COMMENTS TO THE AUTHORS ARE ESSENTIAL TO IMPROVING SCIENCE The section entitled “Comments to the Authors” is where the science of the study is assessed and where most reviewers feel quite comfortable. There are punch-lists (eg, PRISMA) to assist the reviewer in “checking” all the boxes. But it is here that a systemic, insightful assessment is needed. While conceding that there is no perfect study, the editors need to know if the methodology is sufficiently rigorous scientifically to yield a reasonable contribution? Are the gaps in knowledge concisely and correctly described? Does the research objective (question) address the identified gap(s) in knowledge? Is the data set fit to answer the question? That is, does the data set contain the data elements or reasonable proxies thereof required to answer the question? Is the number of observations adequate to yield informative results? Will the data analytic technique appropriately interrogate the data to reveal a reasonable answer? Does the interpretation of the findings flow logically from the analysis and data limitations? Has the research question been answered, at least in part? The reviewer’s approach to answering these questions is important as well as the answers obtained. Which is more informative to the investigator: “The investigators need to get a competent biostatistician to analyze these data” or “The data analysis raises numerous concerns, including….”? Ad hominid remarks serve no useful purpose; beyond reflecting poorly on the character of the reviewer, such remarks often discourage the new investigator and provide little guidance as to the shortcomings that must be corrected. The take-home point is that specificity in the critique is essential to yielding improvement in the science. OPPORTUNITY DISGUISED AS WORK When you are asked to provide a peer review, it is an opportunity to deepen or extend scientific knowledge, and to help develop the next generation of health scientists. For those of us who have been around a while, we know that opportunity almost always comes disguised as work. We recognize that typical reviewers are trying to move forward in their careers and that they have a considerable workload in terms of teaching, conducting their personal research, and performing service activities. They are actively submitting manuscripts for review and desire a timely review by someone competent to evaluate their work. But the authors of the work we are asking to be reviewed come from this same population of active scientists and have the same constraints and the same desires regarding the review of their manuscripts. One of the more frustrating challenges of editors is securing reviewers to do the review, especially around holidays, semester breaks, and in the summer: times when everyone is seeking a break from the routine. Fortunately, there are always those heroic individuals who say, yes, I can do the review. A well-constructed review makes all the difference in the quality of the work published in scientific journals. A constructive review takes considerable effort because it requires specificity; as a reviewer, you must explain your decision-making. Mentoring, when done right, is no easy task. But if our goal is to make an impact and to do so in the near-term, we need to identify and advance or improve the work that will make an impact and, in the process, build the skills of the investigators doing that work. The effort given to the task of peer review may not be immediately remunerated but it will pay dividends to the scientific community and society by improving the body of evidence upon which we base practice and policy. We welcome all who will join us in trying to move the effectiveness of peer review forward. And by the way, we have a manuscript that we would like you to review. If you are interested in signing up as a potential peer reviewer for Medical Care, please visit https://www.editorialmanager.com/mdc/accounts/rev_requirements.html to learn more and register.