Little evidence to support the use of editorial peer review to ensure quality of published research

Abstract

Studies were identified using the following sources: CINAHL (Cumulative Index to nursing and Allied Health Literature), Cochrane Methodology Register, Dissertation Abstracts, Embase, Evidence-based Medicine Reviews, American College of Physicians Journal Club, Medline, PsycINFO and PubMed. Prospective or retrospective comparative studies were included that had two or more comparison groups, generated by random or other appropriate methods, and that reported original research, regardless of publication status. Because of the variety of study designs, unit of randomisation and outcome measures, a descriptive review was carried out. Twenty-eight studies were included. There was no clear-cut evidence of effect of the well-researched practice of reviewer and/ or author concealment on the outcome of the quality assessment process (nine studies). Checklists and other standardisation media have some evidence to support their use (two studies). There is no evidence that referees' training has any effect on the quality of the outcome (one study). Different methods of communicating with reviewers and means of dissemination do not appear to have an effect on quality (three studies). On the basis of one study, little can be said about the ability of the peer-review process to detect bias against unconventional drugs. Validity of peer review was tested by only one small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (two studies), but the evidence for this has very limited generalisability. At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. The methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be launched urgently.