Statistical reviewing policies of medical journals: caveat lector?

Abstract

To describe the current policies regarding statistical review of clinical research in biomedical journals. Cross-sectional survey. Editors of biomedical journals that publish original clinical research. General policies on statistical review, types of persons used for statistical reviewing, compensation of statistical reviewers, percentage of articles subject to such review, percentage of time statistical review makes an important difference, journal circulation, and selectivity. Of 171 journals, 114 (67%) responded to the survey. About one third of journals had policies that guaranteed statistical review for all accepted manuscripts. In approximately half of the journals, articles were sent for statistical review at the discretion of the editor. There was some evidence that statistical review policies differed between journals of different circulation size. In journals in the top quartile of circulation (> 25,000) the probability of definitely having a statistical review before an acceptance decision was 52%, but it was only 27% in journals in the lower three quartiles (p = .09). The probability of a statistical consultant on staff ranged from 31% in the bottom quarter, to 58% in the middle two, to 82% in the highest quarter (p < .001). Editors judged that statistical review resulted in an important change in a manuscript about half of the time. Except in the largest circulation medical journals, the probability of formal methodologic review of original clinical research is fairly low. As readers and researchers depend on the journals to assess the validity of the statistical methods and logic used in published reports, this is potentially a serious problem. This situation may exist because the cost of such statistical review can be considerable, and because finding appropriate reviewers can be difficult. It may also exist partly because editors or publishers may not regard such review as important. The professions of medical publishing, statistics, epidemiology, and other quantitative disciplines should work together to address this problem.