POINT: Statistical analysis in NIH peer review— identifying innovation

Abstract

In this essay/hypothesis, I assess the statistical technology used in the NIH peer review of research proposals. In the NIH system of proposal review, sample sizes are inappropriately small, samples are highly selective or nonrandom which makes the review particularly susceptible to bias—the samples are not independently derived, the scoring precision is unrealistically high, and the arithmetic mean is the only statistical value considered. I propose a complementary peer review system that addresses these issues. Moreover, I hypothesize that innovative grant applications can be assessed in a system with robust statistical procedures by measures of dispersion such as variance and/or kurtosis. The identification of innovation has been a longterm deficiency at NIH. The peer review of grant proposals is an essential part of the management of biomedical research. Consequently, it is reasonable to consider how statistics are used in this peer review process. First, NIH and most funding agencies assess grant applications by requesting reviews from just two or three qualified persons. The number of samples is constrained by the large size of the grant applications and by writing proposals narrowly for extreme experts in narrowly defined fields. Second, peer review does not use random sampling of the relevant population. Instead experts are chosen based on their perceived expertise and qualifications. Unfortunately, this sampling scheme is particularly subject to bias because the extreme experts have a vested interest in the current paradigms. Third, the review process used by NIH and others involves discussions among reviewers. These discussions are meant to provide consensus, not to assess the distribution of sensibilities among the relevant population of scientists. The scores provided in this system are not independently derived, but independence of the observations is a valuable attribute for statistical analysis. Fourth, NIH uses a 41-grade scoring system which suggests an unrealistic degree of precision. A high-precision score is needed to distinguish among the many applications in a situation with only two or three dependent samples being collected. Finally, peer review of grant applications at NIH and most funding agencies uses a single statistic, the arithmetic mean. The use of this single statistic may be reasonable for a normal distribution of parametrically related observations; yet, scores given in peer review are not necessarily normally distributed and are definitely not parametrically related. Although funding agencies use statistical techniques that are constrained and although they largely ignore the distribution of reviews among the relevant peer group, their system has been successful in identifying excellent grant proposals. What NIH and other funding agencies have specifically struggled with is the identification of innovative ideas. Innovation and excellence are easily conflated; nevertheless, their distinction is essential in establishing the most potent funding policies. Excellence is the property that indicates a high degree of certainty that the proposed work will enhance our scientific understanding or capabilities. The trade-off is the inverse relationship between the degree of certainty and the degree of the enhancement. Excellence usually provides us with small steps forward. In contrast, innovation is the property that indicates novelty which carries with it a low degree of certainty of success. Although innovative projects lack certainty, the potential for huge enhancements in our scientific understanding or