Discussion: An estimate of the science-wise false discovery rate and application to the top medical literature

Abstract

It is a pleasure to have the opportunity to comment on this contribution by Jager and Leek (2013), perhaps the first broad-based empirical attempt to examine the now 8-year-old claim that most conclusions in the medical literature are false (Ioannidis, 2005). The latter paper has had considerable traction, reportedly being the most downloaded article in the history of PLoS Medicine. Its potential positive impact was more wariness about blithely accepting the results of published studies, better understanding of the factors that lead to misleading research, and more awareness of the domain of meta-research. The potential negative impact of the claim was an unwarranted degree of skepticism, hopefully not cynicism, about truth claims in medical science. So, exploring that hypothesis further using empirical data was much needed. But before commenting on Jager and Leek’s contribution, it is useful to understand the assumptions and methodology that supported the original claim, outlined in detail in 2007 (Goodman and Greenland, 2007). Ioannidis’s argument was based on simple Bayesian mathematics, the same kind used in diagnostic studies. However, two elements were incorporated that had the mathematical effect of nullifying research evidence. First, the information from an experiment used in his model was simply P <0.05, not the exact P-value. Thus, an epidemiologic study generating a main finding with a P <0.0001 was treated in that model identically to one with P =0.04. Quantitatively, for a study with 80% power, this reduced the maximum Bayes factor—the degree to which an experiment raises the prior odds of the alternative hypothesis— from infinity to 16.