A Simple Question of Reproducibility

Abstract

O of the things that we pride ourselves on at BioTechniques is providing simple methods developments that can be implemented in the lab immediately. Maybe this makes us throwbacks—“old school” so to speak. But it seems that life science research is becoming so complex at the moment that it is now difficult to reproduce major scientific results between labs. At least this is better than the alternative explanation, that fabricating or “massaging” data have become common practices amoung researchers. Either way, the lack of reproducibility is a serious problem at the moment that needs to be addressed by life scientists. The troubling trend of irreproducibility, which has been brewing for some time now, came to a head for many with the publication of a commentary in the journal Nature by Begley and Ellis finding that the results of 47 out of 53 studies could not be replicated. These preclinical studies formed the basis for other research studies and in some instances were the starting points for costly drug studies. Begley and Ellis are not alone in their findings— other reports have surfaced in recent months highlighting the problem of irreproducible studies. Recently, a group called the Global Biological Standards Institute (GBSI) presented a report making a case for biological standards. In interviews with 60 key figures in the life science community, nearly 75% of those interviewed described having to deal with irreproducible data and/or results. The conclusion of the GBSI was that there is a need for more well-defined and consistently used standards, both material (reference reagents and chemicals) and written (optimal practices and methodologies). The Begley and Ellis commentary also brings to light another fundamental problem in life science today: researchers tackling projects beyond their fields of expertise. Preclinical and translational research have become the buzzwords of the moment, leading many investigators to search out ways in which to focus their grants on these types of studies. But what is the impact of this trend? Are these researchers sufficiently trained and equipped to perform the necessary experiments and analyze their findings in a meaningful way when it comes to clinical significance? Or are attempts to include the “right” buzzwords in grant applications in order to obtain funding dictating experimental design? The impact of irreproducible studies is particularly troubling for me as the editor of a methods-oriented journal. The manner in which methods are reported and detailed is critical to reproducing the findings of an article— researchers need to be able to replicate experiments 1, 10, or even 20 years after an article has been published. While it is a great to be able to go to another lab or exchange reagents to verify an experiment or study in the short-term, what happens over time? Researchers need to understand the methods they are using and document the manner in which they are being performed so that future generations can build upon the work that is being done today. In the end, science continues to grow more complex and more interdisciplinary. This is an exciting development, but it creates the need to modify traditional ways of thinking about grants, peer review, publishing, and experimental design. Specific steps, chemicals, and processes need to be documented. At BioTechniques, we will be taking additional steps during the peer review process to ensure that all of the information needed to replicate a method is presented in every report. In addition, we encourage authors to submit articles identifying “challenging” reagents—including the robust characterization of antibodies and or cell lines— as well as other potential roadblocks standing in the way of reproducibility. The creation of new standards in life science research is an important endeavor that requires care and thought. Researchers need to be cautious in designing their experiments and also report full experimental procedures and results. Journals should require complete methods descriptions, even if they appear as online Supplementary Materials, as this will provide a great first step towards reproducibility. It is interesting to note that at this moment of greater irreproducibility in life science, journals continue to minimize the space given to Materials and Methods sections in articles. Reporting the way in which experiments are performed should not be an afterthought. While the goal is simple—eliminating irreproducibility—getting on the right path could prove tough.