SECTION In probabilistic disciplines from psychology to cancer biology and behavioral ecology, a disturbing quantity of empirically derived understanding has been challenged and found wanting (Begley and Ellis, 2012; Carpenter, 2012; Parker, 2013). Recently, it was reported that 47 of 53 “landmark” cancer studies from the past decade could not be reproduced (Begley and Ellis, 2012). Ongoing attempts to replicate results in psychology (Carpenter, 2012) have found that substantial portions do not stand subsequent tests (Reproducibility Project: https://osf. io/ezcuj/). Although some well-publicized cases of data fabrication have plagued that field recently (Vogel, 2011), much of the lack of repeatability is expected to result from less nefarious forms of bias (Ioannidis, 2005). Closer to home, a recent meta-analysis of studies of plumage color in a European songbird has substantially clouded what had been hailed as a model for the understanding of plumage color and sexual selection (Parker, 2013). The crux of the problem is that the published literature, especially in highly probabilistic systems, suffers from inflated type I error (false positive) rates, and careful replication is too rare to reliably separate the robust results from those resulting from error (Ioannidis, 2005; Parker, 2013). Thus, many published results are incorrect, and these results are too rarely discredited. Concerns about problems of empirical error are receiving attention from prestigious journals (e.g., Nature; Nuzzo, 2014), and in the popular press (e.g., Lehrer, 2010; Anonymous, 2013) they have stimulated a discourse that may be eroding public confidence in science. Strategies to reduce the problems of inflated error and infrequent replication are emerging in psychology, neuroscience, and medicine (Baker, 2012; Carpenter, 2012). High rates of type I error and low rates of replication may appear to result primarily from the decisions of individual researchers. These researchers are, however, responding to institutional incentive structures. For instance, funding bodies support novel projects to the exclusion of replications, and high impact journals also place a premium on novelty (Palmer, 2000; Kelly, 2006). As another example, most journals select articles based on study outcome rather than just soundness of hypothesis, predictions, and methods (Chambers, 2013). Thus, researchers often choose to report the most interesting subsets of results or pursue other forms of biased reporting rather than reporting the entire set of outcomes (John et al., 2012). Institutions also promote bias, and possibly even academic dishonesty, by basing professional evaluation and remuneration on number of publications and the stature of the journals in which they are published (Qiu, 2010; John et al., 2012). Thus, effective strategies will come from changes in the institutions that influence our research practices, such as professional societies (including journals) and funding agencies (Parker, 2013). It is precisely at this institutional level that psychology and medicine are tackling the challenge of reducing bias and increasing replication. Initiatives in these other disciplines are not necessarily templates directly transferable to ecology and evolution. Yet, such examples should serve to stimulate discussion and they clearly demonstrate that redesigning incentive structures is possible. Reducing incomplete and biased reporting of results may be accomplished by encouraging or requiring registration of studies at their initiation (Schooler, 2011). Since 2000, the US government has provided a registry for clinical trials of medical interventions (ClinicalTrials.gov). Registration prior to initiation is a requirement of many funding agencies and medical journals, and thus has become “standard practice” (Huser and Cimino, 2013). Although results from approximately half of registered trials end up unpublished, about a third of the unpublished studies post some results in the registry (Ross et al., 2009). Further, the registry facilitates a more precise estimate of reporting bias, and provides contact information for researchers with unpublished work. Thus, the bias in available results has dropped along with our ignorance of this bias. These are highly desirable outcomes. A conceptually similar idea is the “registered report” initiated by the neuroscience journal Cortex in 2013 (Chambers, 2013). To publish in the registered report section of the journal, researchers submit a study plan for peer review and conditional acceptance prior to gathering data (http://www.elsevier.com/journals/cortex/ 0010-9452/guide-for-authors). This counteracts several forms of publication bias, including editors’ preferences for statistically significant or novel outcomes, and
Read full abstract
Jun 14, 2024
Ilona Iłowiecka-Tańska 
Open Access