Replicability in the context of systematic reviews: A call for a framework with considerations regarding duplication, overlap, and intentionality

Abstract

Reproducibility and replicability are considered central to the development and evolution of science, as they should ensure the efficient generation of reliable knowledge [[1]National Academies of Sciences, Engineering, and MedicineReproducibility and Replicability in Science. The National Academies Press, Washington, DC2019https://doi.org/10.17226/25303Crossref Google Scholar]. Much has been written about the “reproducibility crisis” [[2]Nelson NC Ichikawa K Chung J Malik MM. Mapping the discursive dimensions of the reproducibility crisis: a mixed methods analysis.PLoS One. 2021; 16e0254090Crossref Scopus (5) Google Scholar] and “replicability crisis” [[3]Schooler JW. Metascience could rescue the 'replication crisis'.Nature. 2014; 515: 9Crossref PubMed Scopus (88) Google Scholar] as a serious threat to all experimental sciences. However, it has been noted that there are inconsistent and contradictory uses of the terms reproducibility and replicability by different disciplines, institutions, and groups. Other similar terms, such as repeatability, have been used to denote the general concept of confirming the research results of other researchers. Thus, the lack of standard definitions hinders the assessment of reproducibility, and replicability [[1]National Academies of Sciences, Engineering, and MedicineReproducibility and Replicability in Science. The National Academies Press, Washington, DC2019https://doi.org/10.17226/25303Crossref Google Scholar]. Although there are manuscripts pondering the definitions of reproducibility and replicability in general [[4]Vachon B Curran JA Karunananthan S Brehaut J Graham ID Moher D Replication research series-paper 1 : a concept analysis and meta-narrative review established a comprehensive theoretical definition of replication research to improve its use.J Clin Epidemiol. 2021; 129: 176-187Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar], we consider that there are particular considerations regarding these definitions in the context of systematic reviews that deserve further attention. When it comes to systematic reviews, multiple articles offered a definition of terms reproducibility, and replicability. For example, Shokraneh [[5]Shokraneh F. Reproducibility and replicability of systematic reviews.World J Meta-Anal. 2020; 7: 66-71Crossref Google Scholar] used the following definition [quote]: “reproducibility is re-conducting the same study, using the same methods and data by a different researcher or team and the replicability is re-doing the same study to gather new data or recollect the data” [[6]Patil P Peng RD Leek JT. A statistical definition for reproducibility and replicability.bioRxiv. 2016; (066803 https://www.biorxiv.org/content/10.1101/066803v1)https://doi.org/10.1101/066803Crossref Google Scholar]. Others reported that reproducibility involves reanalyzing the data collected in a study using the same computational steps and analytic code as the original study, and replicability is conducting a new study designed to address the same question(s) of a prior study [[7]Page MJ Moher D Fidler FM Higgins JPT Brennan SE Haddaway NR The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of.Evidence Syst Rev. 2021; 10: 112Crossref PubMed Scopus (1) Google Scholar]. The term reproducibility appears to be used consistently in the context of systematic reviews, basically meaning the use of the same data, and analytic methods as the original study (synonymous with computational reproducibility) to try to reproduce the original results. However, the definition of the term replicability in the context of systematic reviews is challenging. “Addressing the same research question” and “re-doing the prior study” may be done with different methods, and in that case, it is difficult to expect true replication as the impact of different methods will hardly be assessable. The same research question can be addressed in various ways, introducing the notion of “replicability with a twist.” In the manuscript describing the consensus checklist for replicating systematic reviews of interventions, Tugwell et al. acknowledged that terminology, and conceptual framework for replication were not standardized. They defined two types of replication of systematic reviews of interventions as either direct or conceptual. Direct replication was defined as a “purposeful repetition to verify the findings of the original research question,“ and conceptual replication as, purposeful broadening or narrowing of the research question in existing systematic reviews (eg, across broader or more focused populations, intervention types, settings, outcomes, or study designs)” [[8]Tugwell P Welch VA Karunananthan S Maxwell LJ Akl EA Avey MT When to replicate systematic reviews of interventions: consensus checklist.BMJ. 2020; 370: m2864Crossref PubMed Scopus (13) Google Scholar]. True replication of a prior systematic review would replicate precisely the same methods and see whether the same results are achieved. Replication is, basically, a duplication [[9]Moher D. The problem of duplicate systematic reviews.BMJ. 2013; 347: f5040Crossref PubMed Scopus (43) Google Scholar]. That would be in line with the definition of “direct replication” [[8]Tugwell P Welch VA Karunananthan S Maxwell LJ Akl EA Avey MT When to replicate systematic reviews of interventions: consensus checklist.BMJ. 2020; 370: m2864Crossref PubMed Scopus (13) Google Scholar]. This should be feasible if the methods of a prior systematic review were fully and transparently reported, i.e., if a systematic review has the necessary attributes of “replicability” [[10]Bouter LM Riet GT. Replication research series-paper 2 : Empirical research must be replicated before its findings can be trusted.J Clin Epidemiol. 2021; 129: 188-190Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar]. However, when authors investigate the same research question with modified methods or modify the research question, it is not realistic to expect true replication. The question is whether the “conceptual replication” should be considered replication at all. In that case, it is realistic to expect to produce overlapping systematic reviews but not replicated/duplicate systematic reviews. When analyzing the literature across different disciplines, Vachon et al. adopted a broad perspective of replication, including the repetition of an index study, the extension of an index study, and the road testing of a theory [[4]Vachon B Curran JA Karunananthan S Brehaut J Graham ID Moher D Replication research series-paper 1 : a concept analysis and meta-narrative review established a comprehensive theoretical definition of replication research to improve its use.J Clin Epidemiol. 2021; 129: 176-187Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar]. However, just like suggested definitions of direct, and conceptual replication of systematic reviews [[8]Tugwell P Welch VA Karunananthan S Maxwell LJ Akl EA Avey MT When to replicate systematic reviews of interventions: consensus checklist.BMJ. 2020; 370: m2864Crossref PubMed Scopus (13) Google Scholar], this implies intentionally replicating a study. An important aspect in all these definition considerations regarding the systematic review replication is the aspect of intentionality. A group of authors may intend to replicate an existing systematic review and plan it a priori as either a direct or a conceptual replication. However, we believe that it can be safely assumed that the vast majority of overlapping systematic reviews are not intentionally replicating previous reviews. The authors may not be aware of existing systematic reviews because they may not have searched literature adequately or analyzed registries such as PROSPERO [[11]Rombey T Puljak L Allers K Ruano J Pieper D. Inconsistent views among systematic review authors toward publishing protocols as peer-reviewed articles: an international survey.J Clin Epidemiol. 2020; 123: 9-17Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar]. In that case, they are not doing a purposeful replication in any sense. Such “non-intentional replication” reviews will then be considered a replication simply because they have a very similar/same research question compared to the existing review(s). In all non-intentional replications of systematic reviews, this brings up the consideration of what we mean by the same/similar question. Even if the clinical question is really exactly the same, there can be differences arising from the PICO and in the next step from the eligibility criteria. Language restrictions, for example, could play a large role here [[12]Pieper D Puljak L. Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary.J Clin Epidemiol. 2021; 132: 146-147Abstract Full Text Full Text PDF PubMed Scopus (3) Google Scholar]. Reproducibility and replicability are expected to reduce research waste and increase the value of evidence [[13]Shokraneh F. Reducing waste and increasing value through embedded replicability and reproducibility in systematic review process and automation.J Clin Epidemiol. 2019; 112: 98-99Abstract Full Text Full Text PDF PubMed Google Scholar]. Although excessive duplication of systematic reviews [[9]Moher D. The problem of duplicate systematic reviews.BMJ. 2013; 347: f5040Crossref PubMed Scopus (43) Google Scholar] is certainly contributing to research waste, there may be legitimate reasons for conducting new, overlapping systematic reviews. But, then, legitimate overlapping systematic reviews should not necessarily be considered replicated, unnecessary, and contributing to research waste. Thus, we should tread carefully when defining replicability in systematic reviews. Due to the nature of systematic reviews and their increased production, the context of replicability in systematic reviews is different than in other studies. The definitions themselves could lead to further confusion and unnecessary labels of research waste. Considering the concepts of duplication, overlap and intentionality, we consider that there is an urgent need for reconsidering definitions, and the development of a coherent framework of replicability in the context of SRs. Livia Puljak: Conceptualization, Writing - original draft, Writing - review & editing, Dawid Pieper: Conceptualization, Writing - original draft, Writing - review & editing.