To replicateor not to replicate? Insights and interpretations from a randomized trial duplication initiative.

Abstract

Journal of Comparative Effectiveness ResearchVol. 10, No. 13 EditorialOpen AccessTo replicate or not to replicate? Insights and interpretations from a randomized trial duplication initiativeMelvin (Skip) Olson, Kristijan H Kahler & Amy E RudolphMelvin (Skip) Olson *Author for correspondence: E-mail Address: melvin.olson@novartis.comhttps://orcid.org/0000-0002-2870-4443Global Medical Affairs, Novartis Pharma AG, Basel, SwitzerlandSearch for more papers by this author, Kristijan H Kahler https://orcid.org/0000-0002-8227-8879Novartis Pharmaceuticals Corporation, East Hanover, NJ 07936-1080, USASearch for more papers by this author & Amy E Rudolph https://orcid.org/0000-0002-9141-246XNovartis Pharmaceuticals Corporation, East Hanover, NJ 07936-1080, USASearch for more papers by this authorPublished Online:22 Jun 2021https://doi.org/10.2217/cer-2021-0137AboutSectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinkedInRedditEmail Keywords: alectinibcausal inferencedata qualityduplicatepalbociclibrandomized controlled trialRCT DUPLICATEreal-world datareal-world evidenceregulatoryreplicationRandomized controlled trials (RCTs) are considered a ‘gold standard’ for evidence generation for several reasons, including their ability to provide causal inferences. However, real-world data (RWD) are increasingly used to complement RCTs for several purposes. These include defining and monitoring the disease landscape, drug development decision-making, informing clinical study design, improving trial efficiency, assessing drug outcomes and supporting regulatory decisions [1–3].All uses of real-world evidence (RWE) require analytical rigor, but they do not need to replicate or replace RCTs. The value of RWE should be recognized distinctly from the purposes and research questions of RCTs; indeed, RWE can both complement RCTs and be used to make causal inferences (Figure 1). However, many decision-makers have voiced concerns about RWE for several reasons, including the observational nature of the data and the lack of randomization, which precludes protection from bias [4]. Nevertheless, the US FDA is focusing more on the use of RWE to complement and inform RCTs [5], following the introduction of the 21st Century Cures Act in 2016 [6], while the EMA is promoting the use of RWE for decision-making while remaining vigilant on testing and validation [7,8].Figure 1. The real-world evidence ‘ecosystem’.RWD: Real-world data; RCT: Randomized controlled trial.Among several initiatives encouraging the use of regulatory-grade RWE is RCT DUPLICATE, which aims to demonstrate that RWE analyses can emulate RCT findings in certain situations. The program will provide reference examples of causal linkage to increase confidence in the use of RWE for decision-making, particularly in the absence of RCT data. This article discusses the work of RCT DUPLICATE, and how the findings should be interpreted to guide the use of RWE for reimbursement and regulatory decision-making and support the implementation of products in clinical practice.Evolution of RWE to complement RCTsRWE can play several vital roles throughout the product life cycle (Figure 1). The use of RWE to complement RCT data for regulatory and reimbursement decision-making is increasing because the findings can be expanded to different patient populations, treatment patterns and stakeholder-relevant end points and valuable insights can be obtained. Examples of RWE supporting regulatory decision-making include the approval of palbociclib in combination with endocrine therapy for the treatment of male breast cancer [9] and the inclusion of RWE in the label for paliperidone palmitate for the treatment of schizophrenia [10]. RWE can also provide the comparative context and value demonstration required for reimbursement decisions. External comparators can provide treatment alternatives that are not included as control in RCTs, potentially accelerating access. A recent example is the use of RWD from electronic health records combined with clinical trial data to support the reimbursement of alectinib for lung cancer [11].The emulation of RCTs using RWD may provide insight into the opportunities and limitations of employing observational studies for regulatory decision-making. Comparison of results between RCTs and rigorously designed RWD studies may increase confidence that the latter can provide reliable evidence on drug effect and, in some situations, even identify causal inferences.The ‘RCT DUPLICATE’ project: what it is & what it is notThe RCT DUPLICATE project started in late 2017 as a collaboration between the FDA, Brigham and Women’s Hospital and the Division of Pharmacoepidemiology at Harvard Medical School. It aims to replicate 30 completed RCTs using a variety of RWD study designs, analyze a large sample of cardiovascular RCTs and predict the results of seven ongoing Phase IV trials [12,13]. This empirical evidence base will provide a reference standard for researchers and increase regulator confidence in the use of observational studies to support their decision-making [14].Initial findings of RCT DUPLICATE have been published [15]. Ten RWD studies were designed to emulate RCTs for cardiovascular outcomes of antidiabetic or antiplatelet medication. The difference in results between RWD and the RCTs was not statistically significant in nine of the ten replications. Eight of the ten RWE findings achieved ‘estimate agreement’ and six of the ten RWE studies reached ‘regulatory agreement’ with the RCTs. One of the primary limitations of RWE was demonstrated – replication of placebo-controlled trials may be challenging, owing to the variance in care across the ‘real world’ and the intrinsic differences between patients receiving medications in clinical practice and those enrolled in RCTs. Three of the four RWD studies that did not reach regulatory agreement were placebo-controlled.The main clinical implication from these findings is that causal treatment effects can be estimated through both randomized trials and nonrandomized RWE studies and that the variations observed can be attributed to emulation differences. Although insurance claims data with suitable study design and analysis were appropriate to estimate treatment effect by emulating those studies, other RCTs may require alternative data sources.If the RCT DUPLICATE initiative demonstrates that nonrandomized studies can match the findings of published RCTs, and predict the results of ongoing trials, this could indicate that RWE can offer causal insights into treatment effect. Building an evidence base of high-quality case studies that demonstrate scenarios in which RWE studies can lead to causal inference may increase confidence in RWE for regulatory decision-making. However, the replication of RCTs is neither a mandated requirement nor a prerequisite for the use of RWE. For example, one interesting use of causal inference with RWE is the treatment comparative effectiveness analysis performed for payer decision-making in the post-launch phase. This is not an RCT replication, but some of the lessons from RCT DUPLICATE could be applied. Additionally, there are uses of RWE that do not rely on causal inference, such as the evaluation of burden of illness, analysis of treatment patterns and trends and the study of medication adherence, pricing strategy and quality of care.Nonetheless, questions may arise regarding why some observational studies fail to replicate the effect of RCTs and when there will be sufficient empirical evidence to predict the validity of observational studies of treatment effectiveness with high certainty.What if observational studies cannot replicate randomized trials?The agreement between RWD studies and RCTs can be interpreted as support of causal inference but the absence of concurrence often generates uncertainty and skepticism. However, disagreement between the results of RWD studies and RCTs does not necessarily invalidate the former and their use for regulatory purposes. This is because an intrinsic ‘efficacy–effectiveness gap’ exists between RCTs and real-world clinical practice. Questioning the value of RWE for failing to align with RCTs has been suggested as potentially harmful to the acceptance of RWE for decision-making purposes [16].RWE can be used to complement the findings of RCTs to allow translation of new technologies into ‘real-world’ practice. However, we suggest that there are unexplored uses of RWE. An initial goal may be to expedite the time to treatment initiation, although complexities of the healthcare system and patient preferences need to be considered. A second aim may be to reduce the 17-year lag from research evidence to clinical adoption [17].To accelerate implementation of medications in clinical practice and subsequently improve patients’ lives, it is important to understand that RWE is just one type of evidence required to address the challenges of the healthcare ecosystem and that integrated and diverse evidence is required to complement RCTs, particularly for regulatory decision-making.ConclusionThe use of RWD to support the regulatory approval of medicinal products is a relatively new concept and uncertainty is still present. RCT DUPLICATE will provide clarity on the use of RWE to support regulatory decision-making. However, the ultimate outcome of these RWE replication exercises is not to encourage regulators to request replication of RCTs because these represent the ‘gold standard’ of causally linked evidence. Rather, the goal is to develop a universal resource of case studies demonstrating causal linkage with RWE. This will increase the validity of RWE for decision-making, particularly when RCTs are unavailable or not applicable for the question at hand.The diversity of RWE use demands that the validity lies in the quality and use of data that are fit for each specific purpose, rather than requiring causal linkage. However, a growing portfolio of successful RCT emulations may provide a foundation for the development of similarly rigorous RWE studies, further increasing confidence in this type of evidence.Author contributionsAll authors have fulfilled the authorship criteria and accept accountability for all content. Each author has significantly contributed to, reviewed and approved the editorial.AcknowledgmentsThe authors are grateful for the valuable input of S Schneeweiss of the Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, MA, USA.Financial & competing interests disclosureDevelopment of this editorial was funded by Novartis Pharma AG and Novartis Pharmaceuticals Corporation. M Olson is an employee of Novartis Pharma AG. KH Kahler and AE Rudolph are employees of Novartis Pharmaceuticals Corporation. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.Medical writing support was provided by Lorenzo Dall'Ava, PhD, of PharmaGenesis London, London, UK, with funding from Novartis Pharma AG.Open accessThis work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/Papers of special note have been highlighted as: • of interest