Reporting transparency and completeness in trials: Paper 1: Introduction - Better reporting for disruptive clinical trials using routinely collected data

Abstract

In 2013, Lauer and D'Agostino wrote that “the randomized registry trial represents a disruptive technology, a technology that transforms existing standards, procedures, and cost structures" [[1]Lauer MS D'Agostino Sr, RB The randomized registry trial–the next disruptive technology in clinical research?.N Engl J Med. 2013; 369: 1579-1581Google Scholar]. Indeed, numerous examples since then have shown that randomized clinical trials using routinely collected data (RCD-RCTs) or data from cohorts can provide insights that were previously possible only at great expense, if at all [[2]Mc Cord KA Al-Shahi Salman R Treweek S et al.Routinely collected data for randomized trials: promises, barriers, and implications.Trials. 2018; 19: 29Google Scholar,[3]Collins R Bowman L Landray M Peto R. The magic of randomization versus the myth of real-world evidence.N Engl J Med. 2020; 382: 674-678Google Scholar]. Routinely collected data (RCD; e.g., from registries, electronic health records, or administrative databases) are data that are typically collected for purposes other than research via established data infrastructures [[4]Spasoff RA. Epidemiologic methods for health policy. Oxford University Press, 1999Google Scholar]. They can be obtained at a much lower cost than would be possible through investigator-initiated platforms, and as data are collected as part of routine care, they facilitate the conduct of studies outside of the somewhat artificial environment of the research setting [[2]Mc Cord KA Al-Shahi Salman R Treweek S et al.Routinely collected data for randomized trials: promises, barriers, and implications.Trials. 2018; 19: 29Google Scholar,[5]Mc Cord KA Ewald H Ladanie A et al.Current use and costs of electronic health records for clinical trial research: a descriptive study.CMAJ Open. 2019; 7: E23-E32Google Scholar]. However, it is not the increasing reflection of the reality of life in this data universe and the ubiquitous availability of such real-world data for research that is disruptive. It is also not the cost-efficiency. The disruption is that these advantages are combined with randomization, thus merging the most powerful method available to provide valid, unbiased assessment of causality with a robust approach to obtaining results directly applicable to real-world health care and decision making [[3]Collins R Bowman L Landray M Peto R. The magic of randomization versus the myth of real-world evidence.N Engl J Med. 2020; 382: 674-678Google Scholar]. However, bridging the previously often distinct and only partially intertwined worlds of traditional clinical trial research with epidemiology and data science presents both opportunities and challenges. For clinical trials, we have traditionally undergone painstaking planning to build a custom-built data framework. Data are collected prospectively for the specific purposes of a trial according to carefully defined procedures with elaborate, often expensive, and sometimes superfluous monitoring components. A high value is placed on data quality and the transparent documentation of the standards and procedures used to obtain data. These requirements also need to be met when using data obtained outside of the traditional clinical trial infrastructure in trials conducted using RCD. To verify internal and external validity, at a minimum, details are expected on the nature of these data in terms of their completeness and accuracy, and from whom, when and under what circumstances they were collected. Other elements that do not play a significant role in conventional clinical trials are often critical for trials conducted using RCD. One such issue, which has been well-documented in nonrandomized RCD studies, relates to data linkage issues [[6]Benchimol EI Smeeth L Guttmann A et al.The reporting of studies conducted using observational routinely-collected health data (RECORD) statement.PLoS Med. 2015; 12e1001885Google Scholar]. Many RCD-RCTs link datasets from different sources, for example hospital databases with insurance claims data. In traditional clinical trials, with data collected on an as-needed basis, patients have not been recruited based on whether data are or will be available for them. In epidemiological analyses, however (e.g., analyses of patients in cancer registries), study populations are often determined by the availability of RCD. These issues can result in unintended consequences for the design and evaluation of RCD-RCTs, as they open space for novel biases or applicability issues not typically considered in the context of trials [[2]Mc Cord KA Al-Shahi Salman R Treweek S et al.Routinely collected data for randomized trials: promises, barriers, and implications.Trials. 2018; 19: 29Google Scholar]. Trial participant populations and outcomes may be, at least partially, defined by the availability of data, and consideration must be given to the degree to which this may move us further away from our target population, ideal choice of outcomes and most importantly, the research question. The research agenda, though, should not be set by the availability of data items. Traditional clinical trials collect detailed information about trial participants and their outcomes, but not about those who do not participate. As a result, they often are not able to address the critical question of how well trial participants represent the wider target population. This is changing through the use of routine data. RCD can be used not only to describe the trial population and its outcomes, but also those who are not recruited or who choose not to participate. They can also be used to target and efficiently recruit study participants by identifying them in data sources, automate randomization, or directly deliver interventions. Investigators working with ongoing cohorts or RCD can precisely predict how many potential participants will meet trial eligibility criteria, removing a degree of uncertainty that has been a perennial challenge, and possibly avoiding research waste by exploring likely feasibility in advance. Finally, trials conducted using existing cohorts or RCD may be able to respond more rapidly and nimbly to critical research needs, evidenced during the COVID-19 pandemic, though they also introduce complexity. Some studies use hybrid approaches that mix traditional tailored data collection with the use of RCD [[7]Mc Cord KA Ewald H Agarwal A et al.Treatment effects in randomised trials using routinely collected data for outcome assessment versus traditional trials: meta-research study.BMJ. 2021; 372: n450Google Scholar]. A recent example is the RECOVERY Trial, which utilizes routinely collected administrative and healthcare data from the UK NHS system for short- to medium-term outcome assessment, combined with short-term custom-built data collection to evaluate treatments that may be beneficial for people hospitalized with suspected or confirmed COVID-19 [[8]RECOVERY definition and derivation of baseline characteristics and outcomes V3.0 2020-01-06. https://www.recoverytrial.net/files/recovery-outcomes-definitions-v3-0.pdf. Accessed June 07, 2021.Google Scholar]. New features and novel challenges in trials conducted with cohorts and RCD not only increase complexity, but they also require adaptation of existing standards and procedures [[9]Macnair A Love SB Murray ML et al.Accessing routinely collected health data to improve clinical trials: recent experience of access.Trials. 2021; 22: 340Google Scholar]. To ensure that trial results can be assessed, used and replicated, complete and transparent reporting is essential. With this goal in mind, the Consolidated Standards of Reporting Trials extension for the reporting of randomized controlled trials conducted using cohorts and routinely collected data (CONSORT-ROUTINE) was recently published [[10]Kwakkenbos L Imran M McCall SJ et al.CONSORT extension for the reporting of randomised controlled trials conducted using cohorts and routinely collected data (CONSORT-ROUTINE): checklist with explanation and elaboration.BMJ. 2021; 373: n857Google Scholar]. CONSORT-ROUTINE was informed by the CONSORT 2010 statement [[11]Schulz KF Altman DG Moher D CONSORT GroupCONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c332Google Scholar] and two reporting guidelines for observational studies, the Strengthening the Reporting of Observational Studies in Epidemiology statement [[12]von Elm E Altman DG Egger M et al.Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies.BMJ. 2007; 335: 806-808Google Scholar] and the Reporting of studies Conducted using Observational Routinely collected Data statement [[6]Benchimol EI Smeeth L Guttmann A et al.The reporting of studies conducted using observational routinely-collected health data (RECORD) statement.PLoS Med. 2015; 12e1001885Google Scholar]. CONSORT-ROUTINE includes eight modifications to items from the CONSORT 2010 statement and five new items. In this issue of the Journal of Clinical Epidemiology, the results of the first systematic assessments of the reporting completeness and transparency of these items are presented (Box 1) [13Mc Cord K Imran M Rice DB et al.Reporting of randomised trials using registries was often inadequate and hindered the interpretation of results.in: In Press (JCEPI-D-20-00367 Reporting Transparency and Completeness in Trials: Paper 2). 2021Google Scholar, 14Imran M Mc Cord K McCall S et al.Trials conducted using administrative databases do not adequately report elements related to use of databases.in: In Press (JCEPI-D-20-00345 Reporting Transparency and Completeness in Trials: Paper 3). 2021Google Scholar, 15McCall SJ Imran M Hemkens LG et al.Reporting of randomised controlled trials conducted using routinely collected electronic records – room for improvement.in: In Press (JCEPI-D-20-00346 Reporting Transparency and Completeness in Trials: Paper 4). 2021Google Scholar]. Separately for randomized trials using registries, administrative databases, or electronic health records, the reviews provide an initial benchmark for reporting of these types of trials.Box 1Series on Reporting Transparency and Completeness in Trials.•Reporting Transparency and Completeness in Trials: Paper 2 - Reporting of randomized trials using registries was often inadequate and hindered the interpretation of results•Reporting Transparency and Completeness in Trials: Paper 3 - Trials conducted using administrative databases do not adequately report elements related to use of databases•Reporting Transparency and Completeness in Trials: Paper 4: Reporting of randomized controlled trials conducted using routinely collected electronic records – room for improvement Open table in a new tab The first review in the series assessed reports of 47 trials that used a registry (23 of them to assess outcomes only, 10 for patient identification only, and 14 for both); most of the trials were conducted in Scandinavia or the United States. The second review assessed reports of 33 trials that used administrative databases (25 of them to assess outcomes only, one for patient identification only, and seven for both); most of the trials were conducted in the United States or Canada. The third review assessed reports of 183 trials that used electronic health records and were mostly conducted in the United States (122 used the electronic health records for identifying participants, 137 for outcome assessment, 139 for delivering the intervention, and 80 (44%) trials for all three functions). A random sample of 60 of the 183 reports was selected for the systematic assessment of the reporting completeness and transparency. Following harmonized procedures, the authors assessed, across the three reviews, a total of 140 trial reports, published between 2011 and 2018. The reporting was found to be inadequate for several critical details that are required for understanding, replication, bias assessment, interpretation and application of trial results. A specific concern was a lack of detail on the validation and completeness of data used for participant recruitment and outcome assessment. Here reporting information on the codes and algorithms used to define or derive outcomes and identify eligible participants, including methods used to assess accuracy and completeness, would ideally be provided. However, very few trials (between 0% and 6% in the three reviews) provided these critical details. Information on data linkage was infrequently provided (0–8%) as were RCD-related details on participant flow (3% to 10%), and funding of data sources (5–6%). The eligibility criteria for participants in the data source (cohort or RCD) was frequently not adequately reported (6–32%), while at least some studies, mostly those using electronic health records, discussed how the use of RCD may have influenced trial design, eligibility criteria, or data collection in ways that require consideration in interpreting results (21–52%). As CONSORT-ROUTINE was not available at the time of publication, it was not expected these items would be particularly well reported. Nonetheless, the results provide a benchmark and underline that there is much improvement needed to fully harness the full potential of this novel trial technology. Lars G. Hemkens: Conceptualization, Writing – original draft and Writing – review & editing. Edmund Juszczak, Brett D. Thombs: Conceptualization, Writing – review & editing. Lars G. Hemkens, Edmund Juszczak, and Brett D. Thombs are the guarantors. All authors attest they meet the ICMJE criteria for authorship and that no others meeting the criteria have been omitted.