ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): a screening tool to assess the potential for real-world data sources to support creation of credible real-world evidence for regulatory decision-making

Abstract

Adoption and use of real-world data (RWD) for decision-making has been complicated by concerns regarding whether RWD was fit-for-purpose or was of sufficient validity to support the creation of credible RWE. This has greater urgency as regulatory agencies begin to use real world evidence (RWE) to inform decisions about treatment effectiveness. Researchers need an efficient and systematic method to screen the quality of RWD sources considered for use in studies of effectiveness and safety. Based on a literature review we developed a listing of screening criteria that have been previously proposed to assess the quality of RWD sources. We also developed an additional criterion based on Modern Validity Theory. While there has occurred some convergence of conceptual frameworks to assess data quality (DQ) and there is much agreement on specific assessment criteria, consensus has yet to emerge on how to assess whether a specific RWD source is reliable and fit-for-purpose. To create a user-friendly tool to assess whether RWD sources may have sufficient quality to support a well-designed RWE study for submission to a regulatory authority, we grouped the quality criteria with a view to harmonize published frameworks and to be consistent with how researchers generally evaluate existing RWD sources for research that they intend to submit to regulatory agencies. Screening data quality criteria were grouped into five dimensions after a comprehensive literature review via PubMed: authenticity, transparency, relevance, accuracy, and track record. The resultant tool was tested for its response burden using a hypothetical administrative claims data source. Providing responses to the screening criteria required only few hours effort by an experienced data source manager. Thus, the tool should not be an onerous burden on data source providers if asked by prospective researchers to provide the required information. Assessing whether a particular data source is fit-for-purpose will be facilitated by the use of this tool, but it will not be sufficient by itself. Fit-for-purpose judgements will still require further careful consideration based on the context and the specific scientific question of interest. Unlike prior DQ frameworks (DQF), the track record dimension of the tool adds the consideration of experience with RWD sources consistent with Modern Validity Theory. However, the tool does not address issues of study design and analysis that are critical to regulatory agencies in evaluating the robustness and credibility of the real-world evidence generated.