Supplementing systematic review findings with healthcare system data: pilot projects from the Agency for Healthcare Research and Quality Evidence-based Practice Center program

Objectives. The Agency for Healthcare Research and Quality (AHRQ), through the Evidence-based Practice Center (EPC) Program, aims to provide health system decision makers with the highest-quality evidence to inform clinical decisions. However, limitations in the literature may lead to inconclusive findings in EPC systematic reviews (SRs). The EPC Program conducted pilot projects to understand the feasibility, benefits, and challenges of utilizing health system data to augment SR findings to support confidence in healthcare decision making based on real-world experiences. Study Design and Setting. Three contractors (each an EPC located at a different health system) selected a recently completed systematic review conducted by their center and identified an evidence gap that electronic health record (EHR) data might address. All pilot project topics addressed clinical questions: infantile epilepsy, migraine, and hip fracture, respectively. EPCs also tracked additional resources needed to conduct supplemental analyses. The workgroup met monthly from 2022 to 2023 to discuss challenges and lessons learned from the pilot projects. Results. Two supplemental data analyses filled an evidence gap identified in the systematic reviews (raised certainty of evidence, improved applicability), and the third filled a health system knowledge gap. Project challenges fell under three themes: regulatory and logistical issues, data collection and analysis, and interpretation and presentation of findings. Limited ability to capture key clinical variables given inconsistent or missing data within the EHR was a major limitation. Conducting supplemental data analysis alongside an SR added considerable time and resources to the review process (estimated total hours to complete pilot projects ranged from 283 to 595 across EPCs), and the increased effort and resources added limited incremental value. Conclusion. Supplementing existing systematic reviews with analyses of EHR data is feasible, but resource intensive, and requires specialized skillsets throughout the process. While using EHR data for research has immense potential to generate real-world evidence and fill knowledge gaps, these data may not yet be ready for routine use alongside systematic reviews.

Objectives. The Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program wants learning health systems (LHSs) to use the evidence from its reports to improve patient care. In 2018, to improve uptake of EPC Program findings, the EPC Program developed a project to enhance LHSs’ adoption of evidence to improve the quality and effectiveness of patient care. AHRQ contracted with the American Institutes for Research (AIR) and its partners to convene a panel of senior leaders from 11 LHSs to guide the development of tools to help health systems use findings from EPC evidence reports. The panel’s contributions led to developing, implementing, and evaluating two electronic tools to make the EPC report findings more accessible. AIR evaluated the LHSs’ use of the tools to understand (1) LHSs’ experiences with and impressions of the tools, (2) how well the tools helped them access evidence, and (3) how well the tools addressed barriers to LHS use of the EPC reports and barriers to applying the evidence from the reports. Data sources. (1) Implementation meetings with 6 LHSs; (2) interviews with 27 health system leaders and clinical staff who used the tools; and (3) website utilization metrics. Results. The tools were efficient and useful sources of summarized evidence to (1) inform systems change, (2) educate trainees and clinicians, (3) inform research, and (4) support shared decision making with patients and families. Clinical leaders appreciated the thoroughness and quality of the evidence reviews and view AHRQ as a trusted source of information. Participants found both tools to be valuable and complementary. Participants suggested optimizing the content for mobile device use to facilitate health system uptake of the tools. In addition, they felt it would be helpful to have training resources about tool navigation and interpreting the statistical content in the tools. Conclusions. The evaluation shows that LHSs find the tools to be useful resources for making the EPC Program reports more accessible to health system leaders. The tools have the potential to meet some, but not all, LHS evidence needs, while exposing health system leaders to AHRQ as a resource to help meet their information needs. The ability of the EPC reports to support LHSs in improving the quality of care is limited by the strength and robustness of the evidence, as well as the relevance of the report topics to patient care challenges faced by LHSs.

Introduction:The US Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program sponsors the development of systematic reviews to inform clinical policy and practice. The EPC program sought to better understand how health systems identify and use this evidence.Methods:Representatives from eleven EPCs, the EPC Scientific Resource Center, and AHRQ developed a semi-structured interview script to query a diverse group of nine Key Informants (KIs) involved in health system quality, safety and process improvement about how they identify and use evidence. Interviews were transcribed and qualitatively summarized into key themes.Results:All KIs reported that their organizations have either centralized quality, safety, and process improvement functions within their system, or they have partnerships with other organizations to conduct this work. There was variation in how evidence was identified, with larger health systems having medical librarians and central bureaus to gather and disseminate information and smaller systems having local chief medical officers or individual clinicians do this work. KIs generally prefer guidelines, especially those with treatment algorithms, because they are actionable. They like systematic reviews because they efficiently condense study results and reconcile conflicting data. They prefer information from systematic reviews to be presented as short digestible summaries with the full report available on demand. KIs preferred systematic reviews from reputable entities and those without commercial bias. Some of the challenges KIs reported include how to resolve conflicting evidence, the generalizability of evidence to local needs, determining whether the evidence is up-to-date, and the length of time required to generate reviews. The topics of greatest interest included predictive analytics, high-value care, advance care planning, and care coordination. To increase awareness of AHRQ EPC reviews, KIs suggest alerting people at multiple levels in a health-system when new evidence reports are available and making reports easier to find in common search engines.Conclusions:Systematic reviews are valued by health system leaders. To be most useful they should be easy to locate and available in different formats targeted to the needs of different audiences.

AHRQ EPC Series on Improving Translation of Evidence: Use of a Clinical Pathway for C. Difficile Treatment to Facilitate the Translation of Research Findings into Practice

Background: Healthcare decision makers strive to operate on the best available evidence. The Agency for Healthcare Research and Quality Evidence-based Practice Center (EPC) Program aims to support healthcare decision makers by producing evidence reviews that rate the strength of evidence. However, the evidence base is often sparse or heterogeneous, or otherwise results in a high degree of uncertainty and insufficient evidence ratings. Objective: To identify and suggest strategies to make insufficient ratings in systematic reviews more actionable. Methods: A workgroup comprising EPC Program members convened throughout 2020. We conducted interative discussions considering information from three data sources: a literature review for relevant publications and frameworks, a review of a convenience sample of past systematic reviews conducted by the EPCs, and an audit of methods used in past EPC technical briefs. Results: Several themes emerged across the literature review, review of systematic reviews, and review of technical brief methods. In the purposive sample of 43 systematic reviews, the use of the term “insufficient” covered both instances of no evidence and instances of evidence being present but insufficient to estimate an effect. The results of the literature review and review of the EPC Program systematic reviews illustrated the importance of clearly stating the reasons for insufficient evidence. Results of both the literature review and review of systematic reviews highlighted the factors decision makers consider when making decisions when evidence of benefits or harms is insufficient, such as costs, values, preferences, and equity. We identified five strategies for supplementing systematic review findings when evidence on benefit or harms is expected to be or found to be insufficient, including: reconsidering eligible study designs, summarizing indirect evidence, summarizing contextual and implementation evidence, modelling, and incorporating unpublished health system data. Conclusion: Throughout early scoping, protocol development, review conduct, and review presentation, authors should consider five possible strategies to supplement potential insufficient findings of benefit or harms. When there is no evidence available for a specific outcome, reviewers should use a statement such as “no studies” instead of “insufficient.” The main reasons for insufficient evidence rating should be explicitly described.

Methods Guide for Authors of Systematic Reviews of Medical Tests: A Collaboration Between the Agency for Healthcare Research and Quality (AHRQ) and the Journal of General Internal Medicine

The Dental Curriculum

Estimation of heterogeneous treatment effects is an essential component of precision medicine. Model and algorithm-based methods have been developed within the causal inference framework to achieve valid estimation and inference. Existing methods such as the A-learner, R-learner, modified covariates method (with and without efficiency augmentation), inverse propensity score weighting, and augmented inverse propensity score weighting have been proposed mostly under the square error loss function. The performance of these methods in the presence of data irregularity and high dimensionality, such as that encountered in electronic health record (EHR) data analysis, has been less studied. In this research, we describe a general formulation that unifies many of the existing learners through a common score function. The new formulation allows the incorporation of least absolute deviation (LAD) regression and dimension reduction techniques to counter the challenges in EHR data analysis. We show that under a set of mild regularity conditions, the resultant estimator has an asymptotic normal distribution. Within this framework, we proposed two specific estimators for EHR analysis based on weighted LAD with penalties for sparsity and smoothness simultaneously. Our simulation studies show that the proposed methods are more robust to outliers under various circumstances. We use these methods to assess the blood pressure-lowering effects of two commonly used antihypertensive therapies.

Improving Access to and Usability of Systematic Review Data for Health Systems Guidelines Development

AHRQ Series on Improving Translation of Evidence: Linking Evidence Reports and Performance Measures to Help Learning Health Systems Use New Information for Improvement

Optimizing Care Transitions: Adapting Evidence-Informed Solutions to Local Contexts

Electronic health record (EHR) data contain longitudinal patient information and standardized diagnostic codes. EHR data may be useful for estimating transition probabilities for state-transition models, but no guidelines exist on appropriate methods. We applied 3 potential methods to estimate transition probabilities from EHR data, using pediatric eating disorders (EDs) as a case study. We obtained EHR data from PEDsnet, which includes 8 US children's hospitals. Data included inpatient, outpatient, and emergency department visits for all patients with an ED. We mapped diagnoses to 3 ED health states: anorexia nervosa, bulimia nervosa, and other specified feeding or eating disorder. We estimated 1-y transition probabilities for males and females using 3 approaches: simple first-last proportions, a multistate Markov (MSM) model, and independent survival models. Transition probability estimates varied widely between approaches. The first-last proportion approach estimated higher probabilities of remaining in the same health state, while the MSM and independent survival approaches estimated higher probabilities of transitioning to a different health state. All estimates differed substantially from published literature. As a source of health state information, EHR data are incomplete and sometimes inaccurate. EHR data were especially challenging for EDs, limiting the estimation and interpretation of transition probabilities. The 3 approaches produced very different transition probability estimates. Estimates varied considerably from published literature and were rescaled and calibrated for use in a microsimulation model. Estimation of transition probabilities from EHR data may be more promising for diseases that are well documented in the EHR. Furthermore, clinicians and health systems should work to improve documentation of ED in the EHR. Further research is needed on methods for using EHR data to inform transition probabilities.

Objective The aim of the study is to identify how academic health centers (AHCs) have established infrastructures to leverage electronic health record (EHR) data to support research and quality improvement (QI). Methods Phone interviews of 18 clinical informaticians with expertise gained over three decades at 24 AHCs were transcribed for qualitative analysis on three levels. In Level I, investigators independently used NVivo software to code and identify themes expressed in the transcripts. In Level II, investigators reexamined coded transcripts and notes and contextualized themes in the learning health system paradigm. In Level III, an informant subsample validated and supplemented findings. Results Level I analysis yielded six key “determinants”—Institutional Relationships, Resource Availability, Data Strategy, Response to Change, Leadership Support, and Degree of Mission Alignment—which, according to local context, affect use of EHR data for research and QI. Level II analysis contextualized these determinants in a practical frame of reference, yielding a model of learning health system maturation, over-arching key concepts, and self-assessment questions to guide AHC progress toward becoming a learning health system. Level III informants validated and supplemented findings. Discussion Drawn from the collective knowledge of experienced informatics professionals, the findings and tools described offer practical support to help clinical informaticians leverage EHR data for research and QI in AHCs. Conclusion The learning health system model builds on the tripartite AHC mission of research, education, and patient care. AHCs must deliberately transform into learning health systems to capitalize fully on EHR data as a staple of health learning.

Objective: To evaluate the continuity and completeness of electronic health record (EHR) data, and the concordance of select clinical outcomes and baseline comorbidities between EHR and linked claims data, from three healthcare delivery systems in Taiwan. Methods: We identified oral hypoglycemic agent (OHA) users from the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), which was linked to the National Health Insurance Research Database (NHIRD), from June 2011 to December 2016. A secondary evaluation involved two additional EHR databases. We created consecutive 90-day periods before and after the first recorded OHA prescription and defined patients as having continuous EHR data if there was at least one encounter or prescription in a 90-day interval. EHR data completeness was measured by dividing the number of encounters in the NTUH-iMD by the number of encounters in the NHIRD. We assessed the concordance between EHR and claims data on three clinical outcomes (cardiovascular events, nephropathy-related events, and heart failure admission). We used individual comorbidities that comprised the Charlson comorbidity index to examine the concordance of select baseline comorbidities between EHRs and claims. Results: We identified 39,268 OHA users in the NTUH-iMD. Thirty-one percent (n = 12,296) of these users contributed to the analysis that examined data continuity during the 6-month baseline and 24-month follow-up period; 31% (n = 3,845) of the 12,296 users had continuous data during this 30-month period and EHR data completeness was 52%. The concordance of major cardiovascular events, nephropathy-related events, and heart failure admission was moderate, with the NTU-iMD capturing 49–55% of the outcome events recorded in the NHIRD. The concordance of comorbidities was considerably different between the NTUH-iMD and NHIRD, with an absolute standardized difference >0.1 for most comorbidities examined. Across the three EHR databases studied, 29–55% of the OHA users had continuous records during the 6-month baseline and 24-month follow-up period. Conclusion: EHR data continuity and data completeness may be suboptimal. A thorough evaluation of data continuity and completeness is recommended before conducting clinical and translational research using EHR data in Taiwan.

6514 Background: Clinical RWD derived from EHRs is becoming increasingly important for clinical research, trial design, regulatory decisions etc. These applications require identification of lines of therapy (LoT) which are typically not captured in EHR and must be abstracted from other clinical and medication data. EHR data has significant missingness which can be complemented with other data sources such as medical claims data. In this study, we demonstrate how our proprietary line of therapy algorithms for solid cancers show significant improvements when built using integrated EHR and claims data when compared to EHR data alone. Methods: For this analysis, ConcertAI’s RWD360 dataset integrated with a large administrative open-claims dataset (>90% overlap) for 14 solid cancer indications (Breast, Bladder, Lung, Prostate, Pancreas, Melanoma, Liver, Head & Neck, Renal, Colorectal, Melanoma, Ovarian, Thyroid, Endometrial) was used. The date of advanced/metastatic diagnosis used as the index date for LoTs was derived from the EHR data and medications from both EHR and claims data were used. We ran our LoT algorithms on EHR data with and without claims data and evaluated the impact of integrating claims data on the quantity and quality of LoT output. Results: The inclusion of medication data from claims significantly increased (7-22%) the number of patients for which LoTs could be extracted from the EHR data. Furthermore, we observed increases in number of lines per patient, length of lines and medications per line across cohorts. The distance between index date and 1st line start date was shortened in a subset (2-12%) of patients as a result. In a small fraction of cases, we even observed removal of false lines as some of the lines moved to adjuvant/neoadjuvant setting by filling in missing medication from claims. Overall, 7-39% patients in the LoT cohorts were impacted by addition of claims. Results for a few cancer types are presented in Table 1. We also compared the top LoTs derived from the integrated dataset against the standard of care for that cancer and observed very good concordance. Conclusions: Deriving LoTs by integrating data from multiple data sources such as EHR and claims can significantly improve its accuracy. [Table: see text]