Evidence-based Practice Center Reports Research Articles

Supplementing systematic review findings with healthcare system data: pilot projects from the Agency for Healthcare Research and Quality Evidence-based Practice Center program

ObjectivesThe US Agency for Healthcare Research and Quality, through the Evidence-based Practice Center (EPC) Program, aims to provide health system decision makers with the highest-quality evidence to inform clinical decisions. However, limitations in the literature may lead to inconclusive findings in EPC systematic reviews (SRs). The EPC Program conducted pilot projects to understand the feasibility, benefits, and challenges of utilizing health system data to augment SR findings to support confidence in healthcare decision-making based on real-world experiences. Study Design and SettingThree contractors (each an EPC located at a different health system) selected a recently completed SR conducted by their center and identified an evidence gap that electronic health record (EHR) data might address. All pilot project topics addressed clinical questions as opposed to care delivery, care organization, or care disparities topics that are common in EPC reports. Topic areas addressed by each EPC included infantile epilepsy, migraine, and hip fracture. EPCs also tracked additional resources needed to conduct supplemental analyses. The workgroup met monthly in 2022-2023 to discuss challenges and lessons learned from the pilot projects. ResultsTwo supplemental data analyses filled an evidence gap identified in the SRs (raised certainty of evidence, improved applicability) and the third filled a health system knowledge gap. Project challenges fell under three themes: regulatory and logistical issues, data collection and analysis, and interpretation and presentation of findings. Limited ability to capture key clinical variables given inconsistent or missing data within the EHR was a major limitation. The workgroup found that conducting supplemental data analysis alongside an SR was feasible but adds considerable time and resources to the review process (estimated total hours to complete pilot projects ranged from 283 to 595 across EPCs), and that the increased effort and resources added limited incremental value. ConclusionSupplementing existing SRs with analyses of EHR data is resource intensive and requires specialized skillsets throughout the process. While using EHR data for research has immense potential to generate real-world evidence and fill knowledge gaps, these data may not yet be ready for routine use alongside SRs.

Updates to Recurrent Uncomplicated Urinary Tract Infections in Women: AUA/CUA/SUFU Guideline.

In 2019 the American Urological Association (AUA) released the evidence-based guideline "Recurrent Uncomplicated Urinary Tract Infections in Women: AUA/CUA/SUFU Guideline." Information supporting the guideline came from a 2019 systematic evidence review prepared for the AUA by the Pacific Northwest Evidence-based Practice Center (EPC). The AUA used evidence found for 11 Key Questions (Appendix C) in the EPC's report to derive 16 Guideline Statements. In 2021 the EPC conducted an Update Literature Review (ULR) assessing abstracts from new studies published since the 2019 systematic review. The AUA asked the EPC to further assess a subset of studies included in the ULR report, to support potential changes to the 2019 guideline. A systematic-review utilized research from the Oregon Health & Science University. Pacific Northwest EPC was used to update the 2019 AUA Guideline on rUTI in women with new evidence published through 2021. Updates were made to reflect changes in literature since 2019. Updates include recent publications on antibiotic prophylaxis, non-antibiotic prophylaxis, and estrogen therapy. The presence of rUTI is crucial to the health of patients and its effects must be considered for the welfare of society. This document will undergo updating as the knowledge regarding current treatments and future treatment options continues to expand. .

Stakeholder-engaged co-design and implementation of web-based tools to enhance the dissemination and implementation of AHRQ EPC reports.

A mission-critical aspect of learning health systems (LHSs) is the provision of evidence-based practice. One source of such evidence is provided by the Agency for Healthcare Research and Quality (AHRQ) through rigorous systematic reviews, termed evidence reports that synthesize available evidence on nominated topics of interest. However, the AHRQ Evidence-based Practice Center (EPC) program recognizes that the production of high-quality evidence reviews does not guarantee or promote their use and usability in practice. To make these reports more relevant to LHSs and promote evidence dissemination, AHRQ awarded a contract to the American Institutes for Research (AIR) and its Kaiser Permanente ACTION (KPNW ACTION) partner to design and implement web-based tools to meet the gap in dissemination and implementation of EPC reports in LHSs. We used a co-production approach to accomplish this work across three phases of activity: planning, co-design, and implementation between 2018 and 2021. We describe the methods and results and discuss implications for future efforts. Web-based information tools that provide clinically relevant summaries with clear visual representations from the AHRQ EPC systematic evidence reports may be used by LHSs to increase awareness and accessibility of EPC reports, formalize and enhance LHSs' evidence review infrastructure, develop system-specific protocols and care pathways, improve practice at the point of care, and train and educate. The co-design of these tools and facilitated implementation generated an approach to making EPC reports more accessible and allows for more widespread application of systematic review results in supporting evidence-based practices in LHSs.

AHRQ EPC Series on Improving Translation of Evidence: Use of a Clinical Pathway for C. Difficile Treatment to Facilitate the Translation of Research Findings into Practice

In 2018 the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program issued a call for strategies to disseminate AHRQ EPC systematic reviews. In this pilot, findings from the 2016 AHRQ EPC report on Clostridioides difficile infection were translated into a treatment pathway and disseminated via a cloud-based platform and electronic health record (EHR). An existing 10-step framework was used for developing and disseminating evidence-based clinical pathways. The development of the EHR intervention was informed by the Five Rights model for clinical decision support and human-computer interaction design heuristics. The researchers used observations and time measurements to describe the impact of the EPC report on pathway development and examined provider adoption using counts of pathway views. Two main themes emerged: (1) discrepancies between the EPC report and existing guidelines prompted critical discussions about available treatments, and (2) lack of guideline and pathway syntheses in the EPC report necessitated a rapid literature review. Pathway development required 340 hours: 205 for the rapid literature review, 63 for pathway development and EHR intervention design, and 5 for technical implementation of the intervention. Pathways were viewed 1,069 times through the cloud-based platform and 47 times through a hyperlink embedded in key EHR ordering screens. Pathways can be an approach for disseminating AHRQ EPC report findings within health care systems; however, reports should include guideline and pathway syntheses to meet their full potential. Embedding hyperlinks to pathway content within the EHR may be a viable and low-effort solution for promoting awareness of evidence-based resources.

AHRQ Series on Improving Translation of Evidence: Linking Evidence Reports and Performance Measures to Help Learning Health Systems Use New Information for Improvement

Evidence-based Practice Center (EPC) reports are lengthy and difficult for health systems to navigate. A quality measure index was created to allow health systems to more efficiently access information relevant to their needs. Two tables were embedded in an EPC report. The first identified quality measures covered by the report with descriptive information. The second contained page numbers in the report's executive summary addressing these quality measures, with hyperlinks to navigate to those pages. The researchers received feedback on the tables from four health system representatives and enhanced the tables. An exercise with two health system-targeted scenarios was then created. Three trainees (one medical fellow and two pharmacy students) and two health system representatives from a large community and a small rural health system completed the exercise, with and without the enhanced tables. They timed how long it took to find answers to scenario questions and provided general feedback. It took 63.4% less time to find quality measure information when the hyperlinked quality measure indexing tables were used (11.0 ± 5.0 vs. 4.0 ± 3.5 minutes; p = 0.002). The health system representatives stated that the quality measure indexed tables were very easy to use and that if these tables were used in future reports they were "somewhat" or "very likely" to use quality measure-indexed EPC reports in the future. A unique concept that can allow EPC reports to be more user friendly to health systems was identified. The refined quality measure-indexed tables enhanced the efficiency of finding information and the overall likability of the report.

Methods Guide for Authors of Systematic Reviews of Medical Tests: A Collaboration Between the Agency for Healthcare Research and Quality (AHRQ) and the Journal of General Internal Medicine

Methods Guide for Authors of Systematic Reviews of Medical Tests: A Collaboration Between the Agency for Healthcare Research and Quality (AHRQ) and the Journal of General Internal Medicine

Two methods provide similar signals for the need to update systematic reviews

ObjectiveApply and compare two methods that identify signals for the need to update systematic reviews, using three Evidence-based Practice Center reports on omega-3 fatty acids as test cases. Study Design and SettingWe applied the RAND method, which uses domain (subject matter) expert guidance, and a modified Ottawa method, which uses quantitative and qualitative signals. For both methods, we conducted focused electronic literature searches of recent studies using the key terms from the original reports. We assessed the agreement between the methods and qualitatively assessed the merits of each system. ResultsAgreement between the two methods was “substantial” or better (kappa>0.62) in three of the four systematic reviews. Overall agreement between the methods was “substantial” (kappa=0.64, 95% confidence interval [CI] 0.45–0.83). ConclusionThe RAND and modified Ottawa methods appear to provide similar signals for the possible need to update systematic reviews in this pilot study. Future evaluation with a broader range of clinical topics and eventual comparisons between signals to update reports and the results of full evidence review updates will be needed. We propose a hybrid approach combining the best features of both methods, which should allow efficient review and assessment of the need to update.

The Science of Continuing Medical Education: Terms, Tools, and Gaps

BackgroundBy its synthesis of a selected portion of the continuing medical education (CME) literature, the evidence-based practice center (EPC) review discovered several major issues in primary study design and in the systematic review process of CME studies. Through this process, the review speaks to the need for clarity in designing, reporting and synthesizing CME trials and provides an opportunity to advance the research agenda in this field.MethodsThe evidence-based guideline (EBG) committee reviewed the methods section of the EPC report and these guidelines in detail, commenting on the search and review process and on the nature of the primary literature and the definitions used within it, comparing these to other published standardized measures.ResultsAlthough the EBG committee noted much strength in the EPC review, limitations of the primary literature and the review methodology were identified and defined. These strengths and limitations hold implications for further research in this area.ConclusionsNoting these limitations and in order to move the field forward, the EBG committee proposes a standard nomenclature of terms in common use in CME; a more rigorous process of searching, distilling, and synthesizing the primary literature in this area; and a common format on which to base the development and description of future trials of CME interventions. By its synthesis of a selected portion of the continuing medical education (CME) literature, the evidence-based practice center (EPC) review discovered several major issues in primary study design and in the systematic review process of CME studies. Through this process, the review speaks to the need for clarity in designing, reporting and synthesizing CME trials and provides an opportunity to advance the research agenda in this field. The evidence-based guideline (EBG) committee reviewed the methods section of the EPC report and these guidelines in detail, commenting on the search and review process and on the nature of the primary literature and the definitions used within it, comparing these to other published standardized measures. Although the EBG committee noted much strength in the EPC review, limitations of the primary literature and the review methodology were identified and defined. These strengths and limitations hold implications for further research in this area. Noting these limitations and in order to move the field forward, the EBG committee proposes a standard nomenclature of terms in common use in CME; a more rigorous process of searching, distilling, and synthesizing the primary literature in this area; and a common format on which to base the development and description of future trials of CME interventions.

Evidence-Based Practice Center network and health technology assessment in the United States: Bridging the cultural gap

The purpose of this study was to identify the Evidence-Based Practice Center (EPC) network participants' perceptions of the characteristics of the EPC process and the relationship of the process to the success of EPC reports. Semistructured interviews were conducted with the three groups involved in the EPC: EPC staff, Agency for Healthcare Research and Quality (AHRQ) staff, and representatives of partner organizations. The analysis of the coded transcripts revealed three related major themes, which form the conceptual basis for the interpretation presented here: the definition of a successful report, the determinants of a successful report, and the role of AHRQ in the process. A successful report is a report that is used. The ultimate success of the core health technology assessment objective, moving from research to policy, depends on balancing two values: excellence and relevance. Our findings are consistent with the "two communities thesis," which postulates the existence of two camps that confer different values to excellence and relevance, with resulting tension. A promising model for approaching this tension is integration or collaboration, which requires linking researchers and policy makers, promoting productive dialogues about the formulation and timing of analysis, and early consideration of how the resulting analysis will be used. This effort suggests that actively blurring the frontiers between these two groups will enhance their interaction. Furthermore, enhancing the role of the AHRQ as scientific broker will maximize the potential of the EPC network.

Dissemination of Evidence-based Practice Center Reports

The Evidence-based Practice Center (EPC) program within the Agency for Healthcare Research and Quality (AHRQ) provides detailed evidence reports for partner organizations that they can translate into activities that improve patient care. A review of these dissemination activities provides a rich opportunity to understand how to create more successful linkages between best evidence and best practice. On the basis of interviews with EPC directors, AHRQ staff, and representatives of public and private users of EPC reports, we summarize the variety of efforts to disseminate the work of the EPCs. We also identify a case example of a successful dissemination of an EPC report. Experience to date reinforces the importance of creating close ties between researchers and the policymakers, clinicians, and other decision makers who use EPC evidence reports; developing a conceptual framework to guide the process; and establishing the resource foundation for the entire effort.

Challenges in Systematic Reviews of Diagnostic Technologies

Diagnostic tests are critical components of effective health care. They help determine treatments that are most beneficial for a given patient. Their assessment is a complex process that includes such challenges as a dearth of studies that evaluate clinical outcomes and lack of data on use of the test in realistic clinical settings. The methodologic quality of studies of diagnostic tests also lags behind the quality of studies of therapeutic interventions. Statistical methods to combine diagnostic accuracy data are more complex and not as well developed, leading to difficulties in the interpretation of results. The Agency for Healthcare Research and Quality Technology Assessment Program has adopted a 6-level framework for evaluating diagnostic technologies. The model emphasizes the need for systematic reviews of diagnostic test studies to go beyond the assessment of technical feasibility and accuracy to examine the impact of the test on health outcomes. In this paper, we use examples from 3 Evidence-based Practice Center reports to illustrate 3 challenges reviewers may face when reviewing diagnostic test literature: finding relevant studies, assessing methodologic quality of diagnostic accuracy studies, and synthesizing studies that evaluate tests in different patient populations or use different outcomes.

Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions

Randomized, controlled trials (RCTs) are firmly established as the standard for determining which medical treatments are effective. In some areas of health care, however, among them surgery, public health, and the organization of health care delivery, most evidence addressing the effectiveness of clinical or policy interventions rests on nonrandomized studies. We examine the use of study designs other than RCTs in Evidence-based Practice Center reports addressing questions of the effectiveness of treatment interventions. These reports offer the opportunity to examine the approaches used and the challenges faced by reviewers when nonrandomized studies are included and their quality assessed. We then offer recommendations for using these studies in systematic reviews of treatment interventions.

The Dental Curriculum

In 2018 the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program issued a call for strategies to disseminate AHRQ EPC systematic reviews. In this pilot, findings from the 2016 AHRQ EPC report on Clostridioides difficile infection were translated into a treatment pathway and disseminated via a cloud-based platform and electronic health record (EHR).An existing 10-step framework was used for developing and disseminating evidence-based clinical pathways. The development of the EHR intervention was informed by the Five Rights model for clinical decision support and human-computer interaction design heuristics. The researchers used observations and time measurements to describe the impact of the EPC report on pathway development and examined provider adoption using counts of pathway views.Two main themes emerged: (1) discrepancies between the EPC report and existing guidelines prompted critical discussions about available treatments, and (2) lack of guideline and pathway syntheses in the EPC report necessitated a rapid literature review. Pathway development required 340 hours: 205 for the rapid literature review, 63 for pathway development and EHR intervention design, and 5 for technical implementation of the intervention. Pathways were viewed 1,069 times through the cloud-based platform and 47 times through a hyperlink embedded in key EHR ordering screens.Pathways can be an approach for disseminating AHRQ EPC report findings within health care systems; however, reports should include guideline and pathway syntheses to meet their full potential. Embedding hyperlinks to pathway content within the EHR may be a viable and low-effort solution for promoting awareness of evidence-based resources.