Evidence-based Practice Center Program Research Articles

Supplementing systematic review findings with healthcare system data: pilot projects from the Agency for Healthcare Research and Quality Evidence-based Practice Center program

ObjectivesThe US Agency for Healthcare Research and Quality, through the Evidence-based Practice Center (EPC) Program, aims to provide health system decision makers with the highest-quality evidence to inform clinical decisions. However, limitations in the literature may lead to inconclusive findings in EPC systematic reviews (SRs). The EPC Program conducted pilot projects to understand the feasibility, benefits, and challenges of utilizing health system data to augment SR findings to support confidence in healthcare decision-making based on real-world experiences. Study Design and SettingThree contractors (each an EPC located at a different health system) selected a recently completed SR conducted by their center and identified an evidence gap that electronic health record (EHR) data might address. All pilot project topics addressed clinical questions as opposed to care delivery, care organization, or care disparities topics that are common in EPC reports. Topic areas addressed by each EPC included infantile epilepsy, migraine, and hip fracture. EPCs also tracked additional resources needed to conduct supplemental analyses. The workgroup met monthly in 2022-2023 to discuss challenges and lessons learned from the pilot projects. ResultsTwo supplemental data analyses filled an evidence gap identified in the SRs (raised certainty of evidence, improved applicability) and the third filled a health system knowledge gap. Project challenges fell under three themes: regulatory and logistical issues, data collection and analysis, and interpretation and presentation of findings. Limited ability to capture key clinical variables given inconsistent or missing data within the EHR was a major limitation. The workgroup found that conducting supplemental data analysis alongside an SR was feasible but adds considerable time and resources to the review process (estimated total hours to complete pilot projects ranged from 283 to 595 across EPCs), and that the increased effort and resources added limited incremental value. ConclusionSupplementing existing SRs with analyses of EHR data is resource intensive and requires specialized skillsets throughout the process. While using EHR data for research has immense potential to generate real-world evidence and fill knowledge gaps, these data may not yet be ready for routine use alongside SRs.

The effectiveness of opioid stewardship interventions in healthcare: A Making Healthcare Safer rapid review

Objective Opioid stewardship interventions promote appropriate use of opioids. We synthesized evidence from recent studies on the effectiveness of opioid stewardship interventions in United States healthcare settings on opioid prescribing and clinical outcomes. Methods We followed the Evidence-based Practice Center Program's rapid review processes. We searched PubMed and Cochrane for original studies meeting specified inclusion criteria from January 2016 (correlating with the Centers for Disease Control and Prevention Pain Guidelines) to April 2023, supplemented by gray literature searches. Findings Our search identified 13 randomized controlled trials and 6 nonrandomized studies. Randomized controlled trials addressed interventions involving multiple components (typically combining prescriber education, care management, and facilitated access to resources), clinical decision support (CDS) or electronic health records (EHR), and patient education and engagement, mainly in ambulatory chronic pain. Multicomponent interventions and opioid stewardship practices involving CDS or EHR were associated with decreased opioid prescribing or reduced doses and no increases in pain, emergency department visits, or hospitalizations (low strength of evidence for all outcomes). Patient engagement and education interventions had mixed results for opioid prescribing outcomes (insufficient strength of evidence) and no increases in pain, emergency department visits, or hospitalizations (low strength of evidence). Conclusions Selected opioid stewardship interventions may be effective for reducing opioid prescribing without adversely affecting clinical outcomes. Interventions to reduce opioid prescribing should monitor unintended consequences and include access to nonpharmacological pain management resources with patient education and engagement.

Qualitative evaluation of two web-based tools to improve accessibility of evidence reports.

The Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program produces evidence reports to assist health systems in improving patient care. To make EPC evidence reports more accessible and usable, AHRQ convened a panel of learning health system (LHS) senior leaders to develop two web-based tools-a table-based tool presenting high-level results, and a graphical display that presented detailed data-to disseminate the reports and test the tools. Researchers examined (1) the context under which interview participants reviewed and used the evidence reported by the tools, (2) their experiences using the tools, (3) the tools' influence on clinical practice, and (4) how the tools could be improved. Researchers collected and inductively analyzed qualitative data from tool implementation meetings with six LHSs and interviews with 27 LHS leaders and clinical staff who used the tools. Researchers used website utilization metrics to augment qualitative results. The tools were efficient, complementary, and useful sources of summarized evidence to promote system change, educate trainees and clinicians, inform research, and support shared decision making with patients and families. Clinical leaders appreciated the evidence review thoroughness and quality and viewed AHRQ as a trusted source of information. However, many felt the tools were not practical for bedside use because of their complex content. Participants also noted the reports had limited evidence strength and robustness. They suggested optimizing the tools for mobile device use to facilitate tool uptake and developing training resources about tool navigation and statistical content interpretation. LHSs found the tools to be useful resources for making the EPC Program reports more accessible to and usable for health system leaders. The tools have the potential to meet some, but not all, LHS evidence needs. Their value depends on reports' usefulness, which ultimately depends on the evidence quality.

Reproductive Health in First-degree Relatives of Patients With Polycystic Ovary Syndrome: A Review and Meta-analysis

Polycystic ovary syndrome (PCOS) is a highly heritable disease. Emerging evidence elucidated the elevated prevalence of reproductive abnormalities in first-degree relatives (FDRs) of patients with PCOS. To explore the reproductive health in FDRs of patients with PCOS. Ten databases were searched in December 2020 (PubMed, Embase, Web of Science, Cochrane Central Register of Controlled Trials, ClinicalTrials.gov, Chinese Biological Medical Literature, Chinese National Knowledge Infrastructure, Chinese Journals Full-text Database, WanFang, and World Health Organization international clinical trials registry platform). This study included cohort, case-control, or cross-sectional studies. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement was followed. Dichotomous data from each of the eligible studies were combined by the Mantel-Haenszel model. Standard mean differences with 95% CIs were assessed. Heterogeneities were assessed using I2 statistics, and the quality of evidence was evaluated by a US Agency for Healthcare Research and Quality Evidence-based Practice Center program and Grading of Recommendations, Assessment, Development and Evaluation approach. Thirty-eight studies were included. The prevalence of PCOS (0.22; 95% CI, 0.16 to 0.29), menstrual irregularities (0.28; 95% CI, 0.22 to 0.34, P < .01), and ovary morphological changes were elevated in female PCOS FDRs. Female FDRs also presented with increased levels of luteinizing hormone, total testosterone (standard mean difference, 0.53; 95% CI, 0.28 to 0.78, P < .01), unconjugated testosterone, free androgen index, dehydroepiandrosterone sulfate (DHEAS), and antimüllerian hormone levels. Subgroup analyses indicated that some of these changes begun in pubertal girls. Furthermore, fathers of PCOS patients had a higher risk of premature baldness. The DHEAS level was elevated in male FDRs. The findings of this analysis suggested that FDRs of patients with PCOS suffered from reproductive endocrinological dysregulations. Thus, more attention should be focused on this population.

Making Narrative Statements to Describe Treatment Effects.

Accurately describing treatment effects using plain language and narrative statements is a critical step in communicating research findings to end users. However, the process of developing these narratives has not been historically guided by a specific framework. The Agency for Healthcare Research and Quality Evidence-based Practice Center Program developed guidance for narrative summaries of treatment effects that identifies five constructs. We explicitly identify these constructs to facilitate developing narrative statements: (1) direction of effect, (2) size of effect, (3) clinical importance, (4) statistical significance, and (5) strength or certainty of evidence. These constructs clearly overlap. It may not always be feasible to address all five constructs. Based on context and intended audience, investigators can determinewhich constructs will be most important to address in narrative statements.

Next Steps in Improving Healthcare Value: AHRQ Evidence-based Practice Center Program-Applying the Knowledge to Practice to Data Cycle to Strengthen the Value of Patient Care.

For more than 20 years, the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program has been identifying and synthesizing evidence to inform evidence-based healthcare. Recognizing that many healthcare settings continue to face challenges in disseminating and implementing evidence into practice, AHRQ's EPC program has also embarked on initiatives to facilitate the translation of evidence into practice and to measure and monitor how practice changes impact health outcomes. The program has structured its efforts around the three phases of the Learning Healthcare System cycle: knowledge, practice, and data. Here, we use a topic relevant to the field of hospital medicine-Clostridium difficile colitis prevention and treatment-as an exemplar of how the EPC program has used this framework to move evidence into practice and develop systems to facilitate continuous learning in healthcare systems.

Effectiveness of indoor allergen reduction in asthma management: A systematic review

This review will inform updated National Asthma Education and Prevention Program clinical practice guidelines. We sought to evaluate the effectiveness of allergen reduction interventions on asthma outcomes. We systematically searched the "gray literature" and 5 bibliographic databases. Eligible studies included systematic reviews, randomized controlled trials, and nonrandomized interventional studies. Risk of bias was assessed by using the Cochrane Risk of Bias instrument and the Newcastle-Ottawa scale. The evidence base was assessed by using the approach of the Agency for Healthcare Research and Quality's Evidence-based Practice Center program. Fifty-nine randomized and 8 nonrandomized trials addressed 8 interventions: acaricide, air purification, carpet removal, high-efficiency particulate air filtration (HEPA) vacuums, mattress covers, mold removal, pest control, and pet removal. Thirty-seven studies evaluated single-component interventions, and 30 studies assessed multicomponent interventions. Heterogeneity precluded meta-analysis. For most interventions and outcomes, the evidence base was inconclusive or showed no effect. No interventions were associated with improvement in validated asthma control measures or pulmonary physiology. Exacerbations were diminished in multicomponent studies that included HEPA vacuums or pest control (moderate strength of evidence [SOE] for both). Quality of life improved in studies of air purifiers (SOE: low) and in multicomponent studies that included HEPA vacuums (SOE: moderate) or pest control (SOE: low). Single interventions were generally not associated with improvement in asthma measures, with most strategies showing inconclusive results or no effect. Multicomponent interventions improved various outcomes, but no combination of specific interventions appears to be more effective. The evidence was often inconclusive because of a lack of studies. Further research is needed comparing the effect of indoor allergen reduction interventions on validated asthma measures, with sufficient population sizes to detect clinically meaningful differences.

PP22 How Do Health System Leaders Use Evidence To Inform Action?

Introduction:The US Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program sponsors the development of systematic reviews to inform clinical policy and practice. The EPC program sought to better understand how health systems identify and use this evidence.Methods:Representatives from eleven EPCs, the EPC Scientific Resource Center, and AHRQ developed a semi-structured interview script to query a diverse group of nine Key Informants (KIs) involved in health system quality, safety and process improvement about how they identify and use evidence. Interviews were transcribed and qualitatively summarized into key themes.Results:All KIs reported that their organizations have either centralized quality, safety, and process improvement functions within their system, or they have partnerships with other organizations to conduct this work. There was variation in how evidence was identified, with larger health systems having medical librarians and central bureaus to gather and disseminate information and smaller systems having local chief medical officers or individual clinicians do this work. KIs generally prefer guidelines, especially those with treatment algorithms, because they are actionable. They like systematic reviews because they efficiently condense study results and reconcile conflicting data. They prefer information from systematic reviews to be presented as short digestible summaries with the full report available on demand. KIs preferred systematic reviews from reputable entities and those without commercial bias. Some of the challenges KIs reported include how to resolve conflicting evidence, the generalizability of evidence to local needs, determining whether the evidence is up-to-date, and the length of time required to generate reviews. The topics of greatest interest included predictive analytics, high-value care, advance care planning, and care coordination. To increase awareness of AHRQ EPC reviews, KIs suggest alerting people at multiple levels in a health-system when new evidence reports are available and making reports easier to find in common search engines.Conclusions:Systematic reviews are valued by health system leaders. To be most useful they should be easy to locate and available in different formats targeted to the needs of different audiences.

OP71 Evidence Grading Systems Used In Health Technology Assessment Practice

Introduction:To facilitate moving from research findings to conclusions when conducting systematic reviews (SRs) and health technology assessments (HTAs), evidence grading systems (EGSs) have been developed to assess the quality of bodies of evidence and communicate (un)certainty about the effects of evaluated technologies. Use of EGSs has become an essential step in conducting SRs and HTAs and those relying on review conclusions should be aware of EGSs’ potential limitations.Methods:This study aims to identify EGSs used in SR and HTA practice, and summarize findings on their inter-rater reliability (IRR). Relevant sources were searched to identify EGSs used in recently published SRs and IRR studies of available EGSs. Members of the International Network of Agencies for Health Technology Assessment were surveyed regarding their current approaches.Results:Preliminary results indicate that only two conceptually similar EGSs are currently used by several organizations in SR and HTA practice: (i) the Grading of Recommendations Assessment, Development and Evaluation (GRADE) and (ii) the Agency for Healthcare Research and Quality Evidence-based Practice Center Program (AHRQ-EPC). Both approaches emphasize a structured and transparent method. However, results from published IRR studies suggest there is a risk for variability in their application due to researchers’ diverse levels of training and experience in using them, and the complexity and heterogeneity of evidence in SRs.Conclusions:Validated EGSs can play a critical role in whether and how research findings are eventually translated into practice. However, our results indicate a low level of uptake of EGSs in HTA practice. Both currently used EGSs are susceptible to misuse that allows different researchers to grade the same body of evidence differently, and their performance has not been robustly explored in terms of IRR. If these results stand up to replication, one cannot rely on conclusions of published SRs, which has implications for the decisions they inform.

Commentary on EPC methods: an exploration of the use of text-mining software in systematic reviews

• The Agency for Healthcare Research and Quality Evidence-based Practice Center program recently published a methods white paper on the effectivehealthcare.ahrq.gov web site which focused on a preliminary exploration of the use of text-mining in evidence synthesis.

Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews.

BackgroundRapid review (RR) products are inherently appealing as they are intended to be less time-consuming and resource-intensive than traditional systematic reviews (SRs); however, there is concern about the rigor of methods and reliability of results. In 2013 to 2014, a workgroup comprising representatives from the Agency for Healthcare Research and Quality’s Evidence-based Practice Center Program conducted a formal evaluation of RRs. This paper summarizes results, conclusions, and recommendations from published review articles examining RRs.MethodsA systematic literature search was conducted and publications were screened independently by two reviewers. Twelve review articles about RRs were identified. One investigator extracted data about RR methods and how they compared with standard SRs. A narrative summary is presented.ResultsA cross-comparison of review articles revealed the following: 1) ambiguous definitions of RRs, 2) varying timeframes to complete RRs ranging from 1 to 12 months, 3) limited scope of RR questions, and 4) significant heterogeneity between RR methods.ConclusionsRR definitions, methods, and applications vary substantially. Published review articles suggest that RRs should not be viewed as a substitute for a standard SR, although they have unique value for decision-makers. Recommendations for RR producers include transparency of methods used and the development of reporting standards.Electronic supplementary materialThe online version of this article (doi:10.1186/s13643-015-0040-4) contains supplementary material, which is available to authorized users.

Future Research Needs for Evaluating the Integration of Mental Health and Substance Abuse Treatment with Primary Care

Research needs are many in the current health care environment. In this article, we describe a novel method developed by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center Program for prioritizing areas for future research. Using a recent- ly published systematic review as a foundation, investigators worked with a diverse group of 10 stakeholders to identify and prioritize research needs. We enumerate 13 high-priority research needs, as determined by stakeholders who represented researchers, funders, health care providers, and patients and families, and discuss considerations for specific study designs. Our findings suggest that future research on integrating mental health and primary care should focus first on a) identifying methods of integrating primary care into specialty mental health settings, b) identifying cross-cutting strategies for integration across multiple mental health diagnostic categories as opposed to a separate strategy for each diagnostic category, and c) examining the use of information technology for integrating mental and general medical health care. Other priorities for consideration include examining the economic and organizational sus- tainability of successful integration models, identifying dissemination methods for various settings, examining the business case for integration as well as methods of payment, assessing the cost-effectiveness of integration, and identifying key components of successful strategies. The importance of sustainability and economic justification for integrated care strategies was a recurring theme in discussions with the stake- holders. The ability to sustain integrated care in everyday practice remains to be proved and will depend in part on the level of incentives and sup- port provided through payment system reform, as well as the ability of practices to provide care efficiently.

Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews

ObjectivesTo examine consistency (interrater reliability) of applying guidance for grading strength of evidence in systematic reviews for the Agency for Healthcare Research and Quality Evidence-based Practice Center program. Study Design and SettingUsing data from two systematic reviews, authors tested the main components of the approach: (1) scoring evidence on the four required domains (risk of bias, consistency, directness, and precision) separately for randomized controlled trials (RCTs) and observational studies and (2) developing an overall strength of evidence grade, given the scores for each of these domains. ResultsConclusions about overall strength of evidence reached by experienced systematic reviewers based on the same evidence can differ greatly, especially for complex bodies of evidence. Current instructions may be sufficient for straightforward quantitative evaluations that use meta-analysis for summarizing RCT findings. In contrast, agreement suffered when evaluations did not lend themselves to meta-analysis and reviewers needed to rely on their own qualitative judgment. Three areas raised particular concern: (1) evidence from a combination of RCTs and observational studies, (2) outcomes with differing measurement, and (3) evidence that appeared to show no differences in outcomes. ConclusionInterrater reliability was highly variable for scoring strength of evidence domains and combining scores to reach overall strength of evidence grades. Future research can help in establishing improved methods for evaluating these complex bodies of evidence.

Identifying Research Needs for Improving Health Care

Insights from systematic reviews can help new studies better meet the priorities and needs of patients and communities. However, systematic reviews unfortunately have not yet achieved this position to direct and guide new research studies. The Agency for Healthcare Research and Quality's Evidence-based Practice Center Program uses systematic reviews to identify gaps in current evidence and has developed a systematic process of prioritizing these gaps with stakeholder input into clearly defined "future research needs." Eight Evidence-based Practice Centers began to apply this effort in 2010 to various clinical and policy topics. Gaps that prevented systematic reviewers from answering central questions of the review may include insufficient studies on subpopulations, insufficient studies with appropriate comparators, lack of appropriate outcomes measured, and methods problems. Stakeholder panels, consisting of advocacy groups, patients, researchers, clinicians, funders, and policymakers, help refine the gaps through multiple conference calls and prioritization exercises. Each report highlights a focused set of 4 to 15 high-priority needs with an accompanying description of possible considerations for study design. Identification of high-priority research needs could potentially speed the development and implementation of high-priority, stakeholder-engaged research.

Methods Guide for Authors of Systematic Reviews of Medical Tests: A Collaboration Between the Agency for Healthcare Research and Quality (AHRQ) and the Journal of General Internal Medicine

Methods Guide for Authors of Systematic Reviews of Medical Tests: A Collaboration Between the Agency for Healthcare Research and Quality (AHRQ) and the Journal of General Internal Medicine

Too soon to judge value of pharmacogenetic testing in selecting antidepressant

The published research on testing for gene-based variations in the metabolism of selective serotonin-reuptake inhibitors (SSRIs) is poor—so poor that government-funded investigators recently could not find enough evidence to determine whether such testing would be useful in individualizing the treatment of depression. While this lack of evidence could be viewed as discouraging, the government took the view that the glass is half full. “This information will help identify the types of studies that are necessary to help better understand various treatment response issues,” said Beth A. Collins Sharp in a statement issued with the public release of the evidence report “Testing for Cytochrome P450 Polymorphisms in Adults With Non-Psychotic Depression Treated With Selective Serotonin Reuptake Inhibitors (SSRIs).” Collins Sharp is the director of the Evidence-based Practice Center program at the Agency for Healthcare Research and Quality (AHRQ). The 157-page report, released in early January, is part of the Evaluation of Genomic Applications in Practice and Prevention pilot project of the Centers for Disease Control and Prevention (CDC).

Challenges in Systematic Reviews of Therapeutic Devices and Procedures

The authors discuss 3 challenges in conducting and interpreting any systematic review that are particularly relevant for systematic reviews of therapeutic devices or surgical procedures: 1) inclusion or exclusion of grey literature, 2) the role of nonrandomized studies, and 3) issues in applying the results to clinical care that are unique to the surgical and therapeutic device literature. The authors also discuss empirical evidence related to these topics and illustrate how reviewers in the Agency for Healthcare Research and Quality's Evidence-based Practice Center program have dealt with these challenges in developing evidence reports for decision makers and clinicians about therapeutic devices or surgical procedures.

Better Information for Better Health Care: The Evidence-based Practice Center Program and the Agency for Healthcare Research and Quality

To provide decision makers with the best available evidence, the Agency for Healthcare Research and Quality established a network of Evidence-based Practice Centers across North America. The centers perform systematic reviews on important questions posed by partner organizations about clinical, organizational, and policy interventions in healthcare. The Agency works closely with partners and other decision maker s to help translate that evidence into practice or policy. In this paper, we review important lessons we have learned over the past 7 years about how to increase the efficiency and impact of systematic reviews. Lessons concern selecting the right topics and scope, working effectively with partners, and balancing consistency and flexibility in methods. We examine continuing evolutions of the program and the impact of planned work on comparative effectiveness performed as part of the Medicare Modernization Act of 2003.

Pharmacologic and Surgical Management of Obesity in Primary Care: A Clinical Practice Guideline from the American College of Physicians

This guideline is based on the evidence report and accompanying background papers developed by the Southern California Evidence-Based Practice Center. The American College of Physicians nominated this topic to the Agency for Healthcare Research and Quality Evidence-Based Practice Center program as part of a concerted effort to complement the guidelines of the U.S. Preventive Services Task Force. The College recommends that all clinicians refer to the Task Force recommendations as part of an overall strategy for managing overweight and obesity, which should always include appropriate diet and exercise for all patients who are overweight or obese. The intent of this guideline is to provide recommendations based on a review of the evidence on pharmacologic and surgical treatments of obesity. The target audience is all clinicians caring for obese patients, defined as a body mass index of 30 kg/m2 or greater. This guideline is not intended to be used by commercial weight loss centers or for direct-to-consumer marketing by manufacturers and does not apply to patients with body mass indices below 30 kg/m2.

Databases for outcomes research: what has 10 years of experience taught us?

This paper describes how the mission of the Agency for Healthcare Research and Quality (AHRQ) is being executed through the many programs that it has developed and implemented. The Evidence-based Practice Center program was developed to provide systematic reviews on common and expensive conditions and health technologies and to ensure that this information is used to improve health care outcomes and costs. The National Guidelines Clearinghouse provides an internet-based source of clinical practice guidelines that are produced by clinical specialty organizations for the primary purpose of improving health care delivery and outcomes. Relevant to this symposium on databases, AHRQ has supported the development of databases to track hospital utilization on a state-by-state basis. The Healthcare Cost and Utilization Project (HCUP) allows comparisons between states and within regions of individual states. New initiatives have been launched to evaluate interventions across systems rather than focusing on the individual patient (Translating Research into Practice--TRIP). The Centers for Education and Research on Therapeutics (CERTs) program was developed to conduct real world evaluations to better understand the benefits and risks of single and combined therapy. Both programs further the mission of the AHRQ to improve the outcomes and quality of health care, with additional focus on the cost-effectiveness, patient safety, and increasing access to care for all. Information on programs developed by the AHRQ is available in more detail at the Agency Web site http://www.ahrq.gov.