Agency For Healthcare Research And Quality Patient Safety Research Articles

Assessing the Sustainability of Initiatives Aimed to Enhance Patient Safety Culture: A > 10-year Experience with Multifaceted Interventions

Culture is a key factor in maintaining high-quality care. We have instituted a series of mutually-reinforcing initiatives aimed to improve operations, and patient and staff safety. We herein review the impact of these initiatives on our Patient Safety Culture over 14 years. We hypothesize that these initiatives can lead to sustained improvements in Culture. In ≈ 2009, our department, with help of dedicated quality improvement coaches from our internal Division of Healthcare Engineering (with knowledge of Lean-Six Sigma and High Reliability methods and tools), instituted (and largely sustained) initiatives aimed to improve our operations, and to improve patient/staff safety; including: (a) daily multidisciplinary team huddles, (b) daily pre-planning/treatment peer review, (c) a robust Incident Learning System, (d) weekly meetings to review reported incidents, (e) monthly department-wide meetings to review the highlights from the weekly incident review meetings, (f) celebration/recognitions for staff participation and (g) leader Gemba walks. Culture was quantitatively assessed via the AHRQ (Agency for Healthcare Research and Quality) Patient Safety Culture Survey (with 51 questions) which was sent to the department every two years (2021 was skipped due to Covid). Changes in the summary survey results over time were assessed using 2-tailed chi-square. Within 2-4 years of starting of our initiatives, there was an increase in the number of survey respondents, and an increase in the % of favorable responses (vs. pre-initiative data from 2009), for most comparisons (see Table with representative data). The % favorable responses plateaued in ≈ 2013-2015, and there was a non-significant decline in % favorable responses in later years (vs. the high scores in 2015). A series of mutually reinforcing initiatives aimed to improve operations, and to improve patient/staff safety, can lead to improvements in Patient Safety Culture; and these improvements can be largely sustained over time. Some of the modest non-significant decline in later years may reflect staff changes, fatigue, increasing practice complexity, financial pressures, and/or Covid-related issues (in the 2022 survey). This speaks to the importance of maintaining quality/safety initiatives through transitions in leadership, staff and external stressors.

457. Relationship Between Healthcare Worker (HCW) Perception of Safety and Rates of Healthcare-Associated Infections (HAI) and Hand Hygiene (HH) Compliance

BackgroundMany facilities complete the Agency for Healthcare Research and Quality (AHRQ) patient safety culture survey. Our goal was to evaluate associations between healthcare worker (HCW) responses to AHRQ patient safety culture survey questions and unit performance on healthcare-associated infections (HAI) and hand hygiene compliance.Methods11,257 HCW across 10 acute care hospitals and four rehabilitation facilities completed the 2016 AHRQ patient safety survey. Unit-level standardized infection ratios (SIRs) were calculated for central line-associated bloodstream infections (CLABSIs) and catheter-associated urinary tract infections (CAUTIs) and standardized utilization ratios (SURs) for Foley catheter use. SIRs and SURs <1 met HAI performance goals. Hand-hygiene compliance was captured via secret-shopper methods and ≥75% used as the cut-off for meeting compliance goals. Unit-aggregated survey responses were compared between units that did vs. did not meet SIR, SUR goals for the year, and HH goals for the quarter, prior to survey distribution, using two-sample t-tests.ResultsFewer HCW on low-HH compliance units (i.e., <75%; n = 179 units) responded positively to questions pertaining to overall perception of safety, frequency of events reported, supervisor/manager expectations/actions promoting safety, organization learning, teamwork within units, communication openness, and nonpunitive response to errors, than HCW on high-compliance units (i.e., >75%; n = 69 units; P < 0.05). More HCW on units with CAUTI SIR <1 (n = 40 units) responded positively to supervisor/manager expectations/actions promoting safety, teamwork across units, and hospital handoffs, compared with HCW on units with SIR ≥1 (n = 20; P < 0.05). Fewer HCW on units with Foley SUR <1 (n = 27 units) responded positively to questions on supervisor/manager expectations/actions promoting safety and teamwork within units, than HCW on units with SUR ≥1 (n = 22; P < 0.05). We observed no associations between CLABSI SIR performance and AHRQ safety survey responses.ConclusionHCW perceptions of unit safety culture can be associated with HAI and HH compliance performance. Unit performance/compliance was most commonly associated with supervisor expectations suggesting a key managerial component to promoting safety culture.Disclosures All authors: No reported disclosures.

Assessment of the Daily Safety and Operations Huddle of a Pharmacy Department

A daily huddle was tested in an academic medical center pharmacy department to see if it resulted in a positive impact on the safety culture and communication. The huddle is a short daily briefing conducted at shift change in a main pharmacy area with a dial in number for those not present in the main pharmacy area. The purpose of the huddle is to raise awareness of any current safety or operational issues. The effectiveness of the huddle was primarily tested through a targeted survey and the Agency for Healthcare Research and Quality (AHRQ) Patient Safety Culture Survey results. A survey evaluating daily huddle effectiveness had a 24% response rate. The majority of respondents indicated positive feelings towards the huddle: 58.7% indicated effective communication of information between shifts, 67% indicated effective communication of safety events, and 63% indicated effective communication of daily operational issues. Additionally, the department's AHRQ Patient Safety Culture Survey results improved following the huddle implementation. Departmental staff gave their work unit an overall safety score of the top possible answer, excellent, (vs. very good, acceptable, poor, failing) 11% of the time before the huddle began. Eight months after the huddle started, the number of respondents answering with the top possible score increased to 23% (p<0.05). This positive impact sustained to the next year as well (25% top possible score). Additional positive results are discussed. Implementation of a safety huddle positively influenced measures of communication and safety culture in the Department of Pharmacy Services.

Tailoring an educational program on the AHRQ Patient Safety Indicators to meet stakeholder needs: lessons learned in the VA

BackgroundGiven that patient safety measures are increasingly used for public reporting and pay-for performance, it is important for stakeholders to understand how to use these measures for improvement. The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) are one particularly visible set of measures that are now used primarily for public reporting and pay-for-performance among both private sector and Veterans Health Administration (VA) hospitals. This trend generates a strong need for stakeholders to understand how to interpret and use the PSIs for quality improvement (QI). The goal of this study was to develop an educational program and tailor it to stakeholders’ needs. In this paper, we share what we learned from this program development process.MethodsOur study population included key VA stakeholders involved in reviewing performance reports and prioritizing and initiating quality/safety initiatives. A pre-program formative evaluation through telephone interviews and web-based surveys assessed stakeholders’ educational needs/interests. Findings from the formative evaluation led to development and implementation of a cyberseminar-based program, which we tailored to stakeholders’ needs/interests. A post-program survey evaluated program participants’ perceptions about the PSI educational program.ResultsInterview data confirmed that the concepts we had developed for the interviews could be used for the survey. Survey results informed us on what program delivery mode and content topics were of high interest. Six cyberseminars were developed—three of which focused on two content areas that were noted of greatest interest: learning how to use PSIs for monitoring trends and understanding how to interpret PSIs. We also used snapshots of VA PSI reports so that participants could directly apply learnings. Although initial interest in the program was high, actual attendance was low. However, post-program survey results indicated that perceptions about the program were positive.ConclusionsConducting a formative evaluation was a highly important process in program development. The useful information that we collected through the interviews and surveys allowed us to tailor the program to stakeholders’ needs and interests. Our experiences, particularly with the formative evaluation process, yielded valuable lessons that can guide others when developing and implementing similar educational programs.

Feasibility of administrative data for studying complications after hip fracture surgery

PurposeThere is limited information in administrative databases on the occurrence of serious but treatable complications after hip fracture surgery. This study sought to determine the feasibility of identifying the occurrence...

Progress at the Intersection of Patient Safety and Medical Liability: Insights from the AHRQ Patient Safety and Medical Liability Demonstration Program.

To identify lessons learned from the experience of the Agency for Healthcare Research and Quality (AHRQ) Patient Safety and Medical Liability (PSML) Demonstration Program. On September 9, 2009, President Obama directed the Secretary of Health and Human Services to authorize demonstration projects that put "patient safety first" with the intent of reducing preventable adverse outcomes and stemming liability costs. Seven demonstration projects received 3years of funding from AHRQ in the summer of 2010, and the program formally came to a close in June 2015. The seven grantees implemented complex, broad-ranging innovations addressing both patient safety and medical liability in "real-world" contexts. Some projects featured novel approaches, while others implemented adaptations of existing models. Each project was funded by AHRQ to collect data on the impact of its interventions. In addition, AHRQ funded a cross-cutting qualitative evaluation focused on lessons learned in implementing PSML interventions. Site visits and follow-up interviews supplemented with material abstracted from formal project reports to AHRQ. The PSML demonstration projects focused on three broad approaches: (1) improving communication around adverse events through disclosure and resolution programs; (2) preventing harm through implementation of clinical "best practices"; and (3) exploring alternative methods of settling claims. Although the demonstration contributed to accumulating evidence that these kinds of interventions can positively affect outcomes, there is also evidence to suggest that these interventions can be difficult to scale. In addition to producing at least preliminary positive outcomes, the demonstration also lends credence to the idea that targeted interventions that improve some aspect of patient safety or malpractice performance may also contribute more broadly to institutional culture and the alignment of all parties around reducing risk and preventing harm. However, more empirical work needs to be carried out to quantify the effect of such interventions.

Using Harm-Based Weights for the AHRQ Patient Safety for Selected Indicators Composite (PSI-90): Does It Affect Assessment of Hospital Performance and Financial Penalties in Veterans Health Administration Hospitals?

To assess whether hospital profiles for public reporting and pay-for-performance, measured by the Agency for Healthcare Research and Quality (AHRQ) Patient Safety for Selected Indicators (PSI-90) composite measure, were affected by using the recently developed harm-based weights. Retrospective analysis of 2012-2014 data from the Veterans Health Administration (VA). The AHRQ PSI software (v5.0) was applied to obtain the original volume-based PSI-90 scores for 132 acute-care hospitals. We constructed a modified PSI-90 using the harm-based weights developed by AHRQ. We compared hospital profiles for public reporting and pay-for-performance between these two PSI-90s and assessed patterns in these changes. The volume-based and the harm-based PSI-90s were strongly correlated (r=0.67, p<.0001). The use of the harm-based PSI-90 had a relatively small impact on public reporting (i.e., 5 percent hospitals changed categorization), but it had a much larger impact on pay-for-performance (e.g., 15 percent of hospitals would have faced different financial penalties under the Medicare Hospital-Acquired Condition Reduction Program). Because of changes in weights of specific PSIs, hospital profile changes occurred systematically. Use of the harm-based weights in PSI-90 has the potential to significantly change payments under pay-for-performance programs. Policy makers should carefully develop transition plans for guiding hospitals through changes in any quality metrics used for pay-for-performance.

The implementation and assessment of a quality and safety culture education program in a large radiation oncology department

In 2010, the American Society for Radiation Oncology launched a national campaign to improve patient safety in radiation therapy. One recommendation included the expansion of educational programs dedicated to quality and safety. We subsequently implemented a quality and safety culture education program (Q-SCEP) in our large radiation oncology department. The purpose of this study is to describe the design, implementation, and impact of this Q-SCEP. In 2010, we instituted a comprehensive Q-SCEP, consisting of a longitudinal series of lectures, meetings, and interactive workshops. Participation was mandatory for all department members across all network locations. Electronic surveys were administered to assess employee engagement, knowledge retention, preferred learning styles, and the program's overall impact. The Agency for Healthcare Research and Quality (AHRQ) Survey on Patient Safety Culture was administered. Analysis of variance was used for statistical analysis. Between 2010 and 2015, 100% of targeted staff participated in Q-SCEP. Thirty-three percent (132 of 400) and 30% (136 of 450) responded to surveys in 2012 and 2014, respectively. Mean scores improved from 73% to 89% (P < .001), with the largest improvement seen among therapists (+21.7%). The majority strongly agreed that safety culture education was critical to performing their jobs well. Full course compliance was achieved despite the sizable number of personnel and treatment centers. Periodic assessments demonstrated high knowledge retention, which significantly improved over time in nearly all department divisions. Additionally, our AHRQ patient safety grade remains high and continues to improve. These results will be used to further enhance ongoing internal safety initiatives and to inform future innovative efforts.

Improved Coding of Postoperative Deep Vein Thrombosis and Pulmonary Embolism in Administrative Data (AHRQ Patient Safety Indicator 12) After Introduction of New ICD-9-CM Diagnosis Codes

Symptomatic venous thromboembolism is a common postoperative complication. The Agency for Healthcare Research and Quality (AHRQ) has developed a Patient Safety Indicator 12 to assist hospitals, payers, and other stakeholders to identify patients who experienced this complication. To determine whether newly created and recently redefined ICD-9-CM codes improved the criterion validity of Patient Safety Indicator 12, based on new samples of records dated after October 2009. Two sources of data were used: (1) UHC retrospective case-control study of risk factors for acute symptomatic venous thromboembolism occurring within 90 days after total knee arthroplasty in teaching hospitals; (2) chart abstraction data by volunteer hospitals participating in the Validation Pilot Project of the AHRQ. In the UHC sample, the positive predictive value (PPV) was 99% (125/126) and the negative predictive value was 99.4% (460/463). In the AHRQ sample, the overall PPV was 81% (126/156). The PPV based on both samples shows substantial improvement compared with the previously reported PPVs of 43%-48%, suggesting that changes in ICD-9-CM code architecture and better coding guidance can improve the usefulness of coded data.

AHRQ Patient Safety Indicators: Time to Include Hemorrhage and Infection During Childbirth

Many Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) partially or completely exclude pregnant women. Both postoperative hemorrhage or hematoma (PSI 9; hemorrhage), and Postoperative Sepsis (PSI 13; infection) appear to be adaptable to pregnancy hospitalizations. Using the 2009 California Patient Discharge Dataset (N [total] = 508,842), the hemorrhage and infection PSIs were examined for their potential to include pregnant women in gynecological, antepartum, postpartum, and delivery subpopulations. The statewide and hospital-level indicator rates were calculated using hierarchical models adjusted for case mix. Only the delivery population had sufficient cases for hospital-level analysis. Both PSIs required major changes to the technical specifications because of pregnancy-specific codes and coding practices. Nevertheless, these revised indicators identified substantial morbidity that varied widely across hospitals. The hemorrhage indicator rate was 2.50% (95% confidence interval [CI], 2.45-2.54) for all deliveries, compared with 0.26% (95% CI, 0.25-0.27) in the AHRQ population and 0.18% (95% CI, 0.15-0.21) for nonpregnant women of reproductive age. Adjusted hospital rates averaged 2.52%, with a midquartile range of 1.16% to 3.09% Although infection rates were lower for all deliveries than for the AHRQ population (0.18% versus 1.20%), they were highly associated with cesarean versus vaginal birth (0.43% versus 0.05%) and ranged from 0% to 1.15% across hospitals. Although codes and coding practices for pregnancy hospitalizations differ from those used for nonpregnant adults, hospital-level measures of childbirth-associated hemorrhage and infection are feasible, vary widely, and demonstrate considerable opportunity for improvement.

Millennial Health Care: Change You Can Believe in

Millennial Health Care: Change You Can Believe in

Designing an Abstraction Instrument: Lessons from Efforts to Validate the AHRQ Patient Safety Indicators

The U.S. Agency for Healthcare Research and Quality (AHRQ) and other organizations have developed quality indicators based on hospital administrative data. Characteristics of effective abstraction instruments were identified for determining both the positive predictive value (PPV) of Patient Safety Indicators (PSIs) and the extent to which hospitals and clinicians could have prevented adverse events. Through an iterative process involving nurse abstractors, physicians, and nurses with quality improvement experience, and health services researchers, 25 abstraction instruments were designed for 12 AHRQ provider-level morbidity PSIs. Data were analyzed from 13 of these instruments, and data are being collected using several more. Common problems in designing the instruments included avoiding uninformative questions and premature termination of the abstraction process, anticipating misinterpretation of questions, allowing an appropriate range of response options; using clear terminology, optimizing the flow of the abstraction process, balancing the utility of data against abstractor burden, and recognizing the needs of end users, such as hospitals and quality improvement professionals and researchers, for the abstracted information. Designing medical record abstraction instruments for quality improvement research involves several potential pitfalls. Understanding how we addressed these challenges might help both investigators and users of outcome indicators to appreciate the strengths and limitations of outcome-based quality indicators and tools designed to validate or investigate such indicators within provider organizations.

Evaluating Risk-Adjusted Cesarean Delivery Rate as a Measure of Obstetric Quality

To validate the risk-adjusted cesarean delivery rate as a measure of obstetric quality through its association with maternal and neonatal outcomes for all pregnancies (model 1) and in primiparous patients with singleton pregnancies (model 2). We constructed a population-based cohort of 845,651 patients from 401 hospitals representing all deliveries in California and Pennsylvania between 2004 and 2005. We used linked birth certificate and hospital admission records for mother and neonate to estimate the correlation between risk-adjusted cesarean delivery and a composite of adverse maternal outcomes, adverse neonatal outcomes, and four obstetric patient safety indicators from the Agency for Healthcare Research and Quality (AHRQ). In both models, risk-adjusted cesarean delivery rates were negatively correlated with both the maternal and neonatal composite outcomes and the AHRQ patient safety indicators for birth trauma, injury with instrumented vaginal delivery, and cesarean delivery. Approximately 60% of the 107 hospitals with lower-than-expected risk-adjusted cesarean delivery rates had higher-than-expected rates of at least one of the six adverse outcomes, compared with 19.6% of the hospitals with higher-than-expected risk-adjusted cesarean delivery rates and 36.1% of the hospitals with expected rates (P<.001). Lower-than-expected risk-adjusted cesarean delivery rates in all patients or when restricted to a more homogeneous group of primiparous patients with term singleton pregnancies are associated with higher-than-expected adverse maternal or neonatal outcomes. Higher-than-expected risk-adjusted cesarean delivery rates do not result in improved outcomes.

How Valid is the ICD-9-CM Based AHRQ Patient Safety Indicator for Postoperative Venous Thromboembolism?

Hospital administrative data are being used to identify patients with postoperative venous thromboembolism (VTE), either pulmonary embolism (PE) or deep-vein thrombosis (DVT). However, few studies have evaluated the accuracy of these ICD-9-CM codes across multiple hospitals. The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator (PSI)-12 was used to identify cases with postoperative VTE in 80 hospitals that volunteered for either an AHRQ or University HealthSystem Consortium (UHC) validation project. Trained abstractors using a standardized tool and guidelines retrospectively verified all coded VTE events. In the combined samples, the positive predictive value of the set of prespecified VTE codes for any acute VTE at any time during the hospitalization was 451 of 573 = 79% (95% CI: 75%-82%). However, the positive predictive value for acute lower extremity DVT or PE diagnosed after an operation was 209 of 452 = 44% (95% CI: 37%-51%) in the UHC sample and 58 of 121 = 48% (95% CI: 42-67%) in the AHRQ sample. Fourteen percent of all cases had an acute upper extremity DVT, 6% had superficial vein thrombosis and 21% had no acute VTE, however, 61% of the latter had a documented prior/chronic VTE. In the UHC cohort, the sensitivity for any acute VTE was 95.5% (95% CI: 86.4%-100%); the specificity was 99.5% (95% CI: 99.4%-99.7%). Current PSI 12 criteria do not accurately identify patients with acute postoperative lower extremity DVT or PE. Modification of the ICD-9-CM codes and implementation of "present on admission" flags should improve the predictive value for clinically important VTE events.

Application of patient safety indicators internationally: a pilot study among seven countries

To explore the potential for international comparison of patient safety as part of the Health Care Quality Indicators project of the Organization for Economic Co-operation and Development (OECD) by evaluating patient safety indicators originally published by the US Agency for Healthcare Research and Quality (AHRQ). A retrospective cross-sectional study. Acute care hospitals in the USA, UK, Sweden, Spain, Germany, Canada and Australia in 2004 and 2005/2006. Routine hospitalization-related administrative data from seven countries were analyzed. Using algorithms adapted to the diagnosis and procedure coding systems in place in each country, authorities in each of the participating countries reported summaries of the distribution of hospital-level and overall (national) rates for each AHRQ Patient Safety Indicator to the OECD project secretariat. Each country's vector of national indicator rates and the vector of American patient safety indicators rates published by AHRQ (and re-estimated as part of this study) were highly correlated (0.821-0.966). However, there was substantial systematic variation in rates across countries. This pilot study reveals that AHRQ Patient Safety Indicators can be applied to international hospital data. However, the analyses suggest that certain indicators (e.g. 'birth trauma', 'complications of anesthesia') may be too unreliable for international comparisons. Data quality varies across countries; undercoding may be a systematic problem in some countries. Efforts at international harmonization of hospital discharge data sets as well as improved accuracy of documentation should facilitate future comparative analyses of routine databases.

Assessment of Contributions to Patient Safety Knowledge by the Agency for Healthcare Research and Quality‐Funded Patient Safety Projects

To characterize the activities of projects funded in Agency for Healthcare Research and Quality (AHRQ)'s patient safety portfolio and assess their aggregate potential to contribute to knowledge development. Information abstracted from proposals for projects funded in AHRQ's patient safety portfolio, information on safety practices from the AHRQ Evidence Report on Patient Safety Practices, and products produced by the projects. This represented one part of the process evaluation conducted as part of a longitudinal evaluation based on the Context–Input–Process–Product model. The 234 projects funded through AHRQ's patient safety portfolio examined a wide variety of patient safety issues and extended their work beyond the hospital setting to less studied parts of the health care system. Many of the projects implemented and tested practices for which the patient safety evidence report identified a need for additional evidence. The funded projects also generated a substantial body of new patient safety knowledge through a growing number of journal articles and other products. The projects funded in AHRQ's patient safety portfolio have the potential to make substantial contributions to the knowledge base on patient safety. The full value of this new knowledge remains to be confirmed through the synthesis of results

Evaluation of the AHRQ patient safety initiative: framework and approach.

Describe the evaluation performed of the patient safety initiative operated by the Agency for Healthcare Research and Quality (AHRQ). AHRQ PATIENT SAFETY INITIATIVE When patient safety became a national priority in 2000, Congress charged and funded AHRQ to improve health care safety. Over the next 6 years, AHRQ funded more than 300 research projects and other activities, addressing diverse patient safety issues and practices. AHRQ contracted with RAND in 2002 to perform a 4-year evaluation of the initiative, which was completed in 2006. This formative evaluation used the CIPP program evaluation model, which emphasizes multiple stakeholders' interests (e.g., patients, providers, funded researchers). We monitored the progress of the patient safety initiative and provided AHRQ annual feedback that assessed each year's activities, identifying issues and offering suggestions for actions by AHRQ. Given the size and complexity of the initiative, the evaluation needed to examine key individual components and synthesize results across them, and it also had to be responsive to changes in the initiative over time. We used a conceptual framework to bring together the disparate pieces to synthesize overall findings. The remaining articles in this issue describe selected results from this evaluation.

Evaluation of the AHRQ patient safety initiative: synthesis of findings.

To present overall findings from the 4-year evaluation of the national patient safety initiative operated by the Agency for Healthcare Research and Quality (AHRQ). Interviews with AHRQ staff, grantees, and other patient safety stakeholders; published materials; and internal AHRQ documents. The evaluation was structured to address a system framework of five components involved in improving safety. The initiative's contributions to improving each system component were assessed qualitatively, comparing results from three separate analyses—AHRQ's achievement of its patient safety goals, our own assessment of the initiative's activities, and independent stakeholder ratings of AHRQ's contributions. AHRQ has faced a daunting challenge for improving patient safety, given the complex problems of the U.S. health care system and the limited resources AHRQ has had to address them. The patient safety initiative achieved strongest progress for its contributions to knowledge of patient safety epidemiology and effective practices, where AHRQ has considerable experience, and to strengthening infrastructure to support adoption of safe practices. Progress was slower in establishing a national monitoring capability and dissemination of safe practices for adoption. AHRQ needs to expand efforts to apply new knowledge for stimulating use of safe practices in the field.

Validity of Selected AHRQ Patient Safety Indicators Based on VA National Surgical Quality Improvement Program Data

To examine the criterion validity of the Agency for Health Care Research and Quality (AHRQ) Patient Safety Indicators (PSIs) using clinical data from the Veterans Health Administration (VA) National Surgical Quality Improvement Program (NSQIP). Fifty five thousand seven hundred and fifty two matched hospitalizations from 2001 VA inpatient surgical discharge data and NSQIP chart-abstracted data. We examined the sensitivities, specificities, positive predictive values (PPVs), and positive likelihood ratios of five surgical PSIs that corresponded to NSQIP adverse events. We created and tested alternative definitions of each PSI. FY01 inpatient discharge data were merged with 2001 NSQIP data abstracted from medical records for major noncardiac surgeries. Sensitivities were 19-56 percent for original PSI definitions; and 37-63 percent using alternative PSI definitions. PPVs were 22-74 percent and did not improve with modifications. Positive likelihood ratios were 65-524 using original definitions, and 64-744 using alternative definitions. "Postoperative respiratory failure" and "postoperative wound dehiscence" exhibited significant increases in sensitivity after modifications. PSI sensitivities and PPVs were moderate. For three of the five PSIs, AHRQ has incorporated our alternative, higher sensitivity definitions into current PSI algorithms. Further validation should be considered before most of the PSIs evaluated herein are used to publicly compare or reward hospital performance.

Impact of date stamping on patient safety measurement in patients undergoing CABG: Experience with the AHRQ Patient Safety Indicators

BackgroundThe Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) provide information on hospital risk-adjusted rates for potentially preventable adverse events. Although designed to work with routine administrative data, it is unknown whether the PSIs can accurately distinguish between complications and pre-existing conditions. The objective of this study is to examine whether the AHRQ PSIs accurately measure hospital complication rates, using the data with present-on-admission (POA) codes to distinguish between complications and pre-existing conditionsMethodsRetrospective cohort study of patients undergoing isolated CABG surgery in California conducted using the 1998–2000 California State Inpatient Database. We calculated the positive predictive value of selected AHRQ PSIs using information from the POA as the gold standard, and the intra-class correlation coefficient to assess the level of agreement between the hospital risk-adjusted PSI rates with and without the information contained in the POA modifier.ResultsThe false positive error rate, defined as one minus the positive predictive value, was greater than or equal to 20% for four of the eight PSIs examined: decubitus ulcer, failure-to-rescue, postoperative physiologic and metabolic derangement, and postoperative pulmonary embolism or deep venous thrombosis. Pairwise comparison of the hospital risk-adjusted PSI rates, with and without POA information, demonstrated almost perfect agreement for five of the eight PSI's. For decubitus ulcer, failure-to-rescue, and postoperative pulmonary embolism or DVT, the intraclass-correlation coefficient ranged between 0.63 to 0.79.ConclusionFor some of the AHRQ Patient Safety Indicators, there are significant differences in the risk-adjusted rates of adverse events depending on whether the POA indicator is used to distinguish between pre-existing conditions and complications. The use of the POA indicator will increase the accuracy of the AHRQ PSIs as measures of adverse outcomes.