Override Rates Research Articles

Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis.

Primary studies have demonstrated that despite being useful, most of the drug-drug interaction (DDI) alerts generated by clinical decision support systems are overridden by prescribers. To provide more information about this issue, we conducted a systematic review and meta-analysis on the prevalence of DDI alerts generated by CDSS and alert overrides by physicians. The search strategy was implemented by applying the terms and MeSH headings and conducted in the MEDLINE/PubMed, EMBASE, Web of Science, Scopus, LILACS, and Google Scholar databases. Blinded reviewers screened 1873 records and 86 full studies, and 16 articles were included for analysis. The overall prevalence of alert generated by CDSS was 13% (CI95% 5-24%, p-value <0.0001, I^2 = 100%), and the overall prevalence of alert override by physicians was 90% (CI95% 85-95%, p-value <0.0001, I^2 = 100%). This systematic review and meta-analysis presents a high rate of alert overrides, even after CDSS adjustments that significantly reduced the number of alerts. After analyzing the articles included in this review, it was clear that the CDSS alerts physicians about potential DDI should be developed with a focus on the user experience, thus increasing their confidence and satisfaction, which may increase patient clinical safety.

The Acceptance of Interruptive Medication Alerts in an Electronic Decision Support System Differs between Different Alert Types.

Through targeted medication alerts, clinical decision support systems (CDSS) help users to identify medication errors such as disregarded drug-drug interactions (DDIs). Override rates of such alerts are high; however, they can be mitigated by alert tailoring or workflow-interrupting display of severe alerts that need active user acceptance or overriding. Yet, the extent to which the displayed alert interferes with the prescribers' workflow showed inconclusive impact on alert acceptance. We aimed to assess whether and how often prescriptions were changed as a potential result of interruptive alerts on different (contraindicated) prescription constellations with particularly high risks for adverse drug events (ADEs). We retrospectively collected data of all interruptive alerts issued between March 2016 and August 2020 in the local CDSS (AiDKlinik) at Heidelberg University Hospital. The alert battery consisted of 31 distinct alerts for contraindicated DDI with simvastatin, potentially inappropriate medication for patients > 65 years (PIM, N = 14 drugs and 36 drug combinations), and contraindicated drugs in hyperkalemia (N = 5) that could be accepted or overridden giving a reason in free-text form. In 935 prescribing sessions of 500 274 total sessions, at least one interruptive alert was fired. Of all interruptive alerts, about half of the sessions were evaluable whereof in total 57.5% (269 of 468 sessions) were accepted while 42.5% were overridden. The acceptance rate of interruptive alerts differed significantly depending on the alert type (p <0.0001), reaching 85.7% for DDI alerts (N = 185), 65.3% for contraindicated drugs in hyperkalemia (N = 98), and 25.1% for PIM alerts (N = 185). A total of 57.5% of the interruptive medication alerts with particularly high risks for ADE in our setting were accepted while the acceptance rate differed according to the alert type with contraindicated simvastatin DDI alerts being accepted most frequently.

Temporal Change in Alert Override Rate with a Minimally Interruptive Clinical Decision Support on a Next-Generation Electronic Medical Record.

Background and objectives: The aim of this study is to describe the temporal change in alert override with a minimally interruptive clinical decision support (CDS) on a Next-Generation electronic medical record (EMR) and analyze factors associated with the change. Materials and Methods: The minimally interruptive CDS used in this study was implemented in the hospital in 2016, which was a part of the new next-generation EMR, Data Analytics and Research Window for Integrated kNowledge (DARWIN), which does not generate modals, ‘pop-ups’ but show messages as in-line information. The prescription (medication order) and alerts data from July 2016 to December 2017 were extracted. Piece-wise regression analysis and linear regression analysis was performed to determine the temporal change and factors associated with it. Results: Overall, 2,706,395 alerts and 993 doctors were included in the study. Among doctors, 37.2% were faculty (professors), 17.2% were fellows, and 45.6% trainees (interns and residents). The overall override rate was 61.9%. There was a significant change in an increasing trend at month 12 (p < 0.001). We found doctors’ positions and specialties, along with the number of alerts and medication variability, were significantly associated with the change. Conclusions: In this study, we found a significant temporal change of alert override. We also found factors associated with the change, which had statistical significance.

Validation and examination of the Ohio Youth Assessment System with juvenile sex offenders

Research SummaryIn this study, we examined the use of an actuarial risk assessment tool—the Ohio Youth Assessment System‐Disposition Tool (OYAS‐Disposition Tool)—with juvenile sex offenders. Specifically, the main goals of the study were to (a) examine the predictive validity of the tool with sex offenders and (b) explore the nature of the use of professional discretion used to override the tool. The sample consisted of 3,235 youth from a large juvenile county court in the Midwest. The results indicated that the OYAS‐Disposition Tool was a significantly better option for predicting general recidivism for sex offenders than it was for non–sex offenders. The tool was also an effective method for predicting sexual recidivism. Most importantly, however, the use of professional overrides significantly reduced the ability of researchers to apply the tool to predict new court petitions and adjudications to nonsignificant levels. Finally, several justifications were commonly used for overrides: treatment needs, offense seriousness, and use of an alternative sex‐offender–specific assessment.Policy ImplicationsThe findings in this study highlight several important policy implications that would improve the assessment process for juvenile sex offenders. First, agencies using specialized risk assessments designed for sex offenders may consider applying a general risk assessment tool to identify a broader set of criminogenic needs and to predict risk of general recidivism. Second, there is a need to evaluate policies and practices that allow for the use of professional discretion with sex offenders given that they reduce the predictive validity of the risk tool evaluated. The high rate of overrides for juvenile sex offenders and justification for their use go against best practices in corrections. For instance, overrides were often justified based on offense seriousness; however, focusing on evidence‐based criminogenic risk factors provides the best accuracy in predicting future offending. In this study, we call into question court policies that allow for overrides based on crime type or based on a practitioner's professional judgment concerning a juvenile's level of service needs. Last, agencies should consider validation research within their agency before full adoption of a general risk assessment tool to quell concerns about the use and accuracy of a tool for special populations like juvenile sex offenders.

Quantitative Metric and Automatic Auditing Program (QMAP) Can be Applied to Regional Sites to Improve Quality and Consistency

Timely completion of each critical clinical task with quality improves the overall safety, quality and efficiency of Radiation Therapy. In our department, an internally developed software system that allows quantitative metric and automatic measurement (QMAP) based on data in the clinical Record and Verify (R&V) system has been used to help manage the processes. The purpose of this work is to report our experience applying this system to newly integrated regional radiation oncology systems. Two regional radiation oncology departments were integrated into the enterprise Planning and R&V system in March 2016. As of May 2017, the two departments fully adopted the QMAP system for workflow management. We study two important metrics monitored by QMAP, table position overrides during delivery and late plan completion. The table position override monitoring started in March 2016. The late plan completion and alerting commenced in May 2017. We studied the respective changes of the override rates and late plan completion rates shortly after the adoption of the QMAP system. Table position overrides in the longitudinal, lateral and vertical directions for any external beam treatment after the first fraction were recorded. Patients whose simulation were completed at least 2 business days before the scheduled treatment were included in the study of late plan completion. Plans ready for therapists to check less than four business hours before the scheduled treatment were considered late. Fisher’s exact test was used for statistical analysis. After the adoption of each QMAP feature by the two departments, root causes for delinquency of each metric were identified and systematic corrective measures were adopted, resulting substantial and sustainable improvement in the respective metrics. From May 2016 to October 2016, 9107 fractions were delivered, with 613 table position overrides. From June 2017 to Nov 2017, 7446 fractions were delivered, with 248 overrides. Override rate was reduced from 6.7% to 3.3% (p<0.01). In the third and fourth quarters of 2017, 97 and 101 plans were completed in the departments, with 27 and 10 plans being late, respectively. The late chart rate was reduced from 27.8% to 9.9% in these two quarters following the integration (p<0.01). QMAP provides an effective means to identify the bottlenecks in the department workflow. It is effective in reducing override and late plan completion shortly after integration into a regional campus’s workflow, which should help the safety and quality of radiation treatments.

An Investigation of Drug-Drug Interaction Alert Overrides at a Pediatric Hospital.

Drug-drug interactions (DDIs) can result in patient harm. DDI alerts are intended to help prevent harm; when the majority of alerts presented to providers are being overridden, their value is diminished. Our objective was to evaluate the overall rates of DDI alert overrides and how rates varied by specialty, clinician type, and patient complexity. A retrospective study of DDI alert overrides that occurred during 2012 and 2013 within the inpatient setting described at the medication-, hospital-, provider-, and patient encounter-specific levels was performed at an urban, quaternary-care, pediatric hospital. There were >41 000 DDI alerts presented to clinicians; ∼90% were overridden. The 5 DDI pairs that were most frequently presented and overridden included the following: potassium chloride-spironolactone, methadone-ondansetron, ketorolac-ibuprofen, cyclosporine-fluconazole, and potassium chloride-enalapril, each with an alert override rate of ≥0.89. Override rates across provider groups ranged between 0.84 and 0.97. In general, patients with high complexity had a higher frequency of alert overrides, but the rates of alert overrides for each DDI pairing did not differ significantly. High rates of DDI alert overrides occur across medications, provider groups, and patient encounters. Methods to decrease DDI alerts which are likely to be overridden exist, but it is also clear that more robust and intelligent tools are needed. Characteristics exist at the medication, hospital, provider, and patient levels that can be used to help specialize and enhance information transmission.

Evaluation of 'Definite' Anaphylaxis Drug Allergy Alert Overrides in Inpatient and Outpatient Settings.

Drug-allergy interaction (DAI) alerts are generated when a known adverse sensitivity-inducing substance is prescribed. A recent study at our institution showed that providers overrode most DAI alerts, including those that warned against potentially life-threatening 'anaphylaxis'. The aim of this study was to determine the rate of anaphylaxis overrides, the reasons for these overrides, whether the overrides were appropriate, and if harm occurred from overrides. All DAI alerts, with a reaction of 'anaphylaxis', were analysed for inpatients and outpatients within our health system between January 2009 and December 2011. Only alerts that were triggered by 'definite' alerts (i.e. same ordered medication as documented allergen) were included. Patient charts were reviewed to assess the appropriateness of overrides and potential harm, according to a predetermined set of criteria. A total of 202 inpatient and 16 outpatient alerts met the inclusion criteria. The rate of overrides for 'definite' anaphylaxis DAI alerts was high (inpatient: n=93, 46.0%; outpatient: n=11, 68.8%) but appropriate for most overrides in the inpatient (n=78, 83.9%) and outpatient settings (n=11, 100%). The most common override reasons in the inpatient and outpatient settings were 'administer per desensitization protocol' (n=64, 31.7%) and 'patient does not have this allergy' (n=7, 63.6%), respectively. No harm was associated with overrides in either setting, particularly because many medications were not administered. Overrides of 'definite' anaphylaxis DAI alerts were common and often appropriate. Most overrides were due to desensitizations. Allergy reconciliation for patients could further improve critical decision support.

Medication-related clinical decision support alert overrides in inpatients.

To define the types and numbers of inpatient clinical decision support alerts, measure the frequency with which they are overridden, and describe providers' reasons for overriding them and the appropriateness of those reasons. We conducted a cross-sectional study of medication-related clinical decision support alerts over a 3-year period at a 793-bed tertiary-care teaching institution. We measured the rate of alert overrides, the rate of overrides by alert type, the reasons cited for overrides, and the appropriateness of those reasons. Overall, 73.3% of patient allergy, drug-drug interaction, and duplicate drug alerts were overridden, though the rate of overrides varied by alert type (P < .0001). About 60% of overrides were appropriate, and that proportion also varied by alert type (P < .0001). Few overrides of renal- (2.2%) or age-based (26.4%) medication substitutions were appropriate, while most duplicate drug (98%), patient allergy (96.5%), and formulary substitution (82.5%) alerts were appropriate. Despite warnings of potential significant harm, certain categories of alert overrides were inappropriate >75% of the time. The vast majority of duplicate drug, patient allergy, and formulary substitution alerts were appropriate, suggesting that these categories of alerts might be good targets for refinement to reduce alert fatigue. Almost three-quarters of alerts were overridden, and 40% of the overrides were not appropriate. Future research should optimize alert types and frequencies to increase their clinical relevance, reducing alert fatigue so that important alerts are not inappropriately overridden.

Understanding physicians' behavior toward alerts about nephrotoxic medications in outpatients: a cross-sectional analysis.

BackgroundAlthough most outpatients are relatively healthy, many have chronic renal insufficiency, and high override rates for suggestions on renal dosing have been observed. To better understand the override of renal dosing alerts in an outpatient setting, we conducted a study to evaluate which patients were more frequently prescribed contraindicated medications, to assess providers’ responses to suggestions, and to examine the drugs involved and the reasons for overrides.MethodsWe obtained data on renal alert overrides and the coded reasons for overrides cited by providers at the time of prescription from outpatient clinics and ambulatory hospital-based practices at a large academic health care center over a period of 3 years, from January 2009 to December 2011. For detailed chart review, a group of 6 trained clinicians developed the appropriateness criteria with excellent inter-rater reliability (κ = 0.93). We stratified providers by override frequency and then drew samples from the high- and low-frequency groups. We measured the rate of total overrides, rate of appropriate overrides, medications overridden, and the reason(s) for override.ResultsA total of 4120 renal alerts were triggered by 584 prescribers in the study period, among which 78.2% (3,221) were overridden. Almost half of the alerts were triggered by 40 providers and one-third was triggered by high-frequency overriders. The appropriateness rates were fairly similar, at 28.4% and 31.6% for high- and low-frequency overriders, respectively. Metformin, glyburide, hydrochlorothiazide, and nitrofurantoin were the most common drugs overridden. Physicians’ appropriateness rates were higher than the rates for nurse practitioners (32.9% vs. 22.1%). Physicians with low frequency override rates had higher levels of appropriateness for metformin than the high frequency overriders (P = 0.005).ConclusionA small number of providers accounted for a large fraction of overrides, as was the case with a small number of drugs. These data suggest that a focused intervention targeting primarily these providers and medications has the potential to improve medication safety.

Overrides of medication-related clinical decision support alerts in outpatients

Electronic prescribing is increasingly used, in part because of government incentives for its use. Many of its benefits come from clinical decision support (CDS), but often too many alerts are displayed, resulting in alert fatigue. To characterize the override rates for medication-related CDS alerts in the outpatient setting, the reasons cited for overrides at the time of prescribing, and the appropriateness of overrides. We measured CDS alert override rates and the coded reasons for overrides cited by providers at the time of prescribing. Our primary outcome was the rate of CDS alert overrides; our secondary outcomes were the rate of overrides by alert type, reasons cited for overrides at the time of prescribing, and override appropriateness for a subset of 600 alert overrides. Through detailed chart reviews of alert override cases, and selective literature review, we developed appropriateness criteria for each alert type, which were modified iteratively as necessary until consensus was reached on all criteria. We reviewed 157,483 CDS alerts (7.9% alert rate) on 2,004,069 medication orders during the study period. 82,889 (52.6%) of alerts were overridden. The most common alerts were duplicate drug (33.1%), patient allergy (16.8%), and drug-drug interactions (15.8%). The most likely alerts to be overridden were formulary substitutions (85.0%), age-based recommendations (79.0%), renal recommendations (78.0%), and patient allergies (77.4%). An average of 53% of overrides were classified as appropriate, and rates of appropriateness varied by alert type (p<0.0001) from 12% for renal recommendations to 92% for patient allergies. About half of CDS alerts were overridden by providers and about half of the overrides were classified as appropriate, but the likelihood of overriding an alert varied widely by alert type. Refinement of these alerts has the potential to improve the relevance of alerts and reduce alert fatigue.

Clinical decision support for atypical orders: detection and warning of atypical medication orders submitted to a computerized provider order entry system

The specificity of medication-related alerts must be improved to overcome the pernicious effects of alert fatigue. A systematic comparison of new drug orders to historical orders could improve alert specificity and relevance. Using historical order data from a computerized provider order entry system, we alerted physicians to atypical orders during the prescribing of five medications: calcium, clopidogrel, heparin, magnesium, and potassium. The percentage of atypical orders placed for these five medications decreased during the 92 days the alerts were active when compared to the same period in the previous year (from 0.81% to 0.53%; p=0.015). Some atypical orders were appropriate. Fifty of the 68 atypical order alerts were over-ridden (74%). However, the over-ride rate is misleading because 28 of the atypical medication orders (41%) were changed. Atypical order alerts were relatively few, identified problems with frequencies as well as doses, and had a higher specificity than dose check alerts.

Evaluation of medication alerts in electronic health records for compliance with human factors principles

Increasing the adoption of electronic health records (EHRs) with integrated clinical decision support (CDS) is a key initiative of the current US healthcare administration. High over-ride rates of CDS alerts strongly limit these potential benefits. As a result, EHR designers aspire to improve alert design to achieve better acceptance rates. In this study, we evaluated drug-drug interaction (DDI) alerts generated in EHRs and compared them for compliance with human factors principles. We utilized a previously validated questionnaire, the I-MeDeSA, to assess compliance with nine human factors principles of DDI alerts generated in 14 EHRs. Two reviewers independently assigned scores evaluating the human factors characteristics of each EHR. Rankings were assigned based on these scores and recommendations for appropriate alert design were derived. The 14 EHRs evaluated in this study received scores ranging from 8 to 18.33, with a maximum possible score of 26. Cohen's κ (κ=0.86) reflected excellent agreement among reviewers. The six vendor products tied for second and third place rankings, while the top system and bottom five systems were home-grown products. The most common weaknesses included the absence of characteristics such as alert prioritization, clear and concise alert messages indicating interacting drugs, actions for clinical management, and a statement indicating the consequences of over-riding the alert. We provided detailed analyses of the human factors principles which were assessed and described our recommendations for effective alert design. Future studies should assess whether adherence to these recommendations can improve alert acceptance.

Bounds for Rating Override Rates

Overrides of credit ratings are important correctives of ratings that are determined by statistical rating models. Financial institutions and banking regulators agree on this because on the one hand errors with ratings of corporates or banks can have fatal consequences for the lending institutions and on the other hand errors by statistical methods can be minimised but not completely avoided. Nonetheless, rating overrides can be misused in order to conceal the real riskiness of borrowers or even entire portfolios. That is why rating overrides usually are strictly governed and carefully recorded. It is not clear, however, which frequency of overrides is appropriate for a given rating model within a predefined time period. This paper argues that there is a natural error rate associated with a statistical rating model that may be used to inform assessment of whether or not an observed override rate is adequate. The natural error rate is closely related to the rating model's discriminatory power and can readily be calculated.

Vetoes, Bargaining, and Boundary Conditions

“Testing Theories of Congressional-Presidential Interaction with Veto Override Rates” (henceforth “Veto Override Rates”) offers several tests of two models of vetoes and finds the models wanting. The paper concludes that something is seriously amiss with the models. In my view, the problem lies not in the models but in the tests. Understanding why the tests miss the mark is helpful in understanding models of veto politics, and more generally in thinking about testing strategies when multiple models analyze different causal mechanisms that hold under different circumstances. I should note immediately that the effort in the paper to think hard about override rates is admirable; it simply does not go far enough.

Testing Theories of Congressional—Presidential Interaction with Veto Override Rates

The ability to veto legislation is the most important formal power of the presidency in the legislative process, and presidents' veto behavior has thus attracted a great deal of theoretical interest. Game-theoretic models of congressional—presidential interactions explain vetoes by incorporating incomplete information over the distribution of presidential or congressional types. In this article, I examine two prominent such theories, as well as a simple “noisy” extension of a complete information theory. I show that each makes strong predictions not only about vetoes themselves but also about the resulting override votes; given that overrides are so closely connected with vetoes, any valid theory of the latter must be able to successfully explain the former. I test these predictions empirically and show that support for each of these theories in presidential veto and override data from 1973 to 2008 is quite weak. This negative result suggests that current models of the veto are incomplete; I sketch some possibilities for extension in the conclusion.

Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems--I-MeDeSA.

Medication-related decision support can reduce the frequency of preventable adverse drug events. However, the design of current medication alerts often results in alert fatigue and high over-ride rates, thus reducing any potential benefits. The authors previously reviewed human-factors principles for relevance to medication-related decision support alerts. In this study, instrument items were developed for assessing the appropriate implementation of these human-factors principles in drug-drug interaction (DDI) alerts. User feedback regarding nine electronic medical records was considered during the development process. Content validity, construct validity through correlation analysis, and inter-rater reliability were assessed. The final version of the instrument included 26 items associated with nine human-factors principles. Content validation on three systems resulted in the addition of one principle (Corrective Actions) to the instrument and the elimination of eight items. Additionally, the wording of eight items was altered. Correlation analysis suggests a direct relationship between system age and performance of DDI alerts (p=0.0016). Inter-rater reliability indicated substantial agreement between raters (κ=0.764). The authors developed and gathered preliminary evidence for the validity of an instrument that measures the appropriate use of human-factors principles in the design and display of DDI alerts. Designers of DDI alerts may use the instrument to improve usability and increase user acceptance of medication alerts, and organizations selecting an electronic medical record may find the instrument helpful in meeting their clinicians' usability needs.

Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs' Computerized Patient Record System

ObjectiveTo measure critical order check override rates in VA Puget Sound Health Care System's computerized practitioner order entry (CPOE) system and to compare 2006 results to a similar 2001 study. DesignAnalysis of ordering and order check data gathered by a post-hoc logging program. Use of Pearson's chi-square contingency table test comparing results from this study and the earlier study. MeasurementsFactors measured were total number of orders, frequency of order check types, frequency of order check overrides by order check type and comparisons of these results with previous results. ResultsA total of 37,040 orders generated 908 (2.5%) critical order checks. Drug-drug critical alert override rate was 74/85 (87%) in 2006 compared to 95/108 (88%) in 2001 (X2=0.04, df=1, p=0.85). The drug-allergy override rate was 341/420 (81%) compared to 72/105 (69%) in 2001 (X2=7.97, df=1, p=0.005). In 2001, 0.25% (105/42,621) orders generated a drug-allergy order check compared to 1.13% (420/37,040) in 2006 (X2=238.45, df=1, p<0.0001). ConclusionOverride rates of critical drug-drug and drug-allergy order checks remain high at VA Puget Sound Health Care System including significant increases in drug-allergy order checks. We recommend that monitoring override rates be regular practice in clinical computing systems and conclude that qualitative research should be carried out to better understand how physicians interact with decision support at the point of ordering.