Healthcare Improvement Initiatives Research Articles

Improving VTE prophylaxis in ward and ICU surgical urology patients: a Six-Sigma DMAIC methodology improvement project

PurposeThis study intends to improve the quality of venous thromboembolism (VTE) prophylaxis practices including proper VTE risk assessment and the appropriate prophylaxis measures for surgical urology patients.Design/methodology/approachThe authors applied the Six-Sigma define, measure, analyze, improve and control (DMAIC) improvement methodology in a pre–post interventional study that involved all adult patients above 18 years old indicated and scheduled for urology surgical interventions including endoscopic urological surgeries in a urology specialized 60-bed hospital. The pre-intervention sample included all patients meeting the inclusion criteria over a period of six months. Post-intervention sample included all patients meeting the inclusion criteria over a period of six months. The improvement areas included both the VTE risk assessment as well as the VTE prophylaxis prescription.FindingsDMAIC methodology has achieved a substantial sustained improvement in surgical urology VTE prophylaxis practices with an average of 70% on both levels; VTE risk assessment practices and VTE prophylaxis prescribing practices were statistically significant. The post-intervention results also showed a statistically controlled process with no special cause variations. Based on the study results, the Six-Sigma DMAIC methodology can be considered of high value when applied in healthcare clinical practice improvement projects.Research limitations/implicationsThe project study includes some pitfalls that can be addressed as follows: 1. The lack of VTE rate incidence tracking. This limitation can be partly refuted when the authors conduct a literature review and explore that the VTE prophylaxis effectiveness had been proven with sufficient evidence to an extent that pushed several scientific societies to develop their own guidelines to support VTE prophylaxis. (Algattas et al., 2018). 2. Another limitation of this study can be that it handled only surgical patients and more specifically surgical urology patients. Of course, VTE prophylaxis is a crucial life-threatening problem not only for the surgical admitted patients but also for all the medical admitted patients either in hospital wards or ICUs. However, the prediction that surgical patients especially surgical urology patients are more prone to VTE development risk as they have -in several cases-two or three main additive risk factors which are age, procedure duration and malignancy in elderly men. (Tikkinen et al., 2014). So, the authors consider the study project to be a prototype that hopefully can be utilized for future study projects that will manage both other surgical specialty patients and medical patients on the national level and can track accurately and effectively report the VTE incidence rates.Practical implicationsSeveral recommendations can be extracted from the research project that is summarized in the following points: Paying focused attention to continuous healthcare quality improvement initiatives and projects as a main approach for healthcare improvement especially for the public health-related problems. This might be achieved through periodic region-specific or specialty-specific focus groups from which public health problems could be addressed and prioritized to be considered as a part of country healthcare campaigns regarding cost-utility and feasibility studies. The adoption of a system thinking approach in dealing with the improvement strategies; all efforts and resources are to be employed to achieve a common objective. This includes the generation of a national-wide electronic health information system that can aid in healthcare resource allocation and direct the healthcare efforts towards the most important, high-priority public health problems. Electronic national-wide health record is really an effort, and resources consuming activity, but actually, it's worth exerting efforts, and its valuable outcomes may be seen several years later. 3. Development of unified national specialized VTE prophylaxis pathways to standardize the patient-specific VTE prophylaxis plans. Standardization of healthcare pathways enables healthcare professionals to follow an evidence-based practice which will be reflected on the improvement of healthcare quality level, cost-effectiveness enhancement, and timely patient care on all levels especially in high critical areas like ER and ICU. 4. Incorporation of VTE prophylaxis costs in the universal health insurance diagnosis-related group (DRG) insurance packages and service pricing. Universal health insurance is a nationwide strategy that is aiming to cover all Egypt residents by the year 2030. Universal health insurance is being following the DRG reimbursement policy that is thought to control all the healthcare-associated costs so, the VTE prophylaxis costs shall be added as the main cost item to encourage all healthcare facilities to follow an evidence-based VTE prophylaxis pathway taking into consideration the high-risk patient categories who will definitely represent a high-cost burden on the long run if they suffer a VTE event.Originality/valueDMAIC improvement methodology applications in healthcare are still relatively limited, especially on the clinical level. The study can be considered one of a kind in Egypt dealing with a comprehensive DMAIC methodology application on the clinical level.

The role of emergent champions in policy implementation for decentralised drug-resistant tuberculosis care in South Africa

ObjectiveChampions are recognised as important to driving organisational change in healthcare quality improvement initiatives in high-income settings. In low-income and middle-income countries with a high disease burden and constrained human...

Preferences of patients with palliative care needs and their families for engagement with service improvement work within the hospital setting: A qualitative study

Background: There is growing recognition of the importance of involving patients and families with lived experiences of illness in healthcare service quality improvement, research and implementation initiatives. Ensuring input from people with palliative care needs is important, but how to enable this is not well understood. Aim: To seek the perspectives of Australian patients with palliative care needs, and their family members, to elicit their views on how to best contribute to inpatient palliative care quality improvement initiatives. Design: An exploratory qualitative study, using semi-structured interviews. Setting/participants: Eligible participants were adult patients with palliative care needs receiving care within a hospital setting, and their family members. Recruitment occurred through: five hospitals in New South Wales, Australia; and snowballing. Results: Fifty participants took part (21 patients and 29 family members). Results confirmed four themes: (1) Mechanisms for providing feedback about care quality need to be supportive and individualised; (2) The clinician-patient/family power imbalance makes real time feedback challenging to provide; (3) Willingness to contribute varies according to diagnosis, timing and role and (4) Face to face feedback is preferred for health service improvement work. Conclusions: Enabling meaningful consumer input to quality improvement requires careful consideration due to the unique requirements of the palliative care population. Embedding tailored outcome and experience measures to inform real-time care provision coupled with focussed opportunities for input into service improvement may best foster improvements in inpatient palliative care, founded in what matters most for people requiring this care.

Do Pediatricians and Nurses Recommend Vaccines for Preterm Infants? A Survey in Italy

To assess the level of knowledge, attitudes, and behaviors regarding vaccination in preterm infants among primary care pediatricians (PCPs) and health care workers (HCWs) in neonatal intensive care units (NICUs). Data were collected from PCPs through a confidential questionnaire distributed by email, whereas the research team distributed a self-administered anonymous questionnaire to all HCWs working in the selected NICUs. Overall, 64.1% of HCWs consider vaccines in preterm infants to be very safe. The majority of HCWs (69.8%) stated that they always recommend that preterm infants' parents vaccinate their children following the same schedule as for term infants. This behavior was significantly more likely among those who know that the vaccination schedule for preterm infants is the same as for term infants, who consider vaccines in preterm infants very effective, and who strongly agree that preterm infants should be vaccinated on the same schedule as term infants. Moreover, PCPs were more likely than NICU pediatricians to always recommend that preterm infants' parents vaccinate their children following the immunization schedule of term infants, whereas this behavior was significantly less frequent among NICU nurses. There is a need for physicians and nurses in the NICU and in the community to counteract missed or delayed immunizations. Engagement of HCWs in healthcare quality improvement initiatives focused on the promotion of timely vaccinations in preterm infants should be encouraged.

A National Network of Safe Havens: Scottish Perspective.

For over a decade, Scotland has implemented and operationalized a system of Safe Havens, which provides secure analytics platforms for researchers to access linked, deidentified electronic health records (EHRs) while managing the risk of unauthorized reidentification. In this paper, a perspective is provided on the state-of-the-art Scottish Safe Haven network, including its evolution, to define the key activities required to scale the Scottish Safe Haven network’s capability to facilitate research and health care improvement initiatives. A set of processes related to EHR data and their delivery in Scotland have been discussed. An interview with each Safe Haven was conducted to understand their services in detail, as well as their commonalities. The results show how Safe Havens in Scotland have protected privacy while facilitating the reuse of the EHR data. This study provides a common definition of a Safe Haven and promotes a consistent understanding among the Scottish Safe Haven network and the clinical and academic research community. We conclude by identifying areas where efficiencies across the network can be made to meet the needs of population-level studies at scale.

Reliability and validity of an instrument to assess pediatric inpatients' experience of care in China.

BackgroundThe Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child HCAHPS) is a standard instrument to measure pediatric inpatients’ experience of care. Currently, no Chinese version of the Child HCAHPS exists for Chinese patients. Therefore, this study aimed to create a Chinese version of the Child HCAHPS and investigate its validity and reliability in a Chinese setting.MethodsUsing the approach recommended in guidelines from the Agency for Healthcare Research and Quality for translating HCAHPS surveys, we produced a Chinese version of the Child HCAHPS. A two-month field test with seven hospitals across five provinces in China was performed to assess its validity. Construct validity was assessed using confirmatory factor analysis. We evaluated convergent validity by factor loading, average variance extracted (AVE), and construct reliability (CR). Cronbach’s alpha and corrected item-total correlation (CITC) were used to reflect hospital-level unit reliabilities for the survey’s item composites. The correlation of the measure score with the overall rating was calculated to evaluate criterion validity.ResultsAn overall response rate of 63% was achieved, and 2,258 respondents completed the questionnaire. Confirmatory factor analysis showed a comparative fit index (CFI) of 0.905, a non-normed fix index of 0.886, and a root mean square error of approximation (RMSEA) of 0.089. Most items had factor loadings over 0.7. Cronbach’s alpha coefficient on the overall level was 0.981, and all measures’ CITC exceeded 0.6, demonstrating good to excellent hospital-level reliability of the composite and single-item measures. All composite measures had good to excellent internal consistency reliability (0.716 to 0.994). Item-to-composite correlation ranged from 0.510 to 0.997. Composite-to-composite correlations ranged from 0.488 to 0.997. According to the survey result, for all the 18 composite or single-item measures, mean top-box scores ranged from 56% (“Involving teens in care”) to 87% (“Informed in Emergency Room”).ConclusionsThe Chinese version of the Child HCAHPS demonstrated acceptable validity and reliability. The application of this tool can help benchmark ongoing healthcare improvement initiatives in China.

From kamishibai card to key card: a family-targeted quality improvement initiative to reduce paediatric central line-associated bloodstream infections

BackgroundCentral line-associated bloodstream infections (CLABSIs) are major contributors to preventable harm in the inpatient paediatric setting. Despite multiple guidelines to reduce CLABSI, sustaining reliable central line maintenance bundle compliance remains...

The influence of contextual factors on healthcare quality improvement initiatives: a realist review

BackgroundRecognising the influence of context and the context-sensitive nature of quality improvement (QI) interventions is crucial to implementing effective improvements and successfully replicating them in new settings, yet context is still poorly understood. To address this challenge, it is necessary to capture generalisable knowledge, first to understand which aspects of context are most important to QI and why, and secondly, to explore how these factors can be managed to support healthcare improvement, in terms of implementing successful improvement initiatives, achieving sustainability and scaling interventions. The research question was how and why does context influence quality improvement initiatives in healthcare?MethodsA realist review explored the contextual conditions that influence healthcare improvement. Realist methodology integrates theoretical understanding and stakeholder input with empirical research findings. The review aimed to identify and understand the role of context during the improvement cycle, i.e. planning, implementation, sustainability and transferability; and distil new knowledge to inform the design and development of context-sensitive QI initiatives. We developed a preliminary theory of the influence of context to arrive at a conceptual and theoretical framework.ResultsThirty-five studies were included in the review, demonstrating the interaction of key contextual factors across healthcare system levels during the improvement cycle. An evidence-based explanatory theoretical model is proposed to illustrate the interaction between contextual factors, system levels (macro, meso, micro) and the stages of the improvement journey. Findings indicate that the consideration of these contextual factors would enhance the design and delivery of improvement initiatives, across a range of improvement settings.ConclusionsThis is the first realist review of context in QI and contributes to a deeper understanding of how context influences quality improvement initiatives. The distillation of key contextual factors offers the potential to inform the design and development of context-sensitive interventions to enhance improvement initiatives and address the challenge of spread and sustainability. Future research should explore the application of our conceptual model to enhance improvement-planning processes.Systematic review registrationPROSPERO CRD42017062135

Continuing Professional Development through the lens of complexity science: Becoming agents of change in the healthcare system.

This article was migrated. The article was marked as recommended. Healthcare improvement initiatives have not led to expected changes in patient population outcomes over the last decade. This can, in part, be explained by overly reductionist approaches in medical practice and education. This article describes some insights and experiences using a Complex Adaptive Systems approach in the conceptualisation of an in-service training program and reflections on elements that may lead to enhanced practice behaviours and system improvements.

Adapting the Goal Attainment Approach for Major Depressive Disorder.

The need for patient-centered care has become a focal point of healthcare improvement initiatives. Shared decision making—in which patients and clinicians communicate about various treatment options and goals and patient input is considered when making treatment decisions—has been associated with improved health and quality of life. A method of treatment evaluation allowing incorporation of patient-specific goals and perspectives is of increasing interest to healthcare providers, payers, and patients. An approach that allows incorporation of shared goal setting is possible via use of an instrument called the Goal Attainment Scale (GAS). This scale provides the structure for measuring progress toward treatment goals set through patient–clinician collaboration. The goal attainment approach has been used as a primary outcomes measure in numerous studies but not in major depressive disorder (MDD). As MDD is a complex, multidimensional disorder affecting each patient differently, the use of GAS methodology is a relevant framework for setting personalized meaningful treatment goals. Initial research into the feasibility of using the GAS in MDD (GAS-D) to measure patient-centric outcomes that may be neglected when more traditional scales are used has been encouraging. The objective of this Commentary is to provide background and rationale for implementation of the GAS-D in clinical practice.Funding Takeda Pharmaceutical Company, Ltd., and Lundbeck LLC.

Evaluating the impact of a national clinical leadership fellow scheme

BackgroundThe drive towards engaging UK doctors in clinical leadership and management has involved a number of initiatives at various levels, including specific fellowships for doctors in training which enable them...

A survey of perioperative clinician’s knowledge and application of the law regarding the classification of deaths in the perioperative period

Background: Doctors complete death notification forms poorly, in South Africa and globally. This reduces accuracy of mortality statistics in South Africa, which is needed by policymakers and clinicians to focus on healthcare improvement initiatives. The law on procedure-related deaths was changed in 2008, directly informing on the practice of anaesthesiologists. Perioperative deaths continue to be misclassified, possibly related to doctors’ lack of knowledge or understanding of the law. This study aimed to determine if perioperative clinicians working in theatre and the Intensive Care Unit (ICU) knew and understood the law pertaining to deaths in the perioperative period. Method: A survey was conducted at eight hospitals in the Durban area amongst specialists and non-specialists who work in theatre and ICU. The questionnaire had three parts: definitions, medico-legal experiences and clinical scenarios. Results: Eighty responses were received, a response rate of 74%. More non-specialists responded (57%). When undecided on how to classify a death, clinicians mostly seek assistance from their private medico-legal insurance or a senior colleague. However, specialists are unable to define or classify unnatural and procedure-related deaths any better than non-specialists. Conclusion: Specialist and non-specialist knowledge and application of the law relating to unnatural deaths in the perioperative period is poor. Forensic pathologist assistance is underutilised. More needs be done to empower doctors on medico-legal issues affecting perioperative clinical practice.

Research paradigm that tackles the complexity of in situ care: video reflexivity

This issue of BMJ Quality & Safety presents a study conducted at the University of Michigan to evaluate ‘video reflexivity’ (VR, also referred to as VRE or ‘video-reflexive ethnography’) as...

The influence of contextual factors on healthcare quality improvement initiatives: what works, for whom and in what setting? Protocol for a realist review

BackgroundContext shapes the effectiveness of knowledge implementation and influences health improvement. Successful healthcare quality improvement (QI) initiatives frequently fail to transfer to different settings, with local contextual factors often cited as the cause. Understanding and overcoming contextual barriers is therefore crucial to implementing effective improvement; yet context is still poorly understood. There is a paucity of information on the mechanisms underlying how and why QI projects succeed or fail in given settings. A realist review of empirical studies of healthcare QI initiatives will be undertaken to examine the influence and impact of contextual factors on quality improvement in healthcare settings and explore whether QI initiatives can work in all contexts.MethodsThe review will explore which contextual factors are important, and how, why, when and for whom they are important, within varied settings. The dynamic nature of context and change over time will be explored by examining which aspects of context impact at key points in the improvement trajectory. The review will also consider the influence of context on improvement outcomes (provider- and patient-level), spread and sustainability. The review process will follow five iterative steps: (1) clarify scope, (2) search for evidence, (3) appraise primary studies and extract data, (4) synthesise evidence and draw conclusions and (5) disseminate findings. The reviewers will consult with experts and stakeholders in the early stages to focus the review and develop a programme theory consisting of explanatory ‘context–mechanism–outcome’ configurations. Searches for primary evidence will be conducted iteratively. Data will be extracted and tested against the programme theory. A review advisory group will oversee the review process. Review findings will follow RAMESES guidelines and will be disseminated via a report, presentations and peer-reviewed publications.DiscussionThe review will update and consolidate evidence on the contextual conditions for effective improvement and distil new knowledge to inform the design and development of context-sensitive QI initiatives. This review ties in with the study of improvement programmes as vehicles of change and the development of an evidence base around healthcare improvement by addressing whether QI initiatives can work in all contexts.Systematic review registrationPROSPERO CRD42017062135

How do stakeholders from multiple hierarchical levels of a large provincial health system define engagement? A qualitative study

BackgroundEngaging stakeholders from varied organizational levels is essential to successful healthcare quality improvement. However, engagement has been hard to achieve and to measure across diverse stakeholders. Further, current implementation science models provide little clarity about what engagement means, despite its importance.The aim of this study was to understand how stakeholders of healthcare improvement initiatives defined engagement.MethodsParticipants (n = 86) in this qualitative thematic study were purposively sampled for individual interviews. Participants included leaders, core members, frontline clinicians, support personnel, and other stakeholders of Strategic Clinical Networks in Alberta Health Services, a Canadian provincial health system with over 108,000 employees. We used an iterative thematic approach to analyze participants’ responses to the question, “How do you define engagement?”ResultsRegardless of their organizational role, participants defined engagement through three interrelated themes. First, engagement was active participation from willing and committed stakeholders, with levels that ranged from information sharing to full decision-making. Second, engagement centered on a shared decision-making process about meaningful change for everyone “around the table,” those who are most impacted. Third, engagement was two-way interactions that began early in the change process, where exchanges were respectful and all stakeholders felt heard and understood.ConclusionsThis study highlights the commonalities of how stakeholders in a large healthcare system defined engagement—a shared understanding and terminology—to guide and improve stakeholder engagement. Overall, engagement was an active and committed decision-making about a meaningful problem through respectful interactions and dialog where everyone’s voice is considered. Our results may be used in conjunction with current implementation models to provide clarity about what engagement means and how to engage various stakeholders.

The Well Organised Working Environment: A mixed methods study

BackgroundThe English National Health Service Institute for Innovation and Improvement designed a series of programmes called The Productive Series. These are innovations designed to help healthcare staff reduce inefficiency and improve quality, and have been implemented in healthcare organisations in at least 14 different countries. This paper examines an implementation of the first module of the Productive Community Services programme called ‘The Well Organised Working Environment’. ObjectiveThe quantitative component aims to identify the quantitative outcomes and impact of the implementation of the Well Organised Working Environment module. The qualitative component aims to describe the contexts, mechanisms and outcomes evident during the implementation, and to consider the implication of these findings for healthcare staff, commissioners and implementation teams. DesignMixed methods explanatory sequential design. SettingsCommunity Healthcare Organisation in East Anglia, England. ParticipantsFor the quantitative data, participants were 73 staff members that completed End of Module Assessments. Data from 25 services that carried out an inventory of stock items stored were also analysed. For the qualitative element, participants were 45 staff members working in the organisation during the implementation, and four members of the Productive Community Services Implementation Team. MethodsStaff completed assessments at the end of the module implementation, and the value of items stored by clinical services was recorded. After the programme concluded, semi-structured interviews with staff and a focus group with members of the Productive Community Services implementation team were analysed using Framework Analysis employing the principles of Realist Evaluation. Results62.5% respondents (n=45) to the module assessment reported an improvement in their working environment, 37.5% (n=27) reported that their working environment stayed the same or deteriorated. The reduction of the value of items stored by services ranged from £4 to £5039 across different services. Results of the qualitative analysis suggests explanations for why the programme worked in some contexts and not others, for instance due to varying levels of management support, and varying levels of resources allocated to carrying out or sustaining the improvement work. ConclusionsQuantitative analysis of data generated during healthcare improvement initiatives can give an impression of the benefits realised, but additional qualitative analysis also provides opportunity for learning to improve future implementations. Targets set by commissioners for innovation should focus on sustaining improvement rather demonstrating one-off benefits, and implementation teams should not let their preconceptions of what will and what will not work prevent them from trying interventions that may benefit staff.

Abstract 139: Building Sustainability: Creating Structure for Improved Heart Failure Patient Outcomes

Background: As heart failure (HF) prevalence continues to climb, patient re-hospitalizations challenge collective healthcare resources across the US. Many centers have implemented care transition strategies to reduce 30-day readmissions, but impacting 90-day readmissions and sustaining those positive outcomes remain daunting. Aim: Create and implement a programmatic structure to embed a care transitions approach to HF patient care at a tertiary care medical center, sustaining reduced patient readmissions and improving the patient care experience. Methods: To effect change in our evidence-based HF care processes, we adopted a multidisciplinary team approach to develop, implement and monitor patient care processes. Post-discharge phone calls to patients proved a particularly impactful intervention. Weekly internal meetings and monthly meetings with patient and community partner participation were implemented, in addition to ongoing, multimodal communication to the frontline care team to share strategies and outcomes data. Actively engaging patients in care improvement efforts is critical to understanding and meeting their needs and inpatient interviews with empathy mapping was used. Patients participated in medication safety workshops and provided feedback on educational tools that were developed. EMR-embedded tools were created to guide workflows and facilitate analytics (outcomes/balancing measures, process measures, automated risk assessment). Routine team review of post-discharge coding, individual patient readmissions and evaluating program impact on care delivery costs were instituted. Results: 12 months of data both pre- and post- program implementation, all-cause 90-day readmissions declined by 23% from baseline 31% (160 of 524) to 24% (78 of 327). HF patient subjective reporting of feeling prepared to manage self-care post discharge improved by 6%, while understanding of medications improved by 5%. Conclusion: Impacting HF readmissions remains challenging. In executing healthcare quality improvement initiatives, implementing programmatic structure in addition to interventions is essential to sustain positive results with manageable resources. Involving the multidisciplinary team, developing community partnerships and engaging patients are key.

Model-driven approach to data collection and reporting for quality improvement

Continuous data collection and analysis have been shown essential to achieving improvement in healthcare. However, the data required for local improvement initiatives are often not readily available from hospital Electronic Health Record (EHR) systems or not routinely collected. Furthermore, improvement teams are often restricted in time and funding thus requiring inexpensive and rapid tools to support their work. Hence, the informatics challenge in healthcare local improvement initiatives consists of providing a mechanism for rapid modelling of the local domain by non-informatics experts, including performance metric definitions, and grounded in established improvement techniques. We investigate the feasibility of a model-driven software approach to address this challenge, whereby an improvement model designed by a team is used to automatically generate required electronic data collection instruments and reporting tools. To that goal, we have designed a generic Improvement Data Model (IDM) to capture the data items and quality measures relevant to the project, and constructed Web Improvement Support in Healthcare (WISH), a prototype tool that takes user-generated IDM models and creates a data schema, data collection web interfaces, and a set of live reports, based on Statistical Process Control (SPC) for use by improvement teams. The software has been successfully used in over 50 improvement projects, with more than 700 users. We present in detail the experiences of one of those initiatives, Chronic Obstructive Pulmonary Disease project in Northwest London hospitals. The specific challenges of improvement in healthcare are analysed and the benefits and limitations of the approach are discussed.

Collaboration, capacity building and co-creation as a new mantra in global health

Interest in global health innovation and healthcare quality improvement has increased among medical professionals worldwide. Over the past 35 years, participation in global health activities among medical graduates in the USA has increased more than 10-fold [1], with similar trends in other countries [2, 3]. This increased interest is not confined to physicians, as evidenced by the ongoing creation of new and professionally diverse degree programs in global health and development in many universities. During this time, the advent of an increased ease of communication and idea sharing across the globe has occurred in conjunction with relative ease of travel to create a technological leap forward that has been called ‘the most transformative single event in development work of our generation [4]’. Despite this interest, efforts to improve health in low-and-middle-income countries (LMIC), while well intentioned, have fallen short in several areas. First, initiatives are often ad hoc , with entrepreneurial stakeholders arriving for a brief tour of an LMIC, providing short-term care, training or a new technology, and then departing without having created a sustainable human and material infrastructure. These healthcare improvement initiatives, developed in the donor country context, may fail to encompass the unique needs of the LMIC population and critical social, cultural and political context within the LMIC. Such an approach emphasizes the strengths and focus of the donor organization instead of the real needs and capacity of the LMIC. At a minimum, this may waste resources. At its worst, …

Recommendations for Evaluation of Health Care Improvement Initiatives

Intensive efforts are underway across the world to improve the quality of health care. It is important to use evaluation methods to identify improvement efforts that work well before they are replicated across a broad range of contexts. Evaluation methods need to provide an understanding of why an improvement initiative has or has not worked and how it can be improved in the future. However, improvement initiatives are complex, and evaluation is not always well aligned with the intent and maturity of the intervention, thus limiting the applicability of the results. We describe how initiatives can be grouped into 1 of 3 improvement phases—innovation, testing, and scale-up and spread—depending on the degree of belief in the associated interventions. We describe how many evaluation approaches often lead to a finding of no effect, consistent with what has been termed Rossi’s Iron Law of Evaluation. Alternatively, we recommend that the guiding question of evaluation in health care improvement be, “How and in what contexts does a new model work or can be amended to work?” To answer this, we argue for the adoption of formative, theory-driven evaluation. Specifically, evaluations start by identifying a program theory that comprises execution and content theories. These theories should be revised as the initiative develops by applying a rapid-cycle evaluation approach, in which evaluation findings are fed back to the initiative leaders on a regular basis. We describe such evaluation strategies, accounting for the phase of improvement as well as the context and setting in which the improvement concept is being deployed. Finally, we challenge the improvement and evaluation communities to come together to refine the specific methods required so as to avoid the trap of Rossi’s Iron Law.