Assessing the performance of ongoing national indicators on healthcare quality and safety: Development of the quid method and results on ten indicators

Abstract

Introduction Since ten years, quality indicators (QI) are mandatorily measured in all French hospitals. They are used for hospital quality improvement, public disclosure and regulation goals. After a decade of use, the Ministry of Health and the French National Authority for Health set up a task force to evaluate these QI. The goal was to provide a decision aid to the Ministry and HAS in terms of suppressing or maintaining their use, or modifying their content. As there was no available integrated tool, the task force started with the development and test of a tool during the year 2015. This tool aimed at evaluating all components of their performance (measurement feasibility, metrological performance and relevance for users) of QIs according to each of the three goals. The objective was to define an operational and structured method allowing the overall appraisal of ongoing national QI in healthcare, to apply it on ten indicators and to study its impact. Methods The national expert workgroup was constituted by clinicians, epidemiologists, healthcare facility managers, quality managers, sociologists and a patients’ representative: step 1: a search for similar experiences in other countries; step 2: a literature review to identify relevant assessment criteria and methods; step 3: a consensus method between the members of the group to select the criteria: Using a modified Rand/UCLA Appropriateness Method, the appropriateness of these criteria was checked for evaluating the indicators according to each of the three goals. The group extracted a list of criteria for each goal. Each criterion was assessed using a quantitative approach or a qualitative approach; step 4: test and implementation of the method on 10 national QI to identify operational issues in the implementation of the method: The indicator's results collected the previous years were analyzed; the scientific soundness of the indicator development was reviewed; field hospital workers (doctors, nurses, managers), health authorities employees and patients were interviewed. All these data results were discussed by the task force, summarized and displayed on spider charts presenting standardized scores. This integrated tool was tested on ten national process QIs; step 5: assessment criteria of impact of the work: implementation of the recommendations by the Ministry and HAS. Results Twelve were selected as appropriate for the evaluation of QIs for regulation use: relevance (importance of the quality characteristic captured for the healthcare system; benefit/ability to take decision based on the results; potential risks/side effects), feasibility (understandability and Interpretability of the indicator and its results; barriers to implementation due to data collection effort; delays related to data production), scientific soundness of the indicator development (indicator evidence; validity; risk adjustment) and current metrological performance (indicator expression can be influenced by providers; discriminatory power; dynamics of change). Among these, 11 were selected for hospital improvement and 7 for public disclosure. The tool was applied on 10 process QIs. We proposed to suppress four of them and to modify or suspend the six others. The four QIs were suppressed, and the others have been or are currently in a process of modification. Conclusion By combining qualitative and quantitative evaluations, the QUID instrument offers a solid method for thoroughly appraising each indicator. The value of the QUID method was supported by the clear-cut conclusions on 10 QIs and subsequently by the decisions of the Ministry and the HAS to follow the recommendations. More research is needed to finalize the tool on other types of indicators.