Review article: What makes a good healthcare quality indicator? A systematic review and validation study

Abstract

Indicators measuring aspects of performance to assess quality of care are often chosen arbitrarily. The present study aimed to determine what should be considered when selecting healthcare quality indicators, particularly focusing on the application to emergency medicine. Structured searches of electronic databases were supplemented by website searches of quality of care and benchmarking organisations, citation searches and discussions with experts. Candidate attributes of 'good' healthcare indicators were extracted independently by two authors. The validity of each attribute was independently assessed by 16 experts in quality of care and emergency medicine. Valid and reliable attributes were included in a critical appraisal tool for healthcare quality indicators, which was piloted by emergency medicine specialists. Twenty-three attributes were identified, and all were rated moderate to extremely important by an expert panel. The reliability was high: alpha = 0.98. Twelve existing tools explicitly stated a median (range) of 14 (8-17) attributes. A critical appraisal tool incorporating all the attributes was developed. This was piloted by four emergency medicine specialists who were asked to appraise and rank a set of six candidate indicators. Although using the tool took more time than implicit gestalt decision making: median (interquartile range) 190 (43-352) min versus 17.5 (3-34) min, their rankings changed after using the tool. To inform the appraisal of quality improvement indicators for emergency medicine, a comprehensive list of indicator attributes was identified, validated, developed into a tool and piloted. Although expert consensus is still required, this tool provides an explicit basis for discussions around indicator selection.