Factors that might undermine the validity of patient and multi-source feedback

Abstract

Multi-source feedback (MSF) and patient feedback (PF) are used increasingly around the world to assess and quality-assure clinical practice. However, concerns about the evidence for their utility pertain to their ability to identify poor performance, the impact of allowing assessees to select their own assessors and the many confounders that may undermine validity. This study was conducted in conjunction with the National Clinical Assessment Service (NCAS) in the UK and used established MSF and PF instruments to assess doctors in potential difficulty. Multi-source feedback assessors were nominated by both the practitioner (Pnom) and the referring body (RBnom). Demographics were collected to elucidate any differences found. Ratings generated by MSF and PF were compared with one another and with findings of a previous study that provided a normative cohort. Using MSF, NCAS-assessed doctors scored significantly lower than the reference cohort. Nineteen (28%) NCAS-assessed doctors achieved scores that were less than satisfactory. This rose to 50% when only RBnom assessors were used. Overall, ratings awarded by RBnom assessors were significantly lower than those awarded by Pnom assessors. Collected demographics did not help to explain the difference. Only one NCAS-assessed doctor scored below average according to PF. Doctors in the NCAS-assessed group did not score significantly lower than the reference cohort in PF. Doctor assessment scores awarded by patients were significantly higher than those awarded by colleagues. Although colleagues appear to report poor performance using MSF, patients fail to report concurrent findings. This challenges the validity of PF as it is currently constructed. Scores in MSF differ significantly depending on whether they are practitioner- or third party-nominated. Previously recognised confounding factors do not help to explain this difference.