A german-language competency-based multisource feedback instrument for residents: development and validity evidence

Eva K Hennel,Sigrid Harendza,Sören Huwendiek,Christoph Berendonk,Ulrike Subotic,Daniel Stricker

doi:10.1186/s12909-020-02259-2

Abstract

BackgroundIn medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.MethodsWe developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, “unable to comment” ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents’ learning goals and the progress as reported via MSF.ResultsOur resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician’s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach’s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.ConclusionsTo support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

Highlights

In medical settings, multisource feedback (MSF) is a recognised method of formative assessment
To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence
All original items of the MSF questionnaire in German (MSF-RG) are presented in their original order

Summary

Introduction

Multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Feedback can effectively support medical training by making individuals aware of gaps in knowledge or insufficient skills [4,5,6] and guide learning. There are several methods of workplace-based assessment: Mini-CEX [7] and DOPS [8] are two examples which focus on one occasion with a patient encounter or one skill rather than on the overarching performance. MSF can promote medical training in the long term by providing regular feedback and supporting the formulation of individual learning goals [6, 11]

Objectives

Methods

Results

Conclusion