Do assessor comments on a multi-source feedback instrument provide learner-centred feedback?

Abstract

Free-text comments in multi-source feedback are intended to facilitate change in the assessee's practice. This study was designed to utilise a large dataset of free-text comments obtained in a national pilot study in order to investigate how helpful these free-text comments may be to assessees. We investigated: (i) which areas of performance are usually addressed by free-text comments; (ii) to what extent assessors' (doctors, nurses, allied health professionals and clerical or managerial staff) comments correspond to assessees' (career-grade doctors) self-assessments, and (iii) whether the comments contain specific behavioural evidence and suggestions for change. Initially comments were read through to identify commonly recurring themes. A strong theme was 'respondent-centredness', which refers to the extent to which comments focus on issues that are of value to the assessor rather than to the assessee's personal development. In response to this, the data were re-evaluated against predefined research questions to assess how constructive comments were for the assessee's personal development. Of 11,483 assessor forms, 4777 (42%) included free-text comments. A total of 513 forms contained at least one below average score and 286 (56%) of these forms contained the assessor's free-text feedback. Free-text comments were mostly rater-centred and addressed the effect of the assessee on the colleague's working life rather than areas of relevance to the assessee's personal development. A total of 1806 assessor/assessee pairs of comments were compared; most demonstrated clear differences of opinion or interpretation. Reliability and supportiveness were over-represented; clinical performance and personal development were under-represented. The comments were unlikely to provide specific behavioural evidence or to address how change might be initiated. Our data indicate that, in their current form, the overwhelming majority of free-text comments add little to facilitate improvement in assessees' personal development and performance.