Multisource Feedback Process Research Articles

Patient involvement in assessment of postgraduate medical learners: A scoping review.

Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study

ObjectivesTo examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.DesignQualitative text analysis...

Optimizing Multisource Feedback Implementation for Australasian Physicians.

Medical regulatory bodies in Australasia are encouraging greater use of continuing professional development activities, such as multisource feedback (MSF), which are practice-based, include facilitated feedback, and improve performance. The aim of this study was to explore the feasibility, effectiveness, and sustainability of an MSF process that includes a telephone/videoconference debrief, to better design future MSF implementation. Thirty-seven Australasian physician participants sought feedback from patients and colleagues and debriefed their feedback report with a trained facilitator. The impact was evaluated using quantitative and qualitative measures including surveys and semistructured interviews. The feasibility of the MSF process was impacted by the level of support for the participant to complete the MSF, the nature and location of participants' work, and the use of telephone/videoconference to deliver the debrief. Regarding effectiveness, the MSF stimulated reflection on practice and action on areas identified for improvement. In addition, the quality of rater feedback and the inclusion of a debrief impacted participant and facilitator reports of effectiveness. The absence of a culture of feedback, the formative nature of the MSF, and the confidentiality of the results were factors impacting sustainability. Optimizing MSF is important to encourage broad uptake in the wider medical community in Australasia. Although several factors were identified as having an impact, it is clear that inclusion of a quality debrief will increase the perceived value and the effectiveness of MSF. Delivering that debrief through telephone/videoconference can be effective and will increase the overall feasibility and sustainability.

Improving the utility of multisource feedback for medical consultants in a tertiary hospital: a study of the psychometric properties of a survey tool.

Objective The aim of this study was to investigate the psychometric properties of a multisource review survey tool for medical consultants in an Australian health care setting. Methods Two sets of survey data from a convenience sample of medical consultants were analysed using SPSS, comprising self-assessment data from 73 consultants and data from 734 peer reviewers. The 20-question survey consisted of three subscales, plus an additional global question for reviewers. Analysis included the reliability coefficient (α) of the scale and the three subscales, inter-rater reliability or agreement and validity of the model, correlation between the single global question, the total performance score and the three survey subfactors (Pearson's), interrater agreement (rWG(J)), the optimal number of peer reviewers required and model-based reliability (ρ). Results The global question, total performance score and the three subfactors were strongly correlated (general scale r=0.81, clinical subscale r=0.78, humanistic subscale r =0.74, management subscale r=0.75; two-tailed P<0.01 for all). The scale showed very good internal consistency, except for the five-question management subscale. Model-based reliability was excellent (ρ=0.93). Confirmatory factor analysis showed the model fit using the 20-item scale was not satisfactory (minimum discrepancy/d.f.=7.70; root mean square error of approximation=0.10; comparative fit index=0.79; Tucker-Lewis index=0.76). A modified 13-item model provided a good fit. Using the 20-item scale, a 99% level of agreement could be achieved with eight to 10 peer reviewers; for the same level of agreement, the number of reviewers increased to >10 using a revised 13-item scale. Conclusions Overall, the 20-item multisource review survey tool showed good internal consistency reliability for both self and peer ratings; however, further investigation using a larger dataset is needed to analyse the robustness of the model and to clarify the role that a single global question may play in future multisource review processes. What is known about the topic? Defining and measuring skills and behaviours that reflect competence in the health setting have proven to be complex, and this has resulted in the development of specific multisource feedback surveys for individual medical specialities. Because little literature exists on multisource reviews in an Australian context, a pilot study of a revised survey tool was undertaken at an Australian tertiary hospital. What does this paper add? The aim of this study was to investigate the psychometric properties of a generic tool (used across specialities) by assessing the validity, reliability and interrater reliability of the scale and to consider the contribution of a single global question to the overall multisource feedback process. This study provides evidence of the validity and reliability of the survey tool under investigation. The strong correlation between the global item, the total performance score and the three subfactors suggests that this is an area requiring further investigation to determine the role that a robust single global question like this may play in future multisource review surveys. Our five-question management skills subscale provides answers to questions relevant to the specific organisation surveyed, and we anticipate that it may serve to stimulate further exploration in this area. What are the implications for practitioners? The survey tool may provide a valid and reliable basis for performance review of medical consultants in an Australian healthcare setting.

Exploring the Effectiveness of Multisource Feedback and Coaching with Nurse Practitioners.

While multisource feedback and coaching have shown promise as effective professional development strategies for physicians, the effectiveness of these interventions with nurse practitioners - a growing profession in Canada - remains unknown. Despite this knowledge gap, multiple nursing colleges in Canada require their nurse practitioner members to participate in multisource feedback processes. An exploratory study was performed with twelve nurse practitioners using an online multisource feedback process (based on the CanMEDS Framework) and an in-person coaching session (using the R2C2 Model). Participants were surveyed immediately post intervention and two months later. Perspectives of the coaches and process coordinators were also assessed. Nearly all participants reported that the intervention was valuable for their professional development, and 63% reported they changed an aspect of their practice because of participating. However, the majority of participants reported difficulty finding colleagues who could provide them with valid feedback. This was because of the independent nature of their practice. Multisource feedback and coaching show promise as effective professional development strategies for nurse practitioners who work in collaborative practices. Further research is needed to confirm these exploratory findings.

Selecting graduates for the interns\u2019 award by using multisource feedback process: does it work?

BackgroundThe purpose of this study is to find a reliable method for choosing graduates for a higher-education award. One such method that has achieved notable popularity is known as multisource feedback. Multisource feedback is assessment tool that uses evaluations of different groups and includes both physicians and non-physicians. It is useful for assessing several domains, including professionalism, communication and collaboration, and therefore is a valuable tool for providing a well-rounded selection of the top interns for postsecondary awards. 16 graduates in Royal College of Surgeons in Ireland-Medical University of Bahrain (RCSI Bahrain) responded to an invitation to participate in the student award, which was conducted by the using the multisource feedback process. 5 individuals from different categories (physicians, nurses, and fellow students), rated each participant in this study. A total of 15 individuals were the proposed number for rating. The ratings were calculated using mean and standard deviation, and the award went to the one of the top score out of the 16 participants. Reliability and internal consistency was calculated using Cronbach’s coefficient, and construct validity was evaluated using factor analysis.Results16 graduates participated in the Royal College of Surgeons in Ireland-Bahrain interns’ award based on the multisource feedback process, giving us a 16.5% response rate. The instrument was found to be suitable for factor analysis and showed 3 factor solutions representing 79.3% of the total variance. Reliability analysis using Cronbach’s α reliability of internal consistency indicated that the full scale of the instrument had high internal consistency (Cronbach’s α 0.98).ConclusionThis study confirmed our hypothesis, finding multisource feedback to be a process for choosing the most suitable graduates for interns’ awards that is both reliable and valid. Unfortunately, there were low response rate, which could mean that multisource feedback is not a realistic way to bring most students into the process.

The View From Over Here: A Framework for Multi-Source Feedback.

Multiple snapshots of a trainee's real-time performance are required for program directors and Clinical Competency Committees to provide a comprehensive performance assessment and deliver formative feedback.1 Multi-source feedback (MSF), or 360-degree evaluation, is a formal questionnaire-based assessment method that relies on workplace-based direct observation of performance by multiple raters from different perspectives.2–5 While decades of MSF studies in medical education support its feasibility and provide validity evidence,2–4 educators wishing to implement MSF programs may be intimidated by issues of time, labor intensity, and scope.1,5,6The MSF approach is best suited for direct observation of real-time trainee performance. Using diverse raters' perspectives over multiple observations yields a more comprehensive assessment of a learner's performance than any single-source approach.2–4 The MSF approach can be powerful for assessing behaviors in nontechnical or nonprocedural domains, such as professionalism, interpersonal and communication skills, and practice-based learning and improvement, that are difficult to capture by traditional methods.1,3,4 Educators planning to develop an MSF tool must adhere to a rigorous development process to optimize instrument feasibility, reliability, and validity as well as to avoid measurement errors. Steps in this process include the development of items or the use of existing items that will yield specific, meaningful feedback to trainees,1,6 the identification of the behaviors that each rater group is best suited to observe and judge, determining who the assessor(s) are, and the training of raters. Validity evidence can be demonstrated by correlations between MSF instruments (concurrent validity) and between MSF ratings and other assessment measures (predictive validity).3The MSF raters can include self, peers, supervisors or preceptors, nonphysician coworkers (eg, nurses, social workers, pharmacists), and patients.3,6 Self-assessment plays an integral role in the MSF process (table). Trainees' self-assessment, when compared with other MSF ratings, may identify gaps needing further attention.2–4 Eight to 20 credible rater assessments (depending on the source) are needed to optimize validity. Residents perceive raters as credible when raters directly observe their performance and are capable of providing meaningful feedback about that performance.1,4 Thus, each rater must observe the trainee on the unique set of domains matched to the scope of their role and relationship with the trainee in order to provide feedback that will be perceived as credible.4–6 Trainees have identified patient feedback as a particularly powerful trigger for behavior change.4,6The MSF approach may be more likely to lead to behavioral change when trainees buy in to the goals and value of MSF from credible raters.2 Facilitated feedback delivery by a mentor or coach may improve learner receptivity and incorporation of MSF.4 Adding narrative comments to numerical item scores makes the feedback more meaningful.4,6 Trainees report that guidance in identifying gaps between self-assessment and MSF results improves MSF acceptance and helps them generate an action plan to change behavior.4

Reliability of the Interprofessional Collaborator Assessment Rubric (ICAR) in Multi Source Feedback (MSF) with post-graduate medical residents

BackgroundIncreased attention on collaboration and teamwork competency development in medical education has raised the need for valid and reliable approaches to the assessment of collaboration competencies in post-graduate medical education. The purpose of this study was to evaluate the reliability of a modified Interprofessional Collaborator Assessment Rubric (ICAR) in a multi-source feedback (MSF) process for assessing post-graduate medical residents’ collaborator competencies.MethodsPost-graduate medical residents (n = 16) received ICAR assessments from three different rater groups (physicians, nurses and allied health professionals) over a four-week rotation. Internal consistency, inter-rater reliability, inter-group differences and relationship between rater characteristics and ICAR scores were analyzed using Cronbach’s alpha, one-way and two-way repeated measures ANOVA, and logistic regression.ResultsMissing data decreased from 13.1% using daily assessments to 8.8% utilizing an MSF process, p = .032. High internal consistency measures were demonstrated for overall ICAR scores (α = .981) and individual assessment domains within the ICAR (α = .881 to .963). There were no significant differences between scores of physician, nurse, and allied health raters on collaborator competencies (F2,5 = 1.225, p = .297, η2 = .016). Rater gender was the only significant factor influencing scores with female raters scoring residents significantly lower than male raters (6.12 v. 6.82; F1,5 = 7.184, p = .008, η 2 = .045).ConclusionThe study findings suggest that the use of the modified ICAR in a MSF assessment process could be a feasible and reliable assessment approach to providing formative feedback to post-graduate medical residents on collaborator competencies.

The construct and criterion validity of the multi-source feedback process to assess physician performance: a meta-analysis

BackgroundThe purpose of this study was to conduct a meta-analysis on the construct and criterion validity of multi-source feedback (MSF) to assess physicians and surgeons in practice.MethodsIn this study, we followed the guidelines for the reporting of observational studies included in a meta-analysis. In addition to PubMed and MEDLINE databases, the CINAHL, EMBASE, and PsycINFO databases were searched from January 1975 to November 2012. All articles listed in the references of the MSF studies were reviewed to ensure that all relevant publications were identified. All 35 articles were independently coded by two authors (AA, TD), and any discrepancies (eg, effect size calculations) were reviewed by the other authors (KA, AD, CV).ResultsPhysician/surgeon performance measures from 35 studies were identified. A random-effects model of weighted mean effect size differences (d) resulted in: construct validity coefficients for the MSF system on physician/surgeon performance across different levels in practice ranged from d=0.14 (95% confidence interval [CI] 0.40–0.69) to d=1.78 (95% CI 1.20–2.30); construct validity coefficients for the MSF on physician/surgeon performance on two different occasions ranged from d=0.23 (95% CI 0.13–0.33) to d=0.90 (95% CI 0.74–1.10); concurrent validity coefficients for the MSF based on differences in assessor group ratings ranged from d=0.50 (95% CI 0.47–0.52) to d=0.57 (95% CI 0.55–0.60); and predictive validity coefficients for the MSF on physician/surgeon performance across different standardized measures ranged from d=1.28 (95% CI 1.16–1.41) to d=1.43 (95% CI 0.87–2.00).ConclusionThe construct and criterion validity of the MSF system is supported by small to large effect size differences based on the MSF process and physician/surgeon performance across different clinical and nonclinical domain measures.

Investigating the Reliability and Validity of Self and Peer Assessment to Measure Medical Students’ Professional Competencies

The use of peer assessment through a multisource feedback process has gained recognition as a reliable and valid method to assess the characteristics of professionals and trainees. A total of 168 first-year medical students completed a 15-item questionnaire to self-assess their professional work habits and interpersonal abilities. Each student was expected to identify 8 first-year classmates to complete a corresponding 15-item peer assessment. Although the self and peer assessment questionnaires had strong reliability (Cronbach’s α = 0.85 and 0.91, respectively), an exploratory factor analysis resulted in a 3- and 2- factor solution, respectively. The third factor was associated with items related to students’ personal attributes. Significantly lower mean score differences for the self-report assessment were found for all 15 items (Cohen’s d = 0.27 to 1.39, p gards to the construct validity and stability of measures between self and peer assessment measures. The need for self-awareness of students’ strengths and limitations, however, is recommended as part of their development in a profession that emphasizes self-regulation.

Educational climate seems unrelated to leadership skills of clinical consultants responsible of postgraduate medical education in clinical departments

BackgroundThe educational climate is crucial in postgraduate medical education. Although leaders are in the position to influence the educational climate, the relationship between leadership skills and educational climate is unknown. This study investigates the relationship between the educational climate in clinical departments and the leadership skills of clinical consultants responsible for education.MethodsThe study was a trans-sectional correlation study. The educational climate was investigated by a survey among all doctors (specialists and trainees) in the departments. Leadership skills of the consultants responsible for education were measured by multi-source feedback scores from heads of departments, peer consultants, and trainees.ResultsDoctors from 42 clinical departments representing 21 specialties participated. The response rate of the educational climate investigation was moderate 52% (420/811), Response rate was high in the multisource-feedback process 84.3% (420/498). The educational climate was scored quite high mean 3.9 (SD 0.3) on a five-point Likert scale. Likewise the leadership skills of the clinical consultants responsible for education were considered good, mean 5.4 (SD 0.6) on a seven-point Likert scale. There was no significant correlation between the scores concerning the educational climate and the scores on leadership skills, r = 0.17 (p = 0.29).ConclusionsThis study found no relation between the educational climate and the leadership skills of the clinical consultants responsible for postgraduate medical education in clinical departments with the instruments used. Our results indicate that consultants responsible for education are in a weak position to influence the educational climate in the clinical department. Further studies are needed to explore, how heads of departments and other factors related to the clinical organisation could influence the educational climate.

What is a good general practitioner (GP)? The development and evaluation of a multi-source feedback instrument for GP appraisal

Although multi-source feedback (MSF) has been used in primary healthcare, the development of an MSF instrument specific to this setting in the UK has not been previously described. The aims of this study were to develop and evaluate an MSF instrument for GPs in Scotland taking part in appraisal. The members of ten primary healthcare teams in the west of Scotland were asked to provide comments in answer to the question, 'What is a good GP?'. The data were reduced and coded by two researchers and questions were devised. Following content validity testing the MSF process was evaluated with volunteers using face-to-face interviews and a postal survey. Thirty-seven statements covering the six domains of communication skills, professional values, clinical care, working with colleagues, personality issues and duties and responsibilities were accepted as relevant by ten primary healthcare teams using a standard of 80 percent agreement. The evaluation found the MSF process to be feasible and acceptable and participants provided some evidence of educational impact. An MSF instrument for GPs has been developed based on the concept of 'the good GP' as described by the primary healthcare team. The evaluation of the resultant MSF process illustrates the potential of MSF, when delivered in the supportive environment of GP appraisal, to provide feedback which has the possibility of improving working relationships between GPs and their colleagues.

How ratings vary by staff group in multi-source feedback assessment of junior doctors

UK doctors-in-training undergo assessments of their professional behaviours. From an analysis of multi-source feedback (MSF) data, we report how ratings of junior doctors (Foundation Programme [FP] doctors and senior house officers [SHOs]) differed by staff group. The MSF data were collected in 2003 and 2005 in hospitals in the West Midlands. Using a single-sided Team Assessment of Behaviour form, 1928 assessors evaluated 226 FP doctors and SHOs in four domains: professional relationship with patients; communication; team-working, and accessibility. The distribution of 'concerns' across the professional groups was explored using a random effects logistic regression model. On average, each trainee received nine assessment forms from a range of staff, most commonly nurses. Although concerns were identified for the minority, ratings varied by staff group. Peers (other FP doctors or SHOs) and administrators or managers were four and three times, respectively, less likely to indicate concern. By contrast, consultants and sisters (senior nurses) were more likely to give concern ratings. Guidance on the selection of assessors in any MSF process should take into account findings that rating behaviour varies by staff group.

Effects of multi-source feedback on developmental plans for leaders of postgraduate medical education

Multi-source feedback (MSF) is a widely used developmental tool for leaders in organisations including those dealing with health care. This study was performed to examine the effects of an MSF process on developmental plans made by leaders of postgraduate medical education (PGME) in clinical departments. An MSF instrument was developed based on literature on the subject and previous investigations. The instrument was used by consultants responsible for PGME in clinical departments (CREs). Apart from CREs' self-ratings, MSF responses were collected from heads of departments, consultants and young doctors-in-training. The MSF process included individual feedback as well as guidance on drafting developmental plans for both the department and the CREs. Themes emerging in the developmental plans were analysed and compared with the areas in need of improvement identified by the MSF process. The MSF instrument was found to be feasible, valid and reliable (Cronbach's alpha = 0.98). The study included 52 CREs from various specialties and 498 MSF respondents. The response rate was 84%. Low ratings and negative gaps between others' ratings and self-ratings were identified for both management and leadership performance areas. The developmental plans mainly focused on management initiatives, whereas plans for the development of leadership performance were few. Areas rated low by all respondents were scarcely represented in CREs' developmental plans. An MSF process might in itself lead to development in administrative areas. However, MSF carried through as a single stand-alone procedure was not sufficient to foster plans for the development of leadership performance.

Multisource feedback: Lessons learned and implications for practice

AbstractOrganizations around the world are using multisource, or 360‐degree, feed‐back. Although many HR practitioners embrace it as an important mecha‐nism for leadership development, organizations must attend to and address several issues in order to maximize the utility of multisource feedback (MSF). We discuss current research findings and highlight issues for managers to consider both before starting a multisource feedback process and after the feedback is given, plus we review potential outcomes of the process. We also describe lessons learned from an intensive three‐year investigation of an MSF implementation in two organizations. © 2007 Wiley Periodicals, Inc.

Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness

Physician performance is comprised of several domains of professional competence. Multisource feedback (MSF) or 360-degree feedback is an approach used to assess these, particularly the humanistic and relational competencies. Research studying responses to performance assessment shows that reactions vary and can influence how performance feedback is used. Improvement does not always result, especially when feedback is perceived as negative. This small qualitative study undertook preliminary exploration of physicians' reactions to MSF, and perceptions influencing these and the acceptance and use of their feedback. We held focus groups with 15 family physicians participating in an MSF pilot study. Qualitative analyses included content and constant comparative analyses. Participants agreed that the purpose of MSF assessment should be to enhance practice and generally agreed with their patients' feedback. However, responses to medical colleague and co-worker feedback ranged from positive to negative. Several participants who responded negatively did not agree with their feedback nor were inclined to use it for practice improvement. Reactions were influenced by perceptions of accuracy, credibility and usefulness of feedback. Factors shaping these perceptions included: recruiting credible reviewers, ability of reviewers to make objective assessments, use of the assessment tool and specificity of the feedback. Physicians' perceptions of the MSF process and feedback can influence how and if they use the feedback for practice improvement. These findings are important, raising the concern that feedback perceived as negative and not useful will have no or negative results, and highlight questions for further study.

Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study.

To describe responses of family physicians, their medical colleagues, and coworker raters to a multisource feedback assessment process. Data collection tools included multisource feedback self-assessment and medical colleague, coworker, and patient rating forms; and program evaluation physician and rater questionnaires. The pilot study included 142 physicians and their raters, with 113 (80%) physicians completing evaluations. Positive correlations were found between familiarity scores and medical colleague and coworker mean ratings. Peer medical colleagues were significantly more familiar with physicians than were consultants. Consultants were unable to rate items most frequently. Physicians disagreed with colleague feedback more frequently. Agreement was positively correlated with scores. Familiarity, ability to observe physicians appropriately to rate them, and physicians' responses to feedback are factors to consider when multisource feedback is used.

Understanding and optimizing multisource feedback

AbstractThis article integrates the most recent research results on the topic of multisource feedback with what researchers have learned over the years about performance feedback in general. We believe that this review and set of recommendations represent the state of the art at this time. We provide practitioners with new ideas about how to continue to improve the multisource feedback process in their organizations. We also suggest ideas for feedback providers and facilitators about how to maximize the success of the feedback process. Additionally, we provide “food for thought” for researchers concerning ideas for future research. © 2002 Wiley Periodicals, Inc.