Multisource Feedback Assessment Research Articles

A quantitative analysis of the perceived barriers to implementation of multisource feedback in surgical training.

The adoption and integration of work-based assessments by surgical units and training programs continues to increase, it is important to identify challenges in their implementation. The authors evaluated the barriers involved in the deployment of a supervisor assessment tool in Australia. A questionnaire was created based on existing literature from international, and non-surgical contexts. The questionnaire covered areas known to impact multisource feedback assessments. The questionnaire was delivered in September 2019 to 34 trainees and 25 supervisors. Participants then rated each item within the questionnaire on a Likert scale. During the survey period, 23 of 25 (92%) supervisors and 31 of 34 (91.1%) trainees completed the questionnaire. The results show that an assessment form is well received and there is a difference between trainees and supervisors on perspectives surrounding impact on trainee-supervisor relationships, how a trainee manages negative feedback, consequences of poor scores, the importance of anonymity, and acceptability of input from allied health and nursing staff. The results provide reassurance that the perception of supervisors who felt that negative ratings may affect trainee-supervisor relationships is unfounded. A digital distribution format of self-assessment and supervisor assessment forms is well received by participants. This method of assessment should provide trainees an additional source of feedback, and opportunities for self-reflection during their training.

Composite Reliability of Workplace Based Assessment of International Medical Graduates.

This article was migrated. The article was marked as recommended. Introduction All developed countries depend on International Medical Graduates (IMGs) to complement their workforce. However, the assessment of their fitness to practice and acculturation into the new system can be challenging. To improve this, we introduced Workplace Based Assessment (WBA), using a programmatic philosophy. This paper reports the reliability of this new approach. Method Over the past 10 years, we have assessed over 250 IMGs, each cohort assessed over a 6-month period. We used Mini-Cex, Case Based Discussions (CBD) and Multi-Source Feedback (MSF) to assess them. We analysed the reliability of each tool and the composite reliability of 12 Mini-Cex, 5 CBDs and 12 MSF assessments in the tool kit. Results A reliability coefficient of 0.78 with a SEM of 0.19 was obtained for the sample of 236 IMGs. We found the MSF to be the most reliable tool. By adding one more MSF to the assessment on two occasions, we can reach a reliability of 0.8 and SEM of 0.18. Conclusions The current assessment methodology has acceptable reliability. By increasing the MSF, we can improve the reliability. The lessons from this study are generalisable to IMG assessment and other medical education programs.

Through My Eyes: African American Women Leaders' View on the Multi- Source Feedback Assessment

Intersectionality is an emerging topic in leadership development research. Intersectionality theory is an expansion of feminism that looks at the effects of overlapping identities, such as gender and race. This current study sought to understand the role of intersectionality on a popular form of leader development, multi- source feedback interventions. Multi-source feedback interventions provide an individual with self-referent information that promotes motivation and energy towards self-growth (Atwater, Waldman, & Brett, 2002). One's social identity often impacts the content of feedback, the process of how feedback is delivered, as well as how feedback is accepted and processed (Toegel & Conger, 2003). Using a hermeneutic phenomenological research design, I interviewed twenty African American women in leadership roles on their experience with the multi-source feedback assessment. Common themes emerged such as fear of confirming certain stereotypes, a desire for personality-based feedback, and the use of the multi-source feedback for leadership development.

A first-year leadership programme for medical students.

The established medical hierarchy, dramatic expansion of scientific knowledge and emphasis on value-based health care means that graduating physicians need to know how to manage and lead positive change. There is a critical gap in the teaching of these skills in undergraduate medical education. Our medical school developed a first-year medical student competency-based leadership curriculum that focused on: leading yourself; teams and teamwork; influence and communication; problem solving; and systems thinking. The course used four methods of teaching leadership: experiential learning; targeted development; reflection; and feedback. The formal curriculum included topics such as developing a leadership agenda, challenging conversations and negotiations. The informal curriculum (learning outside of the structured curriculum) included applying leadership in co-curricular and extracurricular activities (e.g. in a student-run free clinic). Students recorded leadership experiences using a novel reflective assessment tool, obtained multi-source feedback and then articulated a plan for improvement. Course evaluations noted that only one-third of first-year students responded that the curriculum developed skills in communication, and the ability to problem-solve, apply systems thinking and build teams. Students self-reported that they were often building, leading and managing productive teams, and applying influence and communication. The multi-source feedback assessment revealed that students, on average, were rated as competent to proficient. Creating a robust curriculum for medical students in the first year is challenging. Student reactions ranged from affirming to critical. The next steps will focus on increasing interactive teaching and on helping students understand why, where and how leadership is important.

Validation of Multisource Feedback in Assessing Medical Performance: A Systematic Review.

Over the past 10 years, a number of systematic reviews have evaluated the validity of multisource feedback (MSF) to assess and quality-assure medical practice. The purpose of this study is to synthesize the results from existing reviews to provide a holistic overview of the validity evidence. This review identified eight systematic reviews evaluating the validity of MSF published between January 2006 and October 2016. Using a standardized data extraction form, two independent reviewers extracted study characteristics. A framework of validation developed by the American Psychological Association was used to appraise the validity evidence within each systematic review. In terms of validity evidence, each of the eight reviews demonstrated evidence across at least one domain of the American Psychological Association's validity framework. Evidence of assessment validity within the domains of "internal structure" and "relationship to other variables" has been well established. However, the domains of content validity (ie, ensuring that MSF tools measure what they are intended to measure); consequential validity (ie, evidence of the intended or unintended consequences MSF assessments may have on participants or wider society), and response process validity (ie, the process of standardization and quality control in the delivery and completion of assessments) remain limited. Evidence for the validity of MSF has, across a number of domains, been well established. However, the size and quality of the existing evidence remains variable. To determine the extent to which MSF is considered a valid instrument to assess medical performance, future research is required to determine the following: (1) how best to design and deliver MSF assessments that address the identified limitations of existing tools and (2) how to ensure that involvement within MSF supports positive changes in practice. Such research is integral if MSF is to continue to inform medical performance and subsequent improvements in the quality and safety of patient care.

Team assessment of behaviour: a high stakes assessment with potential for poor implementation and impaired validity

Team assessment of behaviour (TAB) is the multi-source feedback assessment of professional behaviours that all UK foundation doctors must engage in twice during their two-year programme. TAB can identify the few underperforming trainees and provide feedback to consolidate the good practice of most. For optimum validity, TAB must be undertaken by a range of assessors, as specified in the national UK Foundation Programme curriculum. This study reports an audit of invalid TAB submissions over a three-year cycle in the West Midlands' Foundation Programme. In 2010, large numbers of TABs were invalid, owing to an incorrect selection or number of assessors. Introduction of validity checking before sign-off greatly improved the numbers of valid assessments in 2011. This was partially sustained in 2012. Assurance of assessment validity is important to ensure delivery of appropriate constructive feedback and to allow early detection and remediation of signs of poor professional behaviours in foundation doctors.

Using the computer-based feedback (CBF) system to investigate the juniorphysicians’s and clinical-instructors perceptions for the benefits of general medicine clinical-instructors training program

Background: The purpose of this study is to examine the clinical-instructors and junior-physicians (residents and interns) perceptions for the general-medicine training program by using bi-directional interactive and self-assessments computer-based feedback (CBF) and paper-based multisource feedback assessment (PBMFA) systems for the efficiency and benefit evaluation.Methods: Between 2011 January to 2013 December, junior-physicians and their clinical-instructors in the same medical team were enrolled consecutively for monitoring the CBF scores gave by each other after each clinical course. A total of 321 residents, 298 interns and 110 clinical-instructors who participated in the core competency general-medicine training program in 6-months period were included in the study. The CBF and PBMFA evaluations are undergone paralleled to gather the suggested information in different levels of Kirkpatrick evolutional theory.Results: The results showed that lecturers, being 5-10 years as attending physicians, internal medicine sub-specialty clinical-instructors are most benefit from the general medicine training program. Accordingly, the CBF scores of junior-physicians was positively correlated with the times (> 3-times) of exposure to the medical teams that leaded by qualified clinical-instructors. Both clinical-instructors and junior-physicians have positive attitude to the value of the general-medicine training program. Interestingly, a good consistency was existed between residents CBF scores and PBMFA grades for their core-competency performance. Comparatively, the overall perception of clinical-instructors and junior-physicians for the general-medicine training was very positive.Conclusions: Clinical-instructors and junior-physicians had positive perception of CBF and PBMFA systems which could give us different information to improve and strength the further core-competency general-medicine training program by appropriate utilization.

Reliability of the Interprofessional Collaborator Assessment Rubric (ICAR) in Multi Source Feedback (MSF) with post-graduate medical residents

BackgroundIncreased attention on collaboration and teamwork competency development in medical education has raised the need for valid and reliable approaches to the assessment of collaboration competencies in post-graduate medical education. The purpose of this study was to evaluate the reliability of a modified Interprofessional Collaborator Assessment Rubric (ICAR) in a multi-source feedback (MSF) process for assessing post-graduate medical residents’ collaborator competencies.MethodsPost-graduate medical residents (n = 16) received ICAR assessments from three different rater groups (physicians, nurses and allied health professionals) over a four-week rotation. Internal consistency, inter-rater reliability, inter-group differences and relationship between rater characteristics and ICAR scores were analyzed using Cronbach’s alpha, one-way and two-way repeated measures ANOVA, and logistic regression.ResultsMissing data decreased from 13.1% using daily assessments to 8.8% utilizing an MSF process, p = .032. High internal consistency measures were demonstrated for overall ICAR scores (α = .981) and individual assessment domains within the ICAR (α = .881 to .963). There were no significant differences between scores of physician, nurse, and allied health raters on collaborator competencies (F2,5 = 1.225, p = .297, η2 = .016). Rater gender was the only significant factor influencing scores with female raters scoring residents significantly lower than male raters (6.12 v. 6.82; F1,5 = 7.184, p = .008, η 2 = .045).ConclusionThe study findings suggest that the use of the modified ICAR in a MSF assessment process could be a feasible and reliable assessment approach to providing formative feedback to post-graduate medical residents on collaborator competencies.

How to Develop, Implement, and Assess Multi-Source Feedback (MSF) for Trainees

Abstract Introduction Multi-source feedback (MSF) imparts feedback from a multidisciplinary group of raters within the trainee's sphere of influence. Faculty involved in residency and fellowship tr...

Registrar feedback on ‘Formative assessments in medical education’

I write in support of the article ‘Formative assessments in medical education’ by Dr Lakasing.1 I love my job as a GP registrar and look forward to qualification in a few months. Despite the contract wranglings, bad press, and the ever-increasing workload, I feel optimistic and enthused about the future. I support most aspects of the ePortfolio process from the AKT and CSA exams to the Case-based Discussion and Consultation Observation Tool assessments. I think the patient satisfaction questionnaires and multisource feedback assessments are crucial aspects of …

Psychometric Characteristics of the 360° Feedback Scales in Professionalism and Interpersonal and Communication Skills Assessment of Surgery Residents in China

The aim of the study was to test the psychometric characteristics of the 360° evaluation instrument for assessing residents' competency in professionalism, interpersonal and communication skills, and develop a feasible, valid, and reliable multisource feedback (MSF) program for surgery residents to address this gap. We carried out an MSF assessment of 149 surgery residents at 19 hospitals all over China. MSF assessment includes 6 surveys with 21, 21, 21, 26, 14, and 15 items and these surveys were developed to assess surgery residents by attending doctor, resident self, peer, nurse, patient, and office staff, respectively, using a 5-point agreement scale with an "unable-to-evaluate" category. Reliability was assessed by Cronbach alpha. Exploratory factor analysis was used to assess validity and determine which items grouped together into scales. A total of 2384 questionnaires were analyzed in this study. The internal consistency reliability of the instruments was a Cronbach alpha of 0.932, 0.936, 0.914, 0.916, 0.939, and 0.903 for attending doctor, resident self, nurse, patient, resident peer, and office staff surveys, respectively. On the attending, resident self, resident peer, nurse, office staff, and patient surveys, the factor analysis identified 2 factors of professionalism and interpersonal and communication skills accounting for 75.62%, 74.81%, 72.65%, 73.38%, 76.11%, and 63.89% of the variance, respectively. Some items, such as Demonstrates respect for my "Sexual Orientation," "Religion," and "Disability," in different surveys had high unable-to-evaluate rates (more than 10%). The data suggest that these instruments developed to assess surgery residents are feasible to administer and provide valid and reliable evidence. Some items in survey need to be adjusted keeping in mind the Chinese culture.

Student Self-Assessment and Multisource Feedback Assessment

It has become common practice for management students to participate in some sort of self-assessment or multisource feedback assessment (MSF; also called 360-degree assessment or multirater assessment) during their management degree program. These assessments provide students invaluable feedback about themselves and assist students in their personal and professional development. This article draws on a conceptualization of the self as multifaceted to examine the benefits and limitations of self-assessment and MSF. Specifically, although an aid to students’ learning, self-assessment tools predominantly treat the self as an individual self, thereby ignoring the relational and collective aspects of the self. Moreover, self-assessments that people make tend to be inflated, unreliable, and biased. MSF improves on self-assessment by including others’ assessment of the self; however, it also possesses three limitations in its current treatment: MSF (a) conceptualizes the self as an individual self, (b) ignores the importance of context, and (c) relies primarily on a self-other rating agreement. These limitations are explored, and remedies are offered to better account for the multifaceted nature of the self.

Impact of Leader Competency Type with Leader Promotability: Extending Self-Other Agreement Research

Leader development research and practice have relied on multisource feedback assessment (MSF or 360-degree assessment) for years. Self-other rating agreement (SOA) is one area of MSF research that sustains particular interest. SOA studies have repeatedly aggregated leadership behaviors together when using SOA as a predictor of performance. As a result, we know little about SOA and its impact on leader outcomes based on the type of leader behavior used for SOA. We hypothesized (1) self-other in-agreement high ratings of work centric behaviors (e.g., confronting problem employees) will have a stronger positive relationship with promotability than non-work centric behaviors (e.g., putting people at ease), and (2) over-estimating self-ratings, compared to others’ ratings, will have a negative relationship with promotability with work centric behaviors but not with non-work centric behaviors. We tested these two hypotheses with 438 leaders and results show that in-agreement high ratings are associated with higher outcomes regardless of the type of behavior, but, as predicted, overrating work centric behaviors negatively relates to promotability. We discuss the important practical and research implications these findings have for organizations.

An analysis of multisource feedback within the foundation programme

Multisource feedback (MSF) is a mandatory assessment in the foundation programme. The current tool that is used is the validated Team Assessment of Behaviour (TAB). This assesses doctors on four domains: maintaining trust and professional relations; verbal communication skills; teamworking; and accessibility. The aims of our retrospective analysis were to analyse the number of doctors who had an MSF with any type of concern, and to assess if there is any relationship between the number of MSFs submitted and the number of concerns and the staff group. Data were obtained from the e-portfolio Malta database. The anonymized MSFs for the doctors in the foundation programme (FP) in Malta between July 2009 and July 2010 were analysed. A total of 1868 MSF assessments from 83 FP doctors (50 foundation year-1 doctors, FY1s, and 33 FY2s) were analysed. Fifty-two FY doctors were female. The majority of assessments (97.86%) did not show any concern in any of the four domains. However, at least one concern was raised for 21.7 percent of the doctors (12 FY1s and 6 FY2s). The chance of there being a concern reported was statistically significantly higher when the MSF was completed by a more senior doctor, rather than by an FY doctor or a nurse (p = 0.016). The aim of the MSF tool is to collate the views from a range of clinical colleagues in order to assess trainee performance. In our opinion, besides indicating when the MSFs should be performed and the minimum number that need to be submitted, it should also be mandatory that a certain proportion of MSFs should be performed by specific people (e.g. by a clinical supervisor, basic specialist trainee or higher specialist trainee working with the team).

The influence of emotional and social competencies on the performance of Peruvian refinery staff

PurposeThe purpose of this paper is to determine the influence of emotional and social competence (ESC) on job performance by considering self‐ratings and the ratings of others (supervisor, peer and subordinate) using a multisource feedback assessment of ESC.Design/methodology/approachThe authors utilized the 2004‐2006 performance evaluations of 36 staff members of the Ilo Copper Refinery, owned by the Southern Peru Corporation. To assess ESC, the Emotional Competence Inventory (ECI 2.0) was administered to the Ilo staff.FindingsThe study reinforces the importance of ESC to workplace performance and provides evidence of ESC's positive influence. The results reveal that 70 percent of the variance of working performance is explained linearly by the total average of ESC, with four significant competencies – self‐confidence, achievement orientation, optimism, and teamwork & collaboration – accounting for 63 percent of the variance.Research limitations/implicationsAs an initial study in Peru, the authors had a small sample size. There is limited independence in the performance evaluations because the evaluators of performance were repeated in several cases. Job performance ratings were based on the following computerized objective assessment: use of abilities, work organization planning, interpersonal relations, results, initiative, aptitude to the work, and creativity.Originality/valueThe paper is aimed at improving understanding of the links between ESC and performance. It is the first study the authors are aware of to examine these relationships in a Peruvian organization. The approach used in this study contributes to and provides evidence of the importance of emotional competence in the workplace.

Assessment of Resident Physicians in Professionalism, Interpersonal and Communication Skills: a Multisource Feedback

Objective: To assess the internal validity and reliability of a multisource feedback (MSF) program by China Medical Board for resident physicians in China.Method: Multisource feedback was used to assess professionalism, interpersonal and communication skills. 258 resident physicians were assessed by attending doctors, self-evaluation, resident peers, nurses, office staffs, and patients who completed a sealed questionnaire at 19 hospitals in China. Cronbach's alpha coefficient was used to assess reliability. Validity was assessed by exploratory factor analyses and by profile ratings.Results: 4128 questionnaires were collected from this study. All responses had high internal consistency and reliability (Cronbach's α> 0.90), which suggests that both questions and form data were internally consistent. The exploratory factor analysis with varimax rotation for the evaluators' questionnaires was able to account for 70 to 74% of the total variance.Conclusion: The current MSF assessment tools are internally valid and reliable for assessing resident physician professionalism and interpersonal and communication skills in China.

Open-Mindedness and Defensiveness in Multisource Feedback Processes

Multisource Feedback Assessment (MSF or 360-degree feedback) used for employee development has become ubiquitous in many organizations. From an organization development (OD) perspective, this article presents a conceptual model to explain reactions to MSF in feedback recipients. In the ideal mode, MSF recipients react to MSF data with open-mindedness, a strong willingness to evaluate self-perceptions by learning from others’ perspectives. In the converse mode of defensiveness, MSF recipients are likely to be closed to and dismissive of others’ perspectives. The usefulness of the model for testing alternative MSF approaches is illustrated through a discussion of two MSF techniques: the Self–Other Comparison (SOC) approach and the Predicted–Other Comparison (POC) approach. Approaches that foster a mode of open-mindedness over defensiveness are more likely to produce the intended learning benefits.

An evaluation of using multi-source feedback (MSF) among junior hospital pharmacists

The mini Peer Assessment Tool (mini-PAT) for pharmacists was introduced in 2006 as a formative method of assessing junior hospital pharmacists in the workplace and is the first widespread application of multi-source feedback (MSF) specifically within a pharmacy setting. The aim of this study was to evaluate the feasibility and measurement characteristics of the assessment method, in order to guide its future application. At the time of the study (September 2008) the assessment had been in place for 3 years. All assessment data from the first 3 years were analysed retrospectively. We evaluated 633 mini-PAT assessments. Over the study period, the assessor response rate remained relatively consistent at 77% and compared favourably with applications of MSF within medicine. Members of the pharmacy team (pharmacists and pharmacy technicians) dominated the assessor nomination lists. It was encouraging to see completed assessment forms returned from nominated doctors and nurses with whom the junior pharmacist had been working. Differences were found between how different occupational groups rated the junior pharmacists against the 16 items on the assessment form (Kruskal-Wallis, df=3, P<0.001). Pharmacist assessors rated the junior pharmacists lowest against all 16 items on the mini-PAT assessment form, whereas nominated doctors rated them the highest. This study demonstrates that an MSF assessment method can successfully be applied to a wide range of junior hospital pharmacists, and that the majority of junior hospital pharmacists assessed meet expectations.

It may not be what you think: Gender differences in predicting emotional and social competence

Women and men leaders have been noted to be similar in terms of emotional and social competence in the workforce. We analyzed gender differences in self-ratings, others’ ratings, and predicted ratings of men and women leaders on a multi-source feedback assessment of emotional and social competence. Findings revealed that there were significant differences between how leaders were rated on emotional and social competence versus how they predicted they would be rated. Furthermore, results indicated that when asked to predict how they were rated by their managers, direct reports, and peers, women were significantly poorer at predicting others’ ratings compared with men. However, women self-rated their demonstration of emotional and social competence similar to men. Age was investigated, but not found to have an effect on men and women’s predictions of others’ ratings. The article concludes with a discussion of the implications of the findings and areas for future research.

EXPLORING CONDITIONS FOR OPENNESS IN MULTISOURCE FEEDBACK ASSESSMENT.

Multi-Source Feedback assessment (MSF or 360-degree feedback) aims to create change through a comparison of self versus others' ratings. We propose that the ability of an MSF respondent to predict ...