Multisource Feedback Assessment Research Articles

How ratings vary by staff group in multi-source feedback assessment of junior doctors

UK doctors-in-training undergo assessments of their professional behaviours. From an analysis of multi-source feedback (MSF) data, we report how ratings of junior doctors (Foundation Programme [FP] doctors and senior house officers [SHOs]) differed by staff group. The MSF data were collected in 2003 and 2005 in hospitals in the West Midlands. Using a single-sided Team Assessment of Behaviour form, 1928 assessors evaluated 226 FP doctors and SHOs in four domains: professional relationship with patients; communication; team-working, and accessibility. The distribution of 'concerns' across the professional groups was explored using a random effects logistic regression model. On average, each trainee received nine assessment forms from a range of staff, most commonly nurses. Although concerns were identified for the minority, ratings varied by staff group. Peers (other FP doctors or SHOs) and administrators or managers were four and three times, respectively, less likely to indicate concern. By contrast, consultants and sisters (senior nurses) were more likely to give concern ratings. Guidance on the selection of assessors in any MSF process should take into account findings that rating behaviour varies by staff group.

Implementation of a Multi-Source Feedback System for Quality Assurance for the College of Medical Radiation Technologists of Ontario

Under the Health Professions procedural code of the Regulated Health Professions Act, 1991 (RHPA), the College of Medical Radiation Technologists of Ontario (CMRTO) is required to provide a means to assess how people in the profession actually perform in practice. As the first step in satisfying the practice assessment requirement of the Quality Assurance program in 2004, the CMRTO retained Edumetrics Ltd. to undertake the development of a multi-source feedback (MSF) program specific to the profession of medical radiation technology. In a comprehensive study led by Dr. Claudio Violato, a professor in the Department of Community Health Sciences, Faculty of Medicine at the University of Calgary and an expert in the development of program evaluations, psychometrics, research methods and statistics, the CMRTO developed a 'multi-source' assessment process that provides a means for peers, colleagues, patients and the assessed person to complete a survey focused on the standards of practice. The assessed person receives a summary assessment report or feedback about his or her performance. The multi-source feedback assessment provides a formative evaluation, providing continuous reaction to the performance of a medical radiation technologist (MRT). It compares the MRT's clinical performance to that of other MRTs. The CMRTO multi-source practice assessment approach was piloted in 2005. A number of strategies were used to recruit 327 MRTs to take part in the pilot study. Recruitment strategies included consultation with MRTs respecting the standards of practice as the basis of the MSF project, a detailed communication strategy and telephone recruitment. Although the CMRTO staff experienced challenges in recruiting MRTs to take part in the pilot study and dealing with the resultant feedback, the pilot study was considered a success.

Why the real self is fundamental to intentional change

Purpose – The purpose of this paper is to explore the role of the real self in intentional change.Design/methodology/approach – Social science theory and research are used to respond to four primary questions: what is the real self, why is the real self essential to intentional change, how does one access the real self, and what are the challenges to accessing the real self with respect to intentional change?Findings – The implications of this paper suggest scholars and practitioners should revisit the assumptions made in employee and leadership competency development efforts by seeking to draw upon a more comprehensive self of the individual being developed. Implications for further research show the need for a new direction in multisource feedback assessment research and use and empirical studies that understand the real self in relation to the other components of intentional change theory (ICT).Practical implications – Implications for practice reveal a need to rethink how executive coaches and human r...

Assessing Medical Radiation Technologists in Practice: A Multi-Source Feedback System for Quality Assurance

Objective To develop and psychometrically evaluate (feasibility, reliability, validity) a questionnaire based multi-source feedback (MSF) system for quality assurance (QA) for medical radiation technologists (MRTs). Methods Surveys with 12, 14, 30 and 31 items were developed for assessment of MRTs by patients, co-workers, MRT colleagues and themselves, respectively, using 5-point scales with an “unable to assess” category. Core competencies assessed were related to legislation, standards of practice and ethics, equipment and materials, diagnostic examinations and radiation treatment, safe practice, relationship with patients, records and reporting, communications, team work and professional development. Each MRT was assessed by 25 patients, 6 MRT colleagues, and 6 co-workers. Feasibility was assessed with response rates for each instrument. Validity was assessed with a table of specifications, the percentage of participants unable to assess the MRT for each item, and exploratory factor analyses to determine which items grouped together into scales. Cronbach's α assessed reliability. Results A sample of 307 MRTs registered with the College of Medical Radiation Technologists of Ontario (CMRTO) volunteered to undergo MSF assessments. The number of respondents for the MRT peer questionnaire was 1,730, for the co-worker questionnaire was 1,745 and for the patient questionnaire were 7,007. The mean ratings ranged from 4 to 5 for each item on each scale. All of the instruments' full scales had very high Cronbach's α > .92. The factor analyses revealed a 5-factor solution (71.28% of the total variance) for the peer and self surveys, a 3-factor solution for the coworker questionnaire (74.28% of the variance), and a 2-factor solution for the patient questionnaire (81.53% of the variance). Conclusion An MSF system employing surveys that have high reliability, validity and feasibility were developed to provide feedback to MRTs on core competencies and skills.

Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness

Physician performance is comprised of several domains of professional competence. Multisource feedback (MSF) or 360-degree feedback is an approach used to assess these, particularly the humanistic and relational competencies. Research studying responses to performance assessment shows that reactions vary and can influence how performance feedback is used. Improvement does not always result, especially when feedback is perceived as negative. This small qualitative study undertook preliminary exploration of physicians' reactions to MSF, and perceptions influencing these and the acceptance and use of their feedback. We held focus groups with 15 family physicians participating in an MSF pilot study. Qualitative analyses included content and constant comparative analyses. Participants agreed that the purpose of MSF assessment should be to enhance practice and generally agreed with their patients' feedback. However, responses to medical colleague and co-worker feedback ranged from positive to negative. Several participants who responded negatively did not agree with their feedback nor were inclined to use it for practice improvement. Reactions were influenced by perceptions of accuracy, credibility and usefulness of feedback. Factors shaping these perceptions included: recruiting credible reviewers, ability of reviewers to make objective assessments, use of the assessment tool and specificity of the feedback. Physicians' perceptions of the MSF process and feedback can influence how and if they use the feedback for practice improvement. These findings are important, raising the concern that feedback perceived as negative and not useful will have no or negative results, and highlight questions for further study.

Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study.

To describe responses of family physicians, their medical colleagues, and coworker raters to a multisource feedback assessment process. Data collection tools included multisource feedback self-assessment and medical colleague, coworker, and patient rating forms; and program evaluation physician and rater questionnaires. The pilot study included 142 physicians and their raters, with 113 (80%) physicians completing evaluations. Positive correlations were found between familiarity scores and medical colleague and coworker mean ratings. Peer medical colleagues were significantly more familiar with physicians than were consultants. Consultants were unable to rate items most frequently. Physicians disagreed with colleague feedback more frequently. Agreement was positively correlated with scores. Familiarity, ability to observe physicians appropriately to rate them, and physicians' responses to feedback are factors to consider when multisource feedback is used.