Global Rating Form Research Articles

Effects of Peyton's Four-Step Approach on Objective Performance Measures in Technical Skills Training: A Controlled Trial

Background: Although skills-lab training is widely used for training undergraduates in technical procedures, the way in which clinical skills are to be used and instructed remains a matter of debate. Purpose: We conducted a randomized controlled trial to evaluate the learning outcome of two different instructional approaches in the context of acquiring procedural–technical skills. Methods: Volunteer 2nd- and 3rd-year medical students were randomly assigned to an intervention group receiving instruction according to Peyton's Four-Step Approach (IG; n = 17) or to a control group receiving standard instruction (CG; n = 17). Both groups were taught gastric-tube insertion using a manikin. Following each of the two forms of instruction, participants’ first independent gastric-tube insertions were video recorded and scored by two independent video assessors using binary checklists and global rating forms. The time required for each instructional approach and for the first independent performance of the skill was measured. Results: A total of 34 students agreed to participate in the trial. There were no statistically significant group differences with regard to age, sex, completed education in a medical profession, or completed medical clerkships. The groups did not differ in terms of correct stepwise performance of the procedure as assessed by a binary checklist (p < .802). However, ratings based on global rating scales assessing professionalism and accompanying patient-doctor communication proved significantly better in IG (both ps < .001). The length of the different instructional approaches did not differ significantly between the two groups (IG: 605 ± 65 s; CG: 572 ± 79 s; p < .122), but the time needed for the first independent performance of gastric-tube placement on the manikin was significantly shorter in IG (IG: 168 ± 30 s; CG: 242 ± 53 s; p < .001). Conclusions: Peyton's Four-Step Approach is superior to standard instruction with respect to professionalism and accompanying doctor–patient communication and leads to faster performance when trainees perform the learned skill for the first time.

Objective Assessment of Manual Skills and Proficiency in Performing Epidural Anesthesia—Video-Assisted Validation

Demand is growing for objective assessment of manual skills and competencies of invasive procedures. The aim of this study was to validate an objective tool for assessing residents' skill in performing epidural anesthesia by use of a global assessment scale and a 3-scale, 27-stage checklist. We wish to demonstrate that this tool can differentiate operators with different levels of training. Second-year anesthesia residents were recruited. Their previous experience was assessed by questionnaire. They were repeatedly videotaped performing epidural anesthesia over a 6-month period. Videotaping was done in a blinded manner that masked the identity and level of training of the residents. Three blinded, independent examiners evaluated each session by use of a specifically devised assessment tool that consisted of a global rating scale and a 3-scale, 27-stage checklist to judge the skill level and grade the videotaped sessions. Twenty-one sessions by 6 residents were videotaped over 6 months. Interrater reliability for the different checklist and global-rating form items shows moderate to high degree of agreement for most stages. Total scores demonstrate almost perfect agreement (kappa/ICC +/- SE = 0.90 +/- 0.03 and 0.83 +/- 0.13, respectively; P < .0001) between examiners. To test whether higher total scores are associated with greater experience, a series of repeated-measures ANCOVAs were performed. In both the global-rating form and the checklist, a significant relation between total scores and epidurals done was found to exist (checklist: P < .0001; global rating: P < .0001). The results of our study show that scores on a system that consists of a global-rating form and a task-specific checklist had a significant relation to the number of epidural insertions performed (i.e., experience). The interrater reliability of these assessment tools was very strong. Evaluation of technical skills by an objective tool under direct observation, as opposed to laboratory setting, may create a more reliable standard of assessment. Furthermore, residency programs could use these evaluations to identify deficiencies in teaching programs and trainees who require extra instruction.

Patient, faculty, and self-assessment of radiology resident performance:A 360-degree method of measuring professionalism and interpersonal/communication skills11

Rationale and objectives To develop and test the reliability, validity, and feasibility of a 360-degree evaluation to measure radiology resident competence in professionalism and interpersonal/communication skills. Materials and methods An evaluation form with 10 Likert-type items related to professionalism and interpersonal/communication skills was completed by a resident, supervising radiologist and patient after resident-patient interactions related to breast biopsy procedures. Residents were also evaluated by faculty, using an end-of-rotation global rating form. Residents, faculty, and technologists were queried regarding their reaction to the assessments after a 7-month period. Results Fifty-six complete 360-degree data sets (range, 2–14 per resident) and seven rotational evaluations for seven residents were analyzed and compared. Internal consistency reliability estimates were 0.85, 0.86, and 0.87 for resident, patient, and faculty 360-degree evaluations, respectively. Correlations between resident-versus-patient, resident-versus-faculty, and patient-versus-faculty ratings for the 56 interactions were −0.06 (P = .64), 0.31 (P < .02), and 0.45 (P < .0006), respectively. Pearson correlation coefficients approached significant correlation (0.70) between the faculty global rating and patient 360-degree scores (P = .08) but not with faculty 360-degree scores. Residents and faculty felt that completing the 360-degree forms was easy, but the requirement for faculty presence during the consent process was burdensome. Conclusion Results from this pilot study suggest that self, faculty, and patient evaluations of resident performance constitutes a valid and reliable assessment of resident competence. Additional data are needed to determine whether the 360-degree assessment should be incorporated into residency programs and how frequently the assessment should be performed. Requiring only a specified number of assessments per rotation would make the process less burdensome for residents and faculty.

Patient, faculty, and self-assessment of radiology resident performance:: A 360-degree method of measuring professionalism and interpersonal/communication skills 1

Rationale and objectives To develop and test the reliability, validity, and feasibility of a 360-degree evaluation to measure radiology resident competence in professionalism and interpersonal/communication skills. Materials and methods An evaluation form with 10 Likert-type items related to professionalism and interpersonal/communication skills was completed by a resident, supervising radiologist and patient after resident-patient interactions related to breast biopsy procedures. Residents were also evaluated by faculty, using an end-of-rotation global rating form. Residents, faculty, and technologists were queried regarding their reaction to the assessments after a 7-month period. Results Fifty-six complete 360-degree data sets (range, 2–14 per resident) and seven rotational evaluations for seven residents were analyzed and compared. Internal consistency reliability estimates were 0.85, 0.86, and 0.87 for resident, patient, and faculty 360-degree evaluations, respectively. Correlations between resident-versus-patient, resident-versus-faculty, and patient-versus-faculty ratings for the 56 interactions were −0.06 ( P = .64), 0.31 ( P < .02), and 0.45 ( P < .0006), respectively. Pearson correlation coefficients approached significant correlation (0.70) between the faculty global rating and patient 360-degree scores ( P = .08) but not with faculty 360-degree scores. Residents and faculty felt that completing the 360-degree forms was easy, but the requirement for faculty presence during the consent process was burdensome. Conclusion Results from this pilot study suggest that self, faculty, and patient evaluations of resident performance constitutes a valid and reliable assessment of resident competence. Additional data are needed to determine whether the 360-degree assessment should be incorporated into residency programs and how frequently the assessment should be performed. Requiring only a specified number of assessments per rotation would make the process less burdensome for residents and faculty.

Current practices in evaluating radiology residents, faculty, and programs: results of a survey of radiology residency program directors 1

Rationale and objectives We surveyed program directors to determine current radiology program practices in evaluating their residents, faculty, and program. Materials and methods In January 2003, a 52-item Web-based survey was made available to program directors of accredited core radiology programs. Responses to the items were tabulated to determine relative frequency distribution. Two-tailed Pearson chi-square tests were used to compare proportions and assess the association between variables. Results A total of 99 (52%) of 192 program directors responded. Programs were largely in compliance with Accreditation Council for Graduate Medical Education (ACGME) requirements. Noncompliance was related to the requirements to evaluate residents at least four times per year in at least 22 (22.2%) of 99 programs and annually evaluate the program in 20 (20.2%) of 99 programs. New program directors (<1-year tenure) were less likely than those with ≥1-year tenure to be using the Association of Program Directors in Radiology Education Committee global rating form (41.2% versus 68.8%, P = .03). Programs that used this form, compared with those that didn’t, were more likely to evaluate resident competence in systems-based practice (88.5% versus 44.0%, P = .001). Being a program director for 1 or more years versus less than 1 year was associated with using a computerized evaluation system (35.8% versus 11.8%, P = .05). Conclusion In general, there is a high degree of compliance among radiology programs in meeting ACGME evaluation requirements. However, some programs do not comply with requirements for frequency of resident evaluation or annual program evaluation. The percentage of new program directors is high and related to not using or knowing about useful evaluation resources. Use of computerized evaluation systems, which have the potential to decrease the work associated with evaluations and provide more dependable and reliable data, is minimal.

A feasibility study comparing checklists and global rating forms to assess resident performance in clinical skills

This study evaluated the feasibility of two different scoring forms for assessing the clinical performance of residents in anaesthesiology. One of the forms had a checklist format including task-specific items and the other was a global rating form with general dimensions of competence including ‘clinical skills’, ‘communication skills’ and ‘knowledge’. Thirty-two clinicians representing 25 (83%) of the 30 training hospitals in the country participated in the study. The clinicians were randomized into two groups, each of which used one of the scoring formats to assess a resident’s performance in four simulated clinical scenarios on videotape. Clinicians’ opinions about the appropriateness of the scoring forms were rated on a scale of 1–5. The checklist format was rated significantly higher compared with the global rating form (mean 4.6, sd 0.5 vs. mean 3.5, sd 1.4, p < 0.001). The inter-rater agreement regarding pass/fail decisions was poor irrespective of the scoring form used. This was explained by clinicians’ leniency as assessors rather than by lack of vigilance in the observations or disagreements on standards for good performance.

The Effect of Candidatesʼ Perceptions of the Evaluation Method on Reliability of Checklist and Global Rating Scores in an Objective Structured Clinical Examination

Process-oriented global ratings, which assess "overall performance" on one or a number of domains, have been purported to capture nuances of expert performance better than checklists. Pilot data indicate that students change behaviors depending on their perceptions of how they are being scored, while experts do not. This study examines the impact of the students' orientation to the rating system on OSCE scores and the interstation reliability of the checklist and global scores. A total of 57 third- and fourth-year medical students at one school were randomly assigned to two groups and performed a ten-station OSCE. Group 1 was told that scores were based on checklists. Group 2 was informed that performance would be rated using global ratings geared toward assessing overall competence. All candidates were scored by physician-examiners who were unaware of the students' orientations to the rating system and who used both checklists and global rating forms. A mixed two-factor ANOVA identified a significant interaction of rating form by group (F(1,55) = 5.5, p <.05), with Group 1 (checklist-oriented) having higher checklist scores but lower global scores than did Group 2 (oriented to global ratings). In addition, Group 1 had higher interstation alpha coefficients than did Group 2 for both global scores (0.74 versus 0.63) and checklist scores (0.63 versus 0.40). The interaction effect on total exam scores suggests that students adapt their behaviors to the system of evaluation. However, the lower reliability coefficients for both forms found in the process-oriented global-rating group suggest that an individual's capacity to adapt to the system of global rating forms is relatively station-specific, possibly depending on his or her expertise in the domain represented in each station.

Exporting a technical skills evaluation technology to other sites

Background: The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination. Methods: An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination. Results: There was good reliability for both the checklist (α = 0.68 for LA, 0.73 for Chicago) and global rating forms (α = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P <0.01). Despite some challenges, the model of central organization and peripheral delivery was effective for the administration of the examinations. Conclusions: Two iterations of the OSATS at remote sites demonstrated psychometric properties that are highly consistent with previously reported data suggesting that the examination is portable. Both faculty and residents indicated satisfaction with the examination experience. A model of central administration with peripheral delivery was feasible and effective.

Exploring the Concept of Acceptability as a Criterion for Evaluating Performance Measures

Although the value of an appraisal system traditionally has been judged according to reliability and validity indexes, more recent discussion suggests that user acceptance may be critical to a system’s successful implementation and continued use. This study examined the notion of using acceptability as a criterion for evaluating performance appraisal techniques. Data analyses indicated that motivation to rate, trust in others, and situational constraints were predictive of acceptability for both supervisors and job incumbents. In addition, differences in acceptability were found across rating sources and rating forms, with supervisors’perceptions more favorable than job incumbents’ and a global rating form significantly less acceptable to all raters. Results are discussed in terms of usefulness of an acceptability criterion in applied research.

Objective structured assessment of technical skill (OSATS) for surgical residents.

The technical skill of surgical trainees is not well assessed. This study aimed (1) to compare the reliability of three scoring systems, (2) to compare live and bench formats and (3) to assess construct validity of a test of operative skill. Parallel examinations of operative skill, one using live animals and one using simulations, were developed. Performance was graded using operation-specific checklists, detailed global rating forms and pass/fail judgements. Twenty surgical residents each took both formats. Disattenuated correlations between live and bench scores were high (0.69-0.72). Mean interrater reliability across stations ranged from 0.64 to 0.72. Internal consistency was moderate to high (alpha: 0.61-0.74) for the live format using the checklist and for live and bench formats using global ratings. Global ratings discriminated between resident levels for both formats (bench: F(2,17) = 4.45, P < 0.05; live: F(2,17) = 3.55, P < 0.05), checklists did not. This preliminary study suggests that the Objective Structured Assessment of Technical Skill can reliably and validly assess surgical skills. Global ratings are a better method of assessment than task-specific checklists. Bench model simulation gives equivalent results to use of live animals for this test format.

Reliability and construct validity of a Structured Technical Skills Assessment Form

Current methods of evaluating technical competence of surgical residents are subjective and potentially unreliable. This study assesses the reliability and construct validity of a new format for the assessment of technical ability, the two part Structured Technical Skills Assessment Form (STSAF). Part I, which is completed while an operation is proceeding consists of approximately 120 essential components of the procedure. Part II, completed at the end of the operation, is a 10-point global rating form. Forty-one operations were evaluated using the STSAF, with multiple observers present at 26. Inter-rater reliability of both Parts I and II were high (.78 and .73, respectively). Statistically significant differences were noted between senior-resident and junior-resident performances, suggesting construct validity. The incorporation of structured guidelines to the assessment of technical skill leads to high inter-rater reliability and construct validity, which ultimately may result in improved and reproducible evaluations of surgical trainees.