Assessing the Reliability and Validity of Technical Skill Assessment Tools and the Impact of Rater Training

Abstract

Competency-based education frameworks are increasingly being used in surgical training. Such frameworks require robust assessments of technical skill, yet the optimal use of assessment tools has not been well defined. We assessed the effect of a rater training (RT) program on the reliability and validity of 4 technical assessment tools. Surgeons were randomized to RT and no-training groups. A pass-fail system, a visual analog scale (VAS), a task-specific checklist, and an OSATS global rating scale (GRS) were used to assess trainees performing a suturing and knot-tying task. Trainee performances were assessed again 2 weeks later. Internal consistency, intra-rater and inter-rater reliability, and construct validity were measured. Forty-seven surgeons were randomized to RT (n = 24) and nontraining (n = 23). The OSATS GRS had superior reliability and validity compared to the other assessment tools regardless of training. Although there were no significant differences in reliability between training groups, the RT group trended to improved reliability for all assessment forms at both assessments (Table).TableAssessment formInter-rater reliabilityIntra-rater reliabilityInitialDelayedPass-fail RT0.22 (0.18–0.26)0.16 (0.13–0.20)0.45 (0.32–0.59) No training0.20 (0.17–0.25)0.11 (0.07–0.15)0.41 (0.26–0.57)VAS RT0.71 (0.50–0.91)0.54 (0.34–0.80)0.71 (0.64–0.77) No training0.46 (0.27–0.75)0.43 (0.24–0.73)0.62 (0.54–0.70)Checklist RT0.46 (0.27–0.75)0.42 (0.23–0.71)0.53 (0.43–0.61) No training0.33 (0.17–0.64)0.27 (0.13–0.57)0.46 (0.35–0.56)GRS RT0.71 (0.52–0.89)0.64 (0.45–0.86)0.73 (0.67–0.79) No training0.61 (0.41–0.85)0.52 (0.32–0.79)0.66 (0.58–0.73) Open table in a new tab RT did not lead to significant or sustained improvements in technical skill assessments. Trainee evaluations should be performed with standardized assessment tools, such as the OSATS GRS, due to their superior reliability and validity over less standardized methods. Despite our findings, training remains an important tool to ensure the most effective technical skill evaluations. More effective training methods or repeated sessions may be required for sustained or significant effects.