Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

Abstract

The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model. Over 18months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.