Use of an Objective Structured Assessment of Technical Skill (OSATS) after a Sports Rotation

Abstract

Objectives:While the acquisition of competent technical skill is a defining characteristic of surgery, it is not measured systematically in residency. As all post-graduate medical training slowly shifts to a competency-based model, effective assessments of surgical and other technical skills after modules will become necessary. However, the best method for of assessing competence in technical skill in this setting is unknown, and is limited by both cost and access to resources. We hypothesized that a multi-station Objective Structured Assessment of Technical skill (OSATS), using sawbones models, would be a valid and reliable method of assessing resident competence in surgical skills after a sports medicine rotation.Methods:At the start of their three-month sports medicine rotation, each resident was provided a list of 10 surgical skills in which they were expected to demonstrate competence (Table 1). At the end of the rotation, each resident undertook an OSATS comprised of six randomly chosen stations - low-fidelity sawbones models were used in all stations. Residents were evaluated by faculty / staff surgeons using a previously validated global rating scale (the Arthroscopic Surgical Skill Evaluation Tool (ASSET)), as well as task-specific checklists created using a modified Delphi procedure, and a final five-point global rating scale (GRS) using the Drefus model of skill acquisition (1=novice, 2=advanced beginner, 3=competent, 4=proficient, 5=expert). All arthroscopic procedures were recorded, and all hand movements were videotaped - the videos were reviewed by a single, blinded observer, and correlation sought between the faculty ratings and the observer ratings.Results:Over 18 months, 27 residents (19 junior, 8 senior) sat the OSATS after their rotation, as well as seven sports medicine staff and seven fellows, for a total of 41 participants. The overall reliability of the OSATS as measured by Cronbach's Alpha was very high (0.9). A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each individual technical skill (p<0.001) - this difference was also seen for all stations. Post hoc analysis demonstrated a significant difference in the total ASSET score between junior (PGY1-3) and senior (PGY4&5) residents, senior residents and fellows, as well as between fellows and faculty (p<0.05)(Figure 1). A high correlation was seen between the faculty assessments and the blinded observer assessments for each station (>0.8).Conclusion:The results of this study demonstrate that an OSATS using dry models is a valid and reliable means of assessing technical skill in orthopaedic residents after a sports medical rotation. Interestingly, junior residents were not able to perform technical skills as well as senior residents despite an identical rotation, suggesting that overall surgical experience and exposure is as important as intensive teaching.