Assessment of resident surgical skills: Is testing feasible?

Abstract

We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results. An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models. Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's alpha were .97 for overall global rating and .95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from .71 to .97 for individual tasks and was .95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was 40 US dollars to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training. Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.