Development and Evaluation of a New Formative Assessment of Surgical Performance

Abstract

Formative assessment of operative performance is a mandatory part of surgical training. Engagement with assessment is limited in part by the time-consuming nature and the high perceived stakes of current assessment tools. Our aims were to develop and collect validity evidence for a new operative assessment tool that addresses barriers to assessment that current trainers and trainees experience. We developed the Generic Operative Supervised Learning Event (GOSLE). Orthopedic trainees were invited to complete GOSLEs with their trainers after surgical procedures. Experienced consultants assessed videotaped operations performed by trainees using the GOSLE. Validity evidence for content, relationships to other scores, internal structure, response process, and consequences of testing were evaluated. A total of 250 GOSLEs were completed. A strong correlation was found between the GOSLE scores and the Procedure-Based Assessment ratings (r = 0.87, p < 0.001). Rasch analysis confirmed satisfactory internal structure of the rating scale, with sequential increases in rating as performance improved. The reproducibility coefficient was 0.88, with 10 assessments of the same trainee who has to achieve a reliability coefficient of 0.8. Over 90% of users found the GOSLE easy to use, with most preferring it to other assessment methods. Feedback quality was higher using the GOSLE than with current assessments. We have collected validity evidence across multiple domains in support of the GOSLE. Its psychometric performance is comparable to that of current assessments. It is preferred by trainers and trainees over existing assessments. It stimulates high-quality, actionable feedback which better supports formative assessment. By addressing issues experienced with existing assessments, we expect engagement among users to be high.