51. Structured assessment format for evaluating operative reports (SAFE-OR) in general surgery

Abstract

This study determined the construct validity, inter-rater reliability and internal consistency of a “Structured Assessment Format for Evaluating Operative Reports” (SAFE-OR) in general surgery. The assessment instrument was developed using consensus criteria set forth by the Canadian Association of General Surgeons. It includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a video-taped laparoscopic sigmoid colectomy. Transcriptions were then graded by blinded, independent faculty evaluators using SAFE-OR. 
 Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, t=0.001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 ± 4.7 vs 35.9 ± 7.6, t=0.006). Inter-class correlation coefficients were 0.98 (95% CI 0.96-0.99) for structured assessment and 0.93 (95% CI 0.83-0.97) for global quality scales. Cronbach’s alpha coefficients for internal consistency were 0.85 for structured assessment and 0.96 for global quality assessment scales.
 SAFE-OR demonstrates significant construct validity, excellent inter-rater reliability and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and ultimately provide a mechanism for implementing, monitoring, and refining curriculum for operative dictation communication skills.
 Moore R. The dictated operative note: important but is it being taught? Journal of the American College of Surgeons 2000; 190(5):639-40.
 Novitsky Y, Sing R, et al. Prospective, blinded evaluation of accuracy of operative reports dictated by surgical residents. The American Surgeon 2005; 71(8):627-31.
 Wanzel K, Ward R, et al. Teaching the surgical craft: From selection to certification. Current Problems in Surgery 2002; 39(6):573-659.