Evaluation of Oral and Maxillofacial Surgery Residents' Operative Skills: Feasibility and Engagement Study Using SIMPL Software for a Mobile Phone

Abstract

There are no universally accepted tools to evaluate operative skills of surgical residents in a timely fashion. The purpose of this study was to determine the feasibility of using a smartphone application, SIMPL (System for Improving and Measuring Procedural Learning), developed by a multi-institutional research collaborative, to achieve a high rate of timely operative evaluations and resident communication and to collect performance data. The authors hypothesized that these goals would be achieved because the process is convenient and efficient. This was a prospective feasibility and engagement study using SIMPL to evaluate residents' operative skills. SIMPL requires the attending surgeon to answer 3 multiple-choice questions: 1) What level of help (Zwisch Scale) was required by the trainee? 2) What was the level of performance? 3) How complex was the case? The evaluator also can dictate a narrative. The sample was composed of 3 faculty members and 3 volunteer senior residents. Predictor variables were the surgeons, trainees, and procedures performed. Outcome variables included number and percentage of procedures performed by faculty-and-resident pairs assessed, time required to complete assessments, time lapsed to submission, percentage of assessments with narratives, and residents' response rates. From March through June 2016, 151 procedures were performed in the operating room by the faculty-and-resident teams. There were 107 assessments submitted (71%). Resident response (self-assessment) to faculty evaluations was 81%. Recorded time to complete assessments (n=75 of 107) was shorter than 2minutes. The time lapsed to submission was shorter than 72hours (100%). Dictations were submitted for 35 evaluations (33%). Data for the type of help, performance, and complexity of cases were collected for each resident. SIMPL facilitates timely intraoperative evaluations of surgical skills, engagement by faculty and residents, and collection of detailed procedural data. Additional prospective trials to assess this tool further are planned.