Proficiency-based Simulator Training Research Articles

Evaluation of different approaches to define expert benchmark scores for new robotic training simulators based on the Medtronic HUGO™ RAS surgical robot experience.

New robot-assisted surgery platforms being developed will be required to have proficiency-based simulation training available. Scoring methodologies and performance feedback for trainees are currently not consistent across all robotic simulator platforms. Also, there are virtually no prior publications on how VR simulation passing benchmarks have been established. This paper compares methods evaluated to determine the proficiency-based scoring thresholds (a.k.a. benchmarks) for the new Medtronic Hugo™ RAS robotic simulator. Nine experienced robotic surgeons from multiple disciplines performed the 49 skills exercises 5 times each. The data were analyzed in 3 different ways: (1) include all data collected, (2) exclude first sessions, (3) exclude outliers. Eliminating the first session discounts becoming familiar with the exercise. Discounting outliers allows removal of potentially erroneous data that may be due to technical issues, unexpected distractions, etc. Outliers were identified using a common statistical technique involving the interquartile range of the data. Using each method above, mean and standard deviations were calculated, and the benchmark was set at a value of 1 standard deviation above the mean. In comparison to including all the data, when outliers are excluded, fewer data points are removed than just excluding first sessions, and the metric benchmarks are made more difficult by an average of 11%. When first sessions are excluded, the metric benchmarks are made easier by an average of about 2%. In comparison with benchmarks calculated using all data points, excluding outliers resulted in the biggest change making the benchmarks more challenging. We determined that this method provided the best representation of the data. These benchmarks should be validated with future clinical training studies.

FES exam outcomes in year two of a proficiency-based endoscopic skills curriculum.

FES certification is required to sit for the ABS Qualifying Exam. Previous work demonstrated a 40% FES pass rate for residents with standard clinical endoscopy training. After implementing a proficiency-based simulation curriculum, our FES pass rate increased to 87%. The purpose of this study was to monitor the success of our curriculum in its second year. We also hypothesized that residents who took the FES exam within 30days of their clinical endoscopy rotation would have superior pass rates to residents who waited longer. PGY4 residents (N = 12) underwent flexible endoscopy training including a 1month clinical rotation plus proficiency-based simulation training using bench-top models (Trus, Operation Targeting Task) and a virtual reality task on the GI Mentor. Residents that passed FES on their first attempt were compared to residents that did not pass based on number of endoscopies logged, hours spent practicing on simulators, and time elapsed between completing their endoscopy rotation and taking the FES exam. FES total scores and section scores were compared to historical controls. Nine residents (75%) passed FES on their first attempt. Overall, 80% of residents who tested within 30days of their endoscopy rotation (n = 5) passed FES while 71% of residents who waited longer (n = 7) passed FES (p = non-significant). Residents that passed FES were not significantly different from residents who did not pass based on number of endoscopies logged or hours spent practicing on simulators. Compared to historical controls, scores on loop reduction improved significantly with the new curriculum. FES pass rates decreased during the second year of our curriculum. Based on other literature, our trainees would benefit from higher volumes of endoscopy and/or a more robust proficiency-based simulation curriculum. Scheduling the FES exam in the month following the endoscopy rotation did not significantly improve pass rates.

MP35-07 DEVELOPMENT AND VALIDATION OF A SIMULATOR-BASED TEST IN TRANSURETHRAL RESECTION OF BLADDER TUMORS ESTABLISHING PASS/FAIL STANDARDS

INTRODUCTION AND OBJECTIVES:To develop a simulator-based test with ability to discriminate between competency levels and establish a pass/fail score for proficiency-based simulator training in TURB...

Performance variability during training on simulators is associated with skill transfer

BackgroundExpert performance is characterized by consistency. The degree of variability of performance from repetition to repetition during proficiency-based simulator training could potentially indicate acquisition of expertise. We hypothesized that learners with less variability in performance during simulator training would achieve greater performance at the end of training and improved transfer of skills to a live, anesthetized, porcine model. MethodsThe performance of 93 subjects (surgery residents and medical students) who had participated in 3 randomized controlled trials was analyzed for variability. All participants had trained in laparoscopic suturing on the Fundamentals of Laparoscopic Surgery (FLS) simulator. Their performance had been assessed on the simulator before (baseline) and after training (posttest) and on a live, anesthetized, porcine model (transfer test). We computed the coefficient of variations of suturing scores during training for each participant. Linear regression was used to assess whether variability in performance during training predicted posttest and transfer-test scores. ResultsDecreased practice variability in performance was associated with greater scores in posttests and transfer tests. For each percent decrease in variability performance, posttest scores increased by 3.8 points (P < .001) and transfer-test scores increased by 3.0 points (P < .001). Greater mean scores during practice were associated with greater scores on the transfer test (P < .001). ConclusionDecreased variability in performance during practice on simulators is associated with improved performance at the end of training and during transfer to a live, anesthetized, porcine model. These findings suggest that variability in performance during simulator training may be used to track the progress and readiness of a trainee for the clinical environment. Further studies are needed to verify the robustness of this potentially new metric of performance.

Development of a Proficiency-Based Simulation Training Model for Laparoscopic Vaginal Cuff Closure

Faculty Advisor: Amy Boardman, MD PURPOSE: To evaluate if a low fidelity simulation model and training video will improve resident surgical performance of laparoscopic vaginal cuff closure. BACKGROUND: Laparoscopic surgical complexity and resident work-hour restrictions have increased the need for surgical simulation in OBGYN residencies. Evidence of transfer of acquired surgical skills from simulation to the operating room (OR) is limited. METHODS: This prospective, randomized study compares resident surgical skills acquisition after low-fidelity simulation training of vaginal cuff closure versus no intervention. Study phases include development of a vaginal cuff closure training video; video documentation of resident lab performance; and video assessment of resident surgical skill by blinded surgical experts using I – GOALS, II – error events, and III – surgical checklist. RESULTS: OBGYN residents (N=18) from MedStar Georgetown and Washington Hospital Center were randomized to control group (N=7) or intervention group (N=11). Intervention group reviewed the training video on laparoscopic vaginal cuff closure. All residents then used the low-fidelity cuff closure model and recorded baseline videos. Three surgeons scored the videos with an inter-rater reliability of 0.585, 0.531, and 0.825 for tests I, II, and III. Wilcoxon Rank Sum test was used to evaluate differences between groups by averaging all raters’ scores. There was significant difference between the control and intervention groups for Test II (P=.01) but no difference between the control and intervention groups for tests I and III. DISCUSSION: Simulation training of laparoscopic vaginal cuff closure reduces error events after immediate exposure. Continued research will evaluate if acquired skills transfer to OR performance.

Validation of a High-Fidelity Electrophysiology Simulator and Development of a Proficiency-Based Simulator Training Program.

Simulator training can potentially provide high-intensity training in electrophysiology (EP) without compromising patient safety. We assessed the construct validity of a novel EP simulator (ANGIO Mentor; Simbionix) and developed proficiency-based scores for clinical EP simulator training. Two European training centers participated. Participants were grouped on the basis of years of EP experience and (for a subset) subjectively scored clinical catheter manipulation skills. Each participant attempted the simulator's 5 modules 3 times. These modules focus on catheter manipulation in 3-dimensional models, ranging from geometric shapes to fluoroscopic contracting cardiac models, with performance scored by the system on the basis of attainment of preset targets. Using these scores, targets were formulated for basic EP training. Twenty-eight participants were included (13 with subjectively scored catheter manipulation). Scores for participants with less than 1-year EP experience (group 1) were significantly lower for each of the attempts at the modules (P < 0.002). For group 1 only, scores improved with subsequent attempts (P < 0.005). In 4 of the 5 modules, scores of group 1 were significantly lower than the more experienced groups (P < 0.0005). Participants with subjectively scored above-average catheter manipulation skills also had higher scores in 4 of the 5 modules (P < 0.05). Target scores for a proficiency-based training program were generated from the median scores for each module for those with 1-year experience or more. Scores attained in the simulator can distinguish those with less than 1-year EP experience and those with above-average catheter manipulation skills. Consequently, target scores have been generated for a proficiency-based training program.

Ensuring Competency of Novice Laparoscopic Surgeons—Exploring Standard Setting Methods and their Consequences

Simulation-based assessment tools have been developed to allow for proficiency-based simulator training in laparoscopy. However, few studies have examined the consequences of different standard setting methods or examined what level of proficiency is considered adequate for trainees. The objectives of the present study were to explore the consequences of different standard setting methods and to examine the proficiency level that surgical trainees are expected to reach, before performing supervised surgery on patients. Study participants undertook the Training and Assessment of Basic Laparoscopic Techniques test. The tests were video-recorded and rated using a simple scoring system based on number of errors and time. Participants were then asked to assess how high a score a novice should reach before performing supervised surgery on a patient. We then compared 3 methods of standard setting: expert performance level, contrasting groups method, and a modified Angoff method. The study was conducted at the Copenhagen Academy for Medical Education and Simulation. The academy provides surgical simulation training in laparoscopy for trainees at the hospitals in the Capital Region and the Zealand Region of Denmark. Participants were recruited among surgical trainees in their first year of specialty training from surgery, gynecology, and urology departments. A total of 40 participants were included and completed the trial. The different standard setting methods resulted in different pass/fail levels. At the expert performance level, the pass/fail level was 474 points-the contrasting groups method resulted in 358 points and the modified Angoff method resulted in 311 points among experienced surgeons, and 386 points among trainees. The different proficiency levels resulted in a failure rate of 0% to 50% of experienced surgeons and a pass rate of 0% to 25% of novices. Novice laparoscopic surgeons set a higher pass/fail level than experienced surgeons did (p = 0.008). Required proficiency levels varies depending on the standard setting method used, which highlights the importance of using an established standard setting method to set the pass/fail level.

When Experts are Oceans Apart: Comparing Expert Performance Values for Proficiency-Based Laparoscopic Simulator Training

Surgical training is moving away from the operating room toward simulation-based skills training facilities. This has led to the development of proficiency-based training courses in which expert performance data are used for feedback and assessment. However, few expert value data sets have been published, and no standard method for generating expert values has been adopted by the field. To investigate the effect of different proficiency value data sets on simulator training courses, we (1) compared 2 published expert performance data sets for the LapSim laparoscopic virtual-reality simulator (by van Dongen et al. and Heinrichs et al.) and (2) assessed the effect of using either set on LapSim training data obtained from 16 local residents in surgery and gynecology. Across all simulator tasks, the experts consulted by van Dongen et al. performed better on motion efficiency, but not on duration or damage control. Applying both proficiency sets to training data collected during a basic skills laparoscopic simulator course, residents would have graduated on an average in 1.5 fewer sessions using the Heinrichs expert values compared with the van Dongen expert values. The selection of proficiency values for proficiency-based simulator training courses affects training length, skills level assessment, and training costs. Standardized, well-controlled methods are necessary to create valid and reliable expert values for use in training and research.

Skills Transfer After Proficiency-Based Simulation Training in Superficial Femoral Artery Angioplasty

The purpose of this study was to explore whether basic endovascular skills acquired using proficiency-based simulation training in superficial femoral artery (SFA) angioplasty translate to real-world performance. Five international experts were invited to evaluate a preliminary 28-item rating scale for SFA angioplasty using a modified Delphi study. To test the procedural scale, 4 experts and 11 final-year medical students then performed 2 SFA angioplasties each on the vascular intervention simulation trainer simulator. Thereafter, 10 general surgical residents (novices) received didactic training in SFA angioplasty. Trainees were then randomized with 5 trainees receiving further training on the vascular intervention simulation trainer simulator up to proficiency level. All 10 trainees then performed 1 SFA angioplasty on a patient within 5 days of training. The trainees' performance was assessed by 1 attending consultant blinded to the trainees' training status, using the developed procedural scale and a global rating scale. Four items were eliminated from the procedural scale after the Delphi study. There were significant differences in the procedural scale scores between the experts and the students in the first trial [mean (SD), 94.25 (2.22) vs. 74.90 (8.79), P = 0.001] and the second trial [95.25 (0.50) vs. 76.82 (9.44), P < 0.001]. Simulation-trained trainees scored higher than the controls on the procedural scale [86.8 (5.4) vs. 67.6 (6), P = 0.001] and the global rating scale [37.2 (4.1) vs. 24.4 (5.3), P = 0.003]. Basic endovascular skills acquired using proficiency-based simulation training in SFA angioplasty do translate to real-world performance.

Development and evaluation of a simulator-based laparoscopic training program for surgical novices

The use of simulation to train novice surgeons in laparoscopic skills is becoming increasingly popular. To maximize benefit from simulation, training needs to be delivered and assessed in a structured manner. This study aimed to define performance goals, demonstrate construct validity of the training program, and evaluate whether novice surgeons could reach the preset performance goals. Nine expert laparoscopic surgeons established performance goals for three basic modules of an augmented-reality laparoscopic simulator. The three laparoscopic modules were used by 40 novice surgeons and 40 surgical trainees (postgraduate years [PGYs] 1-4). The performance outcomes were analyzed across the different groups (novice, PGYs 1 and 2, PGYs 3 and 4, expert) to determine construct validity. Then 26 recruited novices trained on the three modules with the aim of reaching the performance goals. The results demonstrated a significant difference in performance between all levels of experience for time (p < 0.001), motion analysis (p < 0.001), and error score (p < 0.001), thus demonstrating construct validity. All 26 novice surgeons significantly improved in performance with repetition for the metrics of time (p < 0.001) and motion analysis (p < 0.001). For two of the modules, the proficiency goals were reached in fewer than 10 trials by 80% of the study participants. Basic skills in laparoscopic surgery can be learned and improved using proficiency-based simulation training. It is possible for novice surgeons to achieve predefined performance goals in a reasonable time frame.

Increased stress levels may explain the incomplete transfer of simulator-acquired skill to the operating room

Proficiency-based simulator training in laparoscopic suturing leads to improved operative performance, but the skill transfer is incomplete. The objective of this study was to examine the stress level of trainees during the transition from the simulator to the operating room (OR) and its impact on performance. Novices (n = 20) were randomized into training and control groups. After the training group achieved proficiency in laparoscopic suturing, both groups were tested on a live porcine, laparoscopic Nissen fundoplication model. Participant performance was assessed using an objective score. Stress level was evaluated by recording beat-to-beat heart rate (BBHR) and short-term heart rate variability (STHRV) at baseline, after achieving proficiency (only the training group) and in the OR. Repeated measurement analysis of variance (ANOVA) and t test were used for analysis. Baseline simulator performance and data for heart rate variability were similar for both groups. After achieving simulator proficiency, the trained group demonstrated the anticipated decrease in performance (mean average + or - SEM) in the OR (524 + or - 17 vs 290 + or - 95; P < .001), and an increase in BBHR (98 + or - 14 vs115 + or - 18; P < .001) but not STHRV (4.1 + or - 0.8 vs 3.7 + or - 0.9; P = .5). A similar but lesser increase of the BBHR was observed in the control group compared to the study group. BBHR was a more sensitive measure of stress level compared with STHRV. The increased BBHR observed in the OR that reflects stress and performance anxiety may explain the incomplete transfer of simulator-acquired skill in novice learners.

Do shorter training intervals lead to superior skill acquisition during proficiency-based simulator training?

Do shorter training intervals lead to superior skill acquisition during proficiency-based simulator training?

Do Metrics Matter? Time Versus Motion Tracking for Performance Assessment of Proficiency-Based Laparoscopic Skills Training

The purpose of this study was to compare the added value of motion metrics in determining training completion during a proficiency-based simulator curriculum compared with traditional metrics (time). Novices (n=16) practiced on a basic laparoscopic task of a hybrid simulator until expert-derived proficiency levels for time, path length, and smoothness were achieved on two consecutive attempts. The order by which proficiency in each metric was achieved was recorded and correlated to baseline characteristics. Motion metrics were considered valuable if their incorporation led to extension of training duration. Compared with baseline participant performance improved at training completion according to all metrics (time 67+/-17 to 20+/-6 seconds; P<0.001, pathlength 5326+/-1444 to 2339+/-545 cm; P<0.001, and smoothness from 529+/-185 to 133+/-59; P<0.001). Pathlength was the easiest metric to reach the proficiency level and time the most difficult. Four (33%) participants benefited from the motion metrics as their training was prolonged by an average of 50% compared with using time alone. Baseline characteristics did not correlate to the order of achievement of these metrics. Time may be superior to motion tracking metrics for performance assessment during proficiency-based simulator training. Nevertheless, in this study one third of trainees benefited from motion analysis metrics by having their training duration extended. Further study is needed to establish the value of motion metrics during simulator training and their impact on operating room performance improvement.

Objective Assessment of Training Surgical Skills Using Simulated Tissue Interface with Real-Time Feedback

We have shown previously that achieving competent performance of basic laparoscopic skills is possible in difficult conditions. We hypothesize that real-time performance feedback adds to the quality of proficiency-based simulator training for performance and forces applied to conventional surgical tools while tying square knots and running suture throws. A silk suture was preplaced on a simulated skin pad to assess incision closure by tying square knots and in separate trials to evaluate closure with the task of placing a running suture. The order of task performance was assigned randomly, and each task was repeated 5 times before switching to the second task. In all, 10 repetitions per task were performed by each student. After completion of the second surgical task, the cycle was repeated to test adaptation and retention of motor-skill capabilities. Half the participants were provided with a graphic display in dial format to indicate applied force. Bench-top setup of apparatus was performed in a laboratory at Virginia Commonwealth University, Department of Surgery. Twelve second-year medical students with no surgical skills background participated in the study. Results from the knot-tying task indicated that the average force exerted on tissue forceps by the left hand in the blinded group who performed simple knots actually increased over repeated trials, as opposed to what was achieved by the group that had real-time feedback of their forces being applied. For the running suture, the task average force exerted on surgical tools by both hands was greater in the blind group relative to those viewing real-time graphic feedbacks of forces generated over repeated trials. Inclusion of real-time objective assessment in evaluation of surgical skills minimizes subjective evaluation of performance capabilities. A direct correlation between real-time feedback regarding force exerted and extent of surgical task completion was noted.

Proficiency-Based Laparoscopic Simulator Training Leads to Improved Operating Room Skill That Is Resistant to Decay

The aim of this study was to assess skill retention in the operating room following completion of a proficiency-based laparoscopic skills curriculum. Novices (n = 15) were randomized to a control and a training group that practiced to proficiency on the Fundamentals of Laparoscopic Surgery suturing model. The performance of both groups was assessed on the simulator and on a live porcine laparoscopic Nissen fundoplication model at training completion (posttest) and 5 months later (retention test). Training to proficiency required 4.7 +/-1.2 hours and 41 +/- 10 repetitions. Trained participants outperformed controls, and their performance deteriorated slightly between posttests and retention tests on the simulator (505 +/- 22 vs 462 +/- 50, respectively; P < .05) but not in operating room (263 +/- 138 vs 279 +/- 88, respectively; P = .38). Proficiency-based simulator training results in durable improvement in operative skill of trainees even in the absence of practice for up to 5 months. Minute simulator performance changes do not translate to the operating room.

Closing the Gap in Operative Performance Between Novices and Experts: Does Harder Mean Better for Laparoscopic Simulator Training?

We have previously shown that reaching expert performance on an fundamentals of laparoscopic surgery (FLS)-type simulator model for laparoscopic suturing results in measurable improvement during an actual operation; trained novices, however, demonstrate inferior operative performance compared with experts. We hypothesized that simulator training under more difficult and realistic conditions would enhance the operative performance of novices. Medical students (n=32) participated in an IRB-approved, randomized, controlled trial. All participants were pretested in laparoscopic suturing on a previously validated porcine Nissen model and were randomized into three groups: group I (n=6) received no training, group II (n=13) trained on the FLS videotrainer model until a previously published proficiency score (512) was achieved on 2 consecutive and 10 additional attempts, group III (n=13) trained to the same goal but had to practice in a constrained space, with a shorter suture, starting with a dropped needle, and listening to operating room noise. Training workload was measured with the validated NASA-TLX (Task Load Index) questionnaire after each training session. All groups were posttested on the porcine model. Results were compared using ANOVA; p < 0.05 was considered significant. All group II and III participants reached the training goal. At posttesting, group II and group III participants performed similarly, but substantially better than group I did (210+/-140 versus 218+/-139 versus 0+/-0, respectively; p < 0.001). Compared with group II, group III participants trained longer (329+/-71 minutes versus 239+/-69 minutes, p < 0.001), performed more repetitions (81+/-15 versus 59+/-14, p < 0.001), and their workload improved less by the end of training (5% versus 23%, p < 0.001). Proficiency-based simulator training reliably results in improved operative performance. Although increasing the level of training difficulty increased trainees' workload, the strategy we used in this study did not enhance their operative performance. Other methods for curriculum optimization are needed.