Do hybrid simulator-generated metrics correlate with content-valid outcome measures?

Abstract

Hybrid simulators provide objective metrics for laparoscopic task performance. This study aimed to evaluate the correlation between hybrid simulator-generated metrics and content-valid outcome measures. Residents underwent training with a previously validated 5-task simulation model (5-TSM). The resident tasks included vessel clipping and dividing, lesion excision, loop appendectomy, mesh placement with tacks, and suture perforation with intracorporeal knot tying. After training, the residents were tested using the open module of a hybrid simulator (ProMIS) with previously validated passing scores. Content validity was defined as the extent to which outcome measures departed from clinical reality. Content-valid outcome measures (accuracy error, knot slippage, leak, operating time, tissue damage) were evaluated by two blinded raters. The hybrid simulator-generated metrics were path length and smoothness of movements. Values are given as means (standard deviation is not shown). Over 23 months, 20 residents underwent training with 5-TSM. Respectively, for tasks 1 to 5, the path length was 3,895, 3,472, 4,620, 2,408, and 9,089 mm; the smoothness (jerk) was 346, 455, 549, 264, and 910 cm/sec3; the accuracy error was 0.45, 2.20, 0.55, 0.87, and 0.20 mm; and the knot slippage was 5%. There were no leaks. The operating time, respectively, was 54, 61, 135, 43, and 130 s, and the tissue damage was 0, 0.28, 0, 0.8, and 0 mm. The interrater reliability was more than 0.80 for all the outcome measures except accuracy error (k=0.52). There was correlation between path length and operating time (Spearman rho, 0.537-0.709; p<0.05) for all the tasks. There was no correlation between path length and accuracy error, knot slippage, leak, and tissue damage. No correlation was found between smoothness and any of the outcome measures for any of the tasks except operating time (Spearman rho, 0.762-0.958; p<0.05). Although an expected strong correlation was observed between hybrid simulator-generated metrics and operating time, this study showed no correlation between simulator-generated metrics and other content-valid outcome measures.