PD19-03 LARGE-SCALE EVIDENCE OF CONSTRUCT VALIDITY: THE BASIC LAPAROSCOPIC UROLOGIC SKILLS (BLUS) INITIATIVE

Abstract

You have accessJournal of UrologyTechnology & Instruments: Surgical Education & Skills Assessment III1 Apr 2015PD19-03 LARGE-SCALE EVIDENCE OF CONSTRUCT VALIDITY: THE BASIC LAPAROSCOPIC UROLOGIC SKILLS (BLUS) INITIATIVE Timothy Kowalewski, Robert Sweet, Ashleigh Menhadji, Timothy Averch, Geoffrey Box, Timothy Brand, Michael Fearrandino, Jihad Kaouk, Bodo Knudsen, Jamie Landman, Benjamin Lee, Bradley Schwartz, Bryan Comstock, Cory Schaffhausen, Elspeth McDougall, and Thomas Lendvay Timothy KowalewskiTimothy Kowalewski More articles by this author , Robert SweetRobert Sweet More articles by this author , Ashleigh MenhadjiAshleigh Menhadji More articles by this author , Timothy AverchTimothy Averch More articles by this author , Geoffrey BoxGeoffrey Box More articles by this author , Timothy BrandTimothy Brand More articles by this author , Michael FearrandinoMichael Fearrandino More articles by this author , Jihad KaoukJihad Kaouk More articles by this author , Bodo KnudsenBodo Knudsen More articles by this author , Jamie LandmanJamie Landman More articles by this author , Benjamin LeeBenjamin Lee More articles by this author , Bradley SchwartzBradley Schwartz More articles by this author , Bryan ComstockBryan Comstock More articles by this author , Cory SchaffhausenCory Schaffhausen More articles by this author , Elspeth McDougallElspeth McDougall More articles by this author , and Thomas LendvayThomas Lendvay More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2015.02.702AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES The American Urological Association created the Basic Laparoscopic Urologic Skills (BLUS) skills tasks to objectively measure technical skills proficiency of urology trainees. We sought to establish construct validity of the psychomotor skills portion of this curriculum using objective performance metrics and structured assessment by video review. METHODS After IRB approval, (n=454) videos of dry-lab laparoscopic tasks: Peg Transfer (n=110), Cutting (n=110), Intracorporeal Suturing (n=115), and Clip Applying (n=119) tasks were assessed by lay-people from the Amazon.com Mechanical Turk crowd sourcing platform. These performance scores were correlated with tool motion metrics captured from the Electronic Data Generation and Evaluation (EDGE) system (Simulab Corp, Seattle WA) for all tasks. We sought at least 30 Crowd-Sourced Assessment of Technical Skills (C-SATS) ratings via the validated Global Objective Assessment of Laparoscopic Skills (GOALS) survey tool with an added pass/fail domain. We compared the mean of GOALS domain ratings with objective metrics using Spearman's rho correlation statistics. We estimated performance-specific passing probabilities through mean GOALS score cut-offs previously established by blinded faculty reviews of a representative subset (n=24) of Peg Transfer and Suturing videos with receiver operating characteristic (ROC) curves based on Pass/Fail criteria. RESULTS Spearman's rho between composite GOALS scores and the composite EDGE scores ranged from moderately strong to very strong (Peg Transfer 0.84, Cutting 0.72, Suturing, 0.80 and Clip Apply 0.61). ROC curves for crowd-only pass/fail (AUC=.99) and crowd and EDGE score pass/fail (AUC=.94 Peg Transfer,.93 Cutting,.88 Suturing, and.79 Clip Apply) compare favorably with or exceed the ROC characteristics of the original validated McGill Inanimate System for Training and Evaluation of Laparoscopic Skills(MISTELS)/Fundamentals of Laparoscopic Surgery(FLS) work. CONCLUSIONS We conclude that the agreement between crowd-generated composite GOALS scores and EDGE-derived objective metrics over a large corpus of 454 basic laparoscopic technical skills videos provides evidence of construct validity for the BLUS tasks. Table 1. Correlations of n=454 technical skills performances between mean crowd GOALS scores and objective tool motion metrics. Spearman’s rho (p-value) is shown for each task. – indicates not significant. Crowd Scores vs. Peg Transfer Cutting Suturing Clip Apply EDGE Score 0.84 (0.00) 0.72 (0.00) 0.80 (0.00) 0.61 (0.00) Time -0.79 (0.00) -0.55 (0.00) -0.76 (0.00) -0.59 (0.00) Path Length -0.77 (0.00) -0.70 (0.00) -0.78 (0.00) -0.47 (0.00) Jerk Cost -0.57 (0.00) -0.57 (0.00) -0.61 (0.00) -0.28 (0.00) Movement Count -0.55 (0.00) -0.57 (0.00) -0.65 (0.00) -0.39 (0.00) Economy of Motion 0.35 (0.00) -- -- 0.34 (0.00) Force Variance -- -- -- -- Max Force -- -- -- -- © 2015 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 193Issue 4SApril 2015Page: e392 Advertisement Copyright & Permissions© 2015 by American Urological Association Education and Research, Inc.MetricsAuthor Information Timothy Kowalewski More articles by this author Robert Sweet More articles by this author Ashleigh Menhadji More articles by this author Timothy Averch More articles by this author Geoffrey Box More articles by this author Timothy Brand More articles by this author Michael Fearrandino More articles by this author Jihad Kaouk More articles by this author Bodo Knudsen More articles by this author Jamie Landman More articles by this author Benjamin Lee More articles by this author Bradley Schwartz More articles by this author Bryan Comstock More articles by this author Cory Schaffhausen More articles by this author Elspeth McDougall More articles by this author Thomas Lendvay More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...