Simulator and 2 tools: Validation of performance measures from a novel neurosurgery simulation model using the current Standards framework

Abstract

Ventriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation model could also allow for assessment of trainees' proficiencies with measures that align with Accreditation Council for Graduate Medical Education milestones. This study extends previous work to evaluate validity evidence from the simulator, its newly developed performance assessment, the VentricolostomyProcedural Assessment Tool, and the Objective Structured Assessment for Technical Skills. After Institutional Review Board exemption, performance data were collected from 11 novice and 3 expert neurosurgeons (n=14). Participants self-reported their ability to perform tasks on the simulator using the Ventricolostomy Procedural Assessment Tool, an 11-item, step-wise instrument with 5-point rating scales ranging from 1 (unable to perform) to 5 (performs easily and smoothly). De-identified operative performances were videotaped and independently rated by 3 neurosurgeons, using the Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills. We evaluated multiple sources of validity evidence (2014 Standards) to examine psychometric quality of the measures and to test our assumption that the tools could discriminate between novice and expert performances adequately. We used a multifacet Rasch model and traditional indices, such as Cronbach alpha, intraclass correlation, and Wilcoxon signed-rank test estimates. Validity evidence relevant to test content and response processes was supported adequately. Evidence of internal structure was supported by high interitem consistency (n=0.95) and inter-rater agreement for most Ventricolostomy Procedural Assessment Tool items (Intraclass correlation coefficient=[0.00, 0.91]) and all Objective Structured Assessment for Technical Skills items (Intraclass correlation coefficient=[0.80, 0.93]). Overall, novices performed at a lower level than experts on both scales (P<.05), supporting evidence relevant to relationships to other variables. Deeper analysis of novice/expert ratings indicated novices attained lower performances ratings for all Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills items, but statistical significance was only achieved for the Objective Structured Assessment for Technical Skills items (P<.01). Rater bias estimates were favorable, supporting evidence relevant to consequences of testing. Despite a small sample, favorable evidence using current Standards supports the use of the novel simulator and both tools combined for skills training and performance assessment, but challenges (potential threats to validity) should be considered prior to implementation.