Board 315 - Research Abstract Simulation-Based Educational Assessments as Surrogates for Patient-Related Outcomes

Abstract

Introduction/Background Simulation-based assessments could serve as surrogates for workplace based assessments in medical education, decreasing the potential for patient harm and conserving resources by demonstrating learner competence in a safe training environment. While over reliance on surrogates should be avoided, once the necessary links are established educators and researchers can use the surrogate simulation-based measures during day-to-day operations and initial investigations while reserving use of patient-related outcomes for selected situations. Our objective in conducting this review was to synthesize studies evaluating technology enhanced simulation-based outcomes (defined as skills) as surrogates for patient-related outcomes (defined as provider behaviors and patient effects) in healthcare education. Methods We systematically searched key databases through February 26, 2013. Included studies involved assessment of health professionals using technology enhanced simulation and reported associations of these scores with scores from assessments during real patient care (e.g., workplace-based assessments). We distinguished patient-related outcomes of process behaviors (e.g., instructor ratings of performance), time behaviors (e.g., procedure time) and effects on patients (e.g., complications). Working independently and in duplicate, we extracted data on participants, topics, validity evidence, study quality and magnitude of correlation. We pooled correlations using random effects meta-analysis. Results We identified 33 studies with 1,203 participants including postgraduate physicians, practicing physicians, medical students, dentists and nurses. Topics included laparoscopic surgery, anesthesiology and bedside procedures. Pooled correlations with simulation-based outcomes were 0.51 (95% CI, 0.38-0.62; N=27 studies) for process behaviors, 0.44 (95% CI, 0.15-0.66; N=7) for time behaviors and 0.24 (95% CI, -0.02 to 0.47; N=5) for patient effects. Most reported validity evidence was favorable, though incomplete. Three tools (OSATS, GOALS, and FLS), each evaluated in multiple studies, demonstrated large pooled correlations and favorable validity evidence. Conclusion Simulation-based assessments typically correlate positively with patient-related outcomes. While surrogates are imperfect, some tools with established validity evidence may replace workplace based assessment for select procedural skills. Our findings have implications for improving research practices when establishing simulation-based outcomes as surrogates in competency-based curricula (e.g., ACGME milestones). Disclosures None.