The Simulation Team Assessment Tool (STAT): Development, reliability and validation

Abstract

IntroductionSimulation sessions prepare medical professionals for pediatric emergencies. No validated tools exist to evaluate overall team performance. Our objective was to develop and evaluate the inter-rater reliability and validity of a team performance assessment tool during simulated pediatric resuscitations. MethodsWe developed the Simulation Team Assessment Tool (STAT) which evaluated 4 domains: basic assessment skills, airway/breathing, circulation, and human factors. Scoring of each element was behaviorally anchored from 0 to 2 points. Two teams of resuscitation experts and two teams of pediatric residents performed the same simulated pediatric resuscitation. Each team was scored by six raters using the STAT. Intraclass correlation coefficients (ICC) were calculated to assess inter-rater reliability. Overall performance and domain scores between expert and resident teams were compared using repeated measures of analysis of variance to assess construct validity. ResultsICCs for overall performance were 0.81. Domain ICCs were: basic skills 0.73, airway/breathing skills 0.30, circulation skills 0.76, human factors 0.68. Expert versus resident average scores were: overall performance 84% vs. 66% (p=0.02), basic skills 73% vs. 55% (p<0.01); airway 80% vs. 75% (p=0.25), circulation 90% vs. 69% (p=0.02), human factors 89% vs. 66% (p=0.02). ConclusionsThe STAT's overall performance, basic skills, circulation, and human factors domains had good to excellent inter-rater reliability, discriminating well between expert and resident teams. Similar performance in the airway/breathing domain among all teams magnified the impact of a small number of rater disagreements on the ICC. Additional study is needed to better assess the airway/breathing domain.