Assessment of Simulated Emergency Scenarios: Are Trained Observers Necessary?

Abstract

Objectives: Simulation-based medical training is associated with superior educational outcomes and improved cost efficiency. Self- and peer-assessment may be a cost-effective and flexible alternative to expert-led assessment. We compared accuracy of self- and peer-assessment of untrained raters using basic evaluation tools to expert assessment using advanced validation tools including validated questionnaires and post hoc video-based analysis. Methods: Twenty-eight simulated emergency airway management scenarios were observed and video-recorded for further assessment. Participants consisted of 28 emergency physicians who were involved in four different airway management scenarios with different roles: One scenario as a team leader, one as an assisting team member, and two as an observer. Non-technical skills (NTS) and technical skills (TS) were analyzed by three independent groups: 1) the performing team (PT) consisted of the two emergency physicians acting either in the role of team leader or team member (self-assessment); 2) the observing team (OT), consisted of two of the participating emergency physicians not involved in the current clinical scenario (peer-assessment) and assessment occurred during (OT) or directly after (PT) the simulation without prior specific interpretational training but using standardized questionnaires; and 3) the expert team (ET) consisted of two specifically trained external observers (one psychologist and one emergency physician) using video-assisted objective assessment combined with standardized questionnaires. Results: Intragroup reliability demonstrated by intra-class correlation (ICC) was moderate to good for TS (ICC 0.42*) and NTS (ICC 0.55*) in PT and moderate to good for TS (ICC 0.41*) or poor for NTS (ICC 0.27) in OT. ET showed an excellent intragroup reliability for both TS (ICC 0.78*) and NTS (ICC 0.81*). Interrater reliability was significantly different between ET and PT and between ET and OT for both TS and NTS. There was no difference between OT and PT for neither TS nor NTS; *p < 0.05. Conclusions: Expert assessment of simulation-based medical training scenarios using validated checklists and performance of post hoc video-based analysis was superior to self- or peer-assessment of untrained observers for both TS and NTS.