Validity Evidence for a Novel, Comprehensive Bag–Mask Ventilation Assessment Tool

Abstract

To develop a comprehensive competency assessment tool for pediatric bag-mask ventilation (pBMV) and demonstrate multidimensional validity evidence for this tool. A novel pBMV assessment tool was developed consisting of 3 components: a 22-item-based checklist (trichotomized response), global rating scale (GRS, 5-point), and entrustment assessment (4-point). Participants' performance in a realistic simulation scenario was video-recorded and assessed by blinded raters. Multidimensional validity evidence for procedural assessment, including evidence for content, response-process, internal structure, and relation to other variables, was assessed. The scores of each scale were compared with training level. Item-based checklist scores also were correlated with GRS and entrustment scores. Fifty-eight participants (9 medical students, 10 pediatric residents, 18 critical care/neonatology fellows, 21 critical care/neonatology attendings) were evaluated. The pBMV tool was supported by high internal consistency (Cronbach α=0.867). Inter-rater reliability for the item-based checklist component was acceptable (r=0.65, P<.0001). The item-based checklist scores differentiated between medical students and other providers (P<.0001), but not by other trainee level. GRS and entrustment scores significantly differentiated between training levels (P<.001). Correlation between skill item-based checklist and GRS was r=0.489 (P=.0001) and between item-based checklist and entrustment score was r=0.52 (P<.001). This moderate correlation suggested each component measures pBMV skills differently. The GRS and entrustment scores demonstrated moderate inter-rater reliability (0.42 and 0.46). We established evidence of multidimensional validity for a novel entrustment-based pBMV competence assessment tool, incorporating global and entrustment-based assessments. This comprehensive tool can provide learner feedback and aid in entrustment decisions as learners progress through training.