Assessing Clinical Reasoning in Pediatric Emergency Medicine: Validity Evidence for a Script Concordance Test

Abstract

Clinical reasoning is a crucial skill for all residents to acquire during their training. During most patient encounters in pediatric emergency medicine, physicians and trainees are challenged by diagnostic, investigative, and treatment uncertainties. The Script Concordance Test may provide a means to assess reasoning skills in the context of uncertainty in the practice of pediatric emergency medicine. We gathered validity evidence for the use of a pediatric emergency medicine Script Concordance Test to evaluate residents' reasoning skills. A 1-hour test containing 60 questions nested in 38 cases was administered to 53 residents at the end of their pediatric emergency medicine rotation at 1 academic institution. Twelve experienced pediatricians were part of a reference panel to establish the basis for the scoring process. An optimized version of the test, based on positive item discrimination data, contained 30 cases and 50 questions. Scores ranged from 48% to 82%, with a mean score of 69.9 (SD=11.5). The reliability of the optimized test (Cronbach's alpha) was 0.77. Performance on the test increased as the level of experience of the residents increased. The residents considered the Script Concordance Test true to real-life clinical problems and had enough time to complete the test. This pediatric emergency medicine Script Concordance Test was reliable and useful to assess the progression of clinical reasoning during residency training.