Simulating Various Levels of Clinical Challenge in the Assessment of Clinical Procedure Competence

Abstract

Immersive simulation is increasingly used for competency assessment of emergency physicians. This group's concept of hybrid simulation (HS) (combining simulated patients and part-task trainers (a simulator that simulates a limited component of a clinical procedure) to create a multimodal clinical context) requires clinicians to integrate technical and nontechnical skills in a holistic clinical performance for assessment. It also offers the potential to provide authentic simulation of a given clinical procedure across multiple levels of challenge. The aims of this study are to systematically design and validate 2 patient-focused HS scenarios (each combining a simulated patient with a part-task simulator) for assessment of the management of a commonly encountered problem in an emergency department (ED) at different levels of clinical challenge, and to explore the effect of level of challenge of the HS scenario on physicians' performance. A simple (HS1) and a complex (HS2) HS scenario (based on the management of a patient with a traumatic skin laceration within the ED) was developed according to expert opinion through cognitive task analysis. Interns and emergency medicine residents (stratified into expert and novice groups according to experience) were recruited to participate in both scenarios. Participants were randomized to perform either the HS1 or HS2 scenario first. Participants completed a questionnaire for face validity (realism of simulation) and content validity (comprehensiveness of simulation). Performances were assessed by 2 independent raters using validated rating tools modified to the needs of this study: the Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, the Objective Structured Assessment of Technical Skills-Global Rating Score, and the Direct Observation of Procedural Skills. Ten novice and 10 expert clinicians completed both scenarios. Mean face and content validity ratings were high for both HS1 (mean 4.4 [SD 0.52] and 4.2 [SD 0.48], respectively) and HS2 scenarios (mean 4.5 [SD 0.35] and 4.3 [SD 0.43], respectively). In HS1, no difference was found between experts' and novices' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, Objective Structured Assessment of Technical Skills-Global Rating Score, and Direct Observation of Procedural Skills ratings. Experts performed significantly better than novices in HS2 in terms of the 3 tools' ratings. Novices' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist and Direct Observation of Procedural Skills ratings were significantly worse in HS2 compared with HS1, but no difference was found with the Objective Structured Assessment of Technical Skills-Global Rating Score. No statistical difference was found in experts' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, Objective Structured Assessment of Technical Skills-Global Rating Score, and Direct Observation of Procedural Skills ratings between HS2 and HS1 scenarios. Recreating clinical challenge is an important consideration in the design of simulation-based assessment of procedural skills of clinicians. In this study, we have demonstrated a systematic approach to developing HS scenarios, which may be able to recreate various levels of clinical challenge for purpose of assessment of procedural skills.