A Systematic Method for Using Critical Decision Method Interviews to Generate a Garden Path Scenario for Human-in-the-Loop Simulations for Training and Evaluation of New Technologies

Abstract

Strategies for assessing macrocognitive skill must be tailored to work domains and specific tasks. This paper describes one approach for leveraging the critical decision method, a structured interview technique, to develop an assessment method for new technologies. The approach uses garden path scenarios, where initial cues suggest a different (false prime) diagnosis than later cues and thus requires changing the leading diagnosis over time, to assess sensemaking and re-planning skills in the context of tactical combat casualty care. Critical decision method interviews with emergency medicine physicians revealed critical cues specific to particular injuries and common across different injuries, and also provided cases that were used as the foundation for garden path scenarios. Evaluations using this approach with garden path scenarios enables having an objective measure to compare performance with and without a new technology on a continuous variable of the time until landmark events, such as recognition of a critical cue, committing to a likely diagnosis, or ruling out an inaccurate diagnosis. Additional measures include whether or not particular cues are noticed based upon gaze tracking data and think aloud statements, and exams that assess knowledge of anatomy and treatment priorities. Re-planning measures will focus on comparing performance to an expert model such as for tourniquet application, whether or not tasks on a checklist are conducted in the expected order including for prioritizing where to look for patient assessment, action priorities, and the trainee’s ability to link diagnosis to appropriate treatment.