Board 343 - Research Abstract Development and Evaluation of a Contextually Relevant Measure of Cognitive Load for Simulation-Based Psychomotor Skills Training (Submission #951)

Abstract

Introduction/Background Theoretically-based research exploring instructional design in healthcare simulation has emerged as a top priority.1–3 In turn, interest in cognitive load theory as a foundation for empirical investigation of instructional design principles in simulation has grown.1,4 An essential precursor to this line of inquiry is the development and evaluation of cognitive load (CL) measures that are appropriate for the healthcare simulation setting. To be effective, these measures should be unintrusive, sensitive to cognitive demands imposed by the simulated task and natural to the performer.5 The objectives of this study were to: 1) develop contextually relevant measures of CL based on secondary-task methodology and 2) generate preliminary validity evidence6 supporting their use in simulation-based psychomotor skills training. It was hypothesized that: 1) these measures of secondary-task performance would be sensitive to variations in CL within novices as cognitive demands change and between novices and experts when performing a psychomotor primary-task, and 2) similar patterns would be observed between experts and novices on subjective measures of cognitive load and primary-task performance. Methods . We developed a virtual vital signs monitor with a built-in visual stimulus detection secondary-task, in which participants monitor a baseline heart-rate and press a foot-pedal each time a pre-determined change (bradycardia or tachycardia) is observedThe software subsequently records two performance metrics: stimulus-detection error rate (SDER) and recognition reaction time (RRT)To evaluate the sensitivity of these metrics to variations in CL during simulation-based psychomotor skills training, five experts (surgical residents) and seven novices (medical students) completed a baseline stimulus-detection trial and a dual-task trial consisting of one-handed surgical knot tying on a part-task trainer, while monitoring for changes in heart-rateFollowing the dual-task trial, participants also completed a subjective rating of mental effort (SRME) using a previously developed scale.7 Primary-task (knot-tying) performance was assessed by total movements (TM) and time to complete (TC) a square knot.8 The first hypothesis was tested by analyzing differences in RRT and SDER from baseline to dual-task between experts and novices, using 2x2 repeated measures ANOVA and the Tukey test for post-hoc comparisonsThe second hypothesis was tested by analyzing differences between experts and novices SRME and on knot-tying performance, using the Kruskal-Wallis test and independent sample t-test respectively. Results . Analysis of secondary-task performance demonstrated a significant interaction between expertise (novice vsexpert) and task (single vsdual-task) for RRT (F(1,10)=9.947, p<0.01, partial eta2=0.89) and SDER (F(1,10)=81.133, p<0.0001, partial eta2=0.89)Pairwise comparisons revealed a significant increase in RRT and SDER from baseline to dual-task among novices (q=6.18, p<0.025 and q=16.45, p<0.01 respectively) but not among expertsIn addition, experts had significantly lower RRT and SDER compared to novices during dual-tasking (q=5.21, p<0.05 and q=14.88, p<0.01 respectively) but not at baselineSimilarly, compared to novices, experts had significantly lower dual-task SRME (chi2=5.316, p<0.021) and superior primary task performance with respect to TC (t=4.939, p<0.004), and TM (t=4.748, p<0.005). Conclusion We have developed an instrument for assessing CL that employs a contextually relevant secondary task (response to changes in vital signs). The measures generated from this instrument are sensitive to variations in CL among novices as cognitive demands change (i.e. single to dual-tasking) and between novices and experts performing a psychomotor skill. The difference in performance between novices and experts on these measures are similar to those seen on primary task performance (TC and TM) and subjective ratings of cognitive load, demonstrating preliminary validity evidence in the category of “response to other variables”6 for the two CL measures generated by our instrument (RRT and SDER). The Results indicate this instrument may be effective for measuring cognitive load during simulation-based psychomotor skills training of novice learners.