Limitations of subjective cognitive load measures in simulation-based procedural training.

Abstract

The effective implementation of cognitive load theory (CLT) to optimise the instructional design of simulation-based training requires sensitive and reliable measures of cognitive load. This mixed-methods study assessed relationships between commonly used measures of total cognitive load and the extent to which these measures reflected participants' experiences of cognitive load in simulation-based procedural skills training. Two groups of medical residents (n=38) completed three questionnaires after participating in simulation-based procedural skills training sessions: the Paas Cognitive Load Scale; the NASA Task Load Index (TLX), and a cognitive load component (CLC) questionnaire we developed to assess total cognitive load as the sum of intrinsic load (how complex the task is), extraneous load (how the task is presented) and germane load (how the learner processes the task for learning). We calculated Pearson's correlation coefficients to assess agreement among these instruments. Group interviews explored residents' perceptions about how the simulation sessions contributed to their total cognitive load. Interviews were audio-recorded, transcribed and subjected to qualitative content analysis. Total cognitive load scores differed significantly according to the instrument used to assess them. In particular, there was poor agreement between the Paas Scale and the TLX. Quantitative and qualitative findings supported intrinsic cognitive load as synonymous with mental effort (Paas Scale), mental demand (TLX) and task difficulty and complexity (CLC questionnaire). Additional qualitative themes relating to extraneous and germane cognitive loads were not reflected in any of the questionnaires. The Paas Scale, TLX and CLC questionnaire appear to be interchangeable as measures of intrinsic cognitive load, but not of total cognitive load. A more complete understanding of the sources of extraneous and germane cognitive loads in simulation-based training contexts is necessary to determine how best to measure and assess their effects on learning and performance outcomes.