Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work

Per Øivind Braarud

doi:10.1016/j.ergon.2021.103233

Abstract

Subjective workload and situation awareness measures, such as the NASA task load index (TLX) and the situational awareness rating technique (SART), are frequently used in human–system evaluation. However, the interpretation of these ratings is debated. In this study, empirical evidence for the measures' theoretical assumptions was investigated by comparing operators' ratings collected immediately after performing a scenario and ratings collected after operators' acquisition through a video review of the scenario, knowledge of actual system states. Eighteen licensed control room operators participated in the simulator study, running 12 relatively challenging scenarios. It was found that the interpretation of TLX items involving introspection remained stable after operators acquired factual scenario knowledge, while the interpretation of items involving the perception of external events, such as situation awareness and performance, depended on the operators' scenario knowledge. The result shows that operators’ ratings could discriminate between mental effort, performance, frustration, and situation awareness. No clear evidence for the SART index as a measure of situation awareness was found. Instead, a subjective situation awareness measure developed for this study was distinct from workload and related to operator performance, showing that this type of measure warrants future investigation of its validity. The study findings help in developing measurement procedures and interpreting subjective measures. Finally, the study reveals that informing operators about the scenario can provide useful subjective ratings of situation awareness and performance. Future research should include procedures for how to inform participants adequately and efficiently in subjective assessments.

Highlights

Mental workload and situation awareness are important criteria for designing and assessing human–machine systems (Endsley et al, 1998; Salmon et al, 2009; O’Hara et al, 2012)
The lower part of the table shows three indexes—the average of the six task load index (TLX) items, situa tional awareness rating technique (SART) calculated according to its formula (SA = U – (D− S)), and the average of the three SA3 items
The TLX index was substantially lower than the SART and SA3 indexes for all three control room positions and the team average

Summary

Introduction

Mental workload and situation awareness are important criteria for designing and assessing human–machine systems (Endsley et al, 1998; Salmon et al, 2009; O’Hara et al, 2012). Important questions include how human–system design and organisation of work influence workload and situation awareness, and to what extent a given human–machine configuration represents optimal or acceptable levels (Reid and Colle, 1988; Endsley, 2000a; Young et al, 2015). By improving the human-system inter face or increasing the level of expertise, one could reduce the workload and increase situation awareness—a frequent goal of system design ef forts (Vidulich, 2000; Endsley, 2000a; Vidulich and Tsang, 2015). Separate and valid measures of each construct are war ranted (Endsley, 2000a; Parasuraman et al, 2008)

Methods

Results

Discussion

Conclusion