Measuring Situation Awareness in complex systems: Comparison of measures study

Abstract

Assessing operator situation awareness is a key component of sociotechnical system design and evaluation. This article describes a study that was undertaken in order to compare two different situation awareness measures (a freeze probe recall approach and a post trial subjective rating approach) when used to assess participant situation awareness during a military planning task. The results indicate that only the participant situation awareness scores derived via the freeze probe recall method produced a statistically significant correlation with performance on the planning task and also that there was no significant correlation between the two methods, which suggests that they were measuring different aspects of participant situation awareness during the trials. In conclusion, the findings, whilst raising doubts over the validity of the post trial subjective rating approach, offer validation evidence for the use of freeze probe recall approaches to measure situation awareness during simulated tasks. The findings are subsequently discussed with regard to their wider implications for the future measurement of situation awareness in complex collaborative systems. Relevance to industry Situation Awareness is a critical commodity for teams working in industrial systems. Accordingly, designers and analysts require reliable and valid methods for assessing the impact of new systems, interfaces, training programs and procedures on the level of situation awareness held by operators. This article presents a review and comparison of situation awareness measurement approaches for use in complex industrial systems and provides recommendations on the types of methods to use during future situation awareness assessments.