Measuring Situation Awareness of Student Air Traffic Controllers with Online Probe Queries: Are We Asking the Right Questions?

Abstract

Listen

Translate article

ABSTRACTSituation awareness (SA) is a major factor determining operator performance in the current and future airspace systems, so it is not surprising that a considerable body of research on measuring operator SA has been amassed. A commonly used SA measurement method is a probe technique in which the operator is queried about the current state of his/her task environment, either offline (while a computer simulation is paused) or online (while a computer simulation is running.) Although sufficient evidence for the reliability and validity of both techniques is available, one factor that has not been systematically investigated is the efficacy of the questions themselves. We evaluated the types of online probe questions used for predicting the performance of student air traffic controllers in a simulated air traffic management task. These data were obtained from 84 students who completed a 16-week radar internship course in which their situation awareness and performance were assessed at the midpoint and end of the course. We subjected probe question latencies to a principle component analysis (PCA) to explore the variation between questions and grouped them based on the resulting components. The SA question components that emerged from PCA were then used to model the relations between SA factors and performance. The resulting model provides a framework for understanding factors involved in SA, a way to evaluate the validity of individual probe questions as a measure of SA, and a method for determining the efficacy of SA questions in predicting operator performance.