Multimodal Analysis Using Neuroimaging and Eye Movements to Assess Cognitive Workload

Abstract

Air Traffic Control (ATC) specialists work in an environment where the proficient interaction between humans and computer systems is crucial to provide a safe and efficient flow of traffic. The complexity of this system may increase due to planned changes in operator roles and a projected rise in traffic volume. This increase, over an already highly complex system, will exacerbate the mental workload placed on the operator. The emergence of wearable sensors that measure physiological signals enables us to monitor the mental workload in real time without interfering in operational activity. However, the use of a single sensor approach may not provide a comprehensive assessment of cognitive workload while executing a complex task. Therefore, this study implemented a multimodal approach by using two sensors, namely functional near infrared spectroscopy (fNIRS) and eye-tracking, to evaluate the cognitive workload changes experienced by an ATC specialist. Three retired tower controllers with over 20 years of experience, underwent three sessions of experimentation where each individual fulfilled one of the following roles - observer, a Local controller or a Ground controller. During each iteration, the fNIRS and eye tracking sensors were attached to the Local controller while they commanded aircraft through verbal clearances. The task difficulty and complexity were quantified by the number of aircraft and clearances given, respectively. The number of aircraft displayed on the screen increased across time and was positively correlated with oxygenation measures assessed by the fNIRS signals of both the right and left prefrontal cortex. On the other hand, the number of fixations was positively correlated with the number of clearances. These results suggest fNIRS and eye-tracking measures are sensitive to changes in cognitive workload, and indicates that they may be amenable to complement each other for the assessment of the multidimensionality of cognitive workload induced by task difficulty and complexity.