VALIO: Visual attention-based linear temporal logic method for explainable out-of-the-loop identification

Abstract

The phenomenon of being Out-Of-The-Loop (OOTL) can significantly undermine pilots’ performance and pose a threat to aviation safety. Previous attempts to identify OOTL status have primarily utilized “black-box” machine learning techniques, which fail to provide explainable insights into their findings. To address this gap, our study introduces a novel application of Linear Temporal Logic (LTL) methods within a framework named Visual Attention LTLf for Identifying OOTL (VALIO), leveraging eye-tracking technology to non-intrusively capture the pilots’ attentional focus. By encoding Areas of Interest (AOIs) and gaze durations within the cockpit into Visual Attention Traces (VAT), the method captures the spatial and temporal dimensions of visual attention. It enables the LTL methods to generate interpretable formulas that classify pilot behaviors and provide insights into the understanding of the OOTL phenomenon. Through a case study of a simulated flight experiment, we compared the efficacy of this approach using different time windows from 10 s to 75 s. The results demonstrate that VALIO’s performance is stable across all time windows with the best F1 score of 0.815 and the lowest F1 of 0.769. And it significantly outperforms the other machine learning methods when using time windows shorter than 30 s, signifying its ability to detect the OOTL status more in-timely. Moreover, the VALIO elucidates pilot behaviors through the derivation of human-readable LTLf formulas, offering the explainability of the results and insights into OOTL characteristics. Overall, this research proposes the VALIO framework as an improvement for OOTL identification in both performance and explainability.