The Effects of Level of Automation on the Out-of-the-Loop Unfamiliarity in a Complex Dynamic Fault-Management Task during Simulated Spaceflight Operations

Abstract

Out-of-the-loop unfamiliarity (OOTLUF) and information sampling strategies were examined after operator use of intelligent fault management (FM) support in interaction with a generic, autonomous, atmospheric control system. A simulated model-based reasoning agent provided fault diagnosis and recovery at three different levels of automation (LOA). In about 10% of the experimental trials FM support was withdrawn simulating a catastrophic failure. This allowed investigating OOTLUF potentially developed during reliable FM automation and its variation as a function of LOA. Dependent measures were the percentage of time the system was out-of-target, time elapsed until a repair was initiated, alarm reaction time, accuracy of manual tank level recordings, information sampling behavior, and subjective ratings of workload, tension, fatigue, self-confidence and trust in automation. It was found that automated FM support improved overall system performance and reduced subjective operator workload. OOTLUF occurred in terms of longer times until repair initiation only at medium LOA and not as expected at high LOA. A differential effect of LOA on information sampling strategies was observed suggesting that offloading operators from recovery implementation at high LOA during reliable automation enabled engagement in fault assessment activities, thereby maintaining situation awareness. It is concluded that for avoiding OOTLUF in complex, dynamic tasks, involving intelligent FM support, the important factor is not simply the LOA, but the ability of a particular LOA to support the human operator's information sampling and higher-level reasoning activities.