Monitoring Automation Failures: Effects of Automation Reliability and Task Complexity

Abstract

Two studies examined the effects of automation reliability and task complexity on the monitoring of automation failures during performance of a flight-simulation task. In the first study, 24 students performed tracking and resource management tasks while an automation routine monitored for system malfunctions over four 30-minute sessions Detection of automation failures was significantly higher for variable reliability automation (mean = 81.6%) than for constant reliability automation (mean = 32.7%), indicating that constant-reliability automation induced complacency in monitoring. The effect of automation reliability was eliminated when 16 more subjects were required to complete the monitoring task only. Neither group of subjects exhibited a vigilance decrement. In the second study monitoring performance and vigilance decrement were examined for a situation in which only one automation failure occurred during a session. 36 students were randomly assigned to one of three task groups: simple (visual discrimination task), single-complex (monitoring only) or multi-complex (tracking, resource management, and monitoring). In both the simple and the multi-complex tasks, more subjects detected the automation failure in the first ten minutes of a session than in the last ten minutes of a session (67%-17% and 75%-42% respectively). Subjects in the single-complex condition detected the automation failure equally well in both time periods (92%-83%). The results point to two areas of potential costs in the automation of a task: (1) constant patterns of automation reliability can lead to inefficiency in monitoring automation failures; and (2) infrequent automation failures in multi-task conditions can lead to a vigilance decrement. While these costs should not prohibit the implementation of automation, they should be considered in the design of any automated system.