Automation Failures Research Articles

Misuse of Automation: The Impact of System Experience on Complacency and Automation Bias in Interaction with Automated Aids

The study investigates how complacency and automation bias effects in interaction with automated aids are moderated by system experience. Participants performed a supervisory control task supported by an aid for fault identification and management. Groups differed with respect to how long they worked with the aid until eventually an automation failure occurred, and whether this failure was the first or second one the participants were exposed to. Results show that negative experiences, i.e., automation failures, entail stronger effects on subjective trust in automation as well as the level of complacency and automation bias than positive experiences (correct recommendations of the aid). Furthermore, results suggest that commission errors may be due to three different sorts of effects: (1) a withdrawal of attention in terms of incomplete cross-checks of information, (2) an active discounting of contradictory system information, and (3) an inattentive processing of contradictory information analogue to a “looking-but-not-seeing” effect.

Effects of Automation Types on Air Traffic Controller Situation Awareness and Performance

The Joint Planning and Development Office has proposed the introduction of automated systems to help air traffic controllers handle the increasing volume of air traffic in the next two decades (JPDO, 2007). Because fully automated systems leave operators out of the decision-making loop (e.g., Billings, 1991), it is important to determine the right level and type of automation that will keep air traffic controllers in the loop. This study examined the differences in the situation awareness (SA) and collision detection performance of individuals when they worked with information acquisition, information analysis, decision and action selection and action implementation automation to control air traffic (Parasuraman, Sheridan, & Wickens, 2000). When the automation was unreliable, the time taken to detect an upcoming collision was significantly longer for all the automation types compared with the information acquisition automation. This poor performance following automation failure was mediated by SA, with lower SA yielding poor performance. Thus, the costs associated with automation failure are greater when automation is applied to higher order stages of information processing. Results have practical implications for automation design and development of SA training programs.

Performance Consequences of Automated Aids in Supervisory Control: The Impact of Function Allocation

The present study investigates effects of automated decision aids in terms of expected performance benefits of automation support and possible negative performance consequences. The negative consequences include the possible effects of automation bias and performance decrements when returning to manual performance in case of automation failure. Three automated aids that support fault identification and management are compared in a simulated supervisory control task. Results show that primary and secondary task performance improved with automated support compared to manual performance, with effects directly dependent on the level of automation (LOA). Effects of automation bias emerged independent of LOA. For return-to-manual performance, weak indications of an automation-induced skill loss were only found in the highest LOA. These arose in manual fault management, but not in fault identification performance.

Misuse of automated decision aids: Complacency, automation bias and the impact of training experience

The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment ( N=24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behavior, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids.

Recovery from Automation Error after Robot Neglect

Automation is a robot's ability to control itself without operator intervention. Advances in automation have resulted in a phenomenon called robot neglect (Crandall & Goodrich, 2003) in which a robot operator infrequently attends to an automated robot. Robot neglect becomes a problem when automation fails and the robot operator is not attending to the robot. This work outlines a set of guidelines for aiding an operator faced with robot neglect followed by automation failure. The guidelines focus on how video replay of the incidents leading up to the failure may help the operator overcome inattention and regain situation awareness. Complications with this approach in past research are addressed.

On the Independence of Compliance and Reliance: Are Automation False Alarms Worse Than Misses?

Participants performed a tracking task and system monitoring task while aided by diagnostic automation. The goal of the study was to examine operator compliance and reliance as affected by automation failures and to clarify claims regarding independence of these two constructs. Background data revealed a trend toward nonindependence of the compliance-reliance constructs. Thirty-two undergraduate students performed the simulation that presented the visual display while dependent measures were collected. False alarm-prone automation hurt overall performance more than miss-prone automation. False alarm-prone automation also clearly affected both operator compliance and reliance, whereas miss-prone automation appeared to affect only operator reliance. Compliance and reliance do not appear to be entirely independent of each other. False alarms appear to be more damaging to overall performance than misses, and designers must take the compliance-reliance constructs into consideration.

Automation-induced complacency for monitoring highly reliable systems: the role of task complexity, system experience, and operator trust

The increase in quantity and complexity of advanced automated systems has generated new concerns surrounding automation-induced complacency, or the difficulties operators have monitoring the status of automated systems. The present investigation consists of two studies that assessed the impact of system reliability, monitoring complexity, operator trust, and system experience on automation-induced complacency. In both studies, participants operated a manually controlled flight task while monitoring several simulated aircraft displays for failures. The ability of operators to detect a single automation failure over three experimental sessions was also assessed. Results indicated that realistic levels of system reliability severely impaired an operator's ability to monitor effectively. Further, as system experience increased, operator monitoring performance declined. The results also indicated that the complexity of the monitoring task heavily influenced operator monitoring, with poorer performance associated with more cognitively demanding tasks. Finally, results from both studies indicated that operator trust increased and monitoring performance decreased as a function of increasing system reliability. These results suggest that for highly reliable systems, increasing task complexity and extensive experience may severely impair an operator's ability to monitor for unanticipated system states.

A framework for managing system disturbances and insights from air traffic management

System disturbances are likely to be a key factor affecting the acceptance and safety of future automation. Since hardware and software are rarely totally reliable, humans are always required in socio-technical command and control environments such as air traffic management (ATM). Unfortunately, human–automation interaction is known to be problematic, particularly when the human assumes a monitoring or back-up role. Hence an understanding of how humans manage system disturbances is required, together with a method of looking at the problem for new systems. In this paper we outline a contextual framework of the process by which people recover from system disturbances, together with literature data and findings from 31 interviews with ATM personnel. The framework describes the context and causes of a problem, the problem itself, the effect and exposure, the recovery process, and the outcome. The framework, together with the research findings and operational experience, is also the basis for a performance prediction tool called the Recovery from Automation Failure Tool (RAFT).

How Do Automation False Alarms and Misses Affect Operator Compliance and Reliance?

Participants performed a tracking task along with a system and flight parameter monitoring task while aided by diagnostic automation. The goal of the study was to examine operator compliance and reliance as affected by automation failures, and to clarify claims regarding independence of these two constructs. Background data revealed a trend towards non-independence of the compliance-reliance constructs. Thirty-two undergraduate students performed a simulation that presented the visual display and collected dependent measures. False-alarm prone automation hurt overall performance more than did miss-prone automation, while also clearly affecting both operator compliance and reliance. Miss-prone automation only appeared to affect operator reliance.

Misuse of Automated Aids in Process Control: Complacency, Automation Bias and Possible Training Interventions

One important aspect of automation misuse is reflected in an inappropriate monitoring or checking of automated functions, a phenomenon that commonly has been referred to as complacency. The present study investigates complacency effects in interaction with an automated aid in a process control simulation task, as well as its possible performance consequences, i.e. automation bias in terms of commission errors, and impairments of return-to-manual-performance in case of automation breakdown. Furthermore, the effect of a specific training intervention to reduce complacency by exposing participants intentionally to automation failures is investigated. The results provide clear evidence for complacency effects, reflected in an insufficient verification of recommendations provided by the automated aid. Yet, only very high levels of complacency were associated with commission errors, i.e. following an automated advice although it actually is wrong. Exposing operators intentionally to automation failures during training significantly reduced the complacency effect but did not prevent it completely.

Don't Let Me Down: The Role of Operator Expectations in Human-Automation Interaction

This study investigated how user expectations of automation performance influenced reliance and how expectations interacted with types of automation failures. Participants were assigned to a low, high, or neutral expectations group and interacted with an automated collision avoidance system that either committed errors of false detection (false alarms) or incorrect rejection (misses). Participants with high expectations had higher initial reliance than participants in the low and neutral expectations conditions. Overall, the high expectations group exhibited over-reliance behavior choosing to rely at a higher level than participants in the neutral and low conditions. Participants in the neutral condition began to appropriately adjust their behavior according to the type of automation error, however, participants in the low and high conditions did not. These findings have implications for how expectations are managed in future human-automation research.

Automation Failures on Tasks Easily Performed by Operators Undermine Trust in Automated Aids

We tested the hypothesis that automation errors on tasks easily performed by humans undermine trust in automation. Research has revealed that the reliability of imperfect automation is frequently misperceived. We examined the manner in which the easiness and type of imperfect automation errors affect trust and dependence. Participants performed a target detection task utilizing an automated aid. In Study 1, the aid missed targets either on easy trials (easy miss group) or on difficult trials (difficult miss group). In Study 2, we manipulated both easiness and type of error (miss vs. false alarm). The aid erred on either difficult trials alone (difficult errors group) or on difficult and easy trials (easy miss group; easy false alarm group). In both experiments, easy errors led to participants mistrusting and disagreeing more with the aid on difficult trials, as compared with those using aids that generated only difficult errors. This resulted in a downward shift in decision criterion for the former, leading to poorer overall performance. Misses and false alarms led to similar effects. Automation errors on tasks that appear "easy" to the operator severely degrade trust and reliance. Potential applications include the implementation of system design solutions that circumvent the negative effects of easy automation errors.

Age differences in trust and reliance of a medication management system

The present study examined age differences in trust and reliance of an automated decision aid. In Experiment 1, older and younger participants performed a simple mathematical task concurrent with a simulated medication management task. The decision aid was designed to facilitate medication management, but with varying reliability. Trust, self-confidence and usage of the aid were measured. The results indicated that older adults had greater trust in the aid and were less confident in their performance, but they did not calibrate trust differently than younger adults. In Experiment 2, a variant of the same task was used to investigate whether older adults are subject to over-reliance on the automation. Differences in omission and commission errors were examined. The results indicated that older adults were more reliant on the decision aid and committed more automation-related errors. A signal detection analyses indicated that older adults were less sensitive to automation failures. Results are discussed with respect to the perceptual and cognitive factors that influence age differences in the use of fallible automation.

Effects of Automation Failure in a Luggage Screening Task: A Comparison between Direct and Indirect Cueing

The present study investigated the use of two automated aids of different reliabilities in a luggage screening task. A Direct Cue consisting of a green circle around a potential target directs attention to a specific part of the luggage image, while an Indirect Cue, consisting of a green border around an image determined to have a target, does not. Direct Cues offer an advantage in visual inspection tasks because they guide attention to specific areas of the visual image but this can also cause attentional tunneling. Furthermore, the reliance on automation may negatively impact manual performance after the aid is removed or is no longer available. Thus, two issues were investigated in the current study: (1) how do failures in Direct and Indirect Cues affect reliance and (2) how does a complete failure affect performance after operators had the use of an automated aid? Results suggest that reliance patterns were more optimal with the Direct Cue than with the Indirect Cue and performance with a more reliable Indirect Cue was not much better than a less reliable one. The results also suggest that manual performance, when the aid was removed, was better for participants who had used the automated aids compared to control participants who did not have any use of the aid previously. The advantage of previously aided performance on subsequent manual performance was greatest for those who had used the more reliable Direct Cue. Explanations and implications are discussed.

The Role of Automation in Complex System Failures

Although proponents of advanced information technology argue that automation can improve the reliability of health care delivery, the results of introducing new technology into complex systems are mixed. The new forms of failure that accompany automation challenge technical workers, often demanding novel approaches to recovering from failure and restoring system operations. We present a case where automation created a new form of failure that was not foreseen. In this case, human practitioners were the main source of robust and reliable health care delivery and were the primary agents in recovering from a life-threatening automation failure. The features of this case correspond closely to experience with new information technology in other domains and have implications for plans to improve patient safety using technology.

Type of Automation Failure: The Effects on Trust and Reliance in Automation

Past automation research has focused primarily on machine-related factors (e.g., automation reliability) and human-related factors (e.g., accountability). Other machine-related factors such as type of automation errors, misses or false alarms, have been noticeably overlooked. These two automation errors correspond to potential operator errors, omission (misses) and commission (false alarms), which have proven to directly affect operators' trust in automation. This proposed research will examine how automation-error-type affects operator trust and begin to develop baseline trust measures as they relate to error type and participant age. It is expected that participants presented with more automation false alarms than misses will experience a larger degradation of subjective trust than those presented with equal numbers of false alarms and misses or more automation misses than false alarms.

Effects of Automation Reliability on Human Monitoring Performance

Human decisions to rely on automation of a simulated flight task were monitored in the present study. Participants performed the Multi Attribute Task (MAT) battery previously developed by Comstock and Arneguard (1990). Twenty-eight participants were tested for 7 different reliability levels. Decision accuracy, false alarm rates, and reaction time to simulated automation failures were collected for each participant over nine 10-minute sessions. The results indicated that the rate of automation failures detected varied inversely with automation reliability level. As automation level increased, detection rate decreased thereby indicating automation-induced monitoring inefficiency. However, the mean reaction time of correct detection rate decreased as automation reliability level increased. No significant effect of reliability was obtained for the false alarm rate data. The implications for training and systems design are also outlined.

Automation Failures on Tasks Easily Performed by Operators Undermines Trust in Automated Aids

Automation users often distrust diagnostic aids that are imperfectly reliable. The extent to which users' trust in automation is influenced by the simplicity of automation errors was explored. Participants (n = 30) performed 200 trials of a target-detection task using an imperfectly reliable automated aid. For the “easy-miss” group (n = 15) the automated aid missed targets only on easy trials and was accurate on difficult trials. For the “difficult-miss” group (n = 15) the aid missed targets only on difficult trials and was accurate on easy trials. A control group (n = 15) performed the task unaided. The easy-miss group trusted the aid less, was less accurate, and contradicted the aid more than the difficult-miss group, even on trials when the easy-miss aid was more reliable than the difficult-miss aid. Results suggest that the “easiness” of the aid's errors for the easy-miss group undermined automation trust and reliance. Potential future directions include examining whether easy false alarms affect user trust in a manner akin to easy misses.

Physiological Self-Regulation and Adaptive Automation

Adaptive automation has been proposed as a solution to current problems of human-automation interaction. Past research has shown the potential of this advanced form of automation to enhance pilot engagement and lower cognitive workload. However, there have been concerns voiced regarding issues, such as automation surprises, associated with the use of adaptive automation. This study examined the use of psychophysiological self-regulation training with adaptive automation that may help pilots deal with these problems through the enhancement of cognitive resource management skills. Eighteen participants were assigned to 3 groups (self-regulation training, false feedback, and control) and performed resource management, monitoring, and tracking tasks from the Multiple Attribute Task Battery. The tracking task was cycled between 3 levels of task difficulty (automatic, adaptive aiding, manual) on the basis of the electroencephalogram-derived engagement index. The other two tasks remained in automatic mode that had a single automation failure. Those participants who had received self-regulation training performed significantly better and reported lower National Aeronautics and Space Administration Task Load Index scores than participants in the false feedback and control groups. The theoretical and practical implications of these results for adaptive automation are discussed.

Supporting Situation Assessment Through Attention Guidance: A Cost-Benefit and Depth of Processing Analysis

Automated support systems may be useful tools for aiding situation assessment in complex environments such as the military battlefield. These environments are marked by large amounts of information which often must be weighted and integrated into a meaningful judgment or assessment. The present research examines the effects of attention cueing on information integration tasks in static battlefield situations. Sixteen participants completed a resource allocation task for 56 battlefield scenarios (based on perceived threats). For half the trials, an automated system guided their attention to high-threat units. On 2 trials a memory probe was administered to assess the depth of processing of information, and on the final trial an automation failure was presented. Results demonstrated an overall allocation performance advantage for automation but poorer recall for automation-enhanced units. Half of the participants failed to attend to the system failure. Those participants who detected the failure were inferred to have processed the cues more deeply on the memory trials. The costs and benefits of automated cueing are discussed.