Automated Decision Aids Research Articles

Misuse of automated decision aids: Complacency, automation bias and the impact of training experience

The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment ( N=24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behavior, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids.

The impact of cognitive feedback on judgment performance and trust with decision aids

Computerized decision aids are designed to support human operators’ decision-making activities in a variety of contexts including medicine, military command and control, and aviation industry. One common characteristic of these systems is of the role of the decision-aid, integrating a variety of measured information to produce a simple form of more meaningful information that can be used to support human operators’ judgment about environmental states of interest. When these aids malfunction, the decision makers may ignore the aid due to the lack of trust in the aid. This study examines the effect of automated decision aids of varying quality in producing environmental estimates, and investigates the effect of meta-information in supporting judgment performance of human operators’ with the decision aids and calibrating human trust in such aids. A Lens Model based feedback is used to provide meta-information about the decision-aid. An aircraft identification task is performed under varying conditions of aid quality and the presence of meta-information. Results show that performance, as well as assessments of trust in the aid, are affected by the decision aid's quality. More importantly, participants given with the meta-information performed significantly better than those without it. Results indicate that human operators can compensate for a poor performing aid when meta-information is available. Further, operators’ trust was better calibrated corresponding to the decision aid's quality. A practical implication of this study is that meta-information can be useful to human operators in increasing their understanding and appropriate utilization of automated decision aids.

Misuse of Human and Automated Decision AIDS in a Soldier Detection Task

Providing human operators with automated decision aids does not always improve performance. In this study, 24 students viewed terrain slides, half of which included a camouflaged soldier. After viewing each slide, participants could view the decision reached by a human or automated aid, or continue without the help of an aid. Some of the participants were led to believe their aids were experts; others were not. Significantly more participants showed a preference to view human rather than automated aid decisions. However, participants were as likely to misuse an automated aid as they were a human aid. These biases existed regardless of the perceived expertise of the aids. In addition, results of a linguistic analysis of open-ended descriptions of automated and human aids suggest human operators view automated and human aids differently and have implications for training and system development.

Expertise Levels of Human versus Automated Decision Aids Influence Response Biases in a Visual Search Task

Studies have demonstrated that humans appear to apply norms of human-human interaction to interaction with automated decision aids. We examined the differences in perceptions of automation vs. humans when the expertise and reliability of these advisers varied. Participants (n = 180) performed a luggage-screening task with the assistance of human or automated advisers that differed in pedigree (expert vs. novice) and reliability (high vs. low), but had a similar neutral beta setting of 1.0. Shifts in sensitivity, criterion settings and accuracy were assessed. Participants who were presented with a low-reliable “expert” adviser shifted their bias away from the neutral bias of the adviser and more toward optimal beta compared to participants receiving unreliable advice from a ‘novice’. This effect increased across trials for participants using low-reliability automated advisers but not human advisers. The results have implications for the development of models of optimal utilization of decision aids.

Effects of Automated Conflict Cuing and Traffic Density on Air Traffic Controller Performance and Visual Attention in a Datalink Environment

Proposals to enhance the efficiency and capacity of the air traffic control system involve new technologies such as automated decision aids and datalink that could create additional demands for the limited visual attentional resources of controllers. This study investigated whether automated multimodal conflict cues could mitigate these demands, particularly under high-traffic load. Measures of conflict detection performance and eye movements were obtained from full performance controllers in a simulated air traffic control task with datalink. It was hypothesized that multimodal cuing would improve advance notification of conflicts and lead to earlier detection than purely visual cuing by better guiding visual attention. High-traffic density had significantly negative effects on performance. However, although cuing enhanced performance, no specific benefit of multimodal cuing for performance or visual attention was found. Implications for controller visual attention and for the design of automated decision aids are discussed.

Age differences in trust and reliance of a medication management system

The present study examined age differences in trust and reliance of an automated decision aid. In Experiment 1, older and younger participants performed a simple mathematical task concurrent with a simulated medication management task. The decision aid was designed to facilitate medication management, but with varying reliability. Trust, self-confidence and usage of the aid were measured. The results indicated that older adults had greater trust in the aid and were less confident in their performance, but they did not calibrate trust differently than younger adults. In Experiment 2, a variant of the same task was used to investigate whether older adults are subject to over-reliance on the automation. Differences in omission and commission errors were examined. The results indicated that older adults were more reliant on the decision aid and committed more automation-related errors. A signal detection analyses indicated that older adults were less sensitive to automation failures. Results are discussed with respect to the perceptual and cognitive factors that influence age differences in the use of fallible automation.

Modeling the Effectiveness of Tools to Assist Sonar Operators

The task of building the underwater picture from sonar data is made complex by high volumes of noise and multiple data that arrive from a variety of acoustic sources detected at great distances by modern, sonar equipment. Typically, acoustic sources from ships have a complex spectrum consisting of several base frequency components and related harmonics. The task for operators is to analyse the data to determine if there is a pattern that represents the signature of a known source, thereby leading to identification of a vessel. Since the task can be highly labour intensive automated decision aids may be of value to the operator. This project addresses how to predict and optimise the impact of new technologies in system re-design by using a modeling/simulation approach to operator-system functionality. A generic sonar analysis process was simulated and the effectiveness of a decision aid evaluated. The improvement in performance predicted by the aid was then validated experimentally.

The Effect of Perceived Consequence on Automation Reliance in a Human-Automation Collaborative System

As automated systems are entering new environments, some of which involve high-risk decision making, it is critical that we understand in what situations people will or will not rely on the recommendations of automated decision aids. It is theorized that in deciding whether to trust automation people consider perceived consequence, weighing the cost associated with inappropriate action or inaction and the psychological cost associated with verifying the aid. This study will address the effect that perceived consequence has on attitudes and behavior toward decision aids by exposing participants to different levels of consequence, manipulated by the cost associated with making a mistake and the cost needed to verify the aid. It is expected that as the cost of making a mistake increases and the cost of verifying the automation decreases, trust and reliance in a decision aid will decrease.

CFO Intentions of Fraudulent Financial Reporting

This study investigates factors indicating CFO intentions of fraudulent financial reporting. Structural equation modeling is used to analyze survey data obtained from 139 CFOs. We find that an extended reasoned action model fits the data well and explains CFO intentions to report fraudulently. More specifically, we find that CFOs of large companies are more likely to report fraudulently in the financial statements and that compensation structure is not a good indicator of CFO intentions to report fraudulently. Informal and formal audit methods for assessing management attitudes toward fraudulent reporting, such as questionnaires and automated decision aids, are recommended since they offer the best opportunity to improve significantly auditors' ability to predict fraudulent financial reporting. Implications for future research and practice development are considered.

Automation in Future Air Traffic Management: Effects of Decision Aid Reliability on Controller Performance and Mental Workload

Future air traffic management concepts envisage shared decision-making responsibilities between controllers and pilots, necessitating that controllers be supported by automated decision aids. Even as automation tools are being introduced, however, their impact on the air traffic controller is not well understood. The present experiments examined the effects of an aircraft-to-aircraft conflict decision aid on performance and mental workload of experienced, full-performance level controllers in a simulated Free Flight environment. Performance was examined with both reliable (Experiment 1) and inaccurate automation (Experiment 2). The aid improved controller performance and reduced mental workload when it functioned reliably. However, detection of a particular conflict was better under manual conditions than under automated conditions when the automation was imperfect. Potential or actual applications of the results include the design of automation and procedures for future air traffic control systems.

Solution Generation with Qualitative Models of Preferences

We consider automated decision aids that help users select the best solution from a large set of options. For such tools to successfully accomplish their task, eliciting and representing users' decision preferences is a crucial task. It is usually too complex to get a complete and accurate model of their preferences, especially regarding the trade‐offs between different criteria.We consider decision aid tools where users specify their preferences qualitatively: they are only able to state the criteria they consider, but not the precise numerical utility functions. For each criterion, the tool provides a standardized numerical function that is fixed and identical for all users and used to compare solutions. To compensate for the imprecision of this qualitative model, we let the user choose among a displayed set of possibilities rather than a single optimal solution. We consider the probability of finding the most preferred solution as a function of the number of displayed possibilities and the number of preferences. We present a probabilistic analysis, empirical validation on randomly generated configuration problems and a commercial application. We provide mathematical principles for the design of the selection mechanism, guaranteeing that users are able to find the target solution.

Effects of Task Duration and Type of Automation Support on Human Performance and Stress in a Simulated Battlefield Engagement Task

Automation that is meant to assist the operator by reducing stress and workload may actually increase both. Automation can be imperfect, and as a result may add to uncertainty and degrade performance when it fails. In a complex and dynamic task environment, this may lead to increased stress, which in turn may be associated with inefficient use of automation. The present study examined the relative benefits and costs of imperfect information and decision automation in a simulated battlefield engagement task. Performance and stress were evaluated for short- (10 minutes) and long-durations (30 minutes) of the task. The cost of unreliable aiding—a decrease in accuracy of appropriate target engagement selection—was greater for the decision automation than for information automation, irrespective of task length. Unreliability also slowed response times more for decision than for information automation, but only when the task was long. Subjective reports of stress also increased in the longer automated conditions. The results show that automated decision aids can improve performance when reliable. However, the price of imperfect decision automation is reduced accuracy and increased stress compared to information automation, particularly when the task must be performed for long periods.

Effects of Age on Utilization and Perceived Reliability of an Automated Decision-Making aid for Luggage Screening

An experiment examined the effects of age on utilization and perceived reliability of an imperfectly reliable decision-making aid in a luggage x-ray screening task. Forty-five young adults and 45 elderly adults performed a simulated luggage screening task. Some subjects were provided the assistance of an automated decision aid with a hit rate of .90 and a false alarm rate of .25. Others performed the task with no aid. Signal-detection analysis revealed that automation improved sensitivity only for younger participants, suggesting a tendency for older participants to underutilize the aid's recommendations. Data also revealed unique patterns of individual differences in cue reliance among older and younger participants. Perceived reliability of the aid did not differ between age groups. Order of information presentation (with the aid's recommendation coming before or after the raw data) had little effect for either age group.

The effects of fraud risk assessments and a risk analysis decision aid on auditors’ evaluation of evidence and judgment

Using a sample of 258 auditors, two experiments were conducted to study the effects of fraud risk assessments and an automated decision aid on auditors’ evaluation of evidence and judgment. Results indicate that the assessed level of fraud risk systematically affects the evaluation of evidence by auditors. More specifically, auditors facing high levels of assessed fraud risk evaluate audit evidence more thoroughly than auditors facing low levels of assessed fraud risk. In addition, auditors facing high risk assessments exhibit an unexpected bias in their decision processes. These auditors focus on the last evidence received during decision making, and their decisions change when evidence order is altered. The introduction of a computer‐based decision aid promotes consistency in auditor decisions and eliminates the undesirable effects of evidence order, while preserving the benefits of more thorough evidence evaluation.

Which clinical decisions benefit from automation? A task complexity approach

Objective: To describe a model for analysing complex medical decision making tasks and for evaluating their suitability for automation. Method: Assessment of a decision task's complexity in terms of the number of elementary information processes (EIPs) and the potential for cognitive effort reduction through EIP minimisation using an automated decision aid. Results: The model consists of five steps: (1) selection of the domain and relevant tasks; (2) evaluation of the knowledge complexity for tasks selected; (3) identification of cognitively demanding tasks; (4) assessment of unaided and aided effort requirements for this task accomplishment; and (5) selection of computational tools to achieve this complexity reduction. The model is applied to the task of antibiotic prescribing in critical care and the most complex components of the task identified. Decision aids to support these components can provide a significant reduction of cognitive effort suggesting this is a decision task worth automating. Conclusion: We view the role of decision support for complex decision to be one of task complexity reduction, and the model described allows for task automation without lowering decision quality and can assist decision support systems developers.

The role of trust in automation reliance

A recent and dramatic increase in the use of automation has not yielded comparable improvements in performance. Researchers have found human operators often underutilize (disuse) and overly rely on (misuse) automated aids (Parasuraman and Riley, 1997). Three studies were performed with Cameron University students to explore the relationship among automation reliability, trust, and reliance. With the assistance of an automated decision aid, participants viewed slides of Fort Sill terrain and indicated the presence or absence of a camouflaged soldier. Results from the three studies indicate that trust is an important factor in understanding automation reliance decisions. Participants initially considered the automated decision aid trustworthy and reliable. After observing the automated aid make errors, participants distrusted even reliable aids, unless an explanation was provided regarding why the aid might err. Knowing why the aid might err increased trust in the decision aid and increased automation reliance, even when the trust was unwarranted. Our studies suggest a need for future research focused on understanding automation use, examining individual differences in automation reliance, and developing valid and reliable self-report measures of trust in automation.

Judgment and Trust in Conjunction with Automated Decision Aids: A Theoretical Model and Empirical Investigation

The importance of understanding and supporting human operators' judgment and decision-making has been recognized in a variety of contexts. Of interest in these systems is the extent to which operators utilize and trust such systems, particularly where operators do not have any means to access the environment directly. Operators' trust in automated decision aids is the critical component affecting their performance in estimating the environmental states. A model of human operators' judgment with automated decision aids was developed using the Lens Model framework. This model shows not only different characteristics of automated decision aids, but also the representation of operators' judgment policy. An empirical investigation on operators' judgment performance and trust in conjunction with automated decision aids is described in this research. Among the potential characteristics of such systems that may affect operators' judgment performance and trust, three aspects were controlled. Results provide strong support for factors in the theoretical framework and for identification of the types of information to increase the level of operators' understanding.

EFFECTS OF UNRELIABLE AUTOMATION ON DECISION MAKING IN COMMAND AND CONTROL

The effectiveness of automated decision aids used by human operators in command and control systems may depend not only on automation reliability, but also on the type (stage) and level the automated support provides. Automation can be applied to information acquisition, information integration and analysis, decision choice selection, or action implementation (Parasuraman, Sheridan, & Wickens, 2000). The present study examined the effects of variations in the stage of automation support on performance in a “Sensor to Shooter” targeting simulation of command and control. Independent variables included the type and level of automation support (complete listing, priority listing, top choices, and recommendation of decision choice) and the reliability of the automation (60% and 80%). Dependent variables included accuracy and reaction time of target engagement decisions. Compared to manual performance, reliable automation did not affect the accuracy of target engagement decisions but did significantly reduce decision times. When the automation was unreliable, under the higher reliability condition (80%) there was a greater cost in accuracy performance for higher levels of automation aiding (priority listing, top choice, and recommendation) than at a lower level (complete listing). The results support the view that automation unreliability has a greater performance cost for decision automation than for information automation. This performance cost generalizes across a number of different forms of decision-aiding.

The Influence of Feedback on Automation Use, Misuse, and Disuse

The use of automated decision aids is increasing. Unfortunately, the productivity of human-automated work teams is not always superior to that of the human operator or the computer working alone (Parasuraman & Riley, 1997). In order to examine the effects of various kinds of feedback on appropriate automation use, 159 students provided with an automated aid determined whether a camouflaged soldier was present in 300 slides. A 2 (rationale why aid might err) X 2 (continuous feedback) X 2 (aid's relative performance) X 2 (aid's decision) ANOVA was performed with the transformed p(error) separately for the trials in which the target was present and for those in which the target was absent. Results indicated that participants were able to appropriately rely on the automated aid. For example, those paired with a superior aid were more likely to rely on the aid's decisions than ignore them; those paired with an inferior aid were more likely to ignore the aid than rely on it. A three way interaction indicated that this finding was most strong when participants were provided with information concerning why their aid might err, F(1, 51) = 3.27, p <. 08. Implications for future research are discussed.

The perceived utility of human and automated aids in a visual detection task.

Although increases in the use of automation have occurred across society, research has found that human operators often underutilize (disuse) and overly rely on (misuse) automated aids (R. Parasuraman & V. Riley, 1997). Nearly 275 Cameron University students participated in 1 of 3 experiments performed to examine the effects of perceived utility (M. T. Dzindolet, H. P. Beck, L. G. Pierce, & L. A. Dawe, 2001) on automation use in a visual detection task and to compare reliance on automated aids with reliance on humans. Results revealed a bias for human operators to rely on themselves. Although self-report data indicate a bias toward automated aids over human aids, performance data revealed that participants were more likely to disuse automated aids than to disuse human aids. This discrepancy was accounted for by assuming human operators have a "perfect automation" schema. Actual or potential applications of this research include the design of future automateddecision aids and training procedures for operators relying on such aids.