Imperfect Automation Research Articles

Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance.

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

Adapting to Imperfect Automation: The Impact of Experience on Dependence Behavior and Response Strategies

This study examined the impact of experience on individuals’ dependence behavior and response strategies when interacting with imperfect automation. 41 participants used an automated aid to complete a dual-task scenario comprising of a compensatory tracking task and a threat detection task. The entire experiment was divided into four quarters and multi-level models (MLM) were built to investigate the relationship between experience and the dependent variables. Results show that compliance and reliance behaviors and perfor- mance scores significantly increased as participants gained more experience with automation. In addition, as the experiment progressed, a significant number of participants adapted to the automation and resorted to an extreme use response strategy. The findings of this study suggest that automation response strategies are not static and most individual operators eventually follow or discard the automation. Understanding individual response strategies can support the development of individualized automation systems and improve operator training.

The (Im)perfect Automation Schema: Who Is Trusted More, Automated or Human Decision Support?

This study's purpose was to better understand the dynamics of trust attitude and behavior in human-agent interaction. Whereas past research provided evidence for a perfect automation schema, more recent research has provided contradictory evidence. To disentangle these conflicting findings, we conducted an online experiment using a simulated medical X-ray task. We manipulated the framing of support agents (i.e., artificial intelligence (AI) versus expert versus novice) between-subjects and failure experience (i.e., perfect support, imperfect support, back-to-perfect support) within subjects. Trust attitude and behavior as well as perceived reliability served as dependent variables. Trust attitude and perceived reliability were higher for the human expert than for the AI than for the human novice. Moreover, the results showed the typical pattern of trust formation, dissolution, and restoration for trust attitude and behavior as well as perceived reliability. Forgiveness after failure experience did not differ between agents. The results strongly imply the existence of an imperfect automation schema. This illustrates the need to consider agent expertise for human-agent interaction. When replacing human experts with AI as support agents, the challenge of lower trust attitude towards the novel agent might arise.

Achieving agency within imperfect automation: Working customers and self-service technologies

In this article, we propose to treat agency as something which is accomplished in the entanglement of humans with technologies. This redirects our attention away from the question of what distinguishes humans from smart machines and towards querying how people and automated apparatuses join in processes of mutual sociomaterial engagement. To further our argument, we look at self-service kiosks, which are ubiquitous yet largely overlooked components of mediated environments. We reflect on a participant observation in groceries stores and interviews with customers familiar with self-checkout facilities. They make us aware that operating this equipment is not an individual affair but a joint activity by default, taking place in a temporally regimented setting prone to human errors and malfunction when people try to respond to the terminals’ protocol. This sort of imperfect automation has ambivalent ramifications which rely on the capabilities of users and the capacities of an interface and its underlying operations. Agency, we conclude, thus becomes a matter of viable performance in which humans may act machine-like while machines perform an expanding share of activities.

Attention Control and Performance Across Varying Levels of Imperfect Automation

Attention Control and Performance Across Varying Levels of Imperfect Automation

Fully Automated MRI Segmentation and Volumetric Measurement of Intracranial Meningioma Using Deep Learning.

Accurate and rapid measurement of the MRI volume of meningiomas is essential in clinical practice to determine the growth rate of the tumor. Imperfect automation and disappointing performance for small meningiomas of previous automated volumetric tools limit their use in routine clinical practice. To develop and validate a computational model for fully automated meningioma segmentation and volume measurement on contrast-enhanced MRI scans using deep learning. Retrospective. A total of 659 intracranial meningioma patients (median age, 59.0 years; interquartile range: 53.0-66.0 years) including 554 women and 105 men. The 1.0 T, 1.5T, and 3.0 T; three-dimensional, T1 -weighted gradient-echo imaging with contrast enhancement. The tumors were manually segmented by two neurosurgeons, H.K. and C.-K.P., with 10 and 26 years of clinical experience, respectively, for use as the ground truth. Deep learning models based on U-Net and nnU-Net were trained using 459 subjects and tested for 100 patients from a single institution (internal validation set [IVS]) and 100 patients from other 24 institutions (external validation set [EVS]), respectively. The performance of each model was evaluated with the Sørensen-Dice similarity coefficient (DSC) compared with the ground truth. According to the normality of the data distribution verified by the Shapiro-Wilk test, variables with three or more categories were compared by the Kruskal-Wallis test with Dunn's post hoc analysis. A two-dimensional (2D) nnU-Net showed the highest median DSCs of 0.922 and 0.893 for the IVS and EVS, respectively. The nnU-Nets achieved superior performance in meningioma segmentation than the U-Nets. The DSCs of the 2D nnU-Net for small meningiomas less than 1cm3 were 0.769 and 0.780 with the IVS and EVS, respectively. A fully automated and accurate volumetric measurement tool for meningioma with clinically applicable performance for small meningioma using nnU-Net was developed. 3 TECHNICAL EFFICACY: Stage 2.

Challenging presumed technological superiority when working with (artificial) colleagues

Technological advancements are ubiquitously supporting or even replacing humans in all areas of life, bringing the potential for human-technology symbiosis but also novel challenges. To address these challenges, we conducted three experiments in different task contexts ranging from loan assignment over X-Ray evaluation to process industry. Specifically, we investigated the impact of support agent (artificial intelligence, decision support system, or human) and failure experience (one vs. none) on trust-related aspects of human-agent interaction. This included not only the subjective evaluation of the respective agent in terms of trust, reliability, and responsibility, when working together, but also a change in perspective to the willingness to be assessed oneself by the agent. In contrast to a presumed technological superiority, we show a general advantage with regard to trust and responsibility of human support over both technical support systems (i.e., artificial intelligence and decision support system), regardless of task context from the collaborative perspective. This effect reversed to a preference for technical systems when switching the perspective to being assessed. These findings illustrate an imperfect automation schema from the perspective of the advice-taker and demonstrate the importance of perspective when working with or being assessed by machine intelligence.

The Imperfect Automation Schema - Evidence for Increased Trust in Human Support Agents

The Imperfect Automation Schema - Evidence for Increased Trust in Human Support Agents

Decision Aiding for Nautical Collision Avoidance: Trust, Dependence, and Implicit Understanding of the Decision Algorithm

Two experiments examined imperfect automation decision aids for maritime collision avoidance. In Experiment 1, the algorithm was driven purely by safety, recommending turning your ship in the direction that produced the greatest separation from a hazard ship. In Experiment 2, the algorithm incorporated two additional factors known to influence ship collision avoidance maneuvers: efficiency and procedural adherence to established “rules of the road.” In both experiments, results revealed heavy but not total reliance on the aid. A strong influence of categorical rules of the road was indicated by compliance with recommendations in concurrence to those conventions, even as they reduced continuous parameters of safety, particularly in Experiment 2 when the rules were incorporated in the algorithm. This illustrates the powerful influence of categorical procedural algorithm elements over continuous quantitative ones in affecting automation compliance. Results also revealed the dissociation between rated trust in, and behavioral dependence on decision aiding automation.

Automation trust increases under high-workload multitasking scenarios involving risk

Trust is a critical construct that influences human–automation interaction in multitasking workspaces involving imperfect automation. Karpinsky et al. (Appl Ergon, 70, 194–201, 2018) investigated whether trust affects operators’ attention allocation in high-load scenarios using the multi-attribute task battery II (MATB). Results suggested that task load reduces trust towards imperfect automation, then reducing visual attention allocation to the monitoring task aided by the automation. Participants also reported reduced levels of trust in high-load conditions. However, it is possible that the participants in high-load conditions did not trust the system because their poor task performance did not have expressly adverse consequences (i.e., risk). The current experiments aimed to replicate and extend Karpinsky et al. (2018) by asking forty participants to concurrently perform a tracking task and system monitoring task in the MATB II with or without risk. The reliability of the automated aid supporting the system monitoring task was 70%. The study employed a 2 × 2 split-plot design with task load (easy vs. difficult) via magnitude of errors in the tracking task as a within-participant factor and risk (high vs. low) as a between-participant factor. Participants in the high-risk group received an instruction that poor performance would result in a repeat of the experiment, whereas participants in the low-risk group did not receive this instruction. Results showed that trust was comparable between the high- and the low-load conditions, but the high risk elevated trust in the high-load condition. This implies that operators display greater levels of trust when a multitasking environment demands greater attention and they perceive risk of receiving expressly adverse consequence, regardless of the true reliability of automated systems.

Automation trust and attention allocation in multitasking workspace

Previous research suggests that operators with high workload can distrust and then poorly monitor automation, which has been generally inferred from automation dependence behaviors. To test automation monitoring more directly, the current study measured operators' visual attention allocation, workload, and trust toward imperfect automation in a dynamic multitasking environment. Participants concurrently performed a manual tracking task with two levels of difficulty and a system monitoring task assisted by an unreliable signaling system. Eye movement data indicate that operators allocate less visual attention to monitor automation when the tracking task is more difficult. Participants reported reduced levels of trust toward the signaling system when the tracking task demanded more focused visual attention. Analyses revealed that trust mediated the relationship between the load of the tracking task and attention allocation in Experiment 1, an effect that was not replicated in Experiment 2. Results imply a complex process underlying task load, visual attention allocation, and automation trust during multitasking. Automation designers should consider operators’ task load in multitasking workspaces to avoid reduced automation monitoring and distrust toward imperfect signaling systems.

Attention allocation for human multi-robot control: Cognitive analysis based on behavior data and hidden states

Abstract Human multi-robot interaction exploits both the human operator's high-level decision-making skills and the robotic agents’ vigorous computing and motion abilities. While controlling multi-robot teams, an operator's attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator's attentional resources, a robot with self-reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots’ normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot's request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots’ failures) tasks, regardless of the system's reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users’ underlying cognitive strategies and may fail to reflect the user's true intent. As understanding users’ intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators’ underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments.

Implementing Lumberjacks and Black Swans Into Model-Based Tools to Support Human-Automation Interaction.

The objectives were to (a) implement theoretical perspectives regarding human-automation interaction (HAI) into model-based tools to assist designers in developing systems that support effective performance and (b) conduct validations to assess the ability of the models to predict operator performance. Two key concepts in HAI, the lumberjack analogy and black swan events, have been studied extensively. The lumberjack analogy describes the effects of imperfect automation on operator performance. In routine operations, an increased degree of automation supports performance, but in failure conditions, increased automation results in more significantly impaired performance. Black swans are the rare and unexpected failures of imperfect automation. The lumberjack analogy and black swan concepts have been implemented into three model-based tools that predict operator performance in different systems. These tools include a flight management system, a remotely controlled robotic arm, and an environmental process control system. Each modeling effort included a corresponding validation. In one validation, the software tool was used to compare three flight management system designs, which were ranked in the same order as predicted by subject matter experts. The second validation compared model-predicted operator complacency with empirical performance in the same conditions. The third validation compared model-predicted and empirically determined time to detect and repair faults in four automation conditions. The three model-based tools offer useful ways to predict operator performance in complex systems. The three tools offer ways to predict the effects of different automation designs on operator performance.

Modeling Relationships among Workload, Trust, and Visual Scanning in an Automated Flight Task

The present study explored relationships among operators’ workload, subjective trust, and visual scanning patterns during their interaction with imperfect automation in a low-fidelity flight simulation task. Participants performed both a manual tracking task and a secondary system monitoring task (automated; FA- vs. miss-prone) under high and low load conditions manipulated by central task demand. The high load condition produced, 1) less frequent saccades toward system monitoring, 2) greater subjective workload, and 3) lower levels of subjective trust, compared to the low load condition. A mediation analysis revealed that subjective workload mediated the effect of load on visual scanning patterns whereas subjective trust did not. The results imply that, when the central task demands more attention, operators strategically reduce sampling rates of information from the system monitoring task assisted by imperfect automation depending on their subjective workload but not on subjective trust.

Benefits of Imperfect Conflict Resolution Advisory Aids for Future Air Traffic Control

The aim of this study was to examine the human-automation interaction issues and the interacting factors in the context of conflict detection and resolution advisory (CRA) systems. The issues of imperfect automation in air traffic control (ATC) have been well documented in previous studies, particularly in conflict-alerting systems. The extent to which the prior findings can be applied to an integrated conflict detection and resolution system in future ATC remains unknown. Twenty-four participants were evenly divided into two groups corresponding to a medium- and a high-traffic density condition, respectively. In each traffic density condition, participants were instructed to perform simulated ATC tasks under four automation conditions, including reliable, unreliable with short time allowance to secondary conflict (TAS), unreliable with long TAS, and manual conditions. Dependent variables accounted for conflict resolution performance, workload, situation awareness, and trust in and dependence on the CRA aid, respectively. Imposing the CRA automation did increase performance and reduce workload as compared with manual performance. The CRA aid did not decrease situation awareness. The benefits of the CRA aid were manifest even when it was imperfectly reliable and were apparent across traffic loads. In the unreliable blocks, trust in the CRA aid was degraded but dependence was not influenced, yet the performance was not adversely affected. The use of CRA aid would benefit ATC operations across traffic densities. CRA aid offers benefits across traffic densities, regardless of its imperfection, as long as its reliability level is set above the threshold of assistance, suggesting its application for future ATC.

The effects of driving with different levels of unreliable automation on self-reported workload and secondary task performance

Until automated cars function perfectly, drivers will have to take over control when automation fails or reaches its functional limits. Two simulator experiments (N = 24 and 27) were conducted, each testing four automation levels ranging from manual control (MC) to highly automated driving. In both experiments, participants about once every 3 min experienced an event that required intervention. Participants performed a secondary divided attention task while driving. Automation generally resulted in improved secondary task performance and reduced self-reported physical demand and effort as compared to MC. However, automated speed control was experienced as more frustrating than MC. Participants responded quickly to the events when the stimulus was salient (i.e., stop sign, crossing pedestrian, and braking lead car), but often failed to react to an automation failure when their vehicle was driving slowly. In conclusion, driving with imperfect automation can be frustrating, even though mental and physical demands are reduced.

Human Strategies for Multitasking, Search, and Control Improved via Real-Time Memory Aid for Gaze Location

OBJECTIVE: We aimed to elucidate how our domain-general cuing algorithm improved multitasking performance and changed behavioral strategies in human operators. BACKGROUND: Though many gaze-control systems have been designed, previous real-time gaze-aware assistance systems were not both successful and domain general. It is largely unknown what constitutes optimal search efficiency using eyes, or ideal control using mouse. It is unclear what best coordinating strategies are between these two modalities. Our previously developed closed-loop multitasking aid drastically improved multitasking performance, though behavioral mechanisms through which it acted were unknown. METHODS: We performed in-depth analyses and generated novel eye tracking and mouse movement measures, to explore complex effects of our helpful system on gaze and motor behavior. RESULTS: Our overlay cuing algorithm improved control efficiency and reduced well-known biases in search patterns. This system also reduced micromanaging behavior, with humans rationally relying more on imperfect automation in experimental assistance cue conditions. We showed that mouse and gaze were more independently specialized in helpful cuing condition than in control conditions. Specifically, with our aid, gaze performed more global movement, and mouse performed more local clustered movement. Further, gaze shifted towards search over processing with helpful cuing system. We also illustrated a relationship between mouse and gaze, such that in these studies, the hand was quicker than eye. CONCLUSIONS: Overall, results suggested that our cuing system improved performance and reduced short term working memory load on humans by delegating it to computer in real time. Further, it reduced number of required repeated decisions by an estimate of about one per second. It also enabled gaze to specialize for improved visual search behavior, and mouse to specialize for improved control.

Complacency and Automation Bias in the Use of Imperfect Automation.

We examine the effects of two different kinds of decision-aiding automation errors on human-automation interaction (HAI), occurring at the first failure following repeated exposure to correctly functioning automation. The two errors are incorrect advice, triggering the automation bias, and missing advice, reflecting complacency. Contrasts between analogous automation errors in alerting systems, rather than decision aiding, have revealed that alerting false alarms are more problematic to HAI than alerting misses are. Prior research in decision aiding, although contrasting the two aiding errors (incorrect vs. missing), has confounded error expectancy. Participants performed an environmental process control simulation with and without decision aiding. For those with the aid, automation dependence was created through several trials of perfect aiding performance, and an unexpected automation error was then imposed in which automation was either gone (one group) or wrong (a second group). A control group received no automation support. The correct aid supported faster and more accurate diagnosis and lower workload. The aid failure degraded all three variables, but "automation wrong" had a much greater effect on accuracy, reflecting the automation bias, than did "automation gone," reflecting the impact of complacency. Some complacency was manifested for automation gone, by a longer latency and more modest reduction in accuracy. Automation wrong, creating the automation bias, appears to be a more problematic form of automation error than automation gone, reflecting complacency. Decision-aiding automation should indicate its lower degree of confidence in uncertain environments to avoid the automation bias.

Human factors assessment of conflict resolution aid reliability and time pressure in future air traffic control

Though it has been reported that air traffic controllers' (ATCos') performance improves with the aid of a conflict resolution aid (CRA), the effects of imperfect automation on CRA are so far unknown. The main objective of this study was to examine the effects of imperfect automation on conflict resolution. Twelve students with ATC knowledge were instructed to complete ATC tasks in four CRA conditions including reliable, unreliable and high time pressure, unreliable and low time pressure, and manual conditions. Participants were able to resolve the designated conflicts more accurately and faster in the reliable versus unreliable CRA conditions. When comparing the unreliable CRA and manual conditions, unreliable CRA led to better conflict resolution performance and higher situation awareness. Surprisingly, high time pressure triggered better conflict resolution performance as compared to the low time pressure condition. The findings from the present study highlight the importance of CRA in future ATC operations.Practitioner Summary: Conflict resolution aid (CRA) is a proposed automation decision aid in air traffic control (ATC). It was found in the present study that CRA was able to promote air traffic controllers' performance even when it was not perfectly reliable. These findings highlight the importance of CRA in future ATC operations.

Using Modeling and Simulation to Predict Operator Performance and Automation-Induced Complacency With Robotic Automation: A Case Study and Empirical Validation.

The aim of this study was to develop and validate a computational model of the automation complacency effect, as operators work on a robotic arm task, supported by three different degrees of automation. Some computational models of complacency in human-automation interaction exist, but those are formed and validated within the context of fairly simplified monitoring failures. This research extends model validation to a much more complex task, so that system designers can establish, without need for human-in-the-loop (HITL) experimentation, merits and shortcomings of different automation degrees. We developed a realistic simulation of a space-based robotic arm task that could be carried out with three different levels of trajectory visualization and execution automation support. Using this simulation, we performed HITL testing. Complacency was induced via several trials of correctly performing automation and then was assessed on trials when automation failed. Following a cognitive task analysis of the robotic arm operation, we developed a multicomponent model of the robotic operator and his or her reliance on automation, based in part on visual scanning. The comparison of model predictions with empirical results revealed that the model accurately predicted routine performance and predicted the responses to these failures after complacency developed. However, the scanning models do not account for the entire attention allocation effects of complacency. Complacency modeling can provide a useful tool for predicting the effects of different types of imperfect automation. The results from this research suggest that focus should be given to supporting situation awareness in automation development.