Automation Bias Research Articles

Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment

As misinformation increasingly proliferates on social media platforms, it has become crucial to explore how to best convey automated news credibility assessments to end-users, and foster trust in fact-checking AIs. In this paper, we investigate how model-agnostic, natural language explanations influence trust and reliance on a fact-checking AI. We construct explanations from four Conceptualisation Validations (CVs) - namely consensual, expert, internal (logical), and empirical - which are foundational units of evidence that humans utilise to validate and accept new information. Our results show that providing explanations significantly enhances trust in AI, even in a fact-checking context where influencing pre-existing beliefs is often challenging, with different CVs causing varying degrees of reliance. We find consensual explanations to be the least influential, with expert, internal, and empirical explanations exerting twice as much influence. However, we also find that users could not discern whether the AI directed them towards the truth, highlighting the dual nature of explanations to both guide and potentially mislead. Further, we uncover the presence of automation bias and aversion during collaborative fact-checking, indicating how users' previously established trust in AI can moderate their reliance on AI judgements. We also observe the manifestation of a 'boomerang'/backfire effect often seen in traditional corrections to misinformation, with individuals who perceive AI as biased or untrustworthy doubling down and reinforcing their existing (in)correct beliefs when challenged by the AI. We conclude by presenting nuanced insights into the dynamics of user behaviour during AI-based fact-checking, offering important lessons for social media platforms.

Addressing the risks of generative AI for the judiciary: The accountability framework(s) under the EU AI Act

The rapid advancements in natural language processing, particularly the development of generative large language models (LLMs), have renewed interest in using artificial intelligence (AI) for judicial decision-making. While these technological breakthroughs present new possibilities for legal automation, they also raise concerns about over-reliance and automation bias. Drawing insights from the COMPAS case, this paper examines the implications of deploying generative LLMs in the judicial domain. It identifies the persistent factors that contributed to an accountability gap when AI systems were previously used for judicial decision-making. To address these risks, the paper analyses the relevant provisions of the EU Artificial Intelligence Act, outlining a comprehensive accountability framework based on the regulation's risk-based approach. The paper concludes that the successful integration of generative LLMs in judicial decision-making requires a holistic approach addressing cognitive biases. By emphasising shared responsibility and the imperative of AI literacy across the AI value chain, the regulatory framework can help mitigate the risks of automation bias and preserve the rule of law.

Clinical adoption of deep learning target auto-segmentation for radiation therapy: challenges, clinical risks, and mitigation strategies

Abstract Radiation therapy is a localized cancer treatment that relies on precise delineation of the target to be treated and healthy tissues to guarantee optimal treatment effect. This step, known as contouring or segmentation, involves identifying both target volumes and organs at risk on imaging modalities like CT, PET, and MRI to guide radiation delivery. Manual segmentation, however, is time-consuming and highly subjective, despite the presence of contouring guidelines. In recent years, automated segmentation methods, particularly deep learning models, have shown promise in addressing this task. However, challenges persist in their clinical use, including the need for robust quality assurance (QA) processes and addressing clinical risks associated with the use of the models. This review examines the challenges and considerations of the clinical adoption of deep learning target auto-segmentation in radiotherapy, focused on the target volume. We discuss potential clinical risks (e.g. over- and under-segmentation, automation bias, and appropriate trust), mitigation strategies (e.g. human oversight, uncertainty quantification, and education of clinical professionals), and we highlight the importance of expanding QA to include geometric, dose-volume, and outcome-based performance monitoring. While deep learning target auto-segmentation offers significant potential benefits, careful attention to clinical risks and rigorous QA measures are essential for its successful integration in clinical practice.

Dimensionality Reduction and Nearest Neighbors for Improving Out-of-Distribution Detection in Medical Image Segmentation

Clinically deployed deep learning-based segmentation models are known to fail on data outside of their training distributions. While clinicians review the segmentations, these models tend to perform well in most instances, which could exacerbate automation bias. Therefore, detecting out-of-distribution images at inference is critical to warn the clinicians that the model likely failed. This work applied the Mahalanobis distance (MD) post hoc to the bottleneck features of four Swin UNETR and nnU-net models that segmented the liver on T1-weighted magnetic resonance imaging and computed tomography. By reducing the dimensions of the bottleneck features with either principal component analysis or uniform manifold approximation and projection, images the models failed on were detected with high performance and minimal computational load. In addition, this work explored a non-parametric alternative to the MD, a k-th nearest neighbors distance (KNN). KNN drastically improved scalability and performance over MD when both were applied to raw and average-pooled bottleneck features. Our code is available at <a href='https://github.com/mckellwoodland/dimen_reduce_mahal'>https://github.com/mckellwoodland/dimen_reduce_mahal</a>

The future of medicine or a threat? Artificial intelligence representation in Chicago Med

The eighth season of the American medical drama series Chicago Med (2015–) represented the application of artificial intelligence (AI) in a hospital environment with multiple storylines. Born in the 1950s, medical dramas are among the most popular forms of serial television. Traditionally, the genre aims for a certain amount of accuracy and has educational goals. Former studies investigated the entertainment education and cultivation effects of these series, concluding that these dramas have the potential to contribute information and shape viewers’ opinions on various health-related topics. Chicago Med is a long-running broadcast production with a worldwide audience and considerable viewership. This paper analyzes the series’ representation of medical AI and discusses how this portrayal potentially shapes the audience’s opinion. The research started by identifying artificial intelligence-related storylines in the 22 episodes of the season. The analysis focused on the reasons and outcomes of AI applications, the character’s attitudes, and the ethical issues, including transparency, selective adherence, automation bias, responsibility gap, hallucination, unequal access, and political dimensions. The storyline analysis concluded that Chicago Med provided thought-provoking positive and negative scenarios about applying different types of AI in the surgical and emergency departments. The complex portrayal included groundbreaking opportunities, challenges, dangers, and ethical considerations. The main characters’ attitudes varied, from strong support or opposition to more nuanced, shifting opinions. The educative and engaging content has a potential for knowledge transfer and encourages critical thinking about medical AI.

Lethal Autonomous Weapon Systems and Automation Bias

Autonomy in weapon systems is already a genuine concern. States try to come up with their own definitions of these systems and pay utmost effort to impose their own understanding of these systems upon other states. For a fairly high number of states barring a total ban on such weapons would be the ideal solution; however, such states that are anxious about the increase in autonomy in war-making capabilities, as adopts a second-best scenario to contain risks created by the deployment of such systems. To this end, placing them under meaningful human control emerges as an important political and legal objective. The author believes that placing autonomous weapons under human supervision, despite its initial promise, will yield negative results. This is due to the fact that humans tend rather to be too willing to follow the solutions generated by autonomous systems. First observed in other industries of civilian nature like aviation or health, automation bias has the potential to negate most if not all of supervision measures expected to ensure proper implementation of international humanitarian law.

Two paths for health AI governance: paternalism or democracy

This article assesses the cyclical failures of NHS data modernisation programmes, and considers that they fail because they proceed from a faulty – excessively paternalistic – governance model. Bias in algorithmic delivery of healthcare, a demonstrated problem with many existing health applications, is another serious risk. To regain trust and move towards better use of data in the NHS, we should democratise the development of these systems, and de-risk operational systems from issues such as automation bias. As a comparison, the essay explores two approaches to trust and bias problems in other contexts: Taiwan’s digital democracy, and American Airlines’ struggles to overcome automation bias in their pilots.

Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancer

Artificial intelligence (AI) readers of mammograms compare favourably to individual radiologists in detecting breast cancer. However, AI readers cannot perform at the level of multi-reader systems used by screening programs in countries such as Australia, Sweden, and the UK. Therefore, implementation demands human-AI collaboration. Here, we use a large, high-quality retrospective mammography dataset from Victoria, Australia to conduct detailed simulations of five potential AI-integrated screening pathways, and examine human-AI interaction effects to explore automation bias. Operating an AI reader as a second reader or as a high confidence filter improves current screening outcomes by 1.9–2.5% in sensitivity and up to 0.6% in specificity, achieving 4.6–10.9% reduction in assessments and 48–80.7% reduction in human reads. Automation bias degrades performance in multi-reader settings but improves it for single-readers. This study provides insight into feasible approaches for AI-integrated screening pathways and prospective studies necessary prior to clinical adoption.

Pitfalls of Artificial Intelligence in Medicine.

Artificial Intelligence (AI) offers great promise for healthcare, but integrating it comes with challenges. Over-reliance on AI systems can lead to automation bias, necessitating human oversight. Ethical considerations, transparency, and collaboration between healthcare providers and AI developers are crucial. Pursuing ethical frameworks, bias mitigation techniques, and transparency measures is key to advancing AI's role in healthcare while upholding patient safety and quality care.

The "Magical Theory" of AI in Medicine: Thematic Narrative Analysis.

The discourse surrounding medical artificial intelligence (AI) often focuses on narratives that either hype the technology's potential or predict dystopian futures. AI narratives have a significant influence on the direction of research, funding, and public opinion and thus shape the future of medicine. The paper aims to offer critical reflections on AI narratives, with a specific focus on medical AI, and to raise awareness as to how people working with medical AI talk about AI and discharge their "narrative responsibility." Qualitative semistructured interviews were conducted with 41 participants from different disciplines who were exposed to medical AI in their profession. The research represents a secondary analysis of data using a thematic narrative approach. The analysis resulted in 2 main themes, each with 2 other subthemes. Stories about the AI-physician interaction depicted either a competitive or collaborative relationship. Some participants argued that AI might replace physicians, as it performs better than physicians. However, others believed that physicians should not be replaced and that AI should rather assist and support physicians. The idea of excessive technological deferral and automation bias was discussed, highlighting the risk of "losing" decisional power. The possibility that AI could relieve physicians from burnout and allow them to spend more time with patients was also considered. Finally, a few participants reported an extremely optimistic account of medical AI, while the majority criticized this type of story. The latter lamented the existence of a "magical theory" of medical AI, identified with techno-solutionist positions. Most of the participants reported a nuanced view of technology, recognizing both its benefits and challenges and avoiding polarized narratives. However, some participants did contribute to the hype surrounding medical AI, comparing it to human capabilities and depicting it as superior. Overall, the majority agreed that medical AI should assist rather than replace clinicians. The study concludes that a balanced narrative (that focuses on the technology's present capabilities and limitations) is necessary to fully realize the potential of medical AI while avoiding unrealistic expectations and hype.

The perfect technological storm: artificial intelligence and moral complacency

Artificially intelligent machines are different in kind from all previous machines and tools. While many are used for relatively benign purposes, the types of artificially intelligent machines that we should care about, the ones that are worth focusing on, are the machines that purport to replace humans entirely and thereby engage in what Brian Cantwell Smith calls “judgment.” As impressive as artificially intelligent machines are, their abilities are still derived from humans and as such lack the sort of normative commitments that humans have. So while artificially intelligent machines possess a great capacity for “reckoning,” to use Smith’s terminology, i.e., a calculative prowess of extraordinary utility and importance, they still lack the kind of considered human judgment that accompanies the ethical commitment and responsible action we humans must ultimately aspire toward. But there is a perfect technological storm brewing. Artificially intelligent machines are analogous to a perfect storm in that such machines involve the convergence of a number of factors that threaten our ability to behave ethically and maintain meaningful human control over the outcomes of processes involving artificial intelligence. I argue that the storm in the context of artificially intelligent machines makes us vulnerable to moral complacency. That is, this perfect technological storm is capable of lulling people into a state in which they abdicate responsibility for decision-making and behaviour precipitated by the use of artificially intelligent machines, a state that I am calling “moral complacency.” I focus on three salient problems that converge to make us especially vulnerable to becoming morally complacent and losing meaningful human control. The first problem is that of transparency/opacity. The second problem is that of overtrust in machines, often referred to as the automation bias. The third problem is that of ascribing responsibility. I examine each of these problems and how together they threaten to render us morally complacent.

The efficiency-accountability tradeoff in AI integration: Effects on human performance and over-reliance

As artificial intelligence proliferates across various sectors, it is crucial to explore the psychological impacts of over-reliance on these systems. This study examines how different formats of chatbot assistance (instruction-only, answer-only, and combined instruction and answer) influence user performance and reliance over time. In two experiments, participants completed reasoning tests with the aid of a chatbot, "Cogbot," offering varying levels of explanatory detail and direct answers. In Experiment 1, participants receiving direct answers showed higher reliance on the chatbot compared to those receiving instructions, aligning with the practical hypothesis that prioritizes efficiency over explainability. Experiment 2 introduced transfer problems with incorrect AI guidance, revealing that initial reliance on direct answers impaired performance on subsequent tasks when the AI erred, supporting concerns about automation complacency. Findings indicate that while efficiency-focused AI solutions enhance immediate performance, they risk over-assimilation and reduced vigilance, leading to significant performance drops when AI accuracy falters. Conversely, explanatory guidance did not significantly improve outcomes absent of direct answers. These results highlight the complex dynamics between AI efficiency and accountability, suggesting that responsible AI adoption requires balancing streamlined functionality with safeguards against over-reliance.

Single-center outcomes of artificial intelligence in management of pulmonary embolism and pulmonary embolism response team activation.

Multidisciplinary pulmonary embolism response teams (PERTs) have shown that timely triage expedites treatment. The use of artificial intelligence (AI) may help improve pulmonary embolism (PE) management with early CT pulmonary angiogram (CTPA) screening and accelerate PERT coordination. This study aimed to test the clinical validity of an FDA-approved PE AI algorithm. CTPA scan data of 200 patients referred due to automated AI detection of suspected PE were retrospectively reviewed. In our institution, all patients suspected of PE received a CTPA. The AI app was then used to analyze CTPA for the presence of PE and calculate the right-ventricle/left-ventricle (RV/LV) ratio. We compared the AI's output with the radiologists' report. Inclusion criteria included segmental PE with and without RV dysfunction and high-risk PE. The primary endpoint was false positive rate. Secondary end points included clinical outcomes according to the therapy selected, including catheter-directed interventions, systemic thrombolytics, and anticoagulation. Fifty-seven of 200 exams (28.5%) were correctly identified as positive for PE by the algorithm. A total of 143 exams (71.5%) were incorrectly reported as positive. In 8% of cases, PERT was consulted. Four patients (7%) received systemic thrombolytics without any complications. There were six patients (10.5%) who developed high-risk PE and underwent thrombectomy, one of whom died. Among 46 patients with acute PE without right heart strain, 44 (95%) survived. The false positive rate of our AI algorithm was 71.5%, higher than what was reported in the AI's prior clinical validity study (91% sensitivity, 100% specificity). A high rate of discordant AI auto-detection of suspected PE raises concerns about its diagnostic accuracy. This can lead to increased workloads for PERT consultants, alarm/notification fatigue, and automation bias. The AI direct notification process to the PERT team did not improve PERT triage efficacy.

Automation Bias and Complacency in Security Operation Centers

The volume and complexity of alerts that security operation center (SOC) analysts must manage necessitate automation. Increased automation in SOCs amplifies the risk of automation bias and complacency whereby security analysts become over-reliant on automation, failing to seek confirmatory or contradictory information. To identify automation characteristics that assist in the mitigation of automation bias and complacency, we investigated the current and proposed application areas of automation in SOCs and discussed its implications for security analysts. A scoping review of 599 articles from four databases was conducted. The final 48 articles were reviewed by two researchers for quality control and were imported into NVivo14. Thematic analysis was performed, and the use of automation throughout the incident response lifecycle was recognized, predominantly in the detection and response phases. Artificial intelligence and machine learning solutions are increasingly prominent in SOCs, yet support for the human-in-the-loop component is evident. The research culminates by contributing the SOC Automation Implementation Guidelines (SAIG), comprising functional and non-functional requirements for SOC automation tools that, if implemented, permit a mutually beneficial relationship between security analysts and intelligent machines. This is of practical value to human automation researchers and SOCs striving to optimize processes. Theoretically, a continued understanding of automation bias and its components is achieved.

Leveraging Large Language Models for Direct Interaction with SysML v2

AbstractThis paper examines the potential integration of Large Language Models (LLMs) with the Systems Modeling Language version 2 (SysML v2), proposing a novel methodology for systems engineering by capitalizing on the enhanced readability and human‐friendly syntax of SysML v2. Given the emergent sophistication of LLMs and the coincidental development of SysML v2—‐an endeavor that presents a pivot toward naturally articulated model interaction—‐we explore the possibilities and implications of such an intersection. Our investigation posits that LLMs can serve not only as an interpretive layer, allowing for the syntactically simplified manipulation of system models, but also as a catalyst for a knowledge‐driven design approach.We highlight the efficiencies gained by deploying LLMs for SysML v2 interactions, which reduce the dependency on technical expertise traditionally needed for API navigation and model management. Through case studies and analysis, we demonstrate that the conversational engagement with system models facilitated by LLMs can lead to a democratized and accelerated design process. However, this advent is tempered by a critical awareness of potential pitfalls, such as automation bias and overreliance on automated systems—‐underscoring the need for continued human oversight and the examination of ethical considerations.Emphasizing the chance of SysML v2 being inherently English‐based and the parallel maturation of LLMs, this paper suggests that the collaborative utilization of these concurrent advancements may offer an opportune fusion, potentially revolutionizing the way systems are modeled and managed. Future work involves the empirical validation of these approaches and a deeper investigation into interoperability with existing and future systems engineering ecosystems. The ultimate goal is to ensure that this fusion not only complements human expertise but also propels systems engineering into a new era of innovation and holistic design.

Humans and Automation: Augmenting Security Operation Centers

The continuous integration of automated tools into security operation centers (SOCs) increases the volume of alerts for security analysts. This amplifies the risk of automation bias and complacency to the point that security analysts have reported missing, ignoring, and not acting upon critical alerts. Enhancing the SOC environment has predominantly been researched from a technical standpoint, failing to consider the socio-technical elements adequately. However, our research fills this gap and provides practical insights for optimizing processes in SOCs. The synergy between security analysts and automation can potentially augment threat detection and response capabilities, ensuring a more robust defense if effective human-automation collaboration is established. A scoping review of 599 articles from four databases led to a final selection of 49 articles. Thematic analysis resulted in 609 coding references generated across four main themes: SOC automation challenges, automation application areas, implications on analysts, and human factor sentiment. Our findings emphasize the extent to which automation can be implemented across the incident response lifecycle. The SOC Automation Matrix represents our primary contribution to achieving a mutually beneficial relationship between analyst and machine. This matrix describes the properties of four distinct human-automation combinations. This is of practical value to SOCs striving to optimize their processes, as our matrix mentions socio-technical system characteristics for automated tools.

Artificial intelligence in cardiovascular radiology : Image acquisition, image reconstruction and workflow optimization

Artificial intelligence (AI) has the potential to fundamentally change radiology workflow. This review article provides an overview of AI applications in cardiovascular radiology with afocus on image acquisition, image reconstruction, and workflow optimization. First, established applications of AI are presented for cardiovascular computed tomography (CT) and magnetic resonance imaging (MRI). Building on this, we describe the range of applications that are currently being developed and evaluated. The practical benefits, opportunities, and potential risks of artificial intelligence in cardiovascular imaging are critically discussed. The presentation is based on the relevant specialist literature and our own clinical and scientific experience. AI-based techniques for image reconstruction are already commercially available and enable dose reduction in cardiovascular CT and accelerated image acquisition in cardiac MRI. Postprocessing of cardiovascular CT and MRI examinations can already be considerably simplified using established AI-based segmentation algorithms. In contrast, the practical benefits of many AI applications aimed at the diagnosis of cardiovascular diseases are less evident. Potential risks such as automation bias and considerations regarding cost efficiency should also be taken into account. In amarket characterized by great expectations and rapid technical development, it is important to realistically assess the practical benefits of AI applications for your own hospital or practice.

Automation bias in public administration – an interdisciplinary perspective from law and psychology

The objective of this paper is to break down the widely presumed dichotomy, especially in law, between fully automated decisions and human decisions from a psychological and normative perspective. This is particularly relevant as human oversight is seen as an effective means of quality control, including in the current AI Act. The phenomenon of automation bias argues against this assumption. We have investigated this phenomenon of automation bias, as a behavioral effect of and its implications in normative institutional decision-making situations. The phenomenon of automation bias, whereby individuals overly rely on machine-generated decisions or proposals, has far-reaching implications. Excessive reliance may result in a failure to meaningfully engage with the decision at hand, resulting in an inability to detect automation failures, and an overall deterioration in decision quality, potentially up to a net-negative impact of the decision support system. As legal systems emphasize the role of human decisions in ensuring fairness and quality, this paper critically examines the inadequacies of current EU and national legal frameworks in addressing the risks of automation bias. Contributing a novel perspective, this article integrates psychological, technical, and normative elements to analyze automation bias and its legal implications. Anchoring human decisions within legal principles, it navigates the intersections between AI and human-machine interactions from a normative point of view. An exploration of the AI Act sheds light on potential avenues for improvement. In conclusion, our paper proposes four steps aimed at effectively countering the potential perils posed by automation bias. By linking psychological insights, legal analysis, and technical implications, this paper advocates a holistic approach to evolving legal frameworks in an increasingly automated world.

Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance.

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

Understanding and Mitigating Bias in Imaging Artificial Intelligence.

Artificial intelligence (AI) algorithms are prone to bias at multiple stages of model development, with potential for exacerbating health disparities. However, bias in imaging AI is a complex topic that encompasses multiple coexisting definitions. Bias may refer to unequal preference to a person or group owing to preexisting attitudes or beliefs, either intentional or unintentional. However, cognitive bias refers to systematic deviation from objective judgment due to reliance on heuristics, and statistical bias refers to differences between true and expected values, commonly manifesting as systematic error in model prediction (ie, a model with output unrepresentative of real-world conditions). Clinical decisions informed by biased models may lead to patient harm due to action on inaccurate AI results or exacerbate health inequities due to differing performance among patient populations. However, while inequitable bias can harm patients in this context, a mindful approach leveraging equitable bias can address underrepresentation of minority groups or rare diseases. Radiologists should also be aware of bias after AI deployment such as automation bias, or a tendency to agree with automated decisions despite contrary evidence. Understanding common sources of imaging AI bias and the consequences of using biased models can guide preventive measures to mitigate its impact. Accordingly, the authors focus on sources of bias at stages along the imaging machine learning life cycle, attempting to simplify potentially intimidating technical terminology for general radiologists using AI tools in practice or collaborating with data scientists and engineers for AI tool development. The authors review definitions of bias in AI, describe common sources of bias, and present recommendations to guide quality control measures to mitigate the impact of bias in imaging AI. Understanding the terms featured in this article will enable a proactive approach to identifying and mitigating bias in imaging AI. Published under a CC BY 4.0 license. Test Your Knowledge questions for this article are available in the supplemental material. See the invited commentary by Rouzrokh and Erickson in this issue.