Reliance Behavior Research Articles

A closer look at how experience, task domain, and self-confidence influence reliance towards algorithms

Prior research has demonstrated experience with a forecasting algorithm decreases reliance behaviors (i.e., the action of relying on the algorithm). However, the influence of model experience on reliance intentions (i.e., an intention or willingness to rely on the algorithm) has not been explored. Additionally, other factors such as self-confidence and domain knowledge are posited to influence algorithm reliance. The objective of this research was to examine how experience with a statistical model, task domain (used car sales, college grade point average (GPA), GitHub pull requests), and self-confidence influence reliance intentions, reliance behaviors, and perceived accuracy of one's own estimates and the model's estimates. Participants (N = 347) were recruited online and completed a forecasting task. Results indicate that there was a statistically significant effect of self-confidence and task domain on reliance intentions, reliance behaviors, and perceived accuracy. However, unlike previous findings, model experience did not significantly influence reliance behavior, nor did it lead to significant changes in reliance intentions or perceived accuracy of oneself or the model. Our data suggest that factors such as task domain and self-confidence influence algorithm use more so than model experience. Individual differences and situational factors should be considered important aspects that influence forecasters' decisions to rely on predictions from a model or to instead use their own estimates, which can lead to sub-optimal performance.

The Impact of a Priori Information on Drivers’ Mental Models, Attitudes, and Behavior in Interaction with Partial and Conditional Driving Automation

The introduction of automated driving features to the market will fundamentally change driver vehicle interaction. Because most drivers can be considered novices in regard to automated driving systems, it is likely that they do not have complete knowledge on these systems when they encounter them for the first time. For this reason, the present study assessed how information before the first drive affects mental models, behavior, and attitudes towards a combined partially and conditionally automated system. In a driving simulator study with 45 participants, we manipulated the completeness of the information provided before the first interaction with the system. One group, called the complete group, received all information on requirements for activation of the system and system limitations that were encountered in the subsequent thirty-minute simulated drive. A second group, called the incomplete group, only received information on one out of three requirements for activation and system limitations, respectively. Results indicate that, only after the drive with the system, mental models of the incomplete group were as good as mental models of the complete group before the drive. Furthermore, reliance behavior was supported by complete a priori information on the automated function. There were no effects for time to take over or take-over quality. These results emphasize the importance of driver education before the first automated drive to enhance mental models and trust among drivers of automated vehicles.

Sharing Vehicle Situation Awareness Reduces Driver-Initiated Overrides in Urban Environments

Driver assistance technologies have rapidly advanced. However, using partially automated driving systems in urban environments is still challenging. The potential disuse of driving automation is one of the challenges that prevents users from taking full advantage of the system. To address this issue, we investigated whether sharing the vehicle’s situation awareness (SA) information could increase the proper use of driving automation in urban contexts. An Augmented Reality Head-Up Display (AR HUD) was developed to present the vehicle’s SA information, and its effect was tested using a driving simulator. We used a two-part mixed model to analyze driver reliance behavior. The results showed that sharing the vehicle’s SA information decreased override responses when the automation could handle the situation but had no significant effect on the override time. These findings suggest that providing drivers with the vehicle SA information can increase the appropriate use of driving automation in complex urban driving situations.

Modeling Human Trust and Reliance in AI-Assisted Decision Making: A Markovian Approach

The increased integration of artificial intelligence (AI) technologies in human workflows has resulted in a new paradigm of AI-assisted decision making, in which an AI model provides decision recommendations while humans make the final decisions. To best support humans in decision making, it is critical to obtain a quantitative understanding of how humans interact with and rely on AI. Previous studies often model humans' reliance on AI as an analytical process, i.e., reliance decisions are made based on cost-benefit analysis. However, theoretical models in psychology suggest that the reliance decisions can often be driven by emotions like humans' trust in AI models. In this paper, we propose a hidden Markov model to capture the affective process underlying the human-AI interaction in AI-assisted decision making, by characterizing how decision makers adjust their trust in AI over time and make reliance decisions based on their trust. Evaluations on real human behavior data collected from human-subject experiments show that the proposed model outperforms various baselines in accurately predicting humans' reliance behavior in AI-assisted decision making. Based on the proposed model, we further provide insights into how humans' trust and reliance dynamics in AI-assisted decision making is influenced by contextual factors like decision stakes and their interaction experiences.

A Computational Model of Coupled Human Trust and Self-confidence Dynamics

Autonomous systems that can assist humans with increasingly complex tasks are becoming ubiquitous. Moreover, it has been established that a human’s decision to rely on such systems is a function of both their trust in the system and their own self-confidence as it relates to executing the task of interest. Given that both under- and over-reliance on automation can pose significant risks to humans, there is motivation for developing autonomous systems that could appropriately calibrate a human’s trust or self-confidence to achieve proper reliance behavior. In this article, a computational model of coupled human trust and self-confidence dynamics is proposed. The dynamics are modeled as a partially observable Markov decision process without a reward function (POMDP/R) that leverages behavioral and self-report data as observations for estimation of these cognitive states. The model is trained and validated using data collected from 340 participants. Analysis of the transition probabilities shows that the proposed model captures the probabilistic relationship between trust, self-confidence, and reliance for all discrete combinations of high and low trust and self-confidence. The use of the proposed model to design an optimal policy to facilitate trust and self-confidence calibration is a goal of future work.

The Effect of Informing Participants of the Response Bias of an Automated Target Recognition System on Trust and Reliance Behavior.

To determine how changing and informing a user of the false alarm (FA) rate of an automated target recognition (ATR) system affects the user's trust in and reliance on the system and their performance during an underwater mine detection task. ATR systems are designed to operate using a high sensitivity and a liberal decision criterion to reduce the risk of the ATR system missing a target. A high number of FAs in general may lead to a decrease in operator trust and reliance. Participants viewed sonar images and were asked to identify mines in the images. They performed the task without ATR and with ATR at a lower and higher FA rate. The participants were split into two groups-one informed and one uninformed of the changed FA rate. Trust and/or confidence in detecting mines was measured after each block. When not informed of the FA rate, the FA rate had a significant effect on the participants' response bias. Participants had greater trust in the system and a more consistent response bias when informed of the FA rate. Sensitivity and confidence were not influenced by disclosure of the FA rate but were significantly worse for the high FA rate condition compared with performance without the ATR. Informing a user of the FA rate of automation may positively influence the level of trust in and reliance on the aid.

Adaptive trust calibration for human-AI collaboration.

Safety and efficiency of human-AI collaboration often depend on how humans could appropriately calibrate their trust towards the AI agents. Over-trusting the autonomous system sometimes causes serious safety issues. Although many studies focused on the importance of system transparency in keeping proper trust calibration, the research in detecting and mitigating improper trust calibration remains very limited. To fill these research gaps, we propose a method of adaptive trust calibration that consists of a framework for detecting the inappropriate calibration status by monitoring the user’s reliance behavior and cognitive cues called “trust calibration cues” to prompt the user to reinitiate trust calibration. We evaluated our framework and four types of trust calibration cues in an online experiment using a drone simulator. A total of 116 participants performed pothole inspection tasks by using the drone’s automatic inspection, the reliability of which could fluctuate depending upon the weather conditions. The participants needed to decide whether to rely on automatic inspection or to do the inspection manually. The results showed that adaptively presenting simple cues could significantly promote trust calibration during over-trust.

Introduction matters: Manipulating trust in automation and reliance in automated driving

Trust in automation is a key determinant for the adoption of automated systems and their appropriate use. Therefore, it constitutes an essential research area for the introduction of automated vehicles to road traffic. In this study, we investigated the influence of trust promoting (Trust promoted group) and trust lowering (Trust lowered group) introductory information on reported trust, reliance behavior and take-over performance. Forty participants encountered three situations in a 17-min highway drive in a conditionally automated vehicle (SAE Level 3). Situation 1 and Situation 3 were non-critical situations where a take-over was optional. Situation 2 represented a critical situation where a take-over was necessary to avoid a collision. A non-driving-related task (NDRT) was presented between the situations to record the allocation of visual attention. Participants reporting a higher trust level spent less time looking at the road or instrument cluster and more time looking at the NDRT. The manipulation of introductory information resulted in medium differences in reported trust and influenced participants' reliance behavior. Participants of the Trust promoted group looked less at the road or instrument cluster and more at the NDRT. The odds of participants of the Trust promoted group to overrule the automated driving system in the non-critical situations were 3.65 times (Situation 1) to 5 times (Situation 3) higher. In Situation 2, the Trust promoted group's mean take-over time was extended by 1154 ms and the mean minimum time-to-collision was 933 ms shorter. Six participants from the Trust promoted group compared to no participant of the Trust lowered group collided with the obstacle. The results demonstrate that the individual trust level influences how much drivers monitor the environment while performing an NDRT. Introductory information influences this trust level, reliance on an automated driving system, and if a critical take-over situation can be successfully solved.

A framework for explaining reliance on decision aids

This study presents a framework for understanding task and psychological factors affecting reliance on advice from decision aids. The framework describes how informational asymmetries in combination with rational, motivational and heuristic factors explain human reliance behavior. To test hypotheses derived from the framework, 79 participants performed an uncertain pattern learning and prediction task. They received advice from a decision aid either before or after they expressed their own prediction, and received feedback about performance. When their prediction conflicted with that of the decision aid, participants had to choose to rely on their own prediction or on that of the decision aid. We measured reliance behavior, perceived and actual reliability of self and decision aid, responsibility felt for task outcomes, understandability of one's own reasoning and of the decision aid, and attribution of errors. We found evidence that (1) reliance decisions are based on relative trust, but only when advice is presented after people have formed their own prediction; (2) when people rely as much on themselves as on the decision aid, they still perceive the decision aid to be more reliable than themselves; (3) the less people perceive the decision aid's reasoning to be cognitively available and understandable, the less people rely on the decision aid; (4) the more people feel responsible for the task outcome, the more they rely on the decision aid; (5) when feedback about performance is provided, people underestimate both one's own reliability and that of the decision aid; (6) underestimation of the reliability of the decision aid is more prevalent and more persistent than underestimation of one's own reliability; and (7) unreliability of the decision aid is less attributed to temporary and uncontrollable (but not external) causes than one's own unreliability. These seven findings are potentially applicable for the improved design of decision aids and training procedures.

The Effects of Design Features on Users' Trust in and Reliance on a Combat Identification System

In a previous study, we found that users‟ trust in and reliance on an individual combat identification system is influenced by the system‟s reliability as well as users‟ awareness of the reliability. In this exploratory study we test the effects of design features of the same system on users‟ target identification performance as well as their trust in and reliance on the system. In a simulated task environment, we varied the automation activation mode (i.e., automatic vs. manual) and the presentation of the “unknown” feedback (i.e., explicit vs. implicit). Participants responded fastest when the “unknown” feedback was provided automatically with explicit indication. In addition, participants trusted the explicit “unknown” feedback more than the implicit feedback. However, neither reliance behavior nor identification accuracy changed significantly across conditions. This study has implications for the design of combat identification systems to achieve appropriate trust. In addition, the results suggest that when studying trust in automation using a simulation, it is important to simulate the design features.

Decision aid reliance: Modeling the effects of decision aid reliability and pressures to perform on reliance behavior

We investigate the effects of decision aid reliability and pressure to perform on decision aid reliance. A total of 403 students took part in a four (pressures to perform: one through four) by five (decision aid reliability: 50%, 60%, 70%, 80% and 90%) between-participants experiment. We test two competing models of decision aid reliance behavior: pressure-induced rationality and pressure arousal theory. Additionally, we introduce a general model of reliance on a decision aid. We find that pressure arousal theory predominantly explains decision aid reliance behavior at all but the highest level of decision aid reliability tested in this study (90%). Our results indicate that there are reliance peaks across the reliability levels as more pressure to perform is applied, and that continually increasing pressures can eventually lead to decreased, rather than increased, reliance.

Decision Aid Reliance: A Field Study Involving Professional Buy-Side Financial Analysts

Decision aids (DAs) are designed to assist and improve decision-making processes, and they have been incorporated into many professions, such as auditing, law and medicine. Yet, research findings regarding how DAs impact judgments and decisions has been somewhat mixed, often indicating a reluctance of experienced decision-makers to rely on the recommendations of DAs. One of the major criticisms of this line of research is that prior studies examining DA reliance have been conducted in artificial experimental settings using researcher-developed aids; thus, researchers and practitioners know very little about DA reliance in the field. The current study complements and extends prior DA research by examining the DA reliance behavior of professional buy-side financial analysts in a real world environment.A large mutual fund company provided data on buy-side analysts’ earnings forecasts for four consecutive quarters. As part of the decision process, all of the analysts had a DA available to assist in making earnings forecasts. The results indicate that higher task ability is associated with greater DA reliance, increased performance-contingent incentives are related to decreased DA reliance, and more task complexity is associated with more DA reliance. Also, greater confidence in the DA heightened reliance, and DA confidence interacted with performance-contingent incentives. Finally, increased DA reliance was associated with more accurate earnings forecasts. These results provide valuable theoretical and practical insight into the use of an operational DA by professional decision-makers in a natural work environment.

Age-Related Differences in Reliance Behavior Attributable to Costs Within a Human-Decision Aid System

An empirical investigation was done to determine if there are age-related differences attributable to costs in reliance on a decision aid. Costs of reliance on a decision aid may affect reliance on the aid. Older and younger adults may not perceive and respond to a dynamic cost structure equally or objectively. Sixteen older adults (65-74 years) and 16 younger adults (18-28 years) performed a counting task with an imperfect decision aid. Two types of costs were manipulated: (a) cost of error (CoE) and (b) cost of verification (CoV). The percentage of trials in which participants agreed with the decision aid and did not perform the task manually was recorded as reliance. Participants decreased their reliance as the CoE increased and increased their reliance with a lower CoV; however, they tended to underrely on the decision aid. Younger adults tended to change their reliance behavior more than older adults did with the changing cost structure. Older and younger adults appear to interpret costs differently, with older adults being less responsive to changes in costs. Older adults may have been less able to monitor the changing costs and hence not adapt to them as well as younger adults. Designers of decision aids should consider explicitly stating costs associated with reliance on the aid, as individuals may differ in how they interpret and respond to changing costs.

Selecting Methods for the Analysis of Reliance on Automation

Human reliance on imperfect automation has been the subject of many laboratory experiments. Across these studies, a diverse set of indices of reliance have been used, even in studies with similar experiment settings. This inconsistency of reliance measures makes the meta-analysis of experimental findings difficult. Moreover, few researchers have rigorously defined the optimal level of automation reliance in their settings, making it difficult to judge the appropriateness of that reliance. This paper attempts to guide researchers in selecting and interpreting measures of automation reliance behavior. We review the reliance analysis methods in existing research and propose four criteria for selecting among them. It is recommended that, where possible, future studies define optimal reliance to make unequivocal judgment about the appropriateness of reliance. In addition, more reliable and insightful conclusions can be obtained through the use of multiple measures.