Agreeing with automated diagnostic aids: a study of users' concurrence strategies.

Abstract

Automated diagnostic aids that are less than perfectly reliable often produce unwarranted levels of disuse by operators. In the present study, users' tendencies to either agree or disagree with automated diagnostic aids were examined under conditions in which (a) the aids were less than perfectly reliable but aided-diagnosis was still more accurate that unaided diagnosis; and (b) the system was completely opaque, affording users no additional information upon which to base a diagnosis. The results revealed that some users adopted a strategy of always agreeing with the aids, thereby maximizing the number of correct diagnoses made over several trials. Other users, however, adopted a probability-matching strategy in which agreement and disagreement rates matched the rate of correct and incorrect diagnoses of the aids. The probability-matching strategy, therefore, resulted in diagnostic accuracy scores that were lower than was maximally possible. Users who adopted the maximization strategy had higher self-ratings of problem-solving and decision-making skills, were more accurate in estimating aid reliabilities, and were more confident in their diagnosis on trials in which they agreed with the aids. The potential applications of these findings include the design of interface and training solutions that facilitate the adoption of the most effective concurrence strategies by users of automated diagnostic aids.