The impact of model performance history information on users' confidence in decision models: An experimental examination

Abstract

Effective decision support systems must supply decision makers with information that allows them to make correct judgments. Unfortunately, human intuitive judgments are subject to a number of biases. Among the judgments that a user of a decision support system must make is the selection of an appropriate model. When a decision maker is presented with a history of a model's usage and frequency of success during that usage, the decision maker must judge how confident he/she is in the output that comes from that model. We show, in a laboratory setting using 75 student subjects and 48 managers, that decision makers can be manipulated into irrational confidence levels. In a corporate setting, over- and under-confidence will result in either overreliance on unreliable models or failure to take advantage of a useful tool.