Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment

Lyle Brenner,Dale Griffin,Derek J Koehler

doi:10.1016/j.obhdp.2005.02.002

Abstract

We describe four broad characterizations of subjective probability calibration (overconfidence, conservatism, ecologically perfect calibration, and case-based judgment) and show how Random Support Theory (RST) can serve as a tool for representing, evaluating, and discriminating between these perspectives. We present five studies of probability judgment in a simulated stock market setting and analyse the calibration data in terms of RST parameters. The observed pattern of calibration varies with the outcome base rate and cue value diagnosticity, as predicted by case-based judgment. A similar pattern of calibration is found in real-world judgments of experts in various domains. Case-based RST—defined as RST with stable parameter values—provides a parsimonious account of the substantial changes in calibration performance observed across different judgment environments.

Highlights

In modern society, both experts and laypeople are regularly faced with making probabilistic judgments about financial, medical, and personal outcomes
The base rate manipulation influenced the elevation of the calibration curves; for both levels of diagnosticity, the high BR calibration curves are located above the low BR calibration curves
The diagnosticity manipulation, in contrast, influenced the slope of the calibration curves; the high diagnosticity (high D) calibration curves are steeper than the low diagnosticity (low D) calibration curves

Summary

Introduction

Both experts and laypeople are regularly faced with making probabilistic judgments about financial, medical, and personal outcomes. Given a particular outcome base-rate and level of discriminability a, there exist values of b = b* and r = r* within RST that will produce Bayesian and perfectly calibrated judgments These optimal values can be determined by matching the judged probability predicted by the model to the objective Bayesian probability of the outcome derived from the support distributions in (4a) and (4b). The evidence related to the particular case at hand, and is generally insensitive to aggregate statistical properties of the set of judgment items such as outcome base rate and discriminability as determined by the diagnosticity of the available evidence According to this account, the parameters of the RST model that normatively ‘‘ought’’ to reflect the judgeÕs internalization of these aggregate properties for good calibration (b and r) will instead remain roughly constant, despite changes in environmental features that influence the outcome base rate (BR) or evidence diagnosticity (a). Because of their similar designs (and similar results), we report the results of the first two studies together

Method

Results

Discussion

À BR 4 r