On the evaluation of probability judgments: Calibration, resolution, and monotonicity.

Abstract

Judgments of probability are commonly evaluated by two criteria: calibration, namely, the correspondence between stated confidence and rate of occurrence, and resolution, namely, the ability to distinguish between events that do and do not occur. Two representations of probability judgments are contrasted: the designated form that presupposes a particular coding of outcomes (e.g., rain vs. no rain) and the inclusive form that incorporates all events and their complements. It is shown that the indices of calibration and resolution derived from these representations measure different characteristics of judgment. Calibration is distinguished from two types of overconfidence: specific and generic. An ordinal measure of performance is proposed and compared to the standard measures in forecasts of recession and in both numerical and verbal assessments of general knowledge. Much research on judgment under uncertainty has focused on the comparison of probability judgments with the corresponding relative frequency of occurrence. In a typical study, judges are presented with a series of prediction or knowledge problems and asked to assess the probability of the events in question. Judgments of probability or confidence are used both in research (Lichtenstein, Fischhoff, & Phillips, 1982; Wallsten & Budescu, 1983) and in practice. For example, weather forecasters often report the probability of rain (Murphy & Daan, 1985), and economists are sometimes called upon to estimate the chances of recession (Zarnowitz & Lambros, 1987). The two main criteria used to evaluate such judgments are calibration and resolution. A judge is said to be calibrated if his or her probability judgments match the corresponding relative frequency of occurrence. More specifically, consider all events to which the judge assigns a probability p; the judge is calibrated if the proportion of events in that class that actually occur equals p. Calibration is a desirable property, especially for communication, but it does not ensure informativeness. A judge can be properly calibrated and entirely noninformative if, for example, he or she predicts the sex of each newborn with