Reliability of actual and predicted judgments across time

Abstract

In order for predictions derived from judgment models to effectively represent or replace an individual's decision-making behavior, they must show adequate levels of temporal reliability. This study examined the test—retest reliability of judgment models. Thirty-three undergraduates were asked to rate the acceptability of constructing 50 hypothetical public utility plants and to describe their judgment strategy. This decision task was replicated two weeks later. Test—retest reliabilities of (a) actual judgments, (b) predictions from a statistically derived judgment model, and (c) predictions from a subjectively reported judgment model were computed. All average test—retest reliability estimates were significantly different from zero; the average reliability of judgments predicted by regression models was significantly larger than the average reliability of actual judgments and the average reliability of judgments predicted by subjective weighting strategies. Large individual differences in all three indices of reliability were noted. Further analyses showed that groups who completed the decision task before describing their subjective policies showed significantly higher reliabilities for predictions based upon their subjective policies than did most individuals who described their subjective policies before making judgments. Overall, the results support the implicit assumption that most individuals harbor temporally stable judgment policies across time, although some caution is suggested in the use of subjectively described decision strategies.