Probability Judgment Accuracy Research Articles

Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments

Human forecasts and other probabilistic judgments can be improved by elicitation and aggregation methods. Recent work on elicitation shows that deriving probability estimates from relative judgments (the ratio method) is advantageous, whereas other recent work on aggregation shows that it is beneficial to transform probabilities into coherent sets (coherentization) and to weight judges' assessments by their degree of coherence. We report an experiment that links these areas by examining the effect of coherentization and multiple forms of coherence weighting using direct and ratio elicitation methods on accuracy of probability judgments (both forecasts and events with known distributions). We found that coherentization invariably yields improvements to accuracy. Moreover, judges' levels of probabilistic coherence are related to their judgment accuracy. Therefore, coherence weighting can improve judgment accuracy, but the strength of the effect varies among elicitation and weighting methods. As well, the benefit of coherence weighting is stronger on “calibration” items that served as a basis for establishing the weights than for unrelated “test” items. Finally, echoing earlier research, we found overconfidence in judgment, and the degree of overconfidence was comparable between the two elicitation methods.

Culture and Probability Judgment Accuracy

A well-established phenomenon in the judgment and decision-making tradition is the overconfidence one places in the amount of knowledge that one possesses. Overconfidence or probability judgment accuracy varies not only individually but also across cultures. However, research efforts to explain cross-cultural variations in the overconfidence phenomenon have seldom been made. In Study 1, the authors compared the probability judgment accuracy of U.S. Americans (N = 108) and Mexican participants (N = 100). In Study 2, they experimentally primed culture by randomly assigning English/Spanish bilingual Mexican Americans (N = 195) to response language. Results of both studies replicated the cross-cultural variation of probability judgment accuracy previously observed in other cultural groups. U.S. Americans displayed less overconfidence when compared to Mexicans. These results were then replicated in bilingual participants, when culture was experimentally manipulated with language priming. Holistic reasoning did not account for the cross-cultural variation of overconfidence. Suggestions for future studies are discussed.

Los que se arriesgan, ¿saben cuánto se arriesgan?

In a real-world setting, two groups of subjects showing a different level of risk were compared in several measures of probability judgment accuracy. After answering a 20 true-false items exam, undergraduate students were asked to estimate the subjective probability that each item was true. Subjects were split into two groups of risk. Both achieved the very same grades, but the risky group gave a higher number of answers. The obtained data suggest that risk has an effect on calibration and noisiness of the probability judgments. The more risky subjects were more poorly calibrated, i.e., the subjective probabilities they estimated deviated further from the actual proportion of true items: Therefore, it seems that those who risk more know less about how much they risk. Discussion focuses on implications of these findings for gambling behavior. Calibration and covariance graphs are reported.

Training to Improve Calibration and Discrimination: The Effects of Performance and Environmental Feedback

This study investigated whether calibration and discrimination are distinct or related aspects of probability judgment accuracy by examining the effects of two different training techniques. Participants received either performance feedback or environmental feedback, and we measured their improvement in calibration and discrimination as a function of feedback type. Whereas performance feedback reduced participants' overconfidence and environmental feedback improved discrimination, neither type of feedback led to an improvement on the other component. In fact, environmental feedback led to an increase in overconfidence. We take these results as evidence that calibration and discrimination are dissociable abilities that require separate training techniques for improvement.

Cross-Cultural Variations in Probability Judgment Accuracy: Beyond General Knowledge Overconfidence?

Previous studies have revealed surprising and persistent cross-cultural variations in overconfidence, whereby respondents in some Asian cultures (e.g., Chinese) exhibit markedly higher degrees of overconfidence than respondents in other cultures (e.g., in the United States and Japan). Most of those demonstrations have entailed general knowledge tasks (e.g., answering questions such as whether Europe is larger than Australia). The present studies sought to determine whether such cross-cultural variations extend to judgments about the kinds of events that bear upon more common practical decisions and to aspects of accuracy other than overconfidence. Subjects in Taiwan, Japan, and the United States made probabilistic differential diagnoses of fictional diseases in a stochastic artificial ecology. Results revealed that previously observed cross-cultural variations do indeed generalize. The data were also informative about several potential accounts for such variations, e.g., arguing against a proposal that they rest on different emphases on discrimination rather than calibration, but consistent with the influences of culture-specific cognitive customs, including responsiveness to explicitly displayed information, regardless of its presumed validity.

Probability judgment accuracy for general knowledge. Cross‐national differences and assessment methods

AbstractIn this study we compare the probability judgment accuracy of subjects from the United States and Turkey. Three different response modes were employed — numerical probabilities, pie diagrams, and odds. The questions employed in the study were restricted to two‐alternative, general‐knowledge items. The observed pattern of differences in the components of probability judgment accuracy paralleled those of studies that have compared Western and Asian subjects. In particular, Turkish subjects exhibited better discrimination but worse calibration than their US counterparts. This result persisted across all three response modes. These findings lend support to previous assertions that observed cross‐national differences arise from socioeconomic rather than Asian versus Western cultural differences. However, the consistency of the observed differences across response modes refutes a previous assertion that observed cultural differences are merely the result of response bias.

The Accuracy of Future Forecasts and Past Judgments

The aim of this paper is to examine the accuracy of probability judgments of future and past events, an issue which has been debated in a number of studies. Two experiments are reported here. The results of both experiments indicate that for between-subject comparisons, i.e., when the individuals making future forecasts differ from those making past judgments, the Brier score, and in particular calibration and noise, is better for future forecasts than it is for past judgments. Further, the results of the first experiment indicate that calibration and noise of past judgments continue to worsen the more remote the events become. In contrast, for within-subject comparisons, i.e., when the individuals making future forecasts are the same as those making past judgments, the Brier score, and in particular discrimination, is better for past judgments than it is for future forecasts. The implications of these results are discussed.

Probability judgment accuracy: China, Japan, and the United States

Subjects in China, Japan, and the United States reported probability judgments. In Study 1, Chinese and American subjects indicated degrees of certainty about their answers to general-knowledge questions with discrete alternatives, e.g., whether potatoes grow better in warm or in cool climates. In Study 2, Japanese subjects made similar discrete-alternative assessments. In Study 3, subjects in China and the United States reported probability distribution judgments for various quantities, e.g., the maximum temperature on a specified day. Judgment accuracy was evaluated overall and with respect to several underlying accuracy dimensions. The overall quality of discrete-alternative judgments was indistinguishable among the subjects from the three countries. The accuracy component patterns of the Japanese and American subjects were essentially the same. However, the Chinese subjects achieved the common overall accuracy level very differently. On some accuracy dimensions, e.g., calibration, the American and Japanese subjects' judgments were superior. On others, e.g., discrimination, the assessments of the Chinese subjects excelled. Results for quantity judgments were similar to those for discrete-alternative judgments, although there were notable differences. Potential explanations and implications are discussed.

Analyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition

The probability score (PS) can be used to measure the overall accuracy of probability judgments for a single event, e.g., “Rain falls,” or “This patient has cancer.” It has been shown previously how a “covariance decomposition” of the mean of PS over many occasions indexes several distinct aspects of judgment performance (J. F. Yates, Organizational Behavior and Human Performance, 30, 132–156 (1982)). There are many situations in which probability judgments are reported for sample space partitions containing more than one event and its complement, e.g., medical situations in which a patient might suffer from Disease X, Disease Y, or Disease Z, or testing situations in which the correct answer to an item might be any one of alternatives (a) through (e). The probability score for multiple events (PSM) serves as a measure of the overall accuracy of probability judgments for the events in partitions of any size. The present article describes and interprets an extension of the covariance decomposition to the mean of PSM. The decomposition is illustrated with data from two contexts, medicine and education.

Components of probability judgment accuracy: Individual consistency and effects of subject matter and assessment method

An experiment is reported in which subjects assigned probabilities to the outcomes of basketball games and to the truth of general-knowledge items. Three different methods were used for eliciting subjects' probability judgments. Subjects were more successful in selecting answers to the general-knowledge questions than they were in picking basketball game winners. The overall accuracy of their probability judgments for general-knowledge items was superior, too. On the other hand, subjects' judgments about general-knowledge questions were more overconfident, more poorly calibrated, and included greater scatter. One method of probability assessment gave subjects an irrelevant cue. This was found to increase confidence and overconfidence and to hurt calibration. Correlations between measures of performance on general-knowledge questions and basketball predictions showed substantial individual consistency in confidence, but only weak consistency in other components of judgment quality. Theoretical and practical implications are discussed.