Scoring rules in probability assessment and evaluation

Abstract

The purpose of this paper is to briefly discuss some important current questions and problems related to the use of scoring rules (SRs) both in connection with the actual assessment of probabilities and with the evaluation of probability forecasts and probability assessors. With regard to the assessment process, we consider both the case in which the assessor's utility function is linear and the case in which his utility function is nonlinear. Under linear utility, important problems of concern are the sensitivity of SRs to deviations from optimality (with a strictly proper SR, optimality consists of the assessor making his statements correspond to his judgments) and the effect of psychological considerations arising from the use of different SRs. Under nonlinear utility, SRs should be modified to allow for the nonlinearity in such a manner that for a specific utility function, the modified SRs are strictly proper. This introduces the difficult question of the assessment of the assessor's utility function. With regard to the evaluation process (as opposed to the assessment process), we consider the process from an inferential viewpoint and from a decision-theoretic viewpoint. From an inferential viewpoint, attributes such as validity may be of interest, and in certain circumstances these attributes may be related to SRs. The attributes of interest, of course, depend on the framework within which the evaluation process is undertaken. From a decision-theoretic viewpoint, SRs may be related to a decision maker's utilities or expected utilities (under uncertainty about the utilities) if the decision maker uses the assessed probabilities in an actual decision situation. In summary, there are many important questions and problems related to SRs, and the need for future research on these problems seems clear. Such research should lead to a greatly improved understanding of the processes of probability assessment and evaluation.