Quadratic Scoring Rule Research Articles

Knowing that you know that you know? An extreme-confidence heuristic can lead to above-chance discrimination of metacognitive performance.

In daily life, we can not only estimate confidence in our inferences ('I'm sure I failed that exam'), but can also estimate whether those feelings of confidence are good predictors of decision accuracy ('I feel sure I failed, but my feeling is probably wrong; I probably passed'). In the lab, by using simple perceptual tasks and collecting trial-by-trial confidence ratings visual metacognition research has repeatedly shown that participants can successfully predict the accuracy of their perceptual choices. Can participants also successfully evaluate 'confidence in confidence' in these tasks? This is the question addressed in this study. Participants performed a simple, two-interval forced choice numerosity task framed as an exam. Confidence judgements were collected in the form of a 'predicted exam grade'. Finally, we collected 'meta-metacognitive' reports in a two-interval forced-choice design: trials were presented in pairs, and participants had to select that in which they thought their confidence (predicted grade) best matched their accuracy (actual grade), effectively minimizing their quadratic scoring rule (QSR) score. Participants successfully selected trials on which their metacognition was better when metacognitive performance was quantified using area under the type 2 ROC (AUROC2) but not when using the 'gold-standard' measure m-ratio. However, further analyses suggested that participants selected trials on which AUROC2 is lower in part via an extreme-confidence heuristic, rather than through explicit evaluation of metacognitive inferences: when restricting analyses to trials on which participants gave the same confidence rating AUROC2 no longer differed as a function of selection, and likewise when we excluded trials on which extreme confidence ratings were given. Together, our results show that participants are able to make effective metacognitive discriminations on their visual confidence ratings, but that explicit 'meta-metacognitive' processes may not be required.

An alternative approach to quadratic scoring rules using continuous-valued logic

Following tfhe seminal paper of Offerman et al. (2009), in this study, adaptations of constructions of continuous-valued logic to prospect theory are presented. Here, we demonstrate that the so-called kappa function and its special cases are viable alternatives to some elements of quadratic scoring rule prospects theory. After, we present the tau-eta scoring rule prospect and show that it may be treated as a generalization of the quadratic scoring rule prospect. Furthermore, we prove that if this new prospect for an uncertain event is evaluated using specific kappa functions as utility functions, then (1) the weighting measure of the event is a function of the optimal value of its reported probability, (2) the inverse of the latter function, and (3) the (risk-) corrected reported probability of the event, also as a function of the optimal value of its reported probability, all have a common formula. The parameters of the common formula are unambiguously determined by four tuning parameters. Lastly, we show that with our approach, by fitting one of the abovementioned functions to corresponding empirical data, we can immediately obtain the other two functions as well.

A Robust Bayesian Truth Serum for Small Populations

Peer prediction mechanisms allow the truthful elicitation of private signals (e.g., experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any number of agents n >= 2, but relies on a common prior, shared by all agents and the mechanism. The Bayesian Truth Serum (BTS) relaxes this assumption. While BTS still assumes that agents share a common prior, this prior need not be known to the mechanism. However, BTS is only incentive compatible for a large enough number of agents, and the particular number of agents required is uncertain because it depends on this private prior. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for every n >= 3, taking advantage of a particularity of the quadratic scoring rule. The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n >= 3 without relying on knowledge of the common prior. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational.

Consumers’ responses to food fraud risks: an economic experiment

Abstract This artefactual field experiment explores consumers’ willingness-to-pay (WTP) price premiums for fish products to avoid the risk and uncertainty of purchasing inauthentic produce. The influence of subjective probabilistic beliefs, risk and ambiguity preferences is investigated. Participants’ WTP is elicited using experimental auctions, while behavioural factors are elicited using incentivised and incentive-compatible methods: the quadratic scoring rule and multiple price lists. Results show that consumers are willing to pay a premium to avoid food fraud and purchase an authentic fish product. This premium is higher under uncertainty than risk, likely driven by ambiguity preferences which affect consumers’ purchasing under uncertainty.

Replication: Belief elicitation with quadratic and binarized scoring rules

Researchers increasingly elicit beliefs to understand the underlying motivations of decision makers. Two commonly used methods are the quadratic scoring rule (QSR) and the binarized scoring rule (BSR). Hossain and Okui (2013) use a within-subject design to evaluate the performance of these two methods in an environment where subjects report probabilistic beliefs over binary outcomes with objective probabilities. In a near replication of their study, we show that their results continue to hold with a between-subject design. This is an important validation of the BSR given that researchers typically implement only one method to elicit beliefs. In favor of the BSR, reported beliefs are less accurate under the QSR than the BSR. Consistent with theoretical predictions, risk-averse subjects distort their reported beliefs under the QSR.

Work harder or aim lower? Overconfidence and belief updating in the classroom

AbstractBuilding on recent lab experiments on belief updating, this study uses a university‐based field experiment to investigate behaviour change together with the updating of beliefs in response to feedback in a real‐life context. Postgraduate students predict their performance expectations for a core degree course in a probability distribution format where correct expectations are incentivised using a quadratic scoring rule. Behaviour, in the form of weekly time spent on the course, is also reported. Students then receive feedback in the form of graded tests and assignments, following which they repeat both the prediction exercise and the behaviour reporting. Cognitive dissonance theory predicts that where beliefs and performance feedback do not align, people will either change their beliefs to better conform to received feedback or change their behaviour so that performance better approximates their beliefs. In line with findings from lab experiments, results show that beliefs adjust to better predict true performance when feedback is received. Beliefs also become more precise (lower variance) with feedback. Evidence of behaviour change (time spent on the course) in response to feedback is also noted. The latter finding might account for some of the incomplete adjustment of beliefs often noted in the literature.

A Framework for the Cross‐Validation of Categorical Geostatistical Simulations

AbstractThe mapping of subsurface parameters and the quantification of spatial uncertainty requires selecting adequate models and their parameters. Cross‐validation techniques have been widely used for geostatistical model selection for continuous variables, but the situation is different for categorical variables. In these cases, cross‐validation is seldom applied, and there is no clear consensus on which method to employ. Therefore, this paper proposes a systematic framework for the cross‐validation of geostatistical simulations of categorical variables such as geological facies. The method is based on K‐fold cross‐validation combined with a proper scoring rule. It can be applied whenever an observation data set is available. At each cross‐validation iteration, the training set becomes conditioning data for the tested geostatistical model, and the ensemble of simulations is compared to true values. The proposed framework is generic. Its application is illustrated with two examples using multiple‐point statistics simulations. In the first test case, the aim is to identify a training image from a given data set. In the second test case, the aim is to identify the parameters in a situation including nonstationarity for a coastal alluvial aquifer in the south of France. Cross‐validation scores are used as metrics of model performance and quadratic scoring rule, zero‐one score, and balanced linear score are compared. The study shows that the proposed fivefold stratified cross‐validation with the quadratic scoring rule allows ranking the geostatistical models and helps to identify the proper parameters.

Eliciting farmers’ subjective probabilities, risk, and uncertainty preferences using contextualized field experiments

AbstractSubjective probabilities as well as risk and uncertainty preferences influence many farmers’ decisions. Few contextualized field experiments were recently conducted to elicit farmers’ risk preferences. Contextualized field experiments use nonabstract framings that are familiar to subjects. Despite adding of context can undermine internal validity, such experiments are increasingly used in applied economics. Contextualized field experiments were never used to elicit farmers’ uncertainty preferences. This paper aims to fill this gap in the literature. This required the development of a new approach in which uncertainty preferences were estimated while controlling for farmers’ subjective probabilities regarding future agricultural outcomes. The experiment involves Scottish farmers’ decisions to plant traditional or new potato varieties. Monetary incentives and incentive compatible elicitation techniques, such as quadratic scoring rules and certainty equivalent multiple price lists, were used. Results from the estimation of Fechner models using maximum likelihood estimation procedures show that failure to control for subjective probabilities generates an underestimation of estimated uncertainty preferences. Farmers are more averse to uncertainty than risk, and their choices are noisier under uncertainty than risk.

Belief Elicitation with Binary Outcomes: A Comparison of Quadratic and Binarized Scoring Rules

It is becoming increasingly common for researchers to elicit individuals’ beliefs in controlled experiments to understand the underlying motivations of decision makers. We evaluate the performance of the quadratic scoring rule (QSR) and the binarized scoring rule (BSR) in an environment where subjects report probabilistic beliefs over binary outcomes when the probabilities are objectively known to them. We find that reported beliefs are less accurate under the QSR than the BSR at the aggregate level. Consistent with theoretical predictions, risk-averse subjects tend to distort their reported beliefs under the QSR.

Eliciting Truthful Data From Crowdsourced Wireless Monitoring Modules in Cloud Managed Networks

To facilitate efficient cloud managed resource allocation solutions, collection of key wireless metrics from multiple access points (APs) at different locations within a given area is required. In unlicensed shared spectrum bands collection of metric data can be a challenging task for a cloud manager as independent self-interested APs can operate in these bands in the same area. We propose to design an intelligent crowdsourcing solution that incentivizes independent APs to truthfully measure/report data relating to their wireless channel utilization (CU). Our work focuses on challenging scenarios where independent APs can take advantage of recurring patterns in CU data by utilizing distribution aware strategies to obtain higher reward payments. We design truthful reporting methods that utilize logarithmic and quadratic scoring rules for reward payments to the APs. We show that when measurement computation costs are considered then under certain scenarios these scoring rules no longer ensure incentive compatibility. To address this, we present a novel reward function which incorporates a distribution aware penalty cost that charges APs for distorting reports based on recurring patterns. Along with synthetic data, we also use real CU data values crowdsourced using multiple independent measuring/reporting devices deployed by us in the University of Oulu.

Proper scoring rules with general preferences: A dual characterization of optimal reports

Proper scoring rules incentivize truthful probability reports from risk-neutral individuals. For individuals with much more general preferences (including risk and ambiguity aversion) we use duality techniques to characterize the optimal report in any proper scoring rule. We apply this characterization to several well-known families of preferences, and provide several applications. In particular, for the case of CARA and CRRA preferences we (1) can back out true beliefs from optimal reports, (2) show that the quadratic scoring rule minimizes deviations from truthfulness, and (3) provide a two-step procedure in which the agent reports her risk aversion parameter and her belief truthfully.

Effects of different ways of incentivizing price forecasts on market dynamics and individual decisions in asset market experiments

In this study, we investigate (a) whether eliciting future price forecasts influences market outcomes and (b) whether differences in the way in which subjects are incentivized to submit “accurate” price forecasts influence market outcomes as well as the forecasts in an experimental asset market. We consider four treatments: one without forecast elicitation and three with forecast elicitation. In two of the treatments with forecast elicitation, subjects are paid based on their performance in both forecasting and trading, while in the other treatment with forecast elicitations, they are paid based on only one of those factors, which is chosen randomly at the end of the experiment. We found no significant effect of forecast elicitation on market outcomes in the latter case. Thus, to avoid influencing the behavior of subjects and market outcomes by eliciting price forecasts, paying subjects based on either forecasting or trading performance chosen randomly at the end of the experiment is better than paying them based on both. In addition, we consider forecast-only experiments: one in which subjects are rewarded based on the number of accurate forecasts and the other in which they are rewarded based on a quadratic scoring rule. We found no significant difference in terms of forecasting performance between the two.

Proper Proxy Scoring Rules

Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

Belief Elicitation with a Synchronized Lottery Choice Menu That Is Invariant to Risk Attitudes

This paper uses a Bayesian information processing task to compare belief elicitation mechanisms including a quadratic scoring rule, a Becker-DeGroot-Marschak pricing procedure, and a two-stage menu of lottery choices that is structured to identify a precise point of probability indifference. The choice menu yields a higher incidence of correct Bayesian responses and lower belief error averages. Unlike the quadratic scoring rule, the binary payoffs for the lottery choice mechanism are synchronized to provide theoretical incentive-compatibility regardless of risk attitudes. In addition, the choice menu structure is more transparent and intuitive than the Becker-DeGroot-Marschak procedure. (JEL C91, D44, D81, D83)

In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments

In this paper, we use an experimental design to compare the performance of elicitation rules for subjective beliefs. Contrary to previous works in which elicited beliefs are compared to an objective benchmark, we consider a purely subjective belief framework (confidence in one’s own performance in a cognitive task and a perceptual task). The performance of different elicitation rules is assessed according to the accuracy of stated beliefs in predicting success. We measure this accuracy using two main factors: calibration and discrimination. For each of them, we propose two statistical indexes and we compare the rules’ performances for each measurement. The matching probability method provides more accurate beliefs in terms of discrimination, while the quadratic scoring rule reduces overconfidence and the free rule, a simple rule with no incentives, which succeeds in eliciting accurate beliefs. Nevertheless, the matching probability appears to be the best mechanism for eliciting beliefs due to its performances in terms of calibration and discrimination, but also its ability to elicit consistent beliefs across measures and across tasks, as well as its empirical and theoretical properties.

Confidence measurement in the light of signal detection theory

We compare three alternative methods for eliciting retrospective confidence in the context of a simple perceptual task: the Simple Confidence Rating (a direct report on a numerical scale), the Quadratic Scoring Rule (a post-wagering procedure), and the Matching Probability (MP; a generalization of the no-loss gambling method). We systematically compare the results obtained with these three rules to the theoretical confidence levels that can be inferred from performance in the perceptual task using Signal Detection Theory (SDT). We find that the MP provides better results in that respect. We conclude that MP is particularly well suited for studies of confidence that use SDT as a theoretical framework.

A penny for your thoughts: a survey of methods for eliciting beliefs

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.

Eliciting subjective probabilities with binary lotteries

We evaluate a binary lottery procedure for inducing risk neutral behavior in a subjective belief elicitation task. Prior research has shown this procedure to robustly induce risk neutrality when subjects are given a single risk task defined over objective probabilities. Drawing a sample from the same subject population, we find evidence that the binary lottery procedure also induces linear utility in a subjective probability elicitation task using the Quadratic Scoring Rule. We also show that the binary lottery procedure can induce direct revelation of subjective probabilities in subjects with popular non-expected utility preference representations that satisfy weak conditions.

II—Ralph Wedgwood: Doxastic Correctness

If beliefs are subject to a basic norm of correctness—roughly, to the principle that a belief is correct only if the proposition believed is true—how can this norm guide believers in forming their beliefs? Answer: this norm guides believers indirectly: believers are directly guided by requirements of rationality—which are themselves explained by this norm of correctness. The fundamental connection between rationality and correctness is probabilistic . Incorrectness comes in degrees; for beliefs, these degrees of incorrectness are measured by quadratic scoring rules, such as the so-called Brier score. This account is defended against objections; and its implications for suspension of judgement are explored.

The Binarized Scoring Rule

We introduce a simple method for constructing a scoring rule to elicit an agent's belief about a random variable that is incentive compatible irrespective of her risk-preference. The agent receives a fixed prize when her prediction error, defined by a loss function specified in the incentive scheme, is smaller than an independently generated random number and earns a smaller prize otherwise. Adjusting the loss function according to the belief elicitation objective, the scoring rule can be used in a rich assortment of situations. Moreover, the scoring rule can be incentive compatible even when the agent is not an expected utility maximizer. Results from our probability elicitation experiments show that subjects' predictions are closer to the true probability under this scoring rule compared to the quadratic scoring rule. Copyright 2013, Oxford University Press.