Quadratic Score Research Articles

Asymptotic Results for the Estimation of the Quadratic Score of a Clustering

In cluster analysis one often finds several partitions of a data set using different clustering methods and algorithms set with a variety of hyperparameters and tunings. The number of clusters K is one of the most relevant of such hyperparameters. Cluster selection is the task of choosing the desired partitions. The Bootstrap Quadratic Scoring is a recently introduced method where the cluster selection is performed by optimizing a score attached to a partition that is based on the quadratic discriminant function. Previously, we proposed the estimation of this cluster score via bootstrap resampling and investigated the proposed estimator based on numerical experiments and real data applications. However, that earlier work did not provide theoretical guarantees. In this paper, we fill that gap. We study the asymptotic behavior of the scoring method and show that the proposed estimator converges to well-defined population counterparts.

Knowing that you know that you know? An extreme-confidence heuristic can lead to above-chance discrimination of metacognitive performance.

In daily life, we can not only estimate confidence in our inferences ('I'm sure I failed that exam'), but can also estimate whether those feelings of confidence are good predictors of decision accuracy ('I feel sure I failed, but my feeling is probably wrong; I probably passed'). In the lab, by using simple perceptual tasks and collecting trial-by-trial confidence ratings visual metacognition research has repeatedly shown that participants can successfully predict the accuracy of their perceptual choices. Can participants also successfully evaluate 'confidence in confidence' in these tasks? This is the question addressed in this study. Participants performed a simple, two-interval forced choice numerosity task framed as an exam. Confidence judgements were collected in the form of a 'predicted exam grade'. Finally, we collected 'meta-metacognitive' reports in a two-interval forced-choice design: trials were presented in pairs, and participants had to select that in which they thought their confidence (predicted grade) best matched their accuracy (actual grade), effectively minimizing their quadratic scoring rule (QSR) score. Participants successfully selected trials on which their metacognition was better when metacognitive performance was quantified using area under the type 2 ROC (AUROC2) but not when using the 'gold-standard' measure m-ratio. However, further analyses suggested that participants selected trials on which AUROC2 is lower in part via an extreme-confidence heuristic, rather than through explicit evaluation of metacognitive inferences: when restricting analyses to trials on which participants gave the same confidence rating AUROC2 no longer differed as a function of selection, and likewise when we excluded trials on which extreme confidence ratings were given. Together, our results show that participants are able to make effective metacognitive discriminations on their visual confidence ratings, but that explicit 'meta-metacognitive' processes may not be required.

An alternative approach to quadratic scoring rules using continuous-valued logic

Following tfhe seminal paper of Offerman et al. (2009), in this study, adaptations of constructions of continuous-valued logic to prospect theory are presented. Here, we demonstrate that the so-called kappa function and its special cases are viable alternatives to some elements of quadratic scoring rule prospects theory. After, we present the tau-eta scoring rule prospect and show that it may be treated as a generalization of the quadratic scoring rule prospect. Furthermore, we prove that if this new prospect for an uncertain event is evaluated using specific kappa functions as utility functions, then (1) the weighting measure of the event is a function of the optimal value of its reported probability, (2) the inverse of the latter function, and (3) the (risk-) corrected reported probability of the event, also as a function of the optimal value of its reported probability, all have a common formula. The parameters of the common formula are unambiguously determined by four tuning parameters. Lastly, we show that with our approach, by fitting one of the abovementioned functions to corresponding empirical data, we can immediately obtain the other two functions as well.

Calibration of Hernia-Specific Patient-Reported Outcome Measures.

Although there are many patient-reported outcome measures used for ventral hernia (VH), disease-specific instruments, such as the Hernia-related Quality-of-Life (QoL) Survey (HerQLes) and Abdominal Hernia-Q (AHQ), have shown greater accuracy in capturing all VH-related QoL. We present a novel calibration that allows providers to convert scores between the AHQ and HerQLes, enabling better unification of QoL data. Patients with VH were prospectively identified and simultaneously administered both the AHQ and HerQLes pre- and post-operatively. To ensure the validity of the calibration, responses were excluded if patients answered instruments on different dates or if the responses were discordant on corresponding questions within each instrument. The calibration was estimated using a linear mixed effects model, including linear and quadratic scores, timing of survey relative to surgery and their interactions as fixed effects, and patients as random effects to account for multiple surveys from the same patient. In total, 109 patients were included, responding to 300 pairs of surveys (112 preoperative and 188 postoperative), of which 17 (5.6%) were excluded because of discordant responses. Conversion of the HerQLes to AHQ was most accurate when including whether the survey was completed pre- or post-operatively, with a mean squared error of 0.0091. Similarly, converting the AHQ to HerQLes was most accurate when factoring in the timing of survey administration, with a mean squared error of 0.016. We present a novel and accurate method to convert scores between the AHQ and HerQLes. Being able to unify QoL data from different PROMs supports efforts to more broadly integrate PROMs in surgery and to understand patient-defined measures of success.

P086 A NOVEL CALIBRATION ALGORITHM FOR COMPARISON ACROSS PATIENT REPORTED OUTCOMES (PROS) IN VENTRAL HERNIA

Abstract Aim While there are many patient-reported outcome measures (PROMs) used for ventral hernia (VH), disease-specific instruments, like the Hernia-related Quality-of-Life (QoL) Survey (HerQLes) and Abdominal Hernia-Q (AHQ), have greater accuracy in capturing broad VH-related QoL. We present a novel calibration that allows providers to convert scores between the AHQ and HerQLes, enabling unification of QoL data. Material and Methods VH patients were prospectively identified and simultaneously administered the AHQ and HerQLes pre-and post-operatively. To ensure validity of the calibration, responses were excluded if patients answered instruments on different dates or if responses were discordant on corresponding questions on each instrument. The calibration was estimated using a linear mixed-effects model, including linear and quadratic scores, timing of survey relative to surgery and their interactions as fixed effects, and patients as random effects to account for multiple surveys from the same patient. Results In total, 109 patients were included, responding to 300 pairs of surveys (112 pre-operative and 188 post-operative). Seventeen (5.6%) were statistically excluded due to discordant responses. Conversion of the HerQLes to AHQ was most accurate when including whether the survey was completed pre-or post-operatively, with a mean square error (MSE) of 0.0091. Similarly, converting the AHQ to HerQLes was most accurate when factoring in the timing of survey administration, with a MSE of 0.016. Conclusions We present a novel and accurate method to convert scores between the AHQ and HerQLes. Portability of PROMs will be crucial in efforts to more broadly integrate PROMs into routine care in VH.

A Robust Bayesian Truth Serum for Small Populations

Peer prediction mechanisms allow the truthful elicitation of private signals (e.g., experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any number of agents n >= 2, but relies on a common prior, shared by all agents and the mechanism. The Bayesian Truth Serum (BTS) relaxes this assumption. While BTS still assumes that agents share a common prior, this prior need not be known to the mechanism. However, BTS is only incentive compatible for a large enough number of agents, and the particular number of agents required is uncertain because it depends on this private prior. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for every n >= 3, taking advantage of a particularity of the quadratic scoring rule. The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n >= 3 without relying on knowledge of the common prior. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational.

Consumers’ responses to food fraud risks: an economic experiment

Abstract This artefactual field experiment explores consumers’ willingness-to-pay (WTP) price premiums for fish products to avoid the risk and uncertainty of purchasing inauthentic produce. The influence of subjective probabilistic beliefs, risk and ambiguity preferences is investigated. Participants’ WTP is elicited using experimental auctions, while behavioural factors are elicited using incentivised and incentive-compatible methods: the quadratic scoring rule and multiple price lists. Results show that consumers are willing to pay a premium to avoid food fraud and purchase an authentic fish product. This premium is higher under uncertainty than risk, likely driven by ambiguity preferences which affect consumers’ purchasing under uncertainty.

Replication: Belief elicitation with quadratic and binarized scoring rules

Researchers increasingly elicit beliefs to understand the underlying motivations of decision makers. Two commonly used methods are the quadratic scoring rule (QSR) and the binarized scoring rule (BSR). Hossain and Okui (2013) use a within-subject design to evaluate the performance of these two methods in an environment where subjects report probabilistic beliefs over binary outcomes with objective probabilities. In a near replication of their study, we show that their results continue to hold with a between-subject design. This is an important validation of the BSR given that researchers typically implement only one method to elicit beliefs. In favor of the BSR, reported beliefs are less accurate under the QSR than the BSR. Consistent with theoretical predictions, risk-averse subjects distort their reported beliefs under the QSR.

Work harder or aim lower? Overconfidence and belief updating in the classroom

AbstractBuilding on recent lab experiments on belief updating, this study uses a university‐based field experiment to investigate behaviour change together with the updating of beliefs in response to feedback in a real‐life context. Postgraduate students predict their performance expectations for a core degree course in a probability distribution format where correct expectations are incentivised using a quadratic scoring rule. Behaviour, in the form of weekly time spent on the course, is also reported. Students then receive feedback in the form of graded tests and assignments, following which they repeat both the prediction exercise and the behaviour reporting. Cognitive dissonance theory predicts that where beliefs and performance feedback do not align, people will either change their beliefs to better conform to received feedback or change their behaviour so that performance better approximates their beliefs. In line with findings from lab experiments, results show that beliefs adjust to better predict true performance when feedback is received. Beliefs also become more precise (lower variance) with feedback. Evidence of behaviour change (time spent on the course) in response to feedback is also noted. The latter finding might account for some of the incomplete adjustment of beliefs often noted in the literature.

A Framework for the Cross‐Validation of Categorical Geostatistical Simulations

AbstractThe mapping of subsurface parameters and the quantification of spatial uncertainty requires selecting adequate models and their parameters. Cross‐validation techniques have been widely used for geostatistical model selection for continuous variables, but the situation is different for categorical variables. In these cases, cross‐validation is seldom applied, and there is no clear consensus on which method to employ. Therefore, this paper proposes a systematic framework for the cross‐validation of geostatistical simulations of categorical variables such as geological facies. The method is based on K‐fold cross‐validation combined with a proper scoring rule. It can be applied whenever an observation data set is available. At each cross‐validation iteration, the training set becomes conditioning data for the tested geostatistical model, and the ensemble of simulations is compared to true values. The proposed framework is generic. Its application is illustrated with two examples using multiple‐point statistics simulations. In the first test case, the aim is to identify a training image from a given data set. In the second test case, the aim is to identify the parameters in a situation including nonstationarity for a coastal alluvial aquifer in the south of France. Cross‐validation scores are used as metrics of model performance and quadratic scoring rule, zero‐one score, and balanced linear score are compared. The study shows that the proposed fivefold stratified cross‐validation with the quadratic scoring rule allows ranking the geostatistical models and helps to identify the proper parameters.

Eliciting farmers’ subjective probabilities, risk, and uncertainty preferences using contextualized field experiments

AbstractSubjective probabilities as well as risk and uncertainty preferences influence many farmers’ decisions. Few contextualized field experiments were recently conducted to elicit farmers’ risk preferences. Contextualized field experiments use nonabstract framings that are familiar to subjects. Despite adding of context can undermine internal validity, such experiments are increasingly used in applied economics. Contextualized field experiments were never used to elicit farmers’ uncertainty preferences. This paper aims to fill this gap in the literature. This required the development of a new approach in which uncertainty preferences were estimated while controlling for farmers’ subjective probabilities regarding future agricultural outcomes. The experiment involves Scottish farmers’ decisions to plant traditional or new potato varieties. Monetary incentives and incentive compatible elicitation techniques, such as quadratic scoring rules and certainty equivalent multiple price lists, were used. Results from the estimation of Fechner models using maximum likelihood estimation procedures show that failure to control for subjective probabilities generates an underestimation of estimated uncertainty preferences. Farmers are more averse to uncertainty than risk, and their choices are noisier under uncertainty than risk.

Composite leading indicators of economic activity: An application to the upstream oil and gas industry in Rio de Janeiro

This paper proposes a monthly composite leading indicator to anticipate turning points in the economic activity of the upstream oil and gas industry in Rio de Janeiro, from January 2002 to May 2019. Firstly, we build a database with 61 series, and categorize each of them into i) rapidly responsive to economic activities; ii) expectation-sensitive; or iii) prime movers indicators. Afterward, we remove the seasonality of the series through the X-13 ARIMA-SEATS method and use the Bry-Boschan algorithm to identify the cycles. Then, we evaluate the components’ fit to integrate the composite leading indicator through four statistical tests: cross-correlation, quadratic probability score, Granger causality, and probit. The assessment of the composite leading indicator demonstrates that it leads 67% of the peaks and 100% of the troughs in the target series (5/6 of the turning points). Furthermore, the average leading period is 8.4 months, while the median is 9 and the standard error is 2.8 months. We contribute to the literature by creating, to our knowledge, the first leading indicator for the oil and gas industry in Brazil.

Belief Elicitation with Binary Outcomes: A Comparison of Quadratic and Binarized Scoring Rules

It is becoming increasingly common for researchers to elicit individuals’ beliefs in controlled experiments to understand the underlying motivations of decision makers. We evaluate the performance of the quadratic scoring rule (QSR) and the binarized scoring rule (BSR) in an environment where subjects report probabilistic beliefs over binary outcomes when the probabilities are objectively known to them. We find that reported beliefs are less accurate under the QSR than the BSR at the aggregate level. Consistent with theoretical predictions, risk-averse subjects tend to distort their reported beliefs under the QSR.

Eliciting Truthful Data From Crowdsourced Wireless Monitoring Modules in Cloud Managed Networks

To facilitate efficient cloud managed resource allocation solutions, collection of key wireless metrics from multiple access points (APs) at different locations within a given area is required. In unlicensed shared spectrum bands collection of metric data can be a challenging task for a cloud manager as independent self-interested APs can operate in these bands in the same area. We propose to design an intelligent crowdsourcing solution that incentivizes independent APs to truthfully measure/report data relating to their wireless channel utilization (CU). Our work focuses on challenging scenarios where independent APs can take advantage of recurring patterns in CU data by utilizing distribution aware strategies to obtain higher reward payments. We design truthful reporting methods that utilize logarithmic and quadratic scoring rules for reward payments to the APs. We show that when measurement computation costs are considered then under certain scenarios these scoring rules no longer ensure incentive compatibility. To address this, we present a novel reward function which incorporates a distribution aware penalty cost that charges APs for distorting reports based on recurring patterns. Along with synthetic data, we also use real CU data values crowdsourced using multiple independent measuring/reporting devices deployed by us in the University of Oulu.

Proper scoring rules with general preferences: A dual characterization of optimal reports

Proper scoring rules incentivize truthful probability reports from risk-neutral individuals. For individuals with much more general preferences (including risk and ambiguity aversion) we use duality techniques to characterize the optimal report in any proper scoring rule. We apply this characterization to several well-known families of preferences, and provide several applications. In particular, for the case of CARA and CRRA preferences we (1) can back out true beliefs from optimal reports, (2) show that the quadratic scoring rule minimizes deviations from truthfulness, and (3) provide a two-step procedure in which the agent reports her risk aversion parameter and her belief truthfully.

Differences in Attention Levels between Preliminary Nurses and Pre-service Early Childhood Teachers Using ANT (Attentional Network Test) Computer Test

The purpose of this study was to analyze attentional network test (ANT) and to examine the difference of attention level according to the types of brain dominance between nursing college students and preservice early childhood teachers. The subjects of this study were 79 students in 3rd and 4th grades attending nursing department and early childhood education department of U university. Herrmann's BDI was used to measure the type of brain dominance and Ant (attentional network test) computer test was performed to measure the level of attention. The data were analyzed by independent t-test to determine differences in brain dominance and attentional level between preliminary nurse and pre-service early childhood teacher and to analyze differences in attention level according to types of brain dominance. The results of this study were as follows: First, there was no significant difference between the two groups in A, B, and C quadrants. In quadrant D, the pre-early childhood teacher's quadratic thinking score was significantly higher than preliminary nurse quadrant, respectively. Second, there was no significant difference between preliminary nurses and pre-service early childhood teachers in the level of alerting and executive control. At the level of orienting, the preliminary nurse's orienting score was significantly higher than the pre-service early childhood teacher's orienting score. Third, alerting effect attention was not significantly different according to the type of brain dominance, but orienting was highest in students in the quadrant A, and executive control was highest in students in quadrant D quadrant burnt.

Effects of different ways of incentivizing price forecasts on market dynamics and individual decisions in asset market experiments

In this study, we investigate (a) whether eliciting future price forecasts influences market outcomes and (b) whether differences in the way in which subjects are incentivized to submit “accurate” price forecasts influence market outcomes as well as the forecasts in an experimental asset market. We consider four treatments: one without forecast elicitation and three with forecast elicitation. In two of the treatments with forecast elicitation, subjects are paid based on their performance in both forecasting and trading, while in the other treatment with forecast elicitations, they are paid based on only one of those factors, which is chosen randomly at the end of the experiment. We found no significant effect of forecast elicitation on market outcomes in the latter case. Thus, to avoid influencing the behavior of subjects and market outcomes by eliciting price forecasts, paying subjects based on either forecasting or trading performance chosen randomly at the end of the experiment is better than paying them based on both. In addition, we consider forecast-only experiments: one in which subjects are rewarded based on the number of accurate forecasts and the other in which they are rewarded based on a quadratic scoring rule. We found no significant difference in terms of forecasting performance between the two.

Proper Proxy Scoring Rules

Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

Probabilistic Forecasting with Stationary VAR Models

I use a set of vector autoregressive models to forecast some of the main macroeconomic variables in a wide range of countries. The goal is to provide some insight about different forecast accuracy measures in a probabilistic forecasting framework. The countries are selected based on their available data in the International Financial Statistics (IFS) database. The target variables are GDP, CPI and exchange rate growth in yearly, quarterly and monthly frequencies. I specify the model sets by using a list of potentially relevant variables from IFS database. The results are available up to four forecast horizons and for two different classes of forecast accuracy measures: The probabilistic class which contains linear, logarithmic, quadratic, Hyvarinen and continuous ranked probability score rules, and the non-probabilistic class which contains mean absolute and mean square error measures. The results show that the forecasts are sensitive to the choice of a forecast accuracy class, in sense of Hellinger distance. However, within each class, the choice is not critical. Among different probabilistic measures, the logarithmic and quadratic rules are more reliable in practice. The results also show that the forecast power of stationary multivariate models is higher than the univariate ones, for both probabilistic and non-probabilistic accuracy measures.

Belief Elicitation with a Synchronized Lottery Choice Menu That Is Invariant to Risk Attitudes

This paper uses a Bayesian information processing task to compare belief elicitation mechanisms including a quadratic scoring rule, a Becker-DeGroot-Marschak pricing procedure, and a two-stage menu of lottery choices that is structured to identify a precise point of probability indifference. The choice menu yields a higher incidence of correct Bayesian responses and lower belief error averages. Unlike the quadratic scoring rule, the binary payoffs for the lottery choice mechanism are synchronized to provide theoretical incentive-compatibility regardless of risk attitudes. In addition, the choice menu structure is more transparent and intuitive than the Becker-DeGroot-Marschak procedure. (JEL C91, D44, D81, D83)