Scoring Rules Research Articles

Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile versus multivariate GARCH models

Many static and dynamic models exist to forecast Value-at-Risk and other quantile-related metrics used in financial risk management. Industry practice favours simpler, static models such as historical simulation or its variants. Most academic research focuses on dynamic models in the GARCH family. While numerous studies examine the accuracy of multivariate models for forecasting risk metrics, there is little research on accurately predicting the entire multivariate distribution. However, this is an essential element of asset pricing or portfolio optimization problems having non-analytic solutions. We approach this highly complex problem using various proper multivariate scoring rules to evaluate forecasts of eight-dimensional multivariate distributions: exchange rates, interest rates and commodity futures. This way, we test the performance of static models, namely, empirical distribution functions and a new factor-quantile model with commonly used dynamic models in the asymmetric multivariate GARCH class.

Online belief elicitation methods

How well do incentivized belief elicitation procedures work in online settings? We evaluate the quality of beliefs elicited from online respondents, comparing several characteristics of two widely used complex elicitation mechanisms (the Binarized Scoring Rule – BSR – and a stochastic variation of the Becker–deGroot–Marschak mechanism—BDM) against a flat fee baseline for a variety of beliefs (induced probabilities, first-order factual knowledge, second-order knowledge of others). We find that the flat-fee method requires the least amount of time, the BDM is the most difficult to understand, and that there are no differences in the average accuracy of induced beliefs across conditions. However, the methods are significantly different in terms of the frequency of first-order and second-order beliefs reported at exactly 50%: the flat-fee method leads to the most mass on this belief, followed by BDM and BSR. Regarding induced beliefs, we also find that less-educated participants’ accuracy is higher in the complex incentives treatments, and that attention, numeracy, and education are positively associated with the quality of these beliefs across methods. Our results suggest that the quality of beliefs elicited in online environments may depend less on the formal incentive compatibility properties of the elicitation procedure (whether the procedure prevents “dishonest” reporting) than on the difficulty of comprehending the task and how well incentives induce cognitive effort (thereby inducing subjects to quantify or construct their beliefs).

Incentive-Compatible Forecasting Competitions

We initiate the study of incentive-compatible forecasting competitions in which multiple forecasters make predictions about one or more events and compete for a single prize. We have two objectives: (1) to incentivize forecasters to report truthfully and (2) to award the prize to the most accurate forecaster. Proper scoring rules incentivize truthful reporting if all forecasters are paid according to their scores. However, incentives become distorted if only the best-scoring forecaster wins a prize, since forecasters can often increase their probability of having the highest score by reporting more extreme beliefs. In this paper, we introduce two novel forecasting competition mechanisms. Our first mechanism is incentive compatible and guaranteed to select the most accurate forecaster with probability higher than any other forecaster. Moreover, we show that in the standard single-event, two-forecaster setting and under mild technical conditions, no other incentive-compatible mechanism selects the most accurate forecaster with higher probability. Our second mechanism is incentive compatible when forecasters’ beliefs are such that information about one event does not lead to belief updates on other events, and it selects the best forecaster with probability approaching one as the number of events grows. Our notion of incentive compatibility is more general than previous definitions of dominant strategy incentive compatibility in that it allows for reports to be correlated with the event outcomes. Moreover, our mechanisms are easy to implement and can be generalized to the related problems of outputting a ranking over forecasters and hiring a forecaster with high accuracy on future events.This paper was accepted by Yan Chen, behavioral economics and decision analysis.Funding: This work was supported by the European Research Council [Grant ERC StG 307036] and the National Science Foundation [Grant CCF-1445755].

Core and catalyst criteria for disclosing one’s burnout in the workplace

A sample of 142 full-time employees who experienced job burnout during the COVID-19 pandemic completed an online, open-ended survey designed to explore the core and catalyst privacy rule criteria that individuals consider when disclosing or not disclosing their experiences of burnout in the workplace. Participant responses indicate that core criteria consisted of the quality of one’s relationship with colleagues and the organization’s culture. Catalyst criteria consisted of a sense of a shared experience of burnout, the risk of judgment toward the disclosure, and the severity of one’s burnout. These findings suggest that discussing job burnout in one’s workplace can create feelings of vulnerability that can potentially impact one’s job and, thus, employees are careful in choosing confidants and methods of disclosure for job burnout.

Civil Society and Rule of Law Backsliding in the EU

In the realm of the rule of law crisis taking place in several Member States of the European Union (EU), illiberal governments attempt to dilute institutional and non-institutional checks and balances with the aim to unrestrainedly exercise the state power. In this context, civil society, a sphere autonomous from the state where public interests are being expressed and formulated, is constantly undermined. This article tries to illustrate the relation between civil society and the rule of law and its backsliding. In this regard, a theoretical framework indicating the connections of the dominant civil society theories with core rule of law elements will be developed and the points of friction with rule of law backsliding will be indicated. This theoretical framework will, then, be tested in EU institutional practice through the examples of the EU Annual Rule of Law Report and the case Commission v. Hungary (Transparency of Associations, C-78/18). civil society, rule of law backsliding, fundamental rights, participation in decisionmaking, Annual Rule of Law Report

Quantitative phase imaging based on model transfer learning.

Convolutional neural networks have been widely used in optical information processing and the generalization ability of the network depends greatly on the scale and diversity of the datasets, however, the acquisition of mass datasets and later annotation have become a common problem that hinders its further progress. In this study, a model transfer-based quantitative phase imaging (QPI) method is proposed, which fine-tunes the network parameters through loading pre-training base model and transfer learning, enable the network with good generalization ability. Most importantly, a feature fusion method based on moment reconstruction is proposed for training dataset generation, which can construct rich enough datasets that can cover most situations and accurately annotated, it fundamentally solves the problem from the scale and representational ability of the datasets. Besides, a feature distribution distance scoring (FDDS) rule is proposed to evaluate the rationality of the constructed datasets. The experimental results show that this method is suitable for different types of samples to achieve fast and high-accuracy phase imaging, which greatly relieves the pressure of data, tagging and generalization ability in the data-driven method.

Bayesian multiresolution modeling of georeferenced data: An extension of ‘LatticeKrig’

‘LatticeKrig’ (LK) is a spatial model that is often used for modeling multiresolution spatial data with flexible covariance structures. An extension to LK under a Bayesian framework is proposed that uses integrated nested Laplace approximations (INLA). The extension enables the spatial analysis of non-Gaussian responses in latent Gaussian models, joint spatial modeling with structured and unstructured random effects, and native support for multithreaded parallel likelihood computation. The proposed extended LatticeKrig (ELK) model uses a reparameterization of LK so that the parameters and prior selection are intuitive and interpretable. Priors can be used to make inference robust by penalizing more complex models, and integration over model parameters allows for posterior uncertainty estimates that account for uncertainty in covariance parameters. ELK's ability to reliably resolve multiresolution spatial structure for pointwise and areal predictions is demonstrated in both simulation study and two applications with non-Gaussian observations: a set of 188,717 LiDAR forest canopy height observations in Bonanza Creek Experimental Forest in Alaska, and a set of 1,612 clusters containing counts of secondary education completion from the 2014 Kenya demographic health survey. ELK has improved central predictions as well as uncertainty characterization according to the considered scoring rules when compared against a number of other models, particularly in the forest canopy height application, and performed faster than LK in our tests in part due to its support for and use of parallelization.

Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets.

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters.

Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment

SUMMARYOperational earthquake forecasting for risk management and communication during seismic sequences depends on our ability to select an optimal forecasting model. To do this, we need to compare the performance of competing models in prospective experiments, and to rank their performance according to the outcome using a fair, reproducible and reliable method, usually in a low-probability environment. The Collaboratory for the Study of Earthquake Predictability conducts prospective earthquake forecasting experiments around the globe. In this framework, it is crucial that the metrics used to rank the competing forecasts are ‘proper’, meaning that, on average, they prefer the data generating model. We prove that the Parimutuel Gambling score, proposed, and in some cases applied, as a metric for comparing probabilistic seismicity forecasts, is in general ‘improper’. In the special case where it is proper, we show it can still be used improperly. We demonstrate the conclusions both analytically and graphically providing a set of simulation based techniques that can be used to assess if a score is proper or not. They only require a data generating model and, at least two forecasts to be compared. We compare the Parimutuel Gambling score’s performance with two commonly used proper scores (the Brier and logarithmic scores) using confidence intervals to account for the uncertainty around the observed score difference. We suggest that using confidence intervals enables a rigorous approach to distinguish between the predictive skills of candidate forecasts, in addition to their rankings. Our analysis shows that the Parimutuel Gambling score is biased, and the direction of the bias depends on the forecasts taking part in the experiment. Our findings suggest the Parimutuel Gambling score should not be used to distinguishing between multiple competing forecasts, and for care to be taken in the case where only two are being compared.

What if the Forecaster Knew: Assessing Forecast Reliability Via Simulation

The FIFA Men’s World Cup (FWC) is the most important football (soccer) competition, attracting worldwide attention. A popular practice among football fans in Brazil is to organize contests in which each participant informs guesses on the final score of each match. The participants are then ranked according to some scoring rule. Inspired by these contests, we created a website to hold an online contest, in which participants were asked for their probabilities on the outcomes of upcoming matches of the FWC. After each round of the tournament, the ranking of all participants based on a proper scoring rule was published. In this article we estimate, by means of simulations, the ability of the best forecasters of our contest, considering that their good performances could be due to randomness. We also study the performance of some methods to aggregate individual forecasts, in order to study if some sort of wisdom of crowds (WOC) phenomenon was verified in the contest.

Evaluating the discrimination ability of proper multi-variate scoring rules

Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalized discrimination heuristic. Overall, we find that the variogram score with parameter p=0.5\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$p=0.5$$\\end{document} outperforms the energy score and the other two variogram scores.

Simulation input modelling in the absence of historical data for decision support during crises: Experience with assessing demand uncertainties for simulating walk-through testing in the early waves of COVID-19

During the early waves of COVID-19, a frequent modelling challenge for OR practitioners was the lack of relevant historical data on important model inputs, such as uncertainty distributions. This is common and often critical in OR projects that support decision-making during new crises, in the aftermath of disasters, and generally, when modelling systems that are new or currently do not exist. Guidance for OR practitioners on systematic approaches for this modelling challenge is generally limited, and, in particular, omits important aspects of OR projects that support decisions during crises. These include, e.g., requiring urgent decision support, possible extreme parameter values, only a few experts being available. This paper proposes a novel iterative multi-method framework to address this gap in the literature. Our framework combines uncertainty elicitation, mathematical scoring rules and problem structuring with Discrete Event Simulation, with a focus on crisis decision-making. It provides optimised and transparent weighted combinations for simulation parameters. Our case study is motivated by our experience of simulation modelling during the early COVID-19 pandemic. We faced this modelling challenge for arrival number uncertainties when building a simulation of “no appointment necessary” COVID-19 walk-through testing to provide decision-makers with a better understanding of long waiting time risks.

Gale’s Fixed Tax for Exchanging Houses

We consider taxation of exchanges among a set of agents in which each agent owns one object. Agents may have different valuations for the objects, and they need to pay taxes for exchanges. We show that, if a rule satisfies individual rationality, strategy-proofness, constrained efficiency, weak anonymity, and weak consistency, then it is either the no-trade rule or a fixed-tax core rule. For the latter rules, whenever any agent exchanges an object, the agent pays the same fixed tax (a lump sum payment that is identical for all agents) independently of which object the agent consumes. Gale’s top trading cycles algorithm finds the final assignment using the agents’ valuations adjusted with the fixed tax if the induced preferences are strict.

Proper Scoring Rules for Evaluating Density Forecasts with Asymmetric Loss Functions

This article proposes a novel asymmetric continuous probabilistic score (ACPS) for evaluating and comparing density forecasts. It generalizes the proposed score and defines a weighted version, which emphasizes regions of interest, such as the tails or the center of a variable’s range. The (weighted) ACPS extends the symmetric (weighted) CRPS by allowing for asymmetries in the preferences underlying the scoring rule. A test is used to statistically compare the predictive ability of different forecasts. The ACPS is of general use in any situation where the decision-maker has asymmetric preferences in the evaluation of the forecasts. In an artificial experiment, the implications of varying the level of asymmetry in the ACPS are illustrated. Then, the proposed score and test are applied to assess and compare density forecasts of macroeconomic relevant datasets (U.S. employment growth) and of commodity prices (oil and electricity prices) with particular focus on the recent COVID-19 crisis period.

Robust confidence distributions from proper scoring rules

A confidence distribution is a distribution for a parameter of interest based on a parametric statistical model. As such, it serves the same purpose for frequentist statisticians as a posterior distribution for Bayesians, since it allows to reach point estimates, to assess their precision, to set up tests along with measures of evidence, to derive confidence intervals, comparing the parameter of interest with other parameters from other studies, etc. A general recipe for deriving confidence distributions is based on classical pivotal quantities and their exact or approximate distributions. However, in the presence of model misspecifications or outlying values in the observed data, classical pivotal quantities, and thus confidence distributions, may be inaccurate. The aim of this paper is to discuss the derivation and application of robust confidence distributions. In particular, we discuss a general approach based on the Tsallis scoring rule in order to compute a robust confidence distribution. Examples and simulation results are discussed for some problems often encountered in practice, such as the two-sample heteroschedastic comparison, the receiver operating characteristic curves and regression models.

Four Questions Nurses Can Ask to Predict PTSD 1 Year After a Motor Vehicle Crash.

The role of nurses in screening for posttraumatic stress disorder is crucial in trauma units. To create and evaluate an easy and brief tool for nurses to predict chronic posttraumatic stress disorder 1 year after a motor vehicle crash. We performed a 1-year follow-up multicenter study from 2007 to 2015, including 274 patients injured in a motor vehicle crash who were hospitalized in an orthopedic trauma unit. Nurses administered the DEPITAC questionnaire. Posttraumatic stress disorder was measured by the Post-Traumatic Stress Disorder Checklist of symptoms during the first year following the crash. A multivariable logistic regression model was implemented to select items significantly associated with posttraumatic stress disorder to improve the DEPITAC questionnaire. Predictive performance to predict posttraumatic stress disorder 1 year after the motor vehicle crash was examined for these different models. Of 274 patients studied, a total of 75.9% completed the questionnaire at 1 year of follow-up. We found that only two questions and two simple elements of the patient's medical record (other injury or a person dying during the crash, perception of vital threat, number of children, and length of stay in trauma) predicted posttraumatic stress disorder 1 year after a motor vehicle crash. The brevity of this evaluation, simple scoring rules, and screening test performance suggest that this new screening tool can be easily administered in the acute care setting by nurses.

A transition to the American Academy of Sleep Medicine-recommended hypopnea definition in adults: initiatives of the Hypopnea Scoring Rule Task Force.

Berry RB, Abreu AR, Krishnan V, Quan SF, Strollo PJ Jr, Malhotra RK. A transition to the American Academy of Sleep Medicine-recommended hypopnea definition in adults: initiatives of the Hypopnea Scoring Rule Task Force. J Clin Sleep Med. 2022;18(5):1419-1425.

Elicitability of Instance and Object Ranking

Assessing the quality of a forecasting model crucially depends on a proper scoring rule or suitable loss function. As for point forecasts, the existence of a strictly consistent loss function that allows for a fair comparison of competing forecast models has to be guaranteed, which means that the corresponding statistical functional has to be elicitable. We consider instance and object ranking problems that intend to correctly predict the ordering of instances in a data set. A ranking prediction is naturally identified with a point forecast in the respective symmetric group, that is, the forecaster predicts one single permutation of the row indices. We show that, in the presence of ties, this strategy does not allow for strictly consistent scoring functions because of multiple true permutations. Those multiple optima cannot be entirely covered by a single point forecast, which causes all corresponding optima to be minimizers of standard scoring functions that operate on symmetric groups, so these scoring functions are not strictly consistent. As a remedy, we consider accurately accounting for ties. This is done by treating each configuration of clear orderings and ties as an additional category, which induces extended decision spaces with a clearly defined single optimum. Because these decision spaces are still finite, each type of instance ranking problem that we consider in this work and corresponding ranking functional, mapping into a symmetric group, can be identified with a certain classification problem and corresponding classification functional, mapping into one of our extended decision spaces, which is elicitable.

A prediction rule for severe adverse events in all inpatients with community-acquired pneumonia: a multicenter observational study

BackgroundPrediction of inpatients with community-acquired pneumonia (CAP) at high risk for severe adverse events (SAEs) requiring higher-intensity treatment is critical. However, evidence regarding prediction rules applicable to all patients with CAP including those with healthcare-associated pneumonia (HCAP) is limited. The objective of this study is to develop and validate a new prediction system for SAEs in inpatients with CAP.MethodsLogistic regression analysis was performed in 1334 inpatients of a prospective multicenter study to develop a multivariate model predicting SAEs (death, requirement of mechanical ventilation, and vasopressor support within 30 days after diagnosis). The developed ALL-COP-SCORE rule based on the multivariate model was validated in 643 inpatients in another prospective multicenter study.ResultsThe ALL-COP SCORE rule included albumin (< 2 g/dL, 2 points; 2–3 g/dL, 1 point), white blood cell (< 4000 cells/μL, 3 points), chronic lung disease (1 point), confusion (2 points), PaO2/FIO2 ratio (< 200 mmHg, 3 points; 200–300 mmHg, 1 point), potassium (≥ 5.0 mEq/L, 2 points), arterial pH (< 7.35, 2 points), systolic blood pressure (< 90 mmHg, 2 points), PaCO2 (> 45 mmHg, 2 points), HCO3− (< 20 mmol/L, 1 point), respiratory rate (≥ 30 breaths/min, 1 point), pleural effusion (1 point), and extent of chest radiographical infiltration in unilateral lung (> 2/3, 2 points; 1/2–2/3, 1 point). Patients with 4–5, 6–7, and ≥ 8 points had 17%, 35%, and 52% increase in the probability of SAEs, respectively, whereas the probability of SAEs was 3% in patients with ≤ 3 points. The ALL-COP SCORE rule exhibited a higher area under the receiver operating characteristic curve (0.85) compared with the other predictive models, and an ALL-COP SCORE threshold of ≥ 4 points exhibited 92% sensitivity and 60% specificity.ConclusionsALL-COP SCORE rule can be useful to predict SAEs and aid in decision-making on treatment intensity for all inpatients with CAP including those with HCAP. Higher-intensity treatment should be considered in patients with CAP and an ALL-COP SCORE threshold of ≥ 4 points.Trial registrationThis study was registered with the University Medical Information Network in Japan, registration numbers UMIN000003306 and UMIN000009837.

Weighted Scoring Rules and Convex Risk Measures

In Weighted Scoring Rules and Convex Risk Measures, Dr. Zachary J. Smith and Prof. J. Eric Bickel (both at the University of Texas at Austin) present a general connection between weighted proper scoring rules and investment decisions involving the minimization of a convex risk measure. Weighted scoring rules are quantitative tools for evaluating the accuracy of probabilistic forecasts relative to a baseline distribution. In their paper, the authors demonstrate that the relationship between convex risk measures and weighted scoring rules relates closely with previous economic characterizations of weighted scores based on expected utility maximization. As illustrative examples, the authors study two families of weighted scoring rules based on phi-divergences (generalizations of the Weighted Power and Weighted Pseudospherical Scoring rules) along with their corresponding risk measures. The paper will be of particular interest to the decision analysis and mathematical finance communities as well as those interested in the elicitation and evaluation of subjective probabilistic forecasts.