Scoring Rules Research Articles

From specific-source feature-based to common-source score-based likelihood-ratio systems: ranking the stars

This article studies expected performance and practical feasibility of the most commonly used classes of source-level likelihood-ratio (LR) systems when applied to a trace–reference comparison problem. The article compares performance of these classes of LR systems (used to update prior odds) to each other and to the use of prior odds only, using strictly proper scoring rules as performance measures. It also explores practical feasibility of the classes of LR systems. The present analysis allows for a ranking of these classes of LR systems: from specific-source feature-based to common-source anchored or non-anchored score-based. A trade-off between performance and practical feasibility is observed, meaning that the best performing class of LR systems is the hardest to realize in practice, while the least performing class is the easiest to realize in practice. The other classes of LR systems are in between the two extremes. The one positive exception is a common-source feature-based LR system, with good performance and relatively low experimental demands. This article also argues against the claim that some classes of LR systems should not be used, by showing that all systems have merit (when updating prior odds) over just using the prior odds (i.e. not using the LR system).

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

올림픽 태권도 종목 활성화를 위한 경기규칙 개선 방안 연구

The purpose of this study is to examine the overall situation of the rules of competition applied to the 32nd Tokyo Olympics Taekwondo Competition, and to explore the rules of competition matches suitable for the times. After selecting 6 people and 12 people with Taekwondo-related experience in Korea and China, through in-depth interviews, the introduction of electronic slogans/heads for Taekwondo competition, differential scoring system, stadium, referee system, and rules improvement After identifying the advantages and disadvantages of, the future improvement direction was provided.

To Subsidize Or Not to Subsidize: A Comparison of Market Scoring Rules and Continuous Double Auctions for Price Discovery

To Subsidize Or Not to Subsidize: A Comparison of Market Scoring Rules and Continuous Double Auctions for Price Discovery

Enhancing Microcomputer Edge Computing for Autonomous IoT Motion Control

Devices microprocessors, microcontrollers, and Field Programmable Gate Arrays (FPGA) play the core rule at the IoT edge level and it should be right provisioned. For proper controller performance, control algorithms should be implemented near the actuator eliminating the delay effects. In the IoT domain, this means to implement the mentioned algorithm at the edge level and prior data transmitting. The efficient IoT-enabled motion control can be obtained by considering two main factors; the first factor is from the actuator design point of view and the second factor is from the controller performance point of view. Therefore, in this article, the two mentioned factors are treated concerning the microprocessor rule and importance as a core for proper prototype design and as the main platform to implement the control algorithms. A comparison of controller performance indices for both prototypes is done using previously distributed motion control schemes and newly developed schemes after tuning the respective schemes gains in an optimal manner. The scheme with better behavior of both prototypes are selected for the IoT integration process, this scheme ensures optimal edge computing for the distributed motion control, making the implementation of all control computation take place at the IoT-edge level. As a result, the dynamic pipeline stages (DPS) based prototype gives better controller performance indices for most strategies, less power consumption, and optimally utilized resources encouraging the use of the microprocessors with reconfigurable components at the IoT-edge level.

On approximate robust confidence distributions

A confidence distribution is a complete tool for making frequentist inference for a parameter of interest based on an assumed parametric model. Indeed, it provides point estimates, along with confidence intervals, allows to define rejection regions for testing unilateral and bilateral hypotheses, to assign measures of evidence or levels of confidence to prespecified regions of the parameter space, and to compare the parameter of interest with other parameters from other studies. The aim is to discuss robust confidence distributions derived from unbiased M-estimating functions, which provide bounded-influence inference for a parameter of interest, when the assumed central model is just an approximate parametric model or in the presence of deviant values in the observed data. Paralleling likelihood-based results and extending results available for robust scoring rules, two methods are proposed for deriving robust confidence distributions: the first one uses the asymptotic theory of robust pivotal quantities and the second one is based on simulation methods. An application and a simulation study are illustrated in the context of non-inferiority and superiority testing.

Probabilistic spatiotemporal forecasting of wind speed based on multi-network deep ensembles method

Obtaining reliable and high-quality wind speed probability forecast results is of great significance to wind energy utilization and power system management. In this paper, multi-network deep ensemble method, which combines the intelligent optimization algorithm and deep ensemble method, is proposed to deal with the probabilistic prediction problems. This method can effectively integrate a variety of different deep learning neural networks and provide reliable uncertainty estimates for prediction. Furthermore, spatiotemporal multi-network deep ensemble model, which employs multi-network deep ensemble method, is proposed to deal with the probabilistic spatiotemporal wind speed forecasting problems. In the model, three advanced convolutional recurrent neural networks are integrated to capture spatiotemporal information from the underlying meteorological variables. Intelligent optimization algorithm is used to assign weights to each network in the ensemble. In addition, an uncertainty quantification method, which quantify the uncertainty by adjusting the network structure and optimize the uncertainty by utilizing the truncated negative log-likelihood scoring rule, is introduced to provide reliable probability forecasts. The proposed model is applied to a real-world case in the United States. The test results demonstrate that spatiotemporal multi-network deep ensemble model can not only provide high-precision point prediction results, but also provide suitable interval predictions and reliable probability prediction results. Moreover, the impact of input features on model prediction results is also evaluated.

U-Sleep’s resilience to AASM guidelines

AASM guidelines are the result of decades of efforts aiming at standardizing sleep scoring procedure, with the final goal of sharing a worldwide common methodology. The guidelines cover several aspects from the technical/digital specifications, e.g., recommended EEG derivations, to detailed sleep scoring rules accordingly to age. Automated sleep scoring systems have always largely exploited the standards as fundamental guidelines. In this context, deep learning has demonstrated better performance compared to classical machine learning. Our present work shows that a deep learning-based sleep scoring algorithm may not need to fully exploit the clinical knowledge or to strictly adhere to the AASM guidelines. Specifically, we demonstrate that U-Sleep, a state-of-the-art sleep scoring algorithm, can be strong enough to solve the scoring task even using clinically non-recommended or non-conventional derivations, and with no need to exploit information about the chronological age of the subjects. We finally strengthen a well-known finding that using data from multiple data centers always results in a better performing model compared with training on a single cohort. Indeed, we show that this latter statement is still valid even by increasing the size and the heterogeneity of the single data cohort. In all our experiments we used 28528 polysomnography studies from 13 different clinical studies.

Netiquette rules in online learning through the lens of digital citizenship scale in the post-corona era

PurposeThis study aims to investigate the trending term: “Netiquette” as an important element in the effective digital citizenship. The research suggests a systematic framework of netiquette rules in the field of online education based on the classical core rules of netiquette and according to the digital citizenship scale (DCS). The research also studies the corresponding responsibilities of both educators and students to raise awareness towards using technology in a balanced, safe, smart and ethical way as the shift towards the digital activities increased significantly in the post-corona time.Design/methodology/approachThe research used the qualitative data that were based on the everyday observation and analysis of the online education experience at the university of Jordan in the academic year 2020/2021; the online group discussions of students and teachers; and investigating the guidelines of the online learning netiquette rules in various academic institutes. Comparative analysis was conducted to merge and eliminate redundant rules and to add sub rules, and then to cluster them into groups. The suggested clustered groups were distributed into the classical core rules outline of netiquette. In each core rule, the sub rules were reclassified and recategorized according to the DCS by studying the complexity levels and their corresponding factors. The suggested framework updates and adds DCS levels and factors considering the exceptional experience of online education through the pandemic.FindingsThe research finds that “Netiquette” had been neglected in cyber ethics literature, and so it has to be rediscovered through the lens of digital citizenship that becomes very noticeable issue in the post-COVID era. So, the research presents a systematic framework that outlines more than 150 netiquette sub rules in the field of online education, and that were clustered according to DCS and the classical core rules of netiquette. It also adds a new factor to the bottom level of DCS which is the primarily skills and traits, and also updates the internet and political activism fac-tor by adding the social perspective.Originality/valueA novel classification of the classical core rules of netiquette was proposed in the field of online education to serve as a spectrum of identifying the complexity of digital citizenship levels and factors. This research can be a starting point of more works on netiquette research in online education and on other fields such as online business meetings, social media networking and online gaming.

Impact of correlations between prioritized outcomes onthenet benefit and its estimate by generalized pairwise comparisons.

Benefit-risk balance is gaining interest in clinical trials. For the comprehensive assessment of benefits and risks, generalized pairwise comparisons are increasingly used to estimate the net benefit based on multiple prioritized outcomes. Although previous research has demonstrated that the correlations between the outcomes impact the net benefit and its estimate, the direction and magnitude of this impact remain unclear. In this study, we investigated the impact of correlations between two binary or Gaussian variables on the true net benefit values via theoretical and numerical analyses. We also explored the impact of correlations between survival and categorical variables on the net benefit estimates based on four existing methods (Gehan, Péron, Gehan with correction, and Péron with correction) in the presence of right censoring via simulation and application to actual oncology clinical trial data. Our theoretical and numerical analyses revealed that the true net benefit values were impacted by the correlations in various directions depending on the outcome distributions. With binary endpoints, this direction was governed by a simple rule with a threshold of 50% for a favorable outcome. Our simulation showed that the net benefit estimates based on Gehan's or Péron's scoring rule could be substantially biased in the presence of right censoring, and that the direction and magnitude of this bias were associated with the outcome correlations. The recently proposed correction method greatly reduced this bias, even in the presence of strong outcome correlations. The impact of correlations should be carefully considered when interpreting the net benefit and its estimate.

Bidding on Price and Quality: An Experiment on the Complexity of Scoring Rule Auctions

Abstract We experimentally study procurement auctions when both quality and price matter. We compare two treatments where sellers compete on one dimension only (price or quality), with three treatments where sellers submit a price-quality bid and the winner is determined by a scoring rule that combines the two offers. We find that, in the scoring rule treatments, efficiency and buyer's utility are lower than predicted. Estimates from a Quantal Response Equilibrium model suggest that increasing the dimension of the strategy space imposes a complexity burden on sellers, so that a simpler mechanism like a quality-only auction may be preferable.

Local scale invariance and robustness of proper scoring rules

Averages of proper scoring rules are often used to rank probabilistic forecasts. In many cases, the individual terms in these averages are based on observations and forecasts from different distributions. We show that some of the most popular proper scoring rules, such as the continuous ranked probability score (CRPS), give more importance to observations with large uncertainty, which can lead to unintuitive rankings. To describe this issue, we define the concept of local scale invariance for scoring rules. A new class of generalized proper kernel scoring rules is derived and as a member of this class we propose the scaled CRPS (SCRPS). This new proper scoring rule is locally scale invariant and, therefore, works in the case of varying uncertainty. Like the CRPS, it is computationally available for output from ensemble forecasts, and does not require the ability to evaluate densities of forecasts. We further define robustness of scoring rules, show why this also can be an important concept for average scores unless one is specifically interested in extremes, and derive new proper scoring rules that are robust against outliers. The theoretical findings are illustrated in three different applications from spatial statistics, stochastic volatility models and regression for count data.

Probabilistic solar forecasting: Benchmarks, post-processing, verification

Probabilistic solar forecasts may take the form of predictive probability distributions, ensembles, quantiles, or interval forecasts. State-of-the-art approaches build on input from numerical weather prediction (NWP) models and post-processing with statistical and machine learning methods. We propose a probabilistic benchmark based on a deterministic forecast of clear-sky irradiance, introduce new methods for post-processing that merge statistical techniques with modern neural networks, discuss methods for spatio-temporal scenario forecasts, and illustrate the assessment of predictive ability via proper scoring rules and calibration checks. We expect future solar forecasting efforts to be increasingly probabilistic, and encourage continuing close interaction with operational weather prediction, where innovations based on sophisticated neural networks supplement and challenge traditional approaches.

A novel metric for assessing wind and solar power complementarity based on three different fluctuation states and corresponding fluctuation amplitudes

Assessing complementarity is a foundational work to combine wind and solar power to mitigate their fluctuations. Correlation coefficient is the most commonly used index to assess complementarity. But correlation coefficient mainly quantifies the synchronous and reverse correlations between wind and solar power. Moreover, it ignores the fluctuation amplitudes of wind and solar power, which would misestimate the complementarity. To address the issue, a novel complementarity index is proposed considering both the fluctuation states and corresponding fluctuation amplitudes. The present study firstly divides the fluctuations of wind and solar power into synchronous, reverse, and discrepancy fluctuation states. Then the degrees of the three different fluctuation states are obtained according to the score rules considering the fluctuation amplitudes. Finally, the novel complementarity index is jointly determined by these fluctuation degrees. Validation results show that the proposed index successfully avoids the misestimation scenarios caused by using correlation coefficient to assess complementarity. Then the proposed index is applied to analyze the complementarity of wind and solar power in China on hourly, daily, and monthly time scales. The complementarity index is generally −0.11–0 on the hourly time scale in most regions of Jilin, Heilongjiang, Liaoning, Inner Mongolia, northern Gansu, southern Xinjiang, and the North China Plain, showing greater complementarity than other onshore regions. But the complementarity index is generally 0–0.3 in most of the aforementioned regions on the monthly time scale, implying that the complementarity on the monthly time scale is smaller than that on the hourly time scale. The complementarity indices in most offshore regions are −0.1–0, −0.12–0 and −0.2–0, respectively, on the hourly, daily, and monthly time scales. Further analysis reveals that the complementarity between wind and solar power would be overestimated once the fluctuation amplitude is ignored. Additionally, the proposed complementarity index can be used to optimize the installed capacity ratio of wind and solar power in a hybrid system. The proposed complementarity metric contributes to a better and more accurate understanding of the complementarity between wind and solar power. Furthermore, the proposed metric can be readily applied to assess the complementarity of other renewable power generations.

P348 Disease trajectories in ulcerative colitis: Early identification of patients with endoscopic remission at Week 52

Abstract Background Clinical understanding of different disease courses in response to targeted therapies in ulcerative colitis (UC) is lacking. Symptom load over time can help to define different trajectories leading to outcome measures at week 52. This post hoc analysis from the GEMINI 1 study of vedolizumab (VDZ) vs placebo for the treatment of moderately to severely active UC1 aimed to define different clinical patterns of response that led to Week 52 endoscopic remission based on daily collection of symptom data: rectal bleeding (RB) and stool frequency (SF). Methods RB and SF data, each scored 0 to 3, were extracted from the diary cards of patients who were treated with VDZ up to Week 50, had clinical outcome assessed at Week 52 and had sufficient diary data. From the daily scores, six different daily symptom score rules comprising RB alone or various combinations of RB and SF (Table 1) were derived. Each of the six symptom score rules was then accrued over the time to derive cumulative symptom scores (area under the curve [AUC]). The AUCs at Weeks 6, 10, 14 and 22 for each rule were evaluated for sensitivity and specificity to predict endoscopic remission at Week 52, which was defined as Mayo endoscopic subscore of ≤1 point. Results Of the 895 pts included in the GEMINI 1 study, 217 pts were treated with VDZ, had sufficient RB and SF data and Week 52 outcome data. Of these, 157 (72.4%) achieved endoscopic remission at Week 52. Rule 4, with a score of 0 if both RB=0 and SF≤1 or 1 if not, was consistently better at predicting response at Week 52, i.e., had largest area under the receiver-operator characteristic (ROC) curve (Week 10 and Week 14 shown in Table 2). Area under ROC curve for Rule 4 was 0.7317 (0.6578, 0.8057) and 0.7534 (0.6810, 0.8258) at Week 10 and Week 14, respectively (Fig 1). Rule 4 had better predictive performance than the use of RB Mayo score (RB1) alone or the simple dichotomization (RB2, 0 if RB≤1, 1 if RB >1) [Table 2]. AUCs for Rule 4 differed between pts with vs without endoscopic remission at Week 52, with differences evident as early as Week 10: median AUC=41 for pts with endoscopic remission vs 70 for pts without (Fig 2). Conclusion The daily collection of RB and SF symptoms during the first weeks of therapy with VDZ identifies patients with a high likelihood for endoscopic emission at Week 52. Early clinical symptom change is a simple method for tight monitoring of patients with UC. The identification of preferential responder populations in which marked reduction of suffering are combined with hard endpoints could help in establishing treatment optimization algorithms for targeted therapies.

Assessing the accuracy of exponentially weighted moving average models for Value-at-Risk and Expected Shortfall of crypto portfolios

A plethora of academic papers on generalized autoregressive conditional heteroscedasticity (GARCH) models for bitcoin and other cryptocurrencies have been published in academic journals. Yet few, if indeed any, of these are employed by practitioners. Previous academic studies produce results that are fragmented, confusing and conflicting, so there is no commercial incentive to drive an expensive implementation of complex multivariate GARCH models, which anyway would commonly require more data for calibration than are available in the history of most cryptocurrencies, at least at the daily frequency. Consequently, this paper assesses the forecasting accuracy of simple parametric RiskMetrics type volatility and covariance models, with a focus on ad hoc parameter choice instead of a data-intensive calibration procedure. We provide extensive backtests of hourly and daily Value-at-Risk (VaR) and Expected Shortfall (ES) forecasts that are regarded as best practice in the industry and commonly used for regulatory approval. Our results demonstrate that much simpler models in the exponentially weighted moving average (EWMA) class are just as accurate as GARCH models for VaR and ES forecasting, provided they capture an asymmetric volatility response and a heavy-tailed returns distribution. Moreover, on ranking each model's variance and covariance forecasts using average scores generated from proper univariate and multivariate scoring rules, there is no evidence of superior performance of variance and covariance forecasts generated by GARCH models, using either daily or hourly data.

NECESSARY AND SUFFICIENT CONDITIONS FOR DOMINATION RESULTS FOR PROPER SCORING RULES

AbstractScoring rules measure the deviation between a forecast, which assigns degrees of confidence to various events, and reality. Strictly proper scoring rules have the property that for any forecast, the mathematical expectation of the score of a forecast p by the lights of p is strictly better than the mathematical expectation of any other forecast q by the lights of p. Forecasts need not satisfy the axioms of the probability calculus, but Predd et al. [9] have shown that given a finite sample space and any strictly proper additive and continuous scoring rule, the score for any forecast that does not satisfy the axioms of probability is strictly dominated by the score for some probabilistically consistent forecast. Recently, this result has been extended to non-additive continuous scoring rules. In this paper, a condition weaker than continuity is given that suffices for the result, and the condition is proved to be optimal.

From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

There are many ways to elicit honest probabilistic forecasts from experts. Once those forecasts are elicited, there are many ways to aggregate them into a single forecast. Should the choice of elicitation method inform the choice of aggregation method? In “From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation,” Neyman and Roughgarden establish a connection between these two problems. To every elicitation method they associate the aggregation method that improves as much as possible upon the forecast of a randomly chosen expert, in the worst case. This association maps the two most widely used elicitation methods (Brier and logarithmic scoring) to the two most well-known aggregation methods (linear and logarithmic pooling). The authors show a number of interesting properties of this connection, including a natural axiomatization of aggregation methods obtained through the connection, as well as an algorithm for efficient no-regret learning of expert weights.

Special Report: Measurement-Based Care—Using Data to Augment Psychotherapy

Special Report: Measurement-Based Care—Using Data to Augment Psychotherapy

Multivariate Parametric Study of Nailfold Capillary Images for Disease Detection.

Nailfold capillaroscopy is a tool which is non-invasive in nature and can be useful for diagnosis, research, therapeutic study and prognosis. Research shows that specific capillary morphology patterns are identified for diabetic subjects, hypertensive subjects and normal controls. In this study, we have proposed RATHEW approach of classifying these three classes of subjects. RATHEW approach employs a three step process for classifying nailfold images: one, identify six abnormality parameters from the image dataset; two, score these abnormality parameters based on the defined scoring rules; and three, combine them mathematically to segregate them into three classes. This technique can be further enhanced to grade the severity of disease and organ involvement. This can bring in a paradigm shift to the disease detection and therapeutic study mechanism.