Scoring Rules Research Articles

Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence

Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence

On the computational completeness of several descriptional complexity restrictions of semi-conditional grammars

A semi-conditional grammar, introduced by Gheorghe Păun, is a form of regulated rewriting system where each rule consists of a context-free core rule A → w along with two strings w + , w − . The rule is applicable if w + (the positive condition) occurs as a substring of the current sentential form, but w − (the negative condition) does not. The maximum lengths i, j of the positive or negative conditional strings, respectively, give a natural measure of descriptional complexity, known as the degree of such grammars. Employing several normal form results on phrase-structure grammars as derived in particular by Viliam Geffert, we improve on previously obtained results by reducing the number of nonterminals of semi-conditional grammars of a given degree ( i , j ) while maintaining computational completeness of the said mechanisms.

Psychometric Evaluation of the Traumatic Grief Inventory-Self Report Plus (TGI-SR+) in Chinese Bereaved People.

The Traumatic Grief Inventory-Self Report Plus (TGI-SR+) measures the most recent prolonged grief disorder (PGD) symptom sets defined in the 11th edition of the International Statistical of Diseases and Related Health Problems (ICD-11) and the text revision of the fifth edition of the Diagnostical and Statistical Manual of Mental Disorders (DSM-5-TR). However, the TGI-SR+ has not yet been translated and validated in Chinese. This study aims to evaluate the psychometric properties of the Chinese translation of the TGI-SR+. We examined the Chinese TGI-SR+'s factor structure, internal consistency, convergent validity, discriminant validity, known-groups validity, and optimal clinical cut-off scores in 443 Chinese bereaved adults. Confirmatory factor analyses showed that the two-factor models showed the best fit for the Chinese TGI-SR+ items assessing ICD-11 and DSM-5-TR prolonged grief symptoms. Items assessing ICD-11 and DSM-5-TR prolonged grief symptoms demonstrated good internal consistency. Associations of TGI-SR+ scores with symptom levels of prolonged grief (assessed by the International Prolonged Grief Disorder Scale), posttraumatic stress, anxiety, and depression supported convergent and discriminant validity. Associations with background/loss-related variables provided evidence for known-groups validity. Cut-off points for probable ICD-11 PGD (liberal scoring rule), probable ICD-11 PGD (conservative scoring rule), and probable DSM-5-TR PGD were ≥67, ≥75, and ≥68, respectively. The Chinese TGI-SR+ appears to be a reliable and valid measure to assess prolonged grief symptoms per ICD-11 and DSM-5-TR among Chinese bereaved adults.

The Norwegian traumatic grief inventory-self report plus (TGI-SR+): a psychometric evaluation in traumatically bereaved people

ABSTRACT Background: Prolonged grief disorder (PGD) has been added to the ICD-11 and DSM-5-TR. The Traumatic Grief Inventory-Self Report Plus (TGI-SR+) assesses self-rated PGD intensity as defined in ICD-11 and DSM-5-TR. The TGI-SR + is available in multiple languages, but has not been validated yet in Norwegian. Objective: The aim of this study was to evaluate the psychometric properties of the Norwegian TGI-SR + . Method: Bereaved adults (N = 307) whose child or sibling died ≥6 months ago due to a sudden or violent loss completed the TGI-SR + and measures for posttraumatic stress, depression, and precursor PGD symptoms. We examined the factor structure and internal consistency of the ICD-11 and DSM-5-TR PGD items. Convergent validity and known-groups validity was evaluated. Probable PGD cases, pair-wise agreement between diagnostic scoring rules for both PGD criteria-sets, and cut-off scores were calculated. Results: The 1-factor model for ICD-11 and DSM-5-TR PGD showed the best fit and demonstrated good internal consistency. Convergent validity was supported by strong associations between summed ICD-11 and DSM-5-TR PGD scores and summed posttraumatic stress, depression, and precursor prolonged grief scores. Known-groups validity was supported by PGD intensity being related to educational level and time since loss. The perfect pair-wise agreement was reached using the ICD-11 and DSM-5-TR PGD diagnostic scoring rules. The optimal cut-off score for detecting probable PGD cases, when summing all TGI-SR + items, was ≥73. Conclusions: The Norwegian TGI-SR + seems a valid and reliable instrument to assess ICD-11 and DSM-5-TR PGD intensity after losing a child or sibling under traumatic circumstances.

Acute pulmonary embolism pretest probability estimation by d-dimer test, our modified, new ECG score and clinical prediction rules

ObjectivesWe investigated whether a sufficiently sensitive D-dimer test could exclude acute pulmonary embolism (acPE) as a stand-alone diagnostic test and compared our previously published, modified ECG score with the Wells and Geneva scores in the estimation of acPE pretest probability. MethodsWe retrospectively evaluated 345 patients who underwent chest CT angiography (CTA) for the suspicion of acPE. The pretest probability of acPE was assessed in 120 D-dimer negative [DD (−)] and 225 D-dimer positive [DD (+)] patients. ResultsChest CTA verified acPE in 57/345 (16.5 %) patients and in 1/120 (0.8 %) DD (−) patient. In DD (−) patients the test accuracy (TA) and specificity (SP) of the ECG score (98 %, 99 %) were better than those of the Wells score (92.5 %, 92.4 %) (p = 0.063 and p < 0.05 respectively) and the Geneva score (76.7 %, 76.5 %) (p < 0.001 for both), the Wells score TA and SP were greater than those of the Geneva score (p < 0.001 for both). In DD (+) patients the SPs, TAs and positive predictive values (PPV) of the ECG score (94 %, 78.6 %, 69 %) and the Wells score (91.8 %, 75.1 %, 48 %) were greater than those of the Geneva score (71.3 %, 64.9 %, 38.2 %) (p < 0.001 for both SP and TA respectively, and p < 0.001 for PPV of the ECG score vs. the Geneva score and p < 0.05 for PPV of the Wells score vs. Geneva score), their sensitivities (SE) (36.4 %, 23.6 %) were less than that of the Geneva score (47.5 %) (p < 0.05 and p < 0.001 respectively). The ECG score's TA in a trend, its SE and PPV were significantly (p < 0.01 and p < 0.001) better than those of the Wells score. ConclusionIn contrast to the current guidelines, a stand-alone high sensitivity DD (−) test, without prediction rules, could reliably exclude acPE. Our ECG score slightly outperformed the Wells score, the ECG score and Wells score far outperformed the Geneva score in the estimation of acPE pretest probability. An acPE diagnosis with the ECG score, in addition to the supportive diagnosis with the clinical prediction rules, may further increase the chance of true DD positivity.

Dungeons and Dragons meet PowerPoint: an interactive ICT tabletop experience for EFL learners

Background: While wondering and experimenting with different ways to incorporate creative thinking and students’ interests into my English as a foreign language (EFL) lessons, I developed a Tabletop role-playing game (TRPG) that uses PowerPoint and Dungeons and Dragons (D&amp;D) rules/materials to immerse students in an adventure while at the same time developing their English proficiency and confidence. Aim: To craft a straightforward yet enjoyable TRPG accessible to English learners of all levels, fostering the development of their reading, writing, listening, speaking, and critical/creative thinking skills. Methods: My TRPG, hereinafter referred to as EL Adventures, takes the core rules and concepts of D&amp;D and simplifies them, allowing teachers (the Dungeon Masters) to primarily focus on their students (the players) actively participating and using English. The PowerPoint presentations not only enhance the comprehensibility and fun factor of the story for students through pictures, animations, sounds, and text but also alleviate the teacher’s need to memorize the story text. While dice play a pivotal role in determining certain outcomes, the student’s success in the game hinges on direct communication with the teacher, teamwork, and creative problem-solving as they confront challenge after challenge of varying difficulty levels. Results: EL Adventures provided my EFL students with opportunities to practice reading, writing, and speaking English through intricate storylines, character creation, epic battles, and a variety of challenges. Conclusion: Not all lessons can be taught from a textbook; sometimes they are taught on a noble quest in a fantasy world.

A User-Focused Approach to Evaluating Probabilistic and Categorical Forecasts

Abstract A user-focused verification approach for evaluating probability forecasts of binary outcomes (also known as probabilistic classifiers) is demonstrated that (i) is based on proper scoring rules, (ii) focuses on user decision thresholds, and (iii) provides actionable insights. It is argued that when categorical performance diagrams and the critical success index are used to evaluate overall predictive performance, rather than the discrimination ability of probabilistic forecasts, they may produce misleading results. Instead, Murphy diagrams are shown to provide a better understanding of the overall predictive performance as a function of user probabilistic decision threshold. We illustrate how to select a proper scoring rule, based on the relative importance of different user decision thresholds, and how this choice impacts scores of overall predictive performance and supporting measures of discrimination and calibration. These approaches and ideas are demonstrated using several probabilistic thunderstorm forecast systems as well as synthetic forecast data. Furthermore, a fair method for comparing the performance of probabilistic and categorical forecasts is illustrated using the fixed risk multicategorical (FIRM) score, which is a proper scoring rule directly connected to values on the Murphy diagram. While the methods are illustrated using thunderstorm forecasts, they are applicable for evaluating probabilistic forecasts for any situation with binary outcomes. Significance Statement Recently, several papers have presented verification results for probabilistic forecasts using so-called categorical performance diagrams, which summarize multiple verification metrics. While categorical performance diagrams measure discrimination ability, we demonstrate how they can potentially lead to incorrect conclusions when evaluating overall predictive performance of probabilistic forecasts. By reviewing recent advances in the statistical literature, we show a comprehensive approach for the meteorological community that (i) does not reward a forecaster who “hedges” their forecast, (ii) focuses on the importance of the forecast user’s decision threshold(s), and (iii) provides actionable insights. Additionally, we present an approach for fairly comparing the skill of categorical forecasts to probabilistic forecasts.

Safety of treating acute pulmonary embolism at home: an individual patient data meta-analysis.

Home treatment is considered safe in acute pulmonary embolism (PE) patients selected by a validated triage tool (e.g. simplified PE severity index score or Hestia rule), but there is uncertainty regarding the applicability in underrepresented subgroups. The aim was to evaluate the safety of home treatment by performing an individual patient-level data meta-analysis. Ten prospective cohort studies or randomized controlled trials were identified in a systematic search, totalling 2694 PE patients treated at home (discharged within 24 h) and identified by a predefined triage tool. The 14- and 30-day incidences of all-cause mortality and adverse events (combined endpoint of recurrent venous thromboembolism, major bleeding, and/or all-cause mortality) were evaluated. The relative risk (RR) for 14- and 30-day mortalities and adverse events is calculated in subgroups using a random effects model. The 14- and 30-day mortalities were 0.11% [95% confidence interval (CI) 0.0-0.24, I2 = 0) and 0.30% (95% CI 0.09-0.51, I2 = 0). The 14- and 30-day incidences of adverse events were 0.56% (95% CI 0.28-0.84, I2 = 0) and 1.2% (95% CI 0.79-1.6, I2 = 0). Cancer was associated with increased 30-day mortality [RR 4.9; 95% prediction interval (PI) 2.7-9.1; I2 = 0]. Pre-existing cardiopulmonary disease, abnormal troponin, and abnormal (N-terminal pro-)B-type natriuretic peptide [(NT-pro)BNP] at presentation were associated with an increased incidence of 14-day adverse events [RR 3.5 (95% PI 1.5-7.9, I2 = 0), 2.5 (95% PI 1.3-4.9, I2 = 0), and 3.9 (95% PI 1.6-9.8, I2 = 0), respectively], but not mortality. At 30 days, cancer, abnormal troponin, and abnormal (NT-pro)BNP were associated with an increased incidence of adverse events [RR 2.7 (95% PI 1.4-5.2, I2 = 0), 2.9 (95% PI 1.5-5.7, I2 = 0), and 3.3 (95% PI 1.6-7.1, I2 = 0), respectively]. The incidence of adverse events in home-treated PE patients, selected by a validated triage tool, was very low. Patients with cancer had a three- to five-fold higher incidence of adverse events and death. Patients with increased troponin or (NT-pro)BNP had a three-fold higher risk of adverse events, driven by recurrent venous thromboembolism and bleeding.

Validation of point process predictions with proper scoring rules

AbstractWe introduce a class of proper scoring rules for evaluating spatial point process forecasts based on summary statistics. These scoring rules rely on Monte Carlo approximations of expectations and can therefore easily be evaluated for any point process model that can be simulated. In this regard, they are more flexible than the commonly used logarithmic score and other existing proper scores for point process predictions. The scoring rules allow for evaluating the calibration of a model to specific aspects of a point process, such as its spatial distribution or tendency toward clustering. Using simulations, we analyze the sensitivity of our scoring rules to different aspects of the forecasts and compare it to the logarithmic score. Applications to earthquake occurrences in northern California, United States and the spatial distribution of Pacific silver firs in Findley Lake Reserve in Washington highlight the usefulness of our scores for scientific model selection.

Parameterized complexity of candidate nomination for elections based on positional scoring rules

Consider elections where the set of candidates is partitioned into parties, and each party must nominate exactly one candidate. The Possible President problem asks whether some candidate of a given party can become the unique winner of the election for some nominations from other parties. We perform a multivariate computational complexity analysis of Possible President for several classes of elections based on positional scoring rules. We consider the following parameters: the size of the largest party, the number of parties, the number of voters and the number of voter types. We provide a complete computational map of Possible President in the sense that for each choice of the four possible parameters as (i) constant, (ii) parameter, or (iii) unbounded, we classify the computational complexity of the resulting problem as either polynomial-time solvable or NP-complete, and for parameterized versions as either fixed-parameter tractable or W[1]-hard with respect to the parameters considered.

Web Application Firewall (WAF) Design to Detect and Anticipate Hacking in Web-Based Applications

Data leakage cases have recently been rampant in Indonesia. One of the biggest is the leak of user data from BPJS Health in 2021, this data leak is certainly very detrimental to users. This research develops a Web Application Firewall (WAF) using ModSecurity and OWASP Core Rule Set to protect web applications from SQL Injection and XSS attacks. The methodology involves analyzing the functionality of the existing system using UML, with DVWA and WordPress as test objects. Results showed 100% SQL Injection and 99.8% XSS attack detection, with logs recording attacks in real-time. The findings emphasize the importance of WAF integration with web application built-in security, making significant contributions in the design and implementation of resilient WAFs, as well as improving resilience against evolving cyber threats.

Distribution‐Based Model Evaluation and Diagnostics: Elicitability, Propriety, and Scoring Rules for Hydrograph Functionals

AbstractDistribution forecasts P over future quantities or events are routinely made in hydrology but usually traded for a (likelihood‐weighted) mean or median prediction to accommodate error measures or scoring functions such as the mean absolute error or mean squared error. Case in point is the so‐called KG efficiency (KGE) of Gupta et al. (2009, https://doi.org/10.1016/j.jhydrol.2009.08.003) and improvements thereof (Lamontagne et al., 2020, https://doi.org/10.1029/2020wr027101), which have rapidly gained popularity among hydrologists as alternative scoring functions to the commonly used Nash and Sutcliffe (1970, https://doi.org/10.1016/0022‐1694(70)90255‐6) efficiency, but are equally exclusive in how they quantify model performance using only single‐valued output of the quantities of interest. This point‐valued mapping necessarily implies a loss of information about model performance. This paper advocates the use of probabilistic watershed model training, evaluation and diagnostics. Distribution evaluation opens a mature literature on scoring rules whose strong statistical underpinning provides, as we will demonstrate, the theory, context and guidelines necessary for the development of robust information‐theoretically principled metrics for watershed signatures. These so‐called hydrograph functionals are scalar‐valued mappings of major behavioral watershed functions embodied in a strictly proper scoring rule. We discuss past developments that led to the current state‐of‐the‐art of distribution evaluation in hydrology and review scoring rules for dichotomous and categorical events, quantiles (intervals) and density forecasts. We are particularly concerned with elicitable functionals and scoring rule propriety, discuss the decomposition of scoring rules into a sharpness, reliability and entropy term and present diagnostically appealing strictly proper divergence scores of hydrograph functionals for flood frequency analysis, flow duration and recession curves. The usefulness and power of distribution‐based model evaluation and diagnostics by means of scoring rules is demonstrated on simple illustrative problems and discharge distributions simulated with watershed models using random sampling and Bayesian model averaging. The presented theory (a) enables a more complete evaluation of distribution forecasts, (b) offers a statistically principled means for watershed model training, evaluation, diagnostics and selection using hydrograph functionals and/or extreme events and (c) provides a universal framework for metric development of watershed signatures, promoting metric standardization and reproducibility.

Recent Trends in Information Elicitation

This note provides a survey for the Economics and Computation community of some recent trends in the field of information elicitation. At its core, the field concerns the design of incentives for strategic agents to provide accurate and truthful information. Such incentives are formalized as proper scoring rules , and turn out to be the same object as loss functions in machine-learning settings, providing many connections. More broadly, the field concerns the design of mechanisms to obtain information from groups of agents and aggregate it or use it for decision making. Recently, work on information elicitation has expanded and been connected to online no-regret learning, mechanism design, fair division, and more.

Neighborhood Pattern Is Crucial for Graph Convolutional Networks Performing Node Classification.

Graph convolutional networks (GCNs) are widely believed to perform well in the graph node classification task, and homophily assumption plays a core rule in the design of previous GCNs. However, some recent advances on this area have pointed out that homophily may not be a necessity for GCNs. For deeper analysis of the critical factor affecting the performance of GCNs, we first propose a metric, namely, neighborhood class consistency (NCC), to quantitatively characterize the neighborhood patterns of graph datasets. Experiments surprisingly illustrate that our NCC is a better indicator, in comparison to the widely used homophily metrics, to estimate GCN performance for node classification. Furthermore, we propose a topology augmentation graph convolutional network (TA-GCN) framework under the guidance of the NCC metric, which simultaneously learns an augmented graph topology with higher NCC score and a node classifier based on the augmented graph topology. Extensive experiments on six public benchmarks clearly show that the proposed TA-GCN derives ideal topology with higher NCC score given the original graph topology and raw features, and it achieves excellent performance for semi-supervised node classification in comparison to several state-of-the-art (SOTA) baseline algorithms.

Research on content analysis and quality evaluation for online reviews of new energy vehicles

The development of China's new energy vehicle industry has been gradually transformed from past policy support to the survival of enterprises determined by the market, which forces the new energy vehicle industry to accelerate towards the market-oriented direction. We collected review data from online automotive forums and generated a corpus after pre-processing. Next, we extracted the statistical and topic semantic features of online reviews for new energy vehicles and clarified the quality evaluation indicators for online reviews. Then, the entropy weight-TOPSIS method was used to determine the indicators’ weights and assign values to the quality of online reviews. Finally, the quality scoring rules of online reviews were sorted out from multidimensional perspectives. The research results have shown that the statistical features and topic semantic features of online reviews on new energy vehicles can be used as evaluation indicators to construct a quality evaluation system. It was also found that the quality scores of lower-priced vehicle reviews tend to be consistent in different price groups, while the quality scores of mid-priced vehicle reviews are the most dispersed. The quality of online reviews varies significantly among other groups or models. The study results provide a simple criterion for consumers’ cognition of the value of online reviews. The results will enhance the ability of new energy vehicle companies to carry out online marketing with the help of online reviews and transform the purchase intention of potential consumers into actual purchase behavior.

Knowing that you know that you know? An extreme-confidence heuristic can lead to above-chance discrimination of metacognitive performance.

In daily life, we can not only estimate confidence in our inferences ('I'm sure I failed that exam'), but can also estimate whether those feelings of confidence are good predictors of decision accuracy ('I feel sure I failed, but my feeling is probably wrong; I probably passed'). In the lab, by using simple perceptual tasks and collecting trial-by-trial confidence ratings visual metacognition research has repeatedly shown that participants can successfully predict the accuracy of their perceptual choices. Can participants also successfully evaluate 'confidence in confidence' in these tasks? This is the question addressed in this study. Participants performed a simple, two-interval forced choice numerosity task framed as an exam. Confidence judgements were collected in the form of a 'predicted exam grade'. Finally, we collected 'meta-metacognitive' reports in a two-interval forced-choice design: trials were presented in pairs, and participants had to select that in which they thought their confidence (predicted grade) best matched their accuracy (actual grade), effectively minimizing their quadratic scoring rule (QSR) score. Participants successfully selected trials on which their metacognition was better when metacognitive performance was quantified using area under the type 2 ROC (AUROC2) but not when using the 'gold-standard' measure m-ratio. However, further analyses suggested that participants selected trials on which AUROC2 is lower in part via an extreme-confidence heuristic, rather than through explicit evaluation of metacognitive inferences: when restricting analyses to trials on which participants gave the same confidence rating AUROC2 no longer differed as a function of selection, and likewise when we excluded trials on which extreme confidence ratings were given. Together, our results show that participants are able to make effective metacognitive discriminations on their visual confidence ratings, but that explicit 'meta-metacognitive' processes may not be required.

Building Trees for Probabilistic Prediction via Scoring Rules

Decision trees built with data remain in widespread use for nonparametric prediction. Predicting probability distributions is preferred over point predictions when uncertainty plays a prominent role in analysis and decision-making. We study modifying a tree to produce nonparametric predictive distributions. We find the standard method for building trees may not result in good predictive distributions and propose changing the splitting criteria for trees to one based on proper scoring rules. Analysis of both simulated data and several real datasets demonstrates that using these new splitting criteria results in trees with improved predictive properties considering the entire predictive distribution.

Towards explainable traffic signal control for urban networks through genetic programming

The increasing number of vehicles in urban areas draws significant attention to traffic signal control (TSC), which can enhance the efficiency of the entire network by properly switching the phases of each signalized intersection. Fixed and max-pressure methods are commonly used in TSC systems owing to their high simplicity and good interpretability, but they respectively lack dynamic adaptability and automatic rule generation, possibly leading to low solution accuracy in complicated traffic environments. Meanwhile, meta-heuristic and black-box learning methods meet challenges in practice such as extensive computational time and poor interpretability. To this end, this paper proposes a new TSC method based on Genetic Programming (GP) to generate descriptive score rules automatically for switching phases of all signalized intersections in an urban transportation network. In the proposed method, switching phases of each signalized intersection type is formulated as a symbolic regression problem, and effective primitives are defined to facilitate GP to solve the problem. Experiments have been conducted on both synthetic and real-world networks. The results have validated the effectiveness of our proposed GP based method compared to several state-of-the-art TSC methods in terms of accuracy and interpretability.

SUSTAINABILITY TRANSPARENCY IN SOVEREIGN WEALTH FUNDS: BENCHMARK AND CASE STUDY APPROACH

The COVID-19 pandemic and the current food, energy, and safety crisis caused by the war in Ukraine strongly affected the existing gap in Sustainable Development Goals funding. Restoring global sustainability progress and the 2030 Agenda requires reshaping responsible investment architecture and rethinking the role of sovereign wealth funds as the largest investors in this process.The current study explores the ways in which the most transparent sovereign wealth funds act towards sustainability values and incorporate environmental, social, governance, and ethical criteria (ESGE) into their investment strategy, portfolio, and reporting. Also, 16 high-ranked funds by the Sustainability Transparency Index (Cavagnetto et al., 2022) were investigated by using the case study and benchmark analysis. Study results demonstrate the similarity of all funds disclosure within some questionnaire criteria like sustainability policy, scoring (voting) rules for decision-making, ESGE criteria, stakeholder engagements, and assurance of the financial information disclosed. Nevertheless, qualitative differences were identified between the sustainability transparency criteria of A and B-ranked funds. There is also huge variation in the implementation of funds’ ethical criteria, sustainability coherence, relevance of sustainability goals, disclosure of specific climate-related goals, and foundation membership criteria.Policy implications derived from the differences in sovereign funds' benchmarks are aimed at enhancing the sustainability context in funds’ operating frameworks, developing impact measurement metrics, prioritizing the most material stakeholder requests, sustainability verification procedures, and standardizing funds’ sustainability reporting. Specific channels, instruments, and regulatory measures towards greater funds’ sustainability transparency and their involvement in bridging the sustainable development funding gap may be the subject of further investigation.

Scoring and ranking probabilistic seismic hazard models: an application based on macroseismic intensity data

Abstract. A probabilistic seismic hazard model consists of a set of weighted models/branches that describes the center, the body and the range of seismic hazard. Owing to the intrinsic nature of this kind of analysis, the weight of each model/branch represents its scientific credibility. However, practical uses of this model may sometimes require the selection of one or a few hazard curves that are sampled from the whole model, which often consists of thousands of branches. Here we put forward an innovative procedure that facilitates the scoring, ranking and selection of the hazard curves to account for the requirements of a specific application. The approach consists of a careful quality check of the data used for scoring and the adoption of a proper scoring rule. To show the applicability of this approach, we present an example that consists of scoring and ranking a set of multiple models/branches constituting a recent seismic hazard model of Italy. To score these branches, hazard estimates produced by each of them are compared with time series of macroseismic observations available in the Italian macroseismic database for a carefully selected set of localities deemed sufficiently representative, homogeneously distributed in space and complete with respect to time and intensity levels. The proper scoring parameter used for such a comparison is the logarithmic score, which can always be applied independently of the distribution of the data.