Exploring Contingency Skill Scores Based on Event Sizes

Abstract

AbstractSpace weather forecasts are generally made for events with an arbitrary size threshold imposed on an event statistical size distribution which is likely described by a power law. This is the case for solar energetic (E > 10 MeV) particle (SEP) events, which have a differential power law exponent of γ = 1.2. Event forecasts are usually evaluated by skill scores using a contingency table that matches the forecasted events against observed events independently of the event sizes. Each observed event is either a forecasted hit or a miss, and each forecasted event is either an observed hit or a false alarm. However, for SEP events and most other space weather parameters the event size is a critical factor for the user. It is more important that large events be well forecasted than threshold events. In addition, false alarms may be useful when they match observed events just below the forecast threshold. We explore a forecast evaluation scheme to incorporate the event size within the usual format of a binary contingency table to evaluate model performance. The scheme is applied to three different input options of a recently published evaluation of the Proton Prediction System (PPS) for SEP events to show differences between numbers‐based and intensity‐based skill scores of the PPS. We demonstrate how identical skill scores can result from models with extremely different performances of event intensity forecasts. The scheme requires model validation and would benefit from testing with other space weather applications.