Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve

Abstract

AbstractWe apply idealized scatterplot distributions to the sliding threshold of observation for numeric evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point‐spread distribution. The STONE curve is based on the relative operating characteristic (ROC) curve but is developed to work with a continuous‐valued set of observations, sweeping both the observed and modeled event identification threshold simultaneously. This is particularly useful for model predictions of time series data as is the case for much of terrestrial weather and space weather. The identical sweep of both the model and observational thresholds results in changes to both the modeled and observed event states as the quadrant boundaries shift. The changes in a data‐model pair's event status result in nonmonotonic features to appear in the STONE curve when compared to an ROC curve for the same observational and model data sets. Such features reveal characteristics in the underlying distributions of the data and model values. Many idealized data sets were created with known distributions, connecting certain scatterplot features to distinct STONE curve signatures. A comprehensive suite of feature‐signature combinations is presented, including their relationship to several other metrics. It is shown that nonmonotonic features appear if a local spread is more than 0.2 of the full domain or if a local bias is more than half of the local spread. The example of real‐time plasma sheet electron modeling is used to show the usefulness of this technique, especially in combination with other metrics.