Comments on “Finite Samples and Uncertainty Estimates for Skill Measures for Seasonal Prediction”

Abstract

Different skill measures present differing dependence on the intrinsic predictability level as well as differing dependence on sample size. Kumar (2009) computed the expected skill of idealized forecasts with varying levels of predictability using the anomaly correlation (AC), the Heidke skill score (HSS), and the ranked probability skill score (RPSS) as skill measures. An important consideration for seasonal climate forecasts, as demonstrated by Kumar (2009), is that these skill measures may vary significantly from their expected values when they are computed from relatively short forecast histories. In carrying out his skill measure characterizations, Kumar (2009) assumed that the forecast variance was equal to the climatological variance. Additionally, when computing average skill scores, forecast distributions were assumed to have identical means (signals) for a given predictability level. Here we examine some implications of those assumptions. This comment is organized as follows. In section 2 we present a perfect model framework for examining predictability. A natural requirement for predictability is that the forecast and climatological distributions be different. In the case of joint normal distributions, the existence of predictability (positive signal variance) implies that the forecast variance is less than the climatological variance. In this case, inflating the forecast variance to be equal to the climatological variance results in underconfident forecasts and lower probabilistic skill scores. In section 3 we compute the three skill scores for a given forecast signal (single initial condition) and for a set of forecasts with a specified signal to noise variance ratio (multiple initial conditions). The expected skill of a single forecast depends on both signal level and forecast variance. The AC depends on the ratio of squared signal to forecast variance, while the dependence of HSS and RPSS on signal level and forecast variance is more complex. A consequence of this functional dependence is that the AC of a set of forecasts is equal to that of a single forecast with the signal equal to the signal standard deviation. However, there is no similar relation for HSS and RPSS, and assuming a fixed signal as in Kumar (2009) generally overestimates HSS and always overestimates RPSS. We also present a useful approximation relating the RPSS and AC of a set of forecasts.