Between a rock and a hard place: The evaluation of demographic forecasts

Abstract

Forecasting, in general, has been described as an unavoidable yet impossible task. This irony, which comprises the ‘rock’ and the ‘hard place’ in the title, creates a high level of cognitive dissonance, which, in turn, generates stress for those both making and using forecasts that have non-trivial impacts. Why? Because the forecasted numbers that are invariably accorded a high degree of precision inexorably reveal their inevitable imprecision when the numbers forming the actuality finally take place and the numbers comprising the forecast's errors are precisely measured. The current state of the art in demography for dealing with the ‘rock’ and the ‘hard place’ is a less-than-successful strategy because it is based on an acceptance of accuracy as the primary evaluation criterion, which is the source of cognitive dissonance. One way to reduce cognitive dissonance is to change the relationship of the very cognitive elements creating it. We argue that forecast evaluations currently focused on accuracy and based on measures like RMSE and MAPE be refocused to include utility and propose for this purpose the ‘Proportionate Reduction in Error’ (PRE) measure. We illustrate our proposal with examples and discuss its advantages. We conclude that including PRE as an evaluation criterion can reduce stress by reducing cognitive dissonance without, at the same time, either trivializing the evaluation process or substantively altering how forecasts are done and presented.