On the validity of MAPE as a measure of population forecast accuracy

Abstract

The mean absolute percent error (MAPE) is the summary measure most often used for evaluating the accuracy of population forecasts. While MAPE has many desirable criteria, we argue from both normative and relative standpoints that the widespread practice of exclusively using it for evaluating population forecasts should be changed. Normatively, we argue that MAPE does not meet the criterion of validity because as a summary measure it overstates the error found in a population forecast. We base this argument on logical grounds and support it empirically, using a sample of population forecasts for counties. From a relative standpoint, we examine two alternatives to MAPE, both sharing with it, the important conceptual feature of using most of the information about error. These alternatives are symmetrical MAPE (SMAPE) and a class of measures known as M-estimators. The empirical evaluation suggests M-estimators do not overstate forecast error as much as either MAPE or SMAPE and are, therefore, more valid measures of accuracy. We consequently recommend incorporating M-estimators into the evaluation toolkit. Because M-estimators do not meet the desired criterion of interpretative ease as well as MAPE, we also suggest another approach that focuses on nonlinear transformations of the error distribution.