Root mean square error or mean absolute error? Use their ratio as well

Abstract

The key statistical properties of the Root Mean Square Error (RMSE) and the Mean Absolute Error (MAE) estimators were derived in this study for zero mean symmetric error distributions. A density function, named the Approximate Root Normal Distribution (ARND), was developed to approximate the distribution of a square root of a normal random variable. This enabled approximating the distribution of the RMSE estimator. The theoretical derivations and the demonstrations on common distributions, a benchmark time series, and a real world data set (with prediction errors generated from ANN and ARIMA models) lead to the following practically useful findings. When comparing errors having the same distribution type, RMSE was shown to be preferred for platykurtic distributions, MAE for leptokurtic distributions, and either RMSE or MAE for mesokurtic distributions. For different distribution types, however, using the two estimators alone was shown to lead to erroneous conclusions. The revelation that the estimated RMSE/MSE ratio could identify whether the errors came from platykurtic/mesokurtic/leptokurtic distributions was a useful complementary result. Comparison of errors based on, error distributions, sample size and the standard errors of the estimators, was discussed. The proposed procedure for deriving the statistical properties of the two estimators has scope for extension for other distribution types.