Forecasting Residential Water Consumption in California: Rethinking Model Selection

Abstract

AbstractUrban water managers use forecasts of water consumption to determine management decisions and investment choices. Public reports show that water utilities rely on forecast models that are not selected based on their out‐of‐sample prediction performance; further, these reports frequently only present a single forecast instead of a range of forecasts. In our review of the academic literature on forecasting long‐term water consumption, only a few analyses consider out‐of‐sample prediction performance measures to assess prediction ability. In none of these analyses did out‐of‐sample prediction performance drive model selection. Ensemble‐type long‐term forecasts based on multiple models were also lacking. Using annual data on single‐family residential water consumption in Southern California, we show that predictive ability varies drastically depending on how the forecast model is selected. As an illustration of how forecast performance is affected when the criteria for model selection does support forecasting objectives, we compare statistical models with the best in‐sample and out‐of‐sample prediction performance. We find that the models with the best in‐sample performance over‐estimate consumption five years out by 10%‐25% compared to actual consumption. In contrast, the top 1% of models selected based on out‐of‐sample prediction performance came within 1% of actual consumption. Finally, we compare the performance of our ensemble‐type forecasts to those reported in public documents derived from models selected based on non‐out‐of‐sample prediction performance criteria. Our results highlight the benefits of (i) using an out‐of‐sample evaluation criterion to guide model selection and (ii) reporting ensemble forecasts in lieu of a single forecast.