Towards a Better Uncertainty Quantification in Automated Valuation Models

Abstract

AbstractThis study introduces a novel framework for quantifying prediction uncertainty in automated valuation models (AVMs), crucial tools in modern real estate finance. While non-linear AVMs excel in predictive performance, their limited methods for assessing prediction uncertainty reduces reliability and practical utility. We address this gap by proposing an approach for quantifying the uncertainty associated with predicted house prices and by introducing a model-specific AVM uncertainty estimate (AVMU) for AVM comparisons. Using a dataset of 51,747 historical apartment transactions in Oslo, Norway, we train three AVMs (XGBoost, random forest, support vector machine) to predict sales prices. Thereafter, we develop three base uncertainty estimators (direct loss estimation, bootstrap ensemble, quantile regression) and three meta estimators (average regressor, voting regressor, stacked generalization) for uncertainty quantification. Conformal calibration aligns the outputted uncertainty estimates from the six estimators with standard deviations of corresponding prediction errors. Having strong positive correlations with observed absolute prediction errors, the calibrated uncertainty estimators are shown to effectively capture prediction uncertainty. While the direct loss estimation excels among base estimators, the voting regressor and stacked generalization meta estimators consistently outperform it. Furthermore, by using the AVMU estimate from the stacked generalization meta estimator we can successfully identify the best-performing AVM for three separate apartment portfolios without knowing true sales prices. This alignment of the mean estimated prediction uncertainty with observed deviations underlines the utility of pre-factual AVMU estimates for model comparisons. In conclusion, our framework helps bridge prediction accuracy and uncertainty for AVMs, enhancing their reliability and supporting informed decision making for stakeholders.