Quantifying uncertainty of uplift: Trees and T-learners

Abstract

Uplift modeling refers to the task of estimating the causal effect of a treatment on an individual, also known as the conditional average treatment effect. However, uplift models do not usually provide uncertainty estimates of the predictions. We explain why estimating uncertainty of the treatment effect is particularly important in many common use cases and we show how epistemic uncertainty of the uplift estimates can be quantified for T-learners and trees. We tested the methods on three empirical datasets and evaluated them on a simulated dataset. We found that high uncertainty might be the result of both modeling choices and properties of the data. Sometimes there is not enough data or the data is simply not rich enough to identify the treatment effect well resulting in high uncertainty. In addition, our results suggest that one commonly used dataset might not be suitable for benchmarking.