Abstract
Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.
Highlights
Predictive models based on deep learning have seen a dramatic improvement in recent years [1], which has led to widespread adoption in many areas
We focus on marginal coverage over a dataset D 0 for the canonical α value of 0.05, i.e., 95% prediction intervals
Contributions: In this study, we investigate the empirical coverage properties of prediction intervals constructed from a catalog of popular uncertainty quantification techniques, such as ensembling, Monte Carlo dropout, Gaussian processes, and stochastic variational inference
Summary
Predictive models based on deep learning have seen a dramatic improvement in recent years [1], which has led to widespread adoption in many areas. The prediction sets and intervals we propose in this work are an intuitive way to quantify uncertainty in machine learning models and provide interpretable metrics for downstream, nontechnical users. For new samples beyond the first n samples in the training data, there is a 1 − α probability of the true label of the test point being contained in the set Cn ( xn+1 ). This set can be constructed using a variety of procedures. For a given model and test set D , the empirical coverage by assessing whether yn+1 ∈ Cn ( xn+1 ) ∀ xn+1 ∈ D
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have