Abstract
We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods—Bayesian neural networks (BNNs), concrete dropout (CD), and deep ensembles (DEs) — are compared to the standard analytic error propagation. We discuss this comparison in terms endemic to both machine learning (‘epistemic’ and ‘aleatoric’) and the physical sciences (‘statistical’ and ‘systematic’). The comparisons are presented in terms of simulated experimental measurements of a single pendulum—a prototypical physical system for studying measurement and analysis techniques. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.
Highlights
This manuscript has been authored by Fermi Research Alliance, LLC under Contract No
Summary
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.