Parameter-Centric Analysis Grossly Exaggerates Certainty

Abstract

The reason probability models are used is to characterize uncertainty in observables. Typically, certainty in the parameters of fitted models based on their parametric posterior distributions is much greater than the predictive uncertainty of new (unknown) observables. Consequently, when model results are reported, uncertainty in the observable should be reported and not uncertainty in the parameters of these models. If someone mistook the uncertainty in parameters for uncertainty in the observable itself, a large mistake would be made. This mistake is exceedingly common, and almost exclusive in some fields. Reported here are some possible measures of the over-certainty mistake made when parametric uncertainty is swapped with observable uncertainty.