Robust PAC m : Training Ensemble Models Under Misspecification and Outliers.

Abstract

Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. Probably approximately correct (PAC)-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data are affected by outliers. In recent work, PAC-Bayes bounds-referred to as PAC m -were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC m ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification-with respect to both likelihood and prior distribution-and outliers.