Uncertainty estimation for gradient boosting models based on Gaussian graph model and natural gradient

Abstract

We study the multi-output probability regression prediction and uncertainty quantification problem in multi-output regression prediction with gradient boosting models, and propose an uncertainty estimation method based on Gaussian Graph Model (GMM) constraint and natural gradient. Under the assumption of Gaussian distribution, we employ GMM to model the conditional independence relationship of the outputs, and propose a mixed parameterization to represent the subfamily of Gaussian distributions defined by the graph model. We set this mixed parameters as the learning target of the gradient boosting model, and use natural gradient in Riemann space that based on Fisher information matrix (FIM) instead of the vanilla gradient to be the steepest descent direction. The formula to compute FIM and natural gradient for the subfamily defined by GMM is deduced. Experiments show that the proposed method is able to incorporate the priors on the outputs to optimize the search space, therefore, lead to better uncertainty estimation results on multi-output regression for gradient boosting models.