A two-layer aggregation model with effective consistency for large-scale Gaussian process regression

Abstract

To scale full Gaussian process (GP) to large-scale data sets, aggregation models divide the dataset into independent subsets for factorized training, and then aggregate predictions from distributed experts. Some aggregation models have been able to produce consistent predictions which converge to the latent function when data size approaches infinity. However, these consistent predictions will become ineffective due to the limited subset size of experts. Oriented by the transition from theory to practice, the key idea is using Generalized Robust Bayesian Committee Machine (GRBCM) with corrective function to replace experts of Generalized Product of Experts (GPoE) which focuses on global information, in order to get rid of the limitation of the experts’ size. Such a nested two-layer structure enables the proposed Generalized Product of Generalized Robust Bayesian Committee Machine (GPoGRBCM) to provide effective predictions on large-scale datasets and to inherit virtues of aggregations, e.g., a slightly flawed Bayesian inference framework, distributed/parallel computing. Furthermore, we perform comparisons of GPoGRBCM against the state-of-the-art aggregation models on one toy example and six real-world datasets with up to more than 3 million training points, showing dramatic performance improvement on scalability, capability, controllability, and robustness.