Fast and efficient Bayesian semi-parametric curve-fitting and clustering in massive data

Abstract

The problem of curve-fitting and clustering using Bayesian mixture models, treating the number of components as unknown, has received wide attention in the Bayesian statistical community. Among a number of available Bayesian methodologies specialised for the purpose, the approaches proposed in Escobar and West (1995) and Richardson and Green (1997) stand out. But in the case of massive data substantial computational challenges seem to blur the attractive theoretical advantages of such pioneering Bayesian methodologies. Based on a methodology introduced by Bhattacharya (2008), which, as we show, includes the approach of Escobar and West (1995) as a special case, we propose a very fast and efficient curve-fitting and clustering methodology. Our clustering approach is based on a new approach to analysing non-parametric posterior distributions of clusterings first proposed in Mukhopadhyay, Bhattacharya and Dihidar (2011). Significant advantages of our approach over the aforementioned established mixture modeling approaches, particularly in the case of massive data, are demonstrated theoretically and with extensive simulation studies. We also illustrate our methodologies on a real, cosmological data set consisting of 96,307 bivariate observations and demonstrate that the approach of Escobar and West (1995) is infeasible in this example and the approach of Richardson and Green (1997), although implementable, is likely to be inefficient and computationally expensive.