A Cluster Ensemble Framework for Large Data sets

Abstract

Combining multiple clustering solutions is important for obtaining a robust clustering solution, merging distributed clustering solutions, and scaling to large data sets. The combination of multiple clustering solutions within a scalable and robust framework for large data sets is discussed. A scalable framework requires both cluster ensemble creation and merging to be efficient in terms of time and memory complexity. We also introduce the concept of filtering malformed clusters from the ensemble. They result from unfortunate initialization or unbalanced data distribution or noise. Experimental results on real data sets show that this approach will scale and provide cluster partitions which are functionally better or equivalent when compared to clustering all the data at once and clustering solutions contained in the ensemble. We have also compared our algorithm with other ensemble merging and scalable algorithms to point out its strengths and limitations.