Data Summarization Based Fast Hierarchical Clustering Method for Large Datasets

Abstract

The hierarchical clustering methods are not scalable with the size of the dataset and need many database scans. This is potentially a severe problem for large datasets. One way to speed up the hierarchical methods is to summarize the data efficiently and subsequently apply the clustering methods to the summary of the data. In this paper, we propose a new scheme to summarize the dataset called data sphere (DS). Data sphere (DS) collects sufficient statistics applying the leaders clustering method twice on the dataset. The single-link clustering method which is a well known hierarchical clustering method is modified to work with data spheres. The proposed method is called DS-SL method.The DS-SL takes considerably less time compared to the classical single-link method which is applied on the dataset directly. The clustering results produced by DS-SL is very close to the single-link method. We also show that DS-SL outperforms the single-link method using recently proposed data bubble (DB) as a summarization scheme, both at clustering quality and execution time. Experiments are conducted with two synthetic and three real world datasets which shows effectiveness of the proposed method for large datasets.