Cluster validation for subspace clustering on high dimensional data

Abstract

As an important issue in cluster analysis, cluster validation is the process of evaluating performance of clustering algorithms under varying input conditions. Many existing methods address clustering results of low-dimensional data. This paper presents new solution to the problem of cluster validation for subspace clustering on high dimensional data. We first propose two new measurements for the intra-cluster compactness and inter-cluster separation of subspace clusters. Based on these measurements and the conventional indices, three new cluster validity indices that can be applied to subspace clustering are presented. Combining with a soft subspace clustering algorithm, the new indices are used to determine the number of clusters in high dimensional data. The experimental results on synthetic and real world datasets have shown their effectiveness.