On finding the number of clusters

Abstract

We present a novel approach to finding the number of clusters in data based on the minimization of a regularized cost function. Minimization of the proposed cost function results in the minimization of the sum-of-squared distances of the data points from the respective nearest cluster center as well as the sum-of-squared distances of the individual cluster centers from neighborhood cluster centers. Smaller values of the neighborhood encourage the formation of more distinct cluster centers, while larger values of the neighborhood encourage the formation of fewer distinct cluster centers. We identify the neighborhood as a scale parameter and obtain the number of cluster centers at varying values of the scale parameter. The number of cluster centers in the data is then obtained based on persistence over the largest range of the scale parameter. Four simulations are presented to illustrate the efficacy of the proposed algorithm.