A Hierarchical Clustering algorithm based on Silhouette Index for cancer subtype discovery from genomic data

Abstract

Identifying potential novel subtypes of cancers from genomic data requires techniques to estimate the number of natural clusters in the data. Determining the number of natural clusters in a dataset has been a challenging problem in Machine Learning. Employing an internal cluster validity index such as Silhouette Index together with a clustering algorithm has been a widely used technique for estimating the number of natural clusters, which has limitations. We propose a Hierarchical Agglomerative Clustering algorithm which automatically estimates the numbers of natural clusters and gives the associated clustering solutions along with dendrograms for visualizing the clustering structure. The algorithm has a Silhouette Index-based criterion for selecting the pair of clusters to merge, in the process of building the clustering hierarchy. The proposed algorithm could find decent estimates for the number of natural clusters, and the associated clustering solutions when applied to a collection of cancer gene expression datasets and general datasets. The proposed method showed better overall performance when compared to that of a set of prominent methods for estimating the number of natural clusters, which are used for cancer subtype discovery from genomic data. The proposed method is deterministic. It can be a better alternative to contemporary approaches for identifying potential novel subtypes of cancers from genomic data.