Efficient Feature Clustering for High-Dimensional Datasets: A Non-Parametric Approach

Abstract

Clustering methods have become widely popular in the field of data analysis, as they enable the grouping of similar data points. However, the challenge of clustering high-dimensional data remains a significant obstacle due to the ”curse of dimensionality.” Traditional methods like Principal Component Analysis (PCA) and hierarchy-based feature agglomeration have been used to overcome this challenge by reducing the dimensionality of the dataset. However, there is currently no efficient method for clustering features of a high-dimensional dataset. To address this issue, we present a non-parametric clustering method in this paper, specifically designed for identifying features in a high-dimensional dataset. Our proposed feature clustering algorithm utilizes a partitioning algorithm that creates clusters of features. Through experimental results using publicly available datasets, we demonstrate the effectiveness of our method. Our approach offers a valuable tool for researchers and practitioners to better understand and analyze high-dimensional data.