Dimensional scalability of supervised and unsupervised concept drift detection: An empirical study

Abstract

Big Data presents challenges for predictive analytic algorithms due to the possibility of non-stationary populations. Concept drift detection algorithms can be used to detect changes in underlying distribution in order to retrain. Most concept drift detection methods are known to scale to a relatively low number of features (a few hundred). However, in many areas, datasets with thousands or even tens of thousands of features are becoming common. This paper studies the behavior of supervised concept drift detection algorithms (Drift Detection Method (DDM), Early Drift Detection Method (EDDM)) and unsupervised algorithm (Friedman and Rafsky's algorithm) on high-dimensional datatsets. Our goal was to find if these algorithms can scale, first by studying the growth of execution time with the dimension of the dataset, and second by studying their comparative accuracy on high-dimensional datasets. The algorithms were run on datasets consisting of up to 100,000 features. Results show a linear growth of the execution time with respect to the dimension in each algorithm. The performance of unsupervised algorithm degraded significantly on datasets close to 100,000 dimensions. Our results also show that the drift detection accuracy of the three algorithms did not degrade as the number of features increase.