Detecting the Change of Clustering Structure in Categorical Data Streams

Abstract

Analyzing clustering structures in data streams can provide critical information for making decision in realtime. Most research has been focused on clustering algorithms for data streams. We argue that, more importantly, we need to monitor the change of clustering structure online. In this paper, we present a framework for detecting the change of critical clustering structure in categorical data streams, which is indicated by the change of the best number of clusters (Best K) in the data stream. The framework extends the work on determining the best K for static datasets (the BkPlot method) to categorical data streams with the help of a Hierarchical Entropy Tree structure (HE-Tree). HETree can efficiently capture the entropy property of the categorical data streams and allow us to draw precise clustering information from the data stream for highquality BkPLots. The experiments show that with the combination of HE-Tree and the BkPlot method we are able to efficiently and precisely detect the change of critical clustering structure in categorical data streams.