Detecting concept drift: An information entropy based method using an adaptive sliding window

Abstract

Concept drift in data stream poses many challenges and difficulties in mining this tradition-distinct database. In this paper, we focus on detecting concept drift in evolving data stream. We propose a novel method to detect concept drift using entropy over an adaptive sliding window. In the method, the sliding window is not fixed but dynamically determined. Another distinct property is that the method integrates an algorithm to find the exact timestamp for retraining the classifier whenever a concept drift is detected. In the experiments, we evaluate our method on publicly available data streams containing various types of concept drifts, and then compare it with four well known concept drift detection methods. The experimental results show that compared with four benchmarks, the proposed method is better than or comparable with other methods for most cases.