Concept drift detection based on decision distribution in inconsistent information system

Abstract

Concept drift has been a widely concerned problem in data analysis and many important achievements have been obtained. However, there are few discussions on concept drift detection in inconsistent information system that is very common in various application fields. In this paper, a concept drift detection algorithm based on the decision distribution (IDP-DD) is proposed. To begin, we present the notion of decision distribution function by means of inclusion degree and delineate the decision empirical vector. Subsequently, we expound upon decision empirical distance and analyze the limits characteristics of sampling; thirdly, we display the algorithm implementation process of IDP-DD. Lastly, we evaluate the efficacy of IDP-DD with a some commonly employed datasets and a real-word example dataset. Theory analysis and experiment verification indicate that the algorithm IDP-DD can not only quantify the severity of concept drift, but also visually present the region where concept drift occurs. At the same time, combined with sampling method, this algorithm does not need to traverse all samples at one time, so the complexity and calculation cost can be effectively reduced. So IDP-DD is more stable and can effectively facilitate data-driven decision problems.