Detecting and rationalizing concept drift: A feature-level approach for understanding cause–effect relationships in dynamic environments

Abstract

Concept drift detection is essential for data-driven models to adapt to changing data patterns, ensuring accuracy and reliability in dynamic environments. Explaining drift can be even more important for gaining insights into root causes to facilitate informed decision-making. This paper presents a method to detect and rationalize drift, explaining it at the feature level by identifying impactful changes in the feature-outcome relationship from a cause–effect perspective. Drift occurrences therefore fall into four categories, including changes in the cause–effect relationship, comprising two subcategories: Type 1, where specific features no longer act as causes in the new environment, while Type 2 entails previously non-causal features becoming causes in the new environment. If the cause–effect relationship remains unchanged, it indicates changes in the importance of causal features (Type 3) or variations in the intensity of the cause–effect relationship (Type 4). The F1 score is employed to validate the proposed method for detecting and rationalizing recurring sudden drift in both real and virtual drift contexts. The average F1 scores on the Stagger and Agrawal datasets are 89.5% and 74.8%, respectively. The method is then applied to coalmine pressure data to investigate and rationalize changes in the causal relationship between strut pressure and microseismic energy.