Concept Drift Detection Research Articles

Intrusion detection based on concept drift detection and online incremental learning

Purpose The primary purpose of this paper is to introduce the drift detection method-online random forest (DDM-ORF) model for intrusion detection, combining DDM for detecting concept drift and ORF for incremental learning. The paper addresses the challenges of dynamic and nonstationary data, offering a solution that continuously adapts to changes in the data distribution. The goal is to provide effective intrusion detection in real-world scenarios, demonstrated through comprehensive experiments and evaluations using Apache Spark. Design/methodology/approach The paper uses an experimental approach to evaluate the DDM-ORF model. The design involves assessing classification performance metrics, including accuracy, precision, recall and F-measure. The methodology integrates Apache Spark for distributed computing, using metrics such as processed records per second and input rows per second. The evaluation extends to the analysis of IP addresses, ports and taxonomies in the MAWILab data set. This comprehensive design and methodology showcase the model’s effectiveness in detecting intrusions through concept drift detection and online incremental learning on large-scale, heterogeneous data. Findings The paper’s findings reveal that the DDM-ORF model achieves outstanding classification results with 99.96% accuracy, demonstrating its efficacy in intrusion detection. Comparative analysis against a convolutional neural network-based model indicates superior performance in anomalous and suspicious detection rates. The exploration of IP addresses, ports and taxonomies uncovers valuable insights into attack patterns. Apache Spark evaluation attests to the system’s high processing rates. The study emphasizes the scalability, availability and fault tolerance of DDM-ORF, making it suitable for real-world scenarios. Overall, the paper establishes the model’s proficiency in handling dynamic, nonstationary data for intrusion detection. Research limitations/implications The research acknowledges certain limitations, including the potential challenge of DDM detecting only frequency changes in class labels and not complex concept drifts. The incremental random forest’s reliance on memory may pose constraints as the forest size increases, potentially leading to overfitting. Addressing these limitations could involve exploring alternative concept drift detection algorithms and implementing ensemble pruning techniques for memory efficiency. Further research avenues may investigate algorithms balancing accuracy and memory usage, such as compressed random forests, to enhance the model’s effectiveness in evolving data environments. Practical implications The study’s practical implications are noteworthy. The proposed DDM-ORF model, designed for intrusion detection through concept drift detection and online incremental learning, offers a scalable, available and fault-tolerant solution. Leveraging Apache Spark and Microsoft Azure Cloud enhances processing capabilities for large data sets in dynamic, nonstationary scenarios. The model’s applicability to heterogeneous data sets and its achievement of high-accuracy multi-class classification make it suitable for real-world intrusion detection. Moreover, the auto-scaling features of Microsoft Azure Cloud contribute to adaptability, ensuring efficient resource utilization without downtime. These practical implications underscore the model’s relevance and effectiveness in diverse operational contexts. Social implications The DDM-ORF model’s social implications are significant, contributing to enhanced cybersecurity measures. By providing an effective intrusion detection system, it helps safeguard digital ecosystems, preserving user privacy and securing sensitive information. The model’s accuracy in identifying and classifying various intrusion attempts aids in mitigating potential cyber threats, thereby fostering a safer online environment for individuals and organizations. As cybersecurity is paramount in the digital age, the social impact lies in fortifying the resilience of networks, systems and data against malicious activities, ultimately promoting trust and reliability in online interactions. Originality/value The DDM-ORF model introduces a novel approach to intrusion detection by combining drift detection and online incremental learning. This originality lies in its utilization of the DDM-ORF algorithm, offering a dynamic and adaptive system for evolving data. The model’s contribution extends to its scalability, fault-tolerance and suitability for heterogeneous data sets, addressing challenges in dynamic, nonstationary environments. Its application on a large-scale data set and multi-class classification, along with integration with Apache Spark and Microsoft Azure Cloud, enhances the field’s understanding and application of intrusion detection, providing valuable insights for securing digital infrastructures.

Data Poisoning Attack against Neural Network-Based On-Device Learning Anomaly Detector by Physical Attacks on Sensors.

In this paper, we introduce a security approach for on-device learning Edge AIs designed to detect abnormal conditions in factory machines. Since Edge AIs are easily accessible by an attacker physically, there are security risks due to physical attacks. In particular, there is a concern that the attacker may tamper with the training data of the on-device learning Edge AIs to degrade the task accuracy. Few risk assessments have been reported. It is important to understand these security risks before considering countermeasures. In this paper, we demonstrate a data poisoning attack against an on-device learning Edge AI. Our attack target is an on-device learning anomaly detection system. The system adopts MEMS accelerometers to measure the vibration of factory machines and detect anomalies. The anomaly detector also adopts a concept drift detection algorithm and multiple models to accommodate multiple normal patterns. For the attack, we used a method in which measurements are tampered with by exposing the MEMS accelerometer to acoustic waves of a specific frequency. The acceleration data falsified by this method were trained on an anomaly detector, and the result was that the abnormal state could not be detected.

Hoeffding adaptive trees for multi-label classification on data streams

Data stream learning is a very relevant paradigm because of the increasing real-world scenarios generating data at high velocities and in unbounded sequences. Stream learning aims at developing models that can process instances as they arrive, so models constantly adapt to new concepts and the temporal evolution in the stream. In multi-label data stream environments where instances have the peculiarity of belonging simultaneously to more than one class, the problem becomes even more complex and poses unique challenges such as different concept drifts impacting different labels at simultaneous or distinct times, higher class imbalance, or new labels emerging in the stream. This paper proposes a novel approach to multi-label data stream classification called Multi-Label Hoeffding Adaptive Tree (MLHAT). MLHAT leverages the Hoeffding adaptive tree to address these challenges by considering possible relations and label co-occurrences in the partitioning process of the decision tree, dynamically adapting the learner in each leaf node of the tree, and implementing a concept drift detector that can quickly detect and replace tree branches that are no longer performing well. The proposed approach is compared with other 18 online multi-label classifiers on 41 datasets. The results, validated with statistical analysis, show that MLHAT outperforms other state-of-the-art approaches in 12 well-known multi-label metrics.

Detecting and rationalizing concept drift: A feature-level approach for understanding cause–effect relationships in dynamic environments

Concept drift detection is essential for data-driven models to adapt to changing data patterns, ensuring accuracy and reliability in dynamic environments. Explaining drift can be even more important for gaining insights into root causes to facilitate informed decision-making. This paper presents a method to detect and rationalize drift, explaining it at the feature level by identifying impactful changes in the feature-outcome relationship from a cause–effect perspective. Drift occurrences therefore fall into four categories, including changes in the cause–effect relationship, comprising two subcategories: Type 1, where specific features no longer act as causes in the new environment, while Type 2 entails previously non-causal features becoming causes in the new environment. If the cause–effect relationship remains unchanged, it indicates changes in the importance of causal features (Type 3) or variations in the intensity of the cause–effect relationship (Type 4). The F1 score is employed to validate the proposed method for detecting and rationalizing recurring sudden drift in both real and virtual drift contexts. The average F1 scores on the Stagger and Agrawal datasets are 89.5% and 74.8%, respectively. The method is then applied to coalmine pressure data to investigate and rationalize changes in the causal relationship between strut pressure and microseismic energy.

Estimating data complexity and drift through a multiscale generalized impurity approach

The quality of machine learning solutions, and of classifier models in general, depend largely on the performance of the chosen algorithm, and on the intrinsic characteristics of the input data. Although work has been extensive on the former of these aspects, the latter has received comparably less attention. In this paper, we introduce the Multiscale Impurity Complexity Analysis (MICA) algorithm for the quantification of class separability and decision-boundary complexity of datasets. MICA is both model and dimensionality-independent and can provide a measure of separability based on regional impurity values. This makes it so that MICA is sensible to both global and local data conditions. We show MICA to be capable of properly describing class separability in a comprehensive set of both synthetic and real datasets and comparing it against other state-of-the-art methods. After establishing the robustness of the proposed method, alternative applications are discussed, including a streaming-data variant of MICA (MICA-S), that can be repurposed into a model-independent method for concept drift detection.

A benchmark and survey of fully unsupervised concept drift detectors on real-world data streams

A benchmark and survey of fully unsupervised concept drift detectors on real-world data streams

Variance Feedback Drift Detection Method for Evolving Data Streams Mining

Learning from changing data streams is one of the important tasks of data mining. The phenomenon of the underlying distribution of data streams changing over time is called concept drift. In classification decision-making, the occurrence of concept drift will greatly affect the classification efficiency of the original classifier, that is, the old decision-making model is not suitable for the new data environment. Therefore, dealing with concept drift from changing data streams is crucial to guarantee classifier performance. Currently, most concept drift detection methods apply the same detection strategy to different data streams, with little attention to the uniqueness of each data stream. This limits the adaptability of drift detectors to different environments. In our research, we designed a unique solution to address this issue. First, we proposed a variance estimation strategy and a variance feedback strategy to characterize the data stream’s characteristics through variance. Based on this variance, we developed personalized drift detection schemes for different data streams, thereby enhancing the adaptability of drift detection in various environments. We conducted experiments on data streams with various types of drifts. The experimental results show that our algorithm achieves the best average ranking for accuracy on the synthetic dataset, with an overall ranking 1.12 to 1.5 higher than the next-best algorithm. In comparison with algorithms using the same tests, our method improves the ranking by 3 to 3.5 for the Hoeffding test and by 1.12 to 2.25 for the McDiarmid test. In addition, they achieve a good balance between detection delay and false positive rates. Finally, our algorithm ranks higher than existing drift detection methods across the four key metrics of accuracy, CPU time, false positives, and detection delay, meeting our expectations.

An artificial intelligence framework for explainable drift detection in energy forecasting

Accurate energy consumption forecasting is crucial for reducing operational costs, achieving net-zero carbon emissions, and ensuring sustainable buildings and cities of the future. Despite the frequent use of Artificial Intelligence (AI) algorithms for learning energy consumption patterns and predictions in Building Science, relying solely on these techniques for energy demand prediction addresses only a fraction of the challenge. A drift in energy usage can lead to inaccuracies in these AI models and subsequently to poor decision-making and interventions. While drift detection techniques have been reported, a reliable and robust approach capable of explaining identified discrepancies with actionable insights has not been discussed in extant literature. Hence, this paper presents an Artificial Intelligence framework for energy consumption forecasting with explainable drift detection, aimed at addressing these challenges. The proposed framework is composed of energy embeddings, an optimized dimensional model integrated within a data warehouse, and scalable cloud implementation for effective drift detection with explainability capability. The framework is empirically evaluated in the real-world setting of a multi-campus, mixed-use tertiary education setting in Victoria, Australia. The results of these experiments highlight its capabilities in detecting concept drift, adapting forecast predictions, and providing an interpretation of the changes using energy embeddings.

A Unified Framework for Detecting Gradual and Abrupt Concept Drifts in Data Stream Mining: The Concept Drift Detection Framework with Hybrid Meta-Learning (CDDF-HML)

A Unified Framework for Detecting Gradual and Abrupt Concept Drifts in Data Stream Mining: The Concept Drift Detection Framework with Hybrid Meta-Learning (CDDF-HML)

Efficient and scalable covariate drift detection in machine learning systems with serverless computing

As machine learning models are increasingly deployed in production, robust monitoring and detection of concept and covariate drift become critical. This paper addresses the gap in the widespread adoption of drift detection techniques by proposing a serverless-based approach for batch covariate drift detection in ML systems. Leveraging the open-source OSCAR framework and the open-source Frouros drift detection library, we develop a set of services that enable parallel execution of two key components: the ML inference pipeline and the batch covariate drift detection pipeline. To this end, our proposal takes advantage of the elasticity and efficiency of serverless computing for ML pipelines, including scalability, cost-effectiveness, and seamless integration with existing infrastructure. We evaluate this approach through an edge ML use case, showcasing its operation on a simulated batch covariate drift scenario. Our research highlights the importance of integrating drift detection as a fundamental requirement in developing robust and trustworthy AI systems and encourages the adoption of these techniques in ML deployment pipelines. In this way, organizations can proactively identify and mitigate the adverse effects of covariate drift while capitalizing on the benefits offered by serverless computing.

One or two things we know about concept drift-a survey on monitoring in evolving environments. Part A: detecting concept drift.

The world surrounding us is subject to constant change. These changes, frequently described as concept drift, influence many industrial and technical processes. As they can lead to malfunctions and other anomalous behavior, which may be safety-critical in many scenarios, detecting and analyzing concept drift is crucial. In this study, we provide a literature review focusing on concept drift in unsupervised data streams. While many surveys focus on supervised data streams, so far, there is no work reviewing the unsupervised setting. However, this setting is of particular relevance for monitoring and anomaly detection which are directly applicable to many tasks and challenges in engineering. This survey provides a taxonomy of existing work on unsupervised drift detection. In addition to providing a comprehensive literature review, it offers precise mathematical definitions of the considered problems and contains standardized experiments on parametric artificial datasets allowing for a direct comparison of different detection strategies. Thus, the suitability of different schemes can be analyzed systematically, and guidelines for their usage in real-world scenarios can be provided.

Temporal Attention for Few-Shot Concept Drift Detection in Streaming Data

Concept drift describes unforeseeable changes in the underlying distribution of streaming data over time. Concept drift is a phenomenon in which the statistical properties of a target domain change over time in an arbitrary way. These changes might be caused by changes in hidden variables that cannot be measured directly. With the onset of the big data era, domains such as social networks, meteorology, and finance are generating copious amounts of streaming data. Embedded within these data, the issue of concept drift can affect the attributes of streaming data in various ways, leading to a decline in the accuracy and performance of models. There is a pressing need for new models to re-adapt to the changes in streaming data. Traditional concept drift detection algorithms struggle to effectively capture and utilize the key feature points of concept drift within complex time series, thereby failing to maintain the accuracy and efficiency of the models. In light of these challenges, this study introduces a novel concept drift detection method that incorporates a temporal attention mechanism within a prototypical network. By integrating a temporal attention mechanism during the feature extraction process, our approach enhances the capability to process complex time series data, preserves temporal locality, strengthens the learning of key features, and reduces the amount of labeled data required. This method significantly improves the detection accuracy and efficiency of small sample streaming data by better capturing the local features of the data. Experiments conducted across multiple datasets demonstrate that this method exhibits comprehensive leading performance in terms of accuracy and F1-score, with excellent recall and precision, thereby validating its effectiveness in enhancing concept drift detection in streaming data.

A novel framework for concept drift detection using autoencoders for classification problems in data streams

A novel framework for concept drift detection using autoencoders for classification problems in data streams

Application of concept drift detection and adaptive framework for non linear time series data from cardiac surgery

AbstractThe quality of machine learning (ML) models deployed in dynamic environments tends to decline over time due to disparities between the data used for training and the upcoming data available for prediction, which is commonly known as drift. Therefore, it is important for ML models to be capable of detecting any changes or drift in the data distribution and updating the ML model accordingly. This study presents various drift detection techniques to identify drift in the survival outcomes of patients who underwent cardiac surgery. Additionally, this study proposes several drift adaptation strategies, such as adaptive learning, incremental learning, and ensemble learning. Through a detailed analysis of the results, the study confirms the superior performance of ensemble model, achieving a minimum mean absolute error (MAE) of 10.684 and 2.827 for predicting hospital stay and ICU stay, respectively. Furthermore, the models that incorporate a drift adaptive framework exhibit superior performance compared to the models that do not include such a framework.

Online Concept Drift Detector: Optimally Balancing Delay Detection, Runtime, Memory, and Accuracy.

Online learning and real-time data processing are becoming increasingly vital across various domains such as sensor networks, banking, and telecommunications. A significant challenge in this context is concept drift, wherein the statistical properties of the data change over time. Traditional drift detectors often grapple with high memory usage, extended delay in detection, prolonged runtime, and accuracy inconsistencies. This paper introduces a novel Online Drift Detector that meticulously balances these four aspects. By processing data instance-by-instance, our proposed detector optimizes the trade-offs between delay detection, runtime, memory consumption, and accuracy. We incorporate a unique diversity calculation tailored for multi-label problems, ensuring swift drift detection with minimized memory usage and enhanced runtime efficiency. Comparative analyses reveal the dominance of our approach over contemporary drift detection techniques, particularly in the realms of memory efficiency, detection speed, and accuracy. This work substantially augments the field of online data stream processing by offering a refined strategy for timely and efficient concept drift detection across a myriad of applications.

On the Model Update Strategies for Supervised Learning in AIOps Solutions

AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data produced during the operation of large-scale systems and machine learning models to assist software engineers in their system operations. As operation data produced in the field are constantly evolving due to factors such as the changing operational environment and user base, the models in AIOps solutions need to be constantly maintained after deployment. While prior works focus on innovative modeling techniques to improve the performance of AIOps models before releasing them into the field, when and how to update AIOps models remain an under-investigated topic. In this work, we performed a case study on three large-scale public operation data: two trace datasets from the cloud computing platforms of Google and Alibaba and one disk stats dataset from the BackBlaze cloud storage data center. We empirically assessed five different types of model update strategies for supervised learning regarding their performance, updating cost, and stability. We observed that active model update strategies (e.g., periodical retraining, concept drift guided retraining, time-based model ensembles, and online learning) achieve better and more stable performance than a stationary model. Particularly, applying sophisticated model update strategies (e.g., concept drift detection, time-based ensembles, and online learning) could provide better performance, efficiency, and stability than simply retraining AIOps models periodically. In addition, we observed that, although some update strategies (e.g., time-based ensemble and online learning) can save model training time, they significantly sacrifice model testing time, which could hinder their applications in AIOps solutions where the operation data arrive at high pace and volume and where immediate inferences are required. Our findings highlight that practitioners should consider the evolution of operation data and actively maintain AIOps models over time. Our observations can also guide researchers and practitioners in investigating more efficient and effective model update strategies that fit in the context of AIOps.

Concept drift detection methods based on different weighting strategies

Concept drift detection methods based on different weighting strategies

Ensemble framework for concept drift detection and class imbalance in data streams

Ensemble framework for concept drift detection and class imbalance in data streams

A study on concept drift detection algorithms for real-world data streams

The arrival of data has changed in today's digital environment, becoming more dynamic. Dynamic data is characterized by its speed, variety, and infinite size. Data streams are one category of dynamic data. To address the issues with data streams, several strategies and AI models were developed. One such problem is concept drift, which results from changes in the data's distribution and eventually lowers the performance of the AI model. This means that regular updates to the model are required. In our work, we will analyse the performance using evaluation metrics and compare the effectiveness of the current error-based methods with window-based methods for real-world datasets.

Concept Drift Mitigation in Low-Cost Air Quality Monitoring Networks.

Future air quality monitoring networks will integrate fleets of low-cost gas and particulate matter sensors that are calibrated using machine learning techniques. Unfortunately, it is well known that concept drift is one of the primary causes of data quality loss in machine learning application operational scenarios. The present study focuses on addressing the calibration model update of low-cost NO2 sensors once they are triggered by a concept drift detector. It also defines which data are the most appropriate to use in the model updating process to gain compliance with the relative expanded uncertainty (REU) limits established by the European Directive. As the examined methodologies, the general/global and the importance weighting calibration models were applied for concept drift effects mitigation. Overall, for all the devices under test, the experimental results show the inadequacy of both models when performed independently. On the other hand, the results from the application of both models through a stacking ensemble strategy were able to extend the temporal validity of the used calibration model by three weeks at least for all the sensor devices under test. Thus, the usefulness of the whole information content gathered throughout the original co-location process was maximized.