Concept Drift Research Articles

An optimized isolation forest based intrusion detection system for heterogeneous and streaming data in the industrial Internet of Things (IIoT) networks

While conventional Intrusion Detection Systems (IDS) are essential for defending against intruders in the Industrial Internet of Things (IIoT), handling data from heterogeneous and streaming data sources should receive more attention. This work introduces a novel Optimized IForest-based Intrusion Detection System (OIFIDS) which is designed to handle both heterogeneous and streaming data efficiently. The suggested approach employs a collection of optimized binary trees, each of which is trained on a distinctive subset of data, and in which the location of empty leaves determines the anomaly score assigned to a certain data point. Optimizing isolation Forest (iForest) utilizing a modified version of the Harris Hawks Optimization algorithm, which exploits both Exploration factor and Random walk strategies (ERHHO) decreases the dataset's dimension, decreases its learning time, and enhances the detection precision, accuracy, F1-score, FPR, and recall. To demonstrate how effective the proposed approach is, it is evaluated using three datasets: CICIDS-2018, NSL-KDD, and UNSW-NB15. The experimental results prove the ability of the suggested approach in handling both heterogeneous and streaming data efficiently and delivering results that were comparable to the cutting-edge baseline techniques. Moreover, it performs effectively when there are no anomalies in the training sample and when dealing with challenging scenarios with several irrelevant features and high dimensions. Based on the comparison with various state-of-the-art IDSs, the suggested approach is able to detect intrusion with greater accuracies of 95.6%, 94.8%, and 99% than the other approaches on the NSL-KDD, UNSW-NB15, and CICIDS-2018 datasets, respectively. Experiments on heterogenous data reveal that Area Under the ROC Curve (AUC) of OIFIDS beats the baseline approach for UNSW-NB15 dataset and is higher than the second-best method by 8% and 2.4% for NSL-KDD and CICIDS-2018 respectively. Evaluating the proposed system on streaming data illustrates that it can address the concept drift problem well with high AUC value of 0.948, 0.97, and 0.922 on the NSL-KDD, CICIDS-2018, and UNSW-NB15 datasets, respectively.

Overview of Wind and Photovoltaic Data Stream Classification and Data Drift Issues

The development in the fields of clean energy, particularly wind and photovoltaic power, generates a large amount of data streams, and how to mine valuable information from these data to improve the efficiency of power generation has become a hot spot of current research. Traditional classification algorithms cannot cope with dynamically changing data streams, so data stream classification techniques are particularly important. The current data stream classification techniques mainly include decision trees, neural networks, Bayesian networks, and other methods, which have been applied to wind power and photovoltaic power data processing in existing research. However, the data drift problem is gradually highlighted due to the dynamic change in data, which significantly impacts the performance of classification algorithms. This paper reviews the latest research on data stream classification technology in wind power and photovoltaic applications. It provides a detailed introduction to the data drift problem in machine learning, which significantly affects algorithm performance. The discussion covers covariate drift, prior probability drift, and concept drift, analyzing their potential impact on the practical deployment of data stream classification methods in wind and photovoltaic power sectors. Finally, by analyzing examples for addressing data drift in energy-system data stream classification, the article highlights the future prospects of data drift research in this field and suggests areas for improvement. Combined with the systematic knowledge of data stream classification techniques and data drift handling presented, it offers valuable insights for future research.

Estimating data complexity and drift through a multiscale generalized impurity approach

The quality of machine learning solutions, and of classifier models in general, depend largely on the performance of the chosen algorithm, and on the intrinsic characteristics of the input data. Although work has been extensive on the former of these aspects, the latter has received comparably less attention. In this paper, we introduce the Multiscale Impurity Complexity Analysis (MICA) algorithm for the quantification of class separability and decision-boundary complexity of datasets. MICA is both model and dimensionality-independent and can provide a measure of separability based on regional impurity values. This makes it so that MICA is sensible to both global and local data conditions. We show MICA to be capable of properly describing class separability in a comprehensive set of both synthetic and real datasets and comparing it against other state-of-the-art methods. After establishing the robustness of the proposed method, alternative applications are discussed, including a streaming-data variant of MICA (MICA-S), that can be repurposed into a model-independent method for concept drift detection.

Discovering outlying attributes of outliers in data streams

Data streams, continuous sequences of timestamped data points, necessitate real-time monitoring due to their time-sensitive nature. In various data stream applications, such as network security and credit card transaction monitoring, real-time detection of outliers is crucial, as these outliers often signify potential threats. Equally important is the real-time explanation of outliers, enabling users to glean insights and thereby shorten their investigation time. The investigation time for outliers is closely tied to their number of attributes, making it essential to provide explanations that detail which attributes are responsible for the abnormality of a data point, referred to as outlying attributes. However, the unbounded volume of data and concept drift of data streams pose challenges for discovering the outlying attributes of outliers in real time. In response, in this paper we propose EXOS, an algorithm designed for discovering the outlying attributes of multi-dimensional outliers in data streams. EXOS leverages cross-correlations among data streams, accommodates varying data stream schemas and arrival rates, and effectively addresses challenges related to the unbounded volume of data and concept drift. The algorithm is model-agnostic for point outlier detection and provides real-time explanations based on the local context of the outlier, derived from time-based tumbling windows. The paper provides a complexity analysis of EXOS and an experimental analysis comparing EXOS with existing algorithms. The evaluation includes an assessment of performance on both real-world and synthetic datasets in terms of average precision, recall, F1-score, and explanation time. The evaluation results show that, on average, EXOS achieves a 45.6% better F1 Score and is 7.3 times lower in explanation time compared to existing outlying attribute algorithms.

OnsitNet: A memory-capable online time series forecasting model incorporating a self-attention mechanism

Traditional time series (TS) forecasting models are based on fixed, static datasets and lack scalability when faced with the continuous influx of data in real-world scenarios. Real-time online learning of data streams is crucial for improving forecasting efficiency. However, few studies focus on online TS forecasting, and existing approaches have several limitations. Most current online TS forecasting models merely train on data streams and are ineffective in handling concept drift scenarios. Furthermore, they often fail to adequately consider dependencies between variables and do not leverage the robust modeling capabilities of offline models. Therefore, we propose an innovative online learning method called OnsitNet. It consists of multiple learning modules that progressively expand the receptive field of convolutional kernels within the learning modules using an exponentially growing dilation factor, aiding in the capture of multi-scale data features. Within the learning modules, we propose an online learning strategy focusing on memorizing concept drift scenarios, with a fast learner, memorizer, and Pearson trigger. The Pearson trigger activates dynamic interaction between the fast learner and memorizer by detecting new data patterns, facilitating online rapid learning of data streams. To capture the dependencies between variables, we propose a new model, SITransformer, which is a streamlined version of the offline model ITransformer. Unlike the traditional Transformer, it reverses the roles of the feed-forward network and the attention mechanism. This inverted architecture is more effective at learning the correlations between variables. Experimental results on five real-world datasets show OnsitNet achieves lower online prediction errors, enabling timely and effective forecasting of future trends in TS data.

FIDWATCH: Federated incremental distillation for continuous monitoring of IoT security threats

The fast evolutions of Internet of Things (IoT) technologies have been accelerating their applicability in different sectors of life and becoming a pillar for sustainable development. However, this revolutionary expansion led to a substantial increase in attack surface, raising many concerns about security threats and their possible consequences. Machine learning has significantly contributed to designing intrusion detection systems (IDS) but suffers from critical limitations such as data privacy and sovereignty, data imbalance, concept drift, and catastrophic forgetting. This collectively makes existing IDSs an improper choice for securing IoT environments. This paper presents a federated learning approach called FIDWATCH to continuously monitor and detect a broad range of IoT security threats. The local side of FIDWATCH introduces contrastive focal loss to enhance the ability of the local model (teacher) to discriminate between diverse types of IoT security threats while putting an increased emphasis on hard-to-classify samples. A fine-grained Knowledge Distillation (KD) is introduced to allow the client to distill the required teacher's knowledge into a lighter, more compact model termed the pupil model. This greatly assists the competence and flexibility of the model in resource-constrained scenarios. Furthermore, an adaptive incremental updating method is introduced in FIDWATCH to allow the global model to exploit the distilled knowledge and refine the shared dataset. This helps generate global anchors for improving the robustness of the mode against the distributional shift, thereby improving model alignment and compliance with the dynamics of IoT security threats. Proof-of-concept simulations are performed on data from two public datasets (BoT-IoT and ToN-IoT), demonstrating the superiority of FIDWATCH over cutting-edge performance with an average f1-score of 97.07% and 95.63%, respectively.

A benchmark and survey of fully unsupervised concept drift detectors on real-world data streams

A benchmark and survey of fully unsupervised concept drift detectors on real-world data streams

Dynamic Soft Sensor Model for Endpoint Carbon Content and Temperature in BOF Steelmaking Based on Adaptive Feature Matching Variational Autoencoder

The key to endpoint control in basic oxygen furnace (BOF) steelmaking lies in accurately predicting the endpoint carbon content and temperature. However, BOF steelmaking data are complex and change distribution due to variations in raw material batches, process adjustments, and equipment conditions, leading to concept drift and affecting model performance. In order to resolve these problems, this paper proposes a dynamic soft sensor model based on an adaptive feature matching variational autoencoder (VAE-AFM). Firstly, this paper innovatively proposes an adaptive feature matching (AFM) method. This method utilizes the maximum mean discrepancy to calculate the values of the marginal and conditional distributions. Based on the discrepancy between these two values, a dynamic adjustment algorithm is designed to adaptively assign different weights to the two distributions. This approach dynamically and quantitatively evaluates and adjusts the relative importance of different distributions in the domain adaptation process, thereby enhancing the effectiveness of cross-domain data alignment. Secondly, a variational autoencoder (VAE) is employed to process the data, as the VAE model can capture the complex data structures and latent features in the steelmaking process. Finally, the features extracted by the VAE are processed with the adaptive feature matching method, thereby constructing the VAE-AFM dynamic soft sensor model. Experimental studies on actual BOF steelmaking data validate the efficacy of the offered approach, offering a reliable solution to the challenges of high complexity and concept drift in BOF steelmaking data.

Evolving cybersecurity frontiers: A comprehensive survey on concept drift and feature dynamics aware machine and deep learning in intrusion detection systems

Intrusion Detection Systems (IDS) have become pivotal in safeguarding information systems against evolving threats. Concurrently, Concept Drift presents a significant challenge in machine learning, affecting the adaptability and accuracy of predictive models in dynamic environments. Understanding the synergy between IDS and Concept Drift is crucial for developing robust security systems. The motivation behind this survey is driven by the emerging complexities in cyber threats and the dynamic nature of data streams, which necessitate advanced IDS capable of adapting to Concept and Feature Drift. Our analysis reveals a glaring omission in the existing literature—the integration of Concept Drift and Feature Drift within IDS. Most studies have focused on Concept Drift in a general context or on IDS but have yet to comprehensively consider the implications of data dynamics. This oversight has led to a fragmented understanding and suboptimal approaches to tackling modern cyber threats. To address this, we propose a comprehensive review that delves into the role of machine learning in IDS, explicitly focusing on Concept and Feature Drift. We have proposed a framework that includes all the necessary components for a drift-aware IDS. The framework incorporates dynamic feature selection, adaptive learning algorithms, and continuous monitoring techniques to handle Concept Drift and Feature Drift effectively. The survey highlights state-of-the-art methodologies and current challenges in integrating these concepts. The methodology involves an exhaustive analysis of published works from 2019 to 2024, comparing and contrasting various models and approaches. This includes a detailed examination of Concept Drift-aware IDS methods, dynamic feature selection techniques, and the impact of high dimensionality in IDS. These quantitative improvements underscore the necessity for developing adaptive and resilient IDS. The survey uncovers under-represented areas in current research, paving the way for future investigations. By highlighting these gaps and providing comparative data, the survey sets a clear direction for upcoming research efforts to foster the development of more dynamic and adaptable IDS solutions. The quantitative experimental evaluation of the proposed framework is planned to be conducted in a future article, where we will assess its effectiveness and performance in real-world scenarios.

Detecting Process Duration Drift Using Gamma Mixture Models in a Left-Truncated and Right-Censored Environment

Within the realm of business context, process duration signifies time spent by customers between successive activities. This temporal perspective offers important insight to customer behavior, highlighting potential bottlenecks, and influencing business management decisions. The distribution of these process duration often changes over time due to factors such as seasonality, emerging legislation, changes to supply chains, and customer demand. Referred to as concept drift, these variations pose challenges for robust process modeling, understanding, and refinement. Subsequently, gamma mixture models are widely employed to model durations. These source data can, however, become left-truncated and right-censored within any specific observation window thereby necessitating a (well-known) modification to the likelihood function. The approach reported in this article leveraged this adapted likelihood across a series of observation windows, applying the likelihood ratio test to identify duration changes/concept drift. Due to its flexibility in modelling any duration distribution, the gamma mixture model was used with Nelder–Mead optimized likelihood for the left-truncated and right-censored data. The number of gamma components was determined by the Bayesian information criterion. The proposed framework underwent validation through simulated exponential samples, leading to recommendations for its practical application. Subsequently, we applied the methodology to three real-life event logs exhibiting diverse characteristics. Experimental results showcase the effectiveness of our approach in terms of data fitting, as compared to Kaplan–Meier curves, and in detecting instances of drift. This comprehensive validation underscores the practical utility and reliability of our framework for dynamic business scenarios.

Class-imbalanced dynamic financial distress prediction based on random forest from the perspective of concept drift

Class-imbalanced dynamic financial distress prediction based on random forest from the perspective of concept drift

Variance Feedback Drift Detection Method for Evolving Data Streams Mining

Learning from changing data streams is one of the important tasks of data mining. The phenomenon of the underlying distribution of data streams changing over time is called concept drift. In classification decision-making, the occurrence of concept drift will greatly affect the classification efficiency of the original classifier, that is, the old decision-making model is not suitable for the new data environment. Therefore, dealing with concept drift from changing data streams is crucial to guarantee classifier performance. Currently, most concept drift detection methods apply the same detection strategy to different data streams, with little attention to the uniqueness of each data stream. This limits the adaptability of drift detectors to different environments. In our research, we designed a unique solution to address this issue. First, we proposed a variance estimation strategy and a variance feedback strategy to characterize the data stream’s characteristics through variance. Based on this variance, we developed personalized drift detection schemes for different data streams, thereby enhancing the adaptability of drift detection in various environments. We conducted experiments on data streams with various types of drifts. The experimental results show that our algorithm achieves the best average ranking for accuracy on the synthetic dataset, with an overall ranking 1.12 to 1.5 higher than the next-best algorithm. In comparison with algorithms using the same tests, our method improves the ranking by 3 to 3.5 for the Hoeffding test and by 1.12 to 2.25 for the McDiarmid test. In addition, they achieve a good balance between detection delay and false positive rates. Finally, our algorithm ranks higher than existing drift detection methods across the four key metrics of accuracy, CPU time, false positives, and detection delay, meeting our expectations.

Robust online active learning with cluster-based local drift detection for unbalanced imperfect data

With the rapid development of data-driven technologies, a massive amount of actual data emerges from industrial systems, forming data stream. Their data distribution may change over time and outliers may be generated as unbalanced imperfect data due to time-varying working condition, aging equipment, etc. Previous methods struggle with the dual challenges of concept drift and unbalance, however, fail to efficiently distinguishing outliers from a drift under the limited labeling budget, causing the performance degradation. To address the issue, robust online active learning with cluster-based local drift detection is proposed to classify unbalanced imperfect data stream with the above characteristics. The cluster-based local drift detection is first designed to capture a new concept and recognize the corresponding drifted regions. Following that, an improved active learning mechanism is presented to distinguish outliers from a drift, and select most valuable instances for labeling and updating ensemble classifier. Experimental results for eight synthetic and four real-world data streams show that the proposed method outperforms seven comparative methods on classification accuracy and robustness.

An artificial intelligence framework for explainable drift detection in energy forecasting

Accurate energy consumption forecasting is crucial for reducing operational costs, achieving net-zero carbon emissions, and ensuring sustainable buildings and cities of the future. Despite the frequent use of Artificial Intelligence (AI) algorithms for learning energy consumption patterns and predictions in Building Science, relying solely on these techniques for energy demand prediction addresses only a fraction of the challenge. A drift in energy usage can lead to inaccuracies in these AI models and subsequently to poor decision-making and interventions. While drift detection techniques have been reported, a reliable and robust approach capable of explaining identified discrepancies with actionable insights has not been discussed in extant literature. Hence, this paper presents an Artificial Intelligence framework for energy consumption forecasting with explainable drift detection, aimed at addressing these challenges. The proposed framework is composed of energy embeddings, an optimized dimensional model integrated within a data warehouse, and scalable cloud implementation for effective drift detection with explainability capability. The framework is empirically evaluated in the real-world setting of a multi-campus, mixed-use tertiary education setting in Victoria, Australia. The results of these experiments highlight its capabilities in detecting concept drift, adapting forecast predictions, and providing an interpretation of the changes using energy embeddings.

Research on concept drift algorithm based on evolutionary computation

Concept drift (CD) in data streams can significantly impact the performance and stability of data stream classification algorithms, diminishing the generalization capabilities of integrated learning models. This paper addresses CD issues in dichotomous data streams by introducing a novel modeling approach that leverages evolutionary computation techniques. The method entails grouping base learners based on their performance within a sliding window and then evolving the base learning periods using evolutionary techniques. Furthermore, the concept of “gene flow” is introduced to enhance diversity among base learners and improve CD prediction performance. Experimental results on real and artificial datasets demonstrate the superior comprehensive performance of the proposed method. Specifically, the BCDECA algorithm outperforms other similar methods, excelling in accuracy, diversity, convergence rate, and robustness on a range of datasets.

Parsimonious statistical techniques for the detection of drifts toward dangerous operational conditions in tokamaks

The approaching initial operation of major new tokomaks is rendering more pressing the need for effective disruption prediction techniques. The required tools should be not only accurate but also capable of operating with a minimum number of signals, because in the first campaigns of new devices typically only a very limited number of diagnostics is available. The present work reports the results obtained with two advanced statistical techniques, chaos onset and concept drift (CD). These methods are capable of detecting the plasma drifting towards dangerous regions of the operational space using only global signals such as the plasma current or the locked mode amplitude. The obtained performances meet the requirements of mitigation in devices such as JT60-SA and ITER at the beginning of their operation. The computational times of the chaos onset approach are already of the order of 1 ms on laptop computers, whereas the CD technique requires implementation on FPGAs or GPUs to be compatible with real time requirements.

A Unified Framework for Detecting Gradual and Abrupt Concept Drifts in Data Stream Mining: The Concept Drift Detection Framework with Hybrid Meta-Learning (CDDF-HML)

A Unified Framework for Detecting Gradual and Abrupt Concept Drifts in Data Stream Mining: The Concept Drift Detection Framework with Hybrid Meta-Learning (CDDF-HML)

The survey and discussion of research on heart disease prediction based on Apache Spark

Heart is the most important part of the human body, diseases that are related to heart cause a huge thread to human health. In this paper, methods that applied Apache Spark to heart disease related works would be shown and discussed in order to classify these methods and make a conclusion about the innovations and shortcomings of these works. These works are defined into two categories: the ones that adopted traditional machine learning method and the ones that used deep learning methods. By classifying these works into two types, commonalities and similar innovative approaches in the same category of methods can be better observed and summarized, facilitating a clearer comparison of the similarities and differences in the innovative focuses among similar yet distinct methods. By doing so, conclusions were made to show that apart from enhancing operational efficiency and reliability of models for diagnosing and treating heart diseases, current research utilizing Apache Spark in this field also identifies areas for improvement such as expanding sample data representation, speeding up data processing, and addressing concept drift issues with proposed solutions. By addressing these challenges, researchers aim to optimize existing methods using Apache Spark and advanced data analytics techniques to combat heart diseases.

Adaptive bagging-based dynamic ensemble selection in nonstationary environments

The classification of nonstationary data streams presents a substantial challenge mainly due to the simultaneous presence of class imbalance and concept drift. Ensemble learning is a widely acknowledged approach to address these two problems. Nevertheless, most of the existing methods establish independent strategies for each, ignoring their relationship and interactions, which prevents the corresponding strategies from performing as expected. This paper presents a novel algorithm AB-DES (Adaptive Bagging-based Dynamic Ensemble Selection), which employs bagging technique and dynamic ensemble selection system. In AB-DES, instances are stored to balance subsequent data chunks instead of relying solely on resampling methods. Then, all base classifiers are updated using a new dual adaptive sampling bagging strategy, which can help to reduce the classification inaccuracies involving by class imbalance. In the prediction phase, DES system constructs different ensemble models for different instances. The inclusion of a sliding window-based drift detector makes the above two strategies adapt to concept drift. Experimental results show that AB-DES achieves higher classification accuracy and exhibits enhanced robustness compared to 8 state-of-the-art methods on 10 synthetic datasets and 5 real world datasets with different types of concept drift.

Bin.INI: An ensemble approach for dynamic data streams

Class imbalance and concept drifts could deteriorate the performance of classifiers in data stream learning as their co-occurrence presents a complicated learning scenario. This situation becomes more complicated if missing values appear in the data. To tackle this challenge, this paper proposes a novel ensemble method, BINary classification in Incomplete, Non-stationary, and Imbalanced data streams (Bin.INI). As part of Bin.INI, Adapt2Regress builds a linear regression models pool using completely available minority subspaces, monitored under a novel sigmoid-decay function to provide imputation support in a drifting data environment. We use Min++ to balance the data in terms of imbalance complexity, a resampling component proposed earlier. Additionally, the 2-STage Ensemble Pruning technique (2STEP) is proposed which groups the classifiers based upon multiple diversity measures at the first step and selects the best subset using a novel custom metric, Sigmoid-Weighted Imbalance Score, at the second step. By using Adapt2Regress, Min++, and 2STEP, the proposed method can efficiently deal with incomplete, imbalanced, and drifting data in data streams. Experiments are conducted on 350 highly imbalanced data streams containing a variety of concept drifts. 300/350 data streams contain different ratios of missing data (5%–30%) in the minority space. Experiments reveal a noticeable decline in the performance of state-of-the-art approaches when faced with 15% or more missing data, while Bin.INI maintains its performance.