Concept Drifting Data Streams Research Articles

Unsupervised domain adaptation by incremental learning for concept drifting data streams

Unsupervised domain adaptation by incremental learning for concept drifting data streams

Explainable data stream mining: Why the new models are better

Explainable Machine Learning brings expandability, interpretability, and accountability to Data Mining Algorithms. Existing explanation frameworks focus on explaining the decision process of a single model in a static dataset. However, in data stream mining changes in data distribution over time, called concept drift, may require updating the learning models to reflect the current data environment. It is therefore important to go beyond static models and understand what has changed among the learning models before and after a concept drift. We propose a Data Stream Explanability framework (DSE) that works together with a typical data stream mining framework where support vector machine models are used. DSE aims to help non-expert users understand model dynamics in a concept drifting data stream. DSE visualizes differences between SVM models before and after concept drift, to produce explanations on why the new model fits the data better. A survey was carried out between expert and non-expert users on the effectiveness of the framework. Although results showed non-expert users on average responded with less understanding of the issue compared to expert users, the difference is not statistically significant. This indicates that DSE successfully brings the explanability of model change to non-expert users.

A Novel Neural Ensemble Architecture for On-the-fly Classification of Evolving Text Streams

We study on-the-fly classification of evolving text streams in which the relation between the input data and target labels changes over time—i.e., “concept drift.” These variations decrease the model’s performance, as predictions become less accurate over time and they necessitate a more adaptable system. While most studies focus on concept drift detection and handling with ensemble approaches, the application of neural models in this area is relatively less studied. We introduce Adaptive Neural Ensemble Network ( AdaNEN ), a novel ensemble-based neural approach, capable of handling concept drift in data streams. With our novel architecture, we address some of the problems neural models face when exploited for online adaptive learning environments. Most current studies address concept drift detection and handling in numerical streams, and the evolving text stream classification remains relatively unexplored. We hypothesize that the lack of public and large-scale experimental data could be one reason. To this end, we propose a method based on an existing approach for generating evolving text streams by introducing various types of concept drifts to real-world text datasets. We provide an extensive evaluation of our proposed approach using 12 state-of-the-art baselines and 13 datasets. We first evaluate concept drift handling capability of AdaNEN and the baseline models on evolving numerical streams; this aims to demonstrate the concept drift handling capabilities of our method on a general spectrum and motivate its use in evolving text streams. The models are then evaluated in evolving text stream classification. Our experimental results show that AdaNEN consistently outperforms the existing approaches in terms of predictive performance with conservative efficiency.

Multi-type concept drift detection under a dual-layer variable sliding window in frequent pattern mining with cloud computing

The detection of different types of concept drift has wide applications in the fields of cloud computing and security information detection. Concept drift detection can indeed assist in promptly identifying instances where model performance deteriorates or when there are changes in data distribution. This paper focuses on the problem of concept drift detection in order to conduct frequent pattern mining. To address the limitation of fixed sliding windows in adapting to evolving data streams, we propose a variable sliding window frequent pattern mining algorithm, which dynamically adjusts the window size to adapt to new concept drifts and detect them in a timely manner. Furthermore, considering the challenge of existing concept drift detection algorithms that struggle to adapt to different types of drifting data simultaneously, we introduce an additional dual-layer embedded variable sliding window. This approach helps differentiate types of concept drift and incorporates a decay model for drift adaptation. The proposed algorithm can effectively detect different types of concept drift in data streams, perform targeted drift adaptation, and exhibit efficiency in terms of time complexity and memory consumption. Additionally, the algorithm maintains stable performance, avoiding abrupt changes due to window size variations and ensuring overall robustness.

Transfer learning for concept drifting data streams in heterogeneous environments

Transfer learning for concept drifting data streams in heterogeneous environments

Dynamic Ensemble Selection for Imbalanced Data Streams With Concept Drift.

Ensemble learning, as a popular method to tackle concept drift in data stream, forms a combination of base classifiers according to their global performances. However, concept drift generally occurs in local data space, causing significantly different performances of a base classifier at different locations. Thus, employing global performance as a criterion to select base classifier is inappropriate. Moreover, data stream is often accompanied by class imbalance problem, which affects the classification accuracy of ensemble learning on minority instances. To drawback these problems, a dynamic ensemble selection for imbalanced data streams with concept drift (DES-ICD) is proposed. For data arrived in chunk-by-chunk, a novel synthetic minority oversampling technique with adaptive nearest neighbors (AnnSMOTE) is developed to generate new minority instances that conform to the new concept. Following that, DES-ICD creates a base classifier on newly arrived data chunk balanced by AnnSMOTE and merges it with historical base classifiers to form a candidate classifier pool. For each query instance, the optimal combination is constructed in terms of the performance of candidate classifiers in its neighborhood. Experimental results for nine synthetic and five real-world datasets show that the proposed method outperforms seven comparative methods on classification accuracy and tracks new concepts in an imbalanced data stream more preciously.

QuadCDD: A Quadruple-based Approach for Understanding Concept Drift in Data Streams

Concept drift is a prevalent phenomenon in data streams that necessitates detection and in-depth understanding, as it signifies that the statistical properties of a target variable, which the model aims to predict, change over time in unforeseen ways. Existing detection methods predominantly aim to identify the drift start time, which lack comprehensive understanding of data streams, leading to a loss of drift information. In this paper, we present a novel Quadruple-based Approach for Understanding Concept Drift in Data Streams (QuadCDD) framework that not only detects and predicts the concept drift start point but also offers a more detailed analysis of concept drift through the use of quadruples, encompassing drift start, drift end, drift severity, and drift type. Our framework employs quadruples to enable informed decision-making and adopt appropriate actions to handle various concept drifts, effectively maintaining high and stable performance in data streams with concept drift. Experimental results validate the effectiveness of our QuadCDD framework in accurately detecting and understanding concept drifts, as well as in preserving the stability and performance of models in the presence of these drifts.

Model-centric transfer learning framework for concept drift detection

Concept drift refers to the inevitable phenomenon that influences the statistical features of the data stream. Detecting concept drift in data streams quickly and precisely remains challenging, and failure to detect it will render model trained on historical data ineffective. Current drift detection methods suffer from the following problems: detection delay, many false detected drifts, and rarely utilize the deep neural network directly in the field of concept drift detection, which is considerably competent at addressing the classification problems of data stream. Furthermore, the output of the model is usually taken as a metric to detect drift, while changes in the model parameters are often ignored which contain highly useful information. In this paper, we propose a model-centric framework for concept drift detection that uses deep neural network to detect concept drift by focusing on the change in the model itself, rather than the model output. In addition, transfer learning is developed to accelerate the drift detection process and reduce the computational complexity by freezing parts of the network. To further reduce false detected drifts, we propose long and short time windows method to determine the real drift from the potential detected drift. Experiments with real-world and artificial datasets have been undertaken to demonstrate the effectiveness of the proposed framework.

Combining Diverse Meta-Features to Accurately Identify Recurring Concept Drift in Data Streams

Learning from streaming data is challenging as the distribution of incoming data may change over time, a phenomenon known as concept drift. The predictive patterns, or experience learned under one distribution may become irrelevant as conditions change under concept drift, but may become relevant once again when conditions reoccur. Adaptive learning methods adapt a classifier to concept drift by identifying which distribution, or concept , is currently present in order to determine which experience is relevant. Identifying a concept requires some representation to be stored for comparison, with the quality of the representation being key to accurate identification. Existing concept representations are based on meta-features, efficient univariate summaries of a concept. However, no single meta-feature can fully represent a concept, leading to severe accuracy loss when existing representations cannot describe concept drift. To avoid these failure cases, we propose the first general framework for combining a diverse range of meta-features into a single representation. We solve two main challenges, first presenting a method of efficiently computing, storing, and querying an arbitrary set of meta-features as a single representation, showing that a combination of meta-features may successfully avoid failure cases seen with existing methods. Second, we present the first method for dynamically learning which meta-features distinguish concepts in any given dataset, significantly improving performance. Our proposed approach enables state-of-the-art feature selection methods, such as mutual information, to be applied to concept representation meta-features for the first time. We investigate tradeoffs between memory budget and classification performance, observing accuracy increases of up to 16% by dynamically weighting the contribution of each meta-feature.

Intrusion detection in the IoT data streams using concept drift localization

<abstract><p>With the widespread application of smart devices, the security of internet of things (IoT) systems faces entirely new challenges. The IoT data stream operates in a non-stationary, dynamic environment, making it prone to concept drift. This paper focused on addressing the issue of concept drift in data streams, with a key emphasis on introducing an innovative drift detection method-ensemble multiple non-parametric concept localization detectors, abbreviated as EMNCD. EMNCD employs an ensemble of non-parametric statistical methods, including the Kolmogorov-Smirnov, Wilcoxon rank sum and Mann-Kendall tests. By comparing sample distributions within a sliding window, EMNCD accurately detects concept drift, achieving precise localization of drift points, and enhancing overall detection reliability. Experimental results demonstrated the superior performance of EMNCD compared to classical methods on artificial datasets. Simultaneously, to enhance the robustness of data stream processing, we presented an online anomaly detection method based on the isolation forest (iForest). Additionally, we proposedwhale optimization algorithm (WOA)-extreme gradient boosting (XGBoost), a drift adaptation model employing XGBoost as a base classifier. This model dynamically updates using drift points detected by EMNCD and fine-tunes parameters through the WOA. Real-world applications on the edge-industrial IoTset (IIoTset) intrusion dataset explore the impact of concept drift on intrusion detection, where IIoT is a subclass of IoT. In summary, this paper focused on EMNCD, introducing innovative approaches for drift detection, anomaly detection, and drift adaptation. The research provided practical and viable solutions to address concept drift in data streams, enhancing security in IoT systems.</p></abstract>

Effective intrusion detection system using concept drifting data stream and support vector machine

SummaryDue to the widespread distribution of network connectivity, the demand for network security and protection against cyber‐attacks is increasing rapidly. Intrusion detection system (IDS) plays a vital role in the network security. Previously, the concept‐drift based IDS is performed, in which the suspicious activity at specific IP address is detected on the real time network. However, in that research the intrusion detection is performed on single attribute. In this work, an efficient adaptive window with support vector machine (ADWIN‐SVM) is developed to detect the intrusions on large datasets with multiple attributes. The initial process in the proposed system is data cleaning. To clean the input data, the data normalization technique is utilized. Then, the data are streamed as online data by including the timestamp attribute to the dataset. This process assists to extract the patterns and detect the changes in the data. ADWIN algorithm identifies the sudden generation of drift and separates the window based on the split point. The data can be classified as normal and attack by the SVM classifier. For the performance evaluation of the proposed IDS, two datasets such as KDD Cup 1999 and CICIDS 2017 are utilized. Python is the implementation tool and the performance metrics considered for the proposed approach are precision, recall, F1 score, accuracy, and false positive rate. The proposed ADWIN‐SVM performance is compared with various existing techniques such as random forest, decision tree, naive Bayes, logistic regression, AdaBoost, K‐nearest neighbor, and SVM‐rbf. From the comparative analysis, the proposed ADWIN‐SVM is outperformed in intrusion detection.

Ensemble based on Accuracy and Diversity Weighting for Evolving Data Streams

Ensemble classification is an actively researched paradigm that has received much attention due to increasing real-world applications. The crucial issue of ensemble learning is to construct a pool of base classifiers with accuracy and diversity. In this paper, unlike conventional data-streams oriented ensemble methods, we propose a novel Measure via both Accuracy and Diversity (MAD) instead of one of them to supervise ensemble learning. Based on MAD, a novel online ensemble method called Accuracy and Diversity weighted Ensemble (ADE) effectively handles concept drift in data streams. ADE mainly uses the following three steps to construct a concept-drift oriented ensemble: for the current data window, 1) a new base classifier is constructed based on the current concept when drift detect, 2) MAD is used to measure the performance of ensemble members, and 3) a newly built classifier replaces the worst base classifier. If the newly constructed classifier is the worst one, the replacement has not occurred. Comparing with the state-of-art algorithms, ADE exceeds the current best-related algorithm by 2.38% in average classification accuracy. Experimental results show that the proposed method can effectively adapt to different types of drifts.

Stress emotion classification using optimized convolutional neural network for online transfer learning dataset

Nowadays, deep learning methods with transfer learning (TL) makes ease of stress emotion classification tasks. Amongst, an optimized convolutional neural network with TL (OCNNTL) executes OCNN-based classification on emotion and stress data domains to learn high-level features at the top layers. However, it fails to handle the abrupt concept drift in real-time; besides, it end up with huge time complication while on gathering the required data and its transformation. To tackle the aforementioned concerns, a novel online OCNNTL (O2CNNTL) model is proposed; whereas, OCNNTL process initiates in the stress-emotion domain via the prior knowledge acquired by learning the training data both from the stress as well as the emotion domains. Moreover in O2CNNTL model, the concept-drifting data streams are taken into account for solving the online classification by the OCNN classifier; whereas, to enhance the learning efficiency a regularization learning technique is instigated on varied feature spaces. Thus, the proposed O2CNNTL achieves higher efficiency than the state-of-the-art models.

ADAW: Age decay accuracy weighted ensemble method for drifting data stream mining

Dynamic environment data generators are very often in real-world that produce data streams. A data source of a dynamic environment generates data streams in which the underlying data distribution changes very frequently with respect to time and hence results in concept drifts. As compared to the stationary environment, learning in the dynamic environment is very difficult due to the presence of concept drifts. Learning in dynamic environment requires evolutionary and adaptive approaches to be accommodated with the learning algorithms. Ensemble methods are commonly used to build classifiers for learning in a dynamic environment. The ensemble methods of learning are generally described at three very crucial aspects, namely, the learning and testing method employed, result integration method and forgetting mechanism for old concepts. In this paper, we propose a novel approach called Age Decay Accuracy Weighted (ADAW) ensemble architecture for learning in concept drifting data streams. The ADAW method assigned weights to the component classifiers based on its accuracy and its remaining life-time in the ensemble is such a way that ensures maximum accuracy. We empirically evaluated ADAW on benchmark artificial drifting data stream generators and real datasets and compared its performance with ten well-known state-of-the-art existing methods. The experimental results show that ADAW outperforms over the existing methods.

Dynamically adaptive and diverse dual ensemble learning approach for handling concept drift in data streams

AbstractConcept drift refers to the change in data distributions and evolving relationships between input and output variables with the passage of time. To analyze such variations in learning environments and generate models which can accommodate changing performance of predictive systems is one of the challenging machine learning applications. In general, the majority of the existing schemes consider one of the specific drift types: gradual, abrupt, recurring, or mixed, with traditional voting setup. In this work, we propose a novel data stream framework, dynamically adaptive and diverse dual ensemble (DA‐DDE) which responds to multiple drift types in the incoming data streams by combining online and block‐based ensemble techniques. In the proposed scheme, a dual diversified ensemble‐based system is constructed with the combination of active and passive ensembles, updated over a diverse set of resampled input space. The adaptive weight setting method is proposed in this work which utilizes the overall performance of learners on historic as well as recent concepts of distributions. Further a dual voting system has been used for hypothesis generation by considering dynamic adaptive credibility of ensembles in real time. Comparative analysis with 14 state‐of‐the‐art algorithms on 24 artificial and 11 real datasets shows that DA‐DDE is highly effective in handling various drift types.

The impact of data difficulty factors on classification of imbalanced and concept drifting data streams

Class imbalance introduces additional challenges when learning classifiers from concept drifting data streams. Most existing work focuses on designing new algorithms for dealing with the global imbalance ratio and does not consider other data complexities. Independent research on static imbalanced data has highlighted the influential role of local data difficulty factors such as minority class decomposition and presence of unsafe types of examples. Despite often being present in real-world data, the interactions between concept drifts and local data difficulty factors have not been investigated in concept drifting data streams yet. We thoroughly study the impact of such interactions on drifting imbalanced streams. For this purpose, we put forward a new categorization of concept drifts for class imbalanced problems. Through comprehensive experiments with synthetic and real data streams, we study the influence of concept drifts, global class imbalance, local data difficulty factors, and their combinations, on predictions of representative online classifiers. Experimental results reveal the high influence of new considered factors and their local drifts, as well as differences in existing classifiers’ reactions to such factors. Combinations of multiple factors are the most challenging for classifiers. Although existing classifiers are partially capable of coping with global class imbalance, new approaches are needed to address challenges posed by imbalanced data streams.

A clustering and ensemble based classifier for data stream classification

In the era of data mining, the research industry has great attention to data stream mining as well as it has a great impact on a wide range of applications like networking, telecommunication, education, banking, weather forecasting, a stock market, and so on. Because of these data stream mining having more attention from researchers. The handling of concept drifting data streams is one of the major issues and challenges in the data stream mining field. In the presence of the concept drift, the performance of the learning algorithm always degrades. In this paper, a hybrid method has been proposed which are the combination of an ensemble, and grid and density-based clustering methods. The proposed method is tested on both synthetic as well as real data. The proposed method works well in the presence of concept drift and performance is measured in terms of time, accuracy, and memory. As compared with the state-of-art algorithms, the proposed method performed well and gave better accuracy using synthetic datasets like 88.29%, 71.34%, and 75.39% for Hyperplane, RBF, and LED respectively and for real datasets 86.17%, 86.28%, 95.15%, and 99.83% for Adult, Census-Income, KDDCup99%–10%, and Covertype respectively.

Two‐level pruning based ensemble with abstained learners for concept drift in data streams

AbstractMining data streams for predictive analysis is one of the most interesting topics in machine learning. With the drifting data distributions, it becomes important to build adaptive systems which are dynamic and accurate. Although ensembles are powerful in improving accuracy of incremental learning, it is crucial to maintain a set of best suitable learners in the ensemble while considering the diversity between them. By adding diversity‐based pruning to the traditional accuracy‐based pruning, this paper proposes a novel concept drift handling approach named Two‐Level Pruning based Ensemble with Abstained Learners (TLP‐EnAbLe). In this approach, deferred similarity based pruning delays the removal of under performing similar learners until it is assured that they are no longer fit for prediction. The proposed scheme retains diverse learners that are well suited for current concept. Two‐level abstaining monitors performance of learners and chooses the best set of competent learners for participating in decision making. This is an enhancement to traditional majority voting system which dynamically chooses high performing learners and abstains the ones which are not suitable for prediction. In our experiments, it has been demonstrated that TLP‐EnAbLe handles concept drift more effectively than other state‐of‐the‐art algorithms on nineteen artificially drifting and ten real‐world datasets. Further, statistical tests conducted on various drift patterns which include gradual, abrupt, recurring and their combinations prove efficiency of the proposed approach.

Efficient mining of IoT based data streams for advanced computer vision systems

With emergence of Internet of Things (IoT) and subsequent technologies, smart devices are being increasingly used in various domains such as smart homes, smart parking, intelligent transportation etc. Vast amount of image and video data has been produced by IoT based systems in the form of continuous and possibly infinite image and video streams. This demands the development of advanced predictive vision systems which exploits stream mining concepts for intelligent processing of visual data streams. Among other challenges faced by visual data streams, a major challenge is concept drift, which is caused by overtime change in data distribution. In the presence of skewed data, the detection of concept drift becomes more challenging. When analyzing the data generated from smart devices and other performance critical wireless sensors, concept drift affects data integrity and accuracy of prediction results. EWMA for Concept Drift Detection (ECDD) has been proposed in the literature for detecting data streams. However, ECDD has a high prediction error rate which makes it less useful for performance critical data streams generated by imaging and video data streams. In this paper, Vision based Drift Detection Method (VisDDM) is proposed, which systematically handles abrupt and gradual concept drift in data streams. Experiments have been performed using synthetic and real world datasets from different application domains. Our proposed VisDDM algorithm is able to handle abrupt and gradual drift types and outperformed the existing drift detection methods in terms of accuracy and mean evaluation time.

Accurate detecting concept drift in evolving data streams

Abstract Predictive models operating on the evolving data streams are dynamic. The performance of a model will deteriorate eventually when it suffers the effect of concept drift. The learning algorithms require the proper adaptive strategies to cope with the concept drifting data streams. In this paper, we propose a new concept drift detection approach that analyzes the consistency of prequential error rate using Hoeffding’s inequality to detect the concept drifts in data streams. The experimental results show that our proposed method outperforms in comparison with other state-of-the-art detectors in terms of true drift detection, false alarm and delay of detection.