Different Types Of Concept Drift Research Articles

Adaptive bagging-based dynamic ensemble selection in nonstationary environments

The classification of nonstationary data streams presents a substantial challenge mainly due to the simultaneous presence of class imbalance and concept drift. Ensemble learning is a widely acknowledged approach to address these two problems. Nevertheless, most of the existing methods establish independent strategies for each, ignoring their relationship and interactions, which prevents the corresponding strategies from performing as expected. This paper presents a novel algorithm AB-DES (Adaptive Bagging-based Dynamic Ensemble Selection), which employs bagging technique and dynamic ensemble selection system. In AB-DES, instances are stored to balance subsequent data chunks instead of relying solely on resampling methods. Then, all base classifiers are updated using a new dual adaptive sampling bagging strategy, which can help to reduce the classification inaccuracies involving by class imbalance. In the prediction phase, DES system constructs different ensemble models for different instances. The inclusion of a sliding window-based drift detector makes the above two strategies adapt to concept drift. Experimental results show that AB-DES achieves higher classification accuracy and exhibits enhanced robustness compared to 8 state-of-the-art methods on 10 synthetic datasets and 5 real world datasets with different types of concept drift.

Multi-type concept drift detection under a dual-layer variable sliding window in frequent pattern mining with cloud computing

The detection of different types of concept drift has wide applications in the fields of cloud computing and security information detection. Concept drift detection can indeed assist in promptly identifying instances where model performance deteriorates or when there are changes in data distribution. This paper focuses on the problem of concept drift detection in order to conduct frequent pattern mining. To address the limitation of fixed sliding windows in adapting to evolving data streams, we propose a variable sliding window frequent pattern mining algorithm, which dynamically adjusts the window size to adapt to new concept drifts and detect them in a timely manner. Furthermore, considering the challenge of existing concept drift detection algorithms that struggle to adapt to different types of drifting data simultaneously, we introduce an additional dual-layer embedded variable sliding window. This approach helps differentiate types of concept drift and incorporates a decay model for drift adaptation. The proposed algorithm can effectively detect different types of concept drift in data streams, perform targeted drift adaptation, and exhibit efficiency in terms of time complexity and memory consumption. Additionally, the algorithm maintains stable performance, avoiding abrupt changes due to window size variations and ensuring overall robustness.

Autonomic active learning strategy using cluster-based ensemble classifier for concept drifts in imbalanced data stream

Systems are becoming autonomous nowadays. In real-world scenarios, the involvement of experts in generating a new decision has to be paid a certain amount of cost. Thus, in these conditions, creating a system with the least expert involvement and costs is the main destination. As a solution, an autonomic active learning strategy with a cluster-based ensemble classifier for concept drifts in imbalanced data stream (AACE-DI) is proposed in this paper. Initially, a cluster-based initializing formula is considered to select the most informative instances for labeling a dynamically changing ensemble classifier. Next, the automatically adjusting uncertainty strategy is introduced to prioritize the most uncertain, representative, and minority class data from imbalanced data stream, which also ensures the least labeling cost. This automatic formula not only outperforms the system but also makes it autonomic. Experimental evaluations are organized using 6 real-world data streams and 16 synthetic data streams with different types of concept drift and imbalanced ratios. To measure performance, evaluators, such as the prequential area under curve, recall, and geometric mean, are considered. Moreover, the proposed AACE-DI method achieves expected outcomes with the least cost as compared with other state-of-the-art learning methods.

Benchmarking Change Detector Algorithms from Different Concept Drift Perspectives

The stream mining paradigm has become increasingly popular due to the vast number of algorithms and methodologies it provides to address the current challenges of Internet of Things (IoT) and modern machine learning systems. Change detection algorithms, which focus on identifying drifts in the data distribution during the operation of a machine learning solution, are a crucial aspect of this paradigm. However, selecting the best change detection method for different types of concept drift can be challenging. This work aimed to provide a benchmark for four drift detection algorithms (EDDM, DDM, HDDMW, and HDDMA) for abrupt, gradual, and incremental drift types. To shed light on the capacity and possible trade-offs involved in selecting a concept drift algorithm, we compare their detection capability, detection time, and detection delay. The experiments were carried out using synthetic datasets, where various attributes, such as stream size, the amount of drifts, and drift duration can be controlled and manipulated on our generator of synthetic stream. Our results show that HDDMW provides the best trade-off among all performance indicators, demonstrating superior consistency in detecting abrupt drifts, but has suboptimal time consumption and a limited ability to detect incremental drifts. However, it outperforms other algorithms in detection delay for both abrupt and gradual drifts with an efficient detection performance and detection time performance.

Towards Reliable Online Just-in-Time Software Defect Prediction

Throughout its development period, a software project experiences different phases, comprises modules with different complexities and is touched by many different developers. Hence, it is natural that problems such as Just-in-Time Software Defect Prediction (JIT-SDP) are affected by changes in the defect generating process (concept drifts), potentially hindering predictive performance. JIT-SDP also suffers from delays in receiving the labels of training examples (verification latency), potentially exacerbating the challenges posed by concept drift and further hindering predictive performance. However, little is known about what types of concept drift affect JIT-SDP and how they affect JIT-SDP classifiers in view of verification latency. This work performs the first detailed analysis of that. Among others, it reveals that different types of concept drift together with verification latency significantly impair the stability of the predictive performance of existing JIT-SDP approaches, drastically affecting their reliability over time. Based on the findings, a new JIT-SDP approach is proposed, aimed at providing higher and more stable predictive performance (i.e., reliable) over time. Experiments based on ten GitHub open source projects show that our approach was capable of produce significantly more stable predictive performances in all investigated datasets while maintaining or improving the predictive performance obtained by state-of-art methods.

EnHAT — Synergy of a tree-based Ensemble with Hoeffding Adaptive Tree for dynamic data streams mining

The goal of this paper is to improve the predictive accuracy of data streaming algorithms without increasing the processing time of the incoming data. We propose the EnHAT (Ensemble Combined with Hoeffding Adaptive Tree) algorithm, which combines the state-of-the-art Hoeffding Adaptive Tree (HAT) algorithm with an ensemble of J48 decision trees induced from sequential chunks of the data stream. The slack time of HAT adaptation to a new window of incoming records is utilized in parallel for building a decision-tree ensemble. In our experiments on 4 benchmark streaming datasets and 4 synthetic datasets with different types of concept drift, EnHAT has reached the highest predictive accuracy in 23 out of 26 cases compared to an ensemble of J48 trees, HAT, and a single J48 model induced from the last sliding window. Thus, we can conclude that the ensemble/HAT synergy yields better prediction results than each one of the two approaches on its own. The higher accuracy does not come at the expense of any additional computational effort beyond the model induction times of the combined algorithms.

A Diversity Framework for Dealing With Multiple Types of Concept Drift Based on Clustering in the Model Space.

Data stream applications usually suffer from multiple types of concept drift. However, most existing approaches are only able to handle a subset of types of drift well, hindering predictive performance. We propose to use diversity as a framework to handle multiple types of drift. The motivation is that a diverse ensemble can not only contain models representing different concepts, which may be useful to handle recurring concepts, but also accelerate the adaptation to different types of concept drift. Our framework innovatively uses clustering in the model space to build a diverse ensemble and identify recurring concepts. The resulting diversity also accelerates adaptation to different types of drift where the new concept shares similarities with past concepts. Experiments with 20 synthetic and three real-world data streams containing different types of drift show that our diversity framework usually achieves similar or better prequential accuracy than existing approaches, especially when there are recurring concepts or when new concepts share similarities with past concepts.

An extensive study of C-SMOTE, a Continuous Synthetic Minority Oversampling Technique for Evolving Data Streams

Streaming Machine Learning (SML) studies algorithms that update their models, given an unbounded and often non-stationary flow of data performing a single pass. Online class imbalance learning is a branch of SML that combines the challenges of both class imbalance and concept drift. In this paper, we investigate the binary classification problem by rebalancing an imbalanced stream of data in the presence of concept drift, accessing one sample at a time. We propose an extensive comparative study of Continuous Synthetic Minority Oversampling Technique (C-SMOTE), inspired by the popular sampling technique Smote, as a meta-strategy to pipeline with SML classification algorithms. We benchmark C-SMOTE pipelines on both synthetic and real data streams, containing different types of concept drifts, different imbalance levels, and different class distributions. We bring statistical evidence that models learnt with C-SMOTE pipelines improve the minority class performance concerning both the baseline models and the state-of-the-art methods. We also perform a sensitivity analysis to detect the C-SMOTE impact on the majority class performance for the three types of concept drift and several class distributions. Moreover, we show a computational cost analysis in terms of time and memory consumption.

Selective ensemble-based online adaptive deep neural networks for streaming data with concept drift

Concept drift is an important issue in the field of streaming data mining. However, how to maintain real-time model convergence in a dynamic environment is an important and difficult problem. In addition, the current methods have limited ability to deal with the problem of streaming data classification for complex nonlinear problems. To solve these problems, a selective ensemble-based online adaptive deep neural network (SEOA) is proposed to address concept drift. First, the adaptive depth unit is constructed by combining shallow features with deep features and adaptively controls the information flow in the neural network according to changes in streaming data at adjacent moments, which improves the convergence of the online deep learning model. Then, the adaptive depth units of different layers are regarded as base classifiers for ensemble and weighted dynamically according to the loss of each classifier. In addition, a dynamic selection of base classifiers is adopted according to the fluctuation of the streaming data to achieve a balance between stability and adaptability. The experimental results show that the SEOA can effectively contend with different types of concept drift and has good robustness and generalization.

ADES: A New Ensemble Diversity-Based Approach for Handling Concept Drift

Beyond applying machine learning predictive models to static tasks, a significant corpus of research exists that applies machine learning predictive models to streaming environments that incur concept drift. With the prevalence of streaming real-world applications that are associated with changes in the underlying data distribution, the need for applications that are capable of adapting to evolving and time-varying dynamic environments can be hardly overstated. Dynamic environments are nonstationary and change with time and the target variables to be predicted by the learning algorithm and often evolve with time, a phenomenon known as concept drift. Most work in handling concept drift focuses on updating the prediction model so that it can recover from concept drift while little effort has been dedicated to the formulation of a learning system that is capable of learning different types of drifting concepts at any time with minimum overheads. This work proposes a novel and evolving data stream classifier called Adaptive Diversified Ensemble Selection Classifier (ADES) that significantly optimizes adaptation to different types of concept drifts at any time and improves convergence to new concepts by exploiting different amounts of ensemble diversity. The ADES algorithm generates diverse base classifiers, thereby optimizing the margin distribution to exploit ensemble diversity to formulate an ensemble classifier that generalizes well to unseen instances and provides fast recovery from different types of concept drift. Empirical experiments conducted on both artificial and real-world data streams demonstrate that ADES can adapt to different types of drifts at any given time. The prediction performance of ADES is compared to three other ensemble classifiers designed to handle concept drift using both artificial and real-world data streams. The comparative evaluation performed demonstrated the ability of ADES to handle different types of concept drifts. The experimental results, including statistical test results, indicate comparable performances with other algorithms designed to handle concept drift and prove their significance and effectiveness.

Long short-term memory self-adapting online random forests for evolving data stream regression

Evolving data stream, especially with concept drift is generally accepted as a challenging data type for regression task, because it usually makes machine learning models trained on old data not adapting to new data, and leads to dramatic performance degradation as a result. Moreover, the behavior of a data stream may change in different modes and therefore introduces various concept drifts, e.g., abrupt, incremental, gradual, recurring, even more, complex concept drifts. Although there are some algorithms that can adapt to stationary data streams or a specific type of concept drift in non-stationary data streams, a wide range of practical applications call for machine learning regression models to handle multi-type of data streams. In this work, we propose an online learning strategy called adaptive long and short-term memories online Random Forests regression(ALSM-RFR), where an adaptive memory activation mechanism is designed to make the model switch adaptively between long-term and hybrid (long-term plus short-term) memory modes in the face of stationary data streams or non-stationary data streams with different types of concept drift. In particular, leaf and tree weights in random forests are used to learn information at different timescales, namely, long-term and short-term memories. Moreover, we devise an adaptive memory activation mechanism to formulate the switch decision of memory modes as a classification problem. Numerical experiments show remarkable improvements of the proposed method in the adaptability of stream types and predictive accuracy in data streams across several real datasets and synthetic datasets, compared to the state-of-the-art online approaches. Besides, the convergence and the influence of the parameters involved in our method are evaluated.

Supervised learning in the presence of concept drift: a modelling framework

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

An Adaptive Heterogeneous Online Learning Ensemble Classifier for Nonstationary Environments.

In recent years, the prevalence of technological advances has led to an enormous and ever-increasing amount of data that are now commonly available in a streaming fashion. In such nonstationary environments, the underlying process generating the data stream is characterized by an intrinsic nonstationary or evolving or drifting phenomenon known as concept drift. Given the increasingly common applications whose data generation mechanisms are susceptible to change, the need for effective and efficient algorithms for learning from and adapting to evolving or drifting environments can hardly be overstated. In dynamic environments associated with concept drift, learning models are frequently updated to adapt to changes in the underlying probability distribution of the data. A lot of work in the area of learning in nonstationary environments focuses on updating the learning predictive model to optimize recovery from concept drift and convergence to new concepts by adjusting parameters and discarding poorly performing models while little effort has been dedicated to investigate what type of learning model is suitable at any given time for different types of concept drift. In this paper, we investigate the impact of heterogeneous online ensemble learning based on online model selection for predictive modeling in dynamic environments. We propose a novel heterogeneous ensemble approach based on online dynamic ensemble selection that accurately interchanges between different types of base models in an ensemble to enhance its predictive performance in nonstationary environments. The approach is known as Heterogeneous Dynamic Ensemble Selection based on Accuracy and Diversity (HDES-AD) and makes use of models generated by different base learners to increase diversity to circumvent problems associated with existing dynamic ensemble classifiers that may experience loss of diversity due to the exclusion of base learners generated by different base algorithms. The algorithm is evaluated on artificial and real-world datasets with well-known online homogeneous online ensemble approaches such as DDD, AFWE, and OAUE. The results show that HDES-AD performed significantly better than the other three homogeneous online ensemble approaches in nonstationary environments.

Adaptation Strategies for Automated Machine Learning on Evolving Data.

Automated Machine Learning (AutoML) systems have been shown to efficiently build good models for new datasets. However, it is often not clear how well they can adapt when the data evolves over time. The main goal of this study is to understand the effect of concept drift on the performance of AutoML methods, and which adaptation strategies can be employed to make them more robust to changes in the underlying data. To that end, we propose 6 concept drift adaptation strategies and evaluate their effectiveness on a variety of AutoML approaches for building machine learning pipelines, including Bayesian optimization, genetic programming, and random search with automated stacking. These are evaluated empirically on real-world and synthetic data streams with different types of concept drift. Based on this analysis, we propose ways to develop more sophisticated and robust AutoML techniques.

Ensemble Method to classify multi class with concept drift

Real world scenarios is expected to have more number of classes, evolution of new class, change in concept, different distribution of data etc., This paper focus on the joint problem of multi class imbalance problem and different types of concept drift detection. The proposed work Enhanced Concept drift based Resampling ensemble (ECDRE) algorithm addresses the combined issues of Multi Class Imbalance problem with different types of Concept Drifts using the Ensemble Relearning Model. In this framework, EMUOB and EMOOB algorithm helps to re-balance the multi class distribution of the latest block. Window based strategy is used to classify the multi class imbalance dataset with different types of drifts. A group of classifiers form an ensemble and members belonging to the ensembles are updated frequently, where weights are calculated to emphasis misclassification cost. Meanwhile, ECDRE is swiftly adaptable to new conditions. Observed studies proof that ECDRE algorithm is more effective for learning non-stationary environment which suffers from multi class imbalance with different types of concept drift. CDRE algorithm was implemented to deal with multi-class problem and concept drift, but it is not much efficient to handle all types of concept drift. Even in situations of Data Streams which suffer from drifts such as real, virtual and recurring types, ECDRE works well and shows overall better results of above 80% accuracy.

A Novel Concept Drift Detection Method for Incremental Learning in Nonstationary Environments.

We present a novel method for concept drift detection, based on: 1) the development and continuous updating of online sequential extreme learning machines (OS-ELMs) and 2) the quantification of how much the updated models are modified by the newly collected data. The proposed method is verified on two synthetic case studies regarding different types of concept drift and is applied to two public real-world data sets and a real problem of predicting energy production from a wind plant. The results show the superiority of the proposed method with respect to alternative state-of-the-art concept drift detection methods. Furthermore, updating the prediction model when the concept drift has been detected is shown to allow improving the overall accuracy of the energy prediction model and, at the same time, minimizing the number of model updatings.

Exploiting fractal dimension and a distributed evolutionary approach to classify data streams with concept drifts

Evolutionary algorithms, i.e., Genetic Programming (GP), have been successfully used for the task of classification, mainly because they are less likely to get stuck in the local optimum, can operate on chunks of data and allow to compute more solutions in parallel. Ensemble techniques are usually more accurate than component learners constituting the ensemble and can be built in an incremental way, improving flexibility, adapting to changes and maintaining part of the history present in the data. This paper proposes a framework based on a distributed GP ensemble algorithm for coping with different types of concept drift for the task of classification of large data streams. The framework is able to detect changes in a very efficient way using only a detection function based on the fractal dimension, which can also works on new incoming unclassified data. Thus, a distributed GP algorithm is performed only when a change is detected in order to improve classification accuracy and this, together with the exploitation of an adaptive procedure, permits to answer in short time to these changes. Experiments are conducted on a real and on some artificial datasets in order to assess the capacity of the framework to detect the drift and quickly respond to it.

GOOWE

Designing adaptive classifiers for an evolving data stream is a challenging task due to the data size and its dynamically changing nature. Combining individual classifiers in an online setting, the ensemble approach, is a well-known solution. It is possible that a subset of classifiers in the ensemble outperforms others in a time-varying fashion. However, optimum weight assignment for component classifiers is a problem, which is not yet fully addressed in online evolving environments. We propose a novel data stream ensemble classifier, called Geometrically Optimum and Online-Weighted Ensemble (GOOWE), which assigns optimum weights to the component classifiers using a sliding window containing the most recent data instances. We map vote scores of individual classifiers and true class labels into a spatial environment. Based on the Euclidean distance between vote scores and ideal-points, and using the linear least squares (LSQ) solution, we present a novel, dynamic, and online weighting approach. While LSQ is used for batch mode ensemble classifiers, it is the first time that we adapt and use it for online environments by providing a spatial modeling of online ensembles. In order to show the robustness of the proposed algorithm, we use real-world datasets and synthetic data generators using the Massive Online Analysis (MOA) libraries. First, we analyze the impact of our weighting system on prediction accuracy through two scenarios. Second, we compare GOOWE with eight state-of-the-art ensemble classifiers in a comprehensive experimental environment. Our experiments show that GOOWE provides improved reactions to different types of concept drift compared to our baselines. The statistical tests indicate a significant improvement in accuracy, with conservative time and memory requirements.

Knowledge-maximized ensemble algorithm for different types of concept drift

Knowledge extraction from data streams has attracted attention in recent years due to its wide range of applications, including sensor networks, web clickstreams, and user interest analysis. Concept drift is one of the most important research topics in data stream mining. Many algorithms that can adapt to concept drift have been proposed. However, most of them specialize in only one type of concept drift and can rarely be used in the environments with a large number of unavailable sample labels. In this study, we propose a new data stream classifier called knowledge-maximized ensemble (KME). First, supervised and unsupervised knowledge are leveraged to detect concept drift, recognize recurrent concepts, and evaluate the weights of ensemble members. Second, the preserved labelled instances in past blocks can be reused to enhance the recognition ability of the candidate member. The final decision for an incoming observation is derived from all the prediction results of the component classifiers. Accordingly, the maximum utilization of the relevant information in a data stream can be achieved, which is critical to models with limited training data. Third, KME can react to multiple types of concept drift by combining the mechanisms of online and chunk-based ensembles. Finally, we compare KME with eight state-of-the-art classifiers on several synthetic and real-world datasets. The comparison demonstrates the effectiveness of KME in various types of concept drift scenarios.

Online Active Learning with Drifted Data Streams Using Paired Ensemble Framework

In learning to classify data streams, it is impractical and expensive to label all of the instances. Online active learning over streaming data poses additional challenges for its increasing volumes and concept drifts. We propose a new online paired ensemble active learning framework consisting of a stable classifier and a timely substituted dynamic classifier to react to different types of concept drifts. Classifiers are built in block based way and will learn new instances incrementally online. According to a combination strategy of uncertainty strategy and random strategy, the decision whether to label the incoming instance for the updating of the stable classifier and the dynamic classifier will be made. Experimental evaluation results on real datasets show the advantage of the proposed work in comparison with other approaches.