Types Of Concept Drift Research Articles

An extensive study of C-SMOTE, a Continuous Synthetic Minority Oversampling Technique for Evolving Data Streams

Streaming Machine Learning (SML) studies algorithms that update their models, given an unbounded and often non-stationary flow of data performing a single pass. Online class imbalance learning is a branch of SML that combines the challenges of both class imbalance and concept drift. In this paper, we investigate the binary classification problem by rebalancing an imbalanced stream of data in the presence of concept drift, accessing one sample at a time. We propose an extensive comparative study of Continuous Synthetic Minority Oversampling Technique (C-SMOTE), inspired by the popular sampling technique Smote, as a meta-strategy to pipeline with SML classification algorithms. We benchmark C-SMOTE pipelines on both synthetic and real data streams, containing different types of concept drifts, different imbalance levels, and different class distributions. We bring statistical evidence that models learnt with C-SMOTE pipelines improve the minority class performance concerning both the baseline models and the state-of-the-art methods. We also perform a sensitivity analysis to detect the C-SMOTE impact on the majority class performance for the three types of concept drift and several class distributions. Moreover, we show a computational cost analysis in terms of time and memory consumption.

Heuristic ensemble for unsupervised detection of multiple types of concept drift in data stream classification

Real-world data stream classification often deals with multiple types of concept drift, categorized by change characteristics such as speed, distribution, and severity. When labels are unavailable, traditional concept drift detection algorithms, used in stream classification frameworks, are often focused on only one type of concept drift. To overcome the limitations of traditional detection algorithms, this study proposed a Heuristic Ensemble Framework for Drift Detection (HEFDD). HEFDD aims to detect all types of concept drift by employing an ensemble of selected concept drift detection algorithms, each capable of detecting at least one type of concept drift. Experimental results show HEFDD provides significant improvement based on the z-score test when comparing detection accuracy with state-of-the-art individual algorithms. At the same time, HEFDD is able to reduce false alarms generated by individual concept drift detection algorithms.

DME: An Adaptive and Just-in-Time Weighted Ensemble Learning Method for Classifying Block-Based Concept Drift Steam

This study proposes a novel incremental learning algorithm called distribution matching ensemble (DME) in context of adaptive weighted ensemble learning. In particular, DME estimates the distribution of each received data block by Gaussian mixture model (GMM) and reserves the corresponding distribution information, as well it maintains a group of classifiers in a buffer. When we receive a new data block which is required to be predicted, the similarity between its distribution and each reserved distribution will be calculated by Kullback-Leibler (KL) divergence, and then the similarities can be used to guide the weight assignment of each corresponding classifier to further make adaptive ensemble decision. DME gets rid of the underlying hypothesis that the most recent labeled data block always has the most similar distribution with the current unlabeled data block. In addition, to avoid infinite extension of ensemble buffer during incremental learning, we also develop two dynamic classifier update rules. Experiments results on some synthetic and real-world streaming datasets show that the proposed DME algorithm is able to track and adapt to various types of concept drift just in time. Especially, on data stream with frequent reoccurring drifts, the DME significantly outperforms to several state-of-the-art algorithms, indicating its superiority.

Concept drift type identification based on multi-sliding windows

Concept drift is a common and important issue in streaming data analysis and mining. Thus far, many concept drift detection methods have been proposed but may not be able to identify the type of concept drift, which will result in some difficulties, such as extracting the wrong key information, inadequate model learning and poor detection efficiency. To solve these problems, a concept drift type identification method is proposed based on multi-sliding windows (CDT_MSW). This method consists of three processes. During the first detection process, the drift position is detected by sliding the basic window forward. Then, in the growth process, the drift length is detected using the growth of the adjoint window, and the drift category is identified according to the drift length. Finally, during tracking process, the drift subcategory can be accurately identified according to the different tracking flow ratio curves generated during window tracking. Experimental results show that the proposed method can effectively identify the type of concept drift, accurately analyze the key information during online learning and improve the efficiency and generalization performance of streaming data analysis and mining.

Selective ensemble-based online adaptive deep neural networks for streaming data with concept drift

Concept drift is an important issue in the field of streaming data mining. However, how to maintain real-time model convergence in a dynamic environment is an important and difficult problem. In addition, the current methods have limited ability to deal with the problem of streaming data classification for complex nonlinear problems. To solve these problems, a selective ensemble-based online adaptive deep neural network (SEOA) is proposed to address concept drift. First, the adaptive depth unit is constructed by combining shallow features with deep features and adaptively controls the information flow in the neural network according to changes in streaming data at adjacent moments, which improves the convergence of the online deep learning model. Then, the adaptive depth units of different layers are regarded as base classifiers for ensemble and weighted dynamically according to the loss of each classifier. In addition, a dynamic selection of base classifiers is adopted according to the fluctuation of the streaming data to achieve a balance between stability and adaptability. The experimental results show that the SEOA can effectively contend with different types of concept drift and has good robustness and generalization.

ADES: A New Ensemble Diversity-Based Approach for Handling Concept Drift

Beyond applying machine learning predictive models to static tasks, a significant corpus of research exists that applies machine learning predictive models to streaming environments that incur concept drift. With the prevalence of streaming real-world applications that are associated with changes in the underlying data distribution, the need for applications that are capable of adapting to evolving and time-varying dynamic environments can be hardly overstated. Dynamic environments are nonstationary and change with time and the target variables to be predicted by the learning algorithm and often evolve with time, a phenomenon known as concept drift. Most work in handling concept drift focuses on updating the prediction model so that it can recover from concept drift while little effort has been dedicated to the formulation of a learning system that is capable of learning different types of drifting concepts at any time with minimum overheads. This work proposes a novel and evolving data stream classifier called Adaptive Diversified Ensemble Selection Classifier (ADES) that significantly optimizes adaptation to different types of concept drifts at any time and improves convergence to new concepts by exploiting different amounts of ensemble diversity. The ADES algorithm generates diverse base classifiers, thereby optimizing the margin distribution to exploit ensemble diversity to formulate an ensemble classifier that generalizes well to unseen instances and provides fast recovery from different types of concept drift. Empirical experiments conducted on both artificial and real-world data streams demonstrate that ADES can adapt to different types of drifts at any given time. The prediction performance of ADES is compared to three other ensemble classifiers designed to handle concept drift using both artificial and real-world data streams. The comparative evaluation performed demonstrated the ability of ADES to handle different types of concept drifts. The experimental results, including statistical test results, indicate comparable performances with other algorithms designed to handle concept drift and prove their significance and effectiveness.

Long short-term memory self-adapting online random forests for evolving data stream regression

Evolving data stream, especially with concept drift is generally accepted as a challenging data type for regression task, because it usually makes machine learning models trained on old data not adapting to new data, and leads to dramatic performance degradation as a result. Moreover, the behavior of a data stream may change in different modes and therefore introduces various concept drifts, e.g., abrupt, incremental, gradual, recurring, even more, complex concept drifts. Although there are some algorithms that can adapt to stationary data streams or a specific type of concept drift in non-stationary data streams, a wide range of practical applications call for machine learning regression models to handle multi-type of data streams. In this work, we propose an online learning strategy called adaptive long and short-term memories online Random Forests regression(ALSM-RFR), where an adaptive memory activation mechanism is designed to make the model switch adaptively between long-term and hybrid (long-term plus short-term) memory modes in the face of stationary data streams or non-stationary data streams with different types of concept drift. In particular, leaf and tree weights in random forests are used to learn information at different timescales, namely, long-term and short-term memories. Moreover, we devise an adaptive memory activation mechanism to formulate the switch decision of memory modes as a classification problem. Numerical experiments show remarkable improvements of the proposed method in the adaptability of stream types and predictive accuracy in data streams across several real datasets and synthetic datasets, compared to the state-of-the-art online approaches. Besides, the convergence and the influence of the parameters involved in our method are evaluated.

Genetic programming-based regression for temporal data

Various machine learning techniques exist to perform regression on temporal data with concept drift occurring. However, there are numerous nonstationary environments where these techniques may fail to either track or detect the changes. This study develops a genetic programming-based predictive model for temporal data with a numerical target that tracks changes in a dataset due to concept drift. When an environmental change is evident, the proposed algorithm reacts to the change by clustering the data and then inducing nonlinear models that describe generated clusters. Nonlinear models become terminal nodes of genetic programming model trees. Experiments were carried out using seven nonstationary datasets and the obtained results suggest that the proposed model yields high adaptation rates and accuracy to several types of concept drifts. Future work will consider strengthening the adaptation to concept drift and the fast implementation of genetic programming on GPUs to provide fast learning for high-speed temporal data.

Supervised learning in the presence of concept drift: a modelling framework

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm.

The Multi-Armed Bandit (MAB) problem has been extensively studied in order to address real-world challenges related to sequential decision making. In this setting, an agent selects the best action to be performed at time-step t, based on the past rewards received by the environment. This formulation implicitly assumes that the expected payoff for each action is kept stationary by the environment through time. Nevertheless, in many real-world applications this assumption does not hold and the agent has to face a non-stationary environment, that is, with a changing reward distribution. Thus, we present a new MAB algorithm, named f-Discounted-Sliding-Window Thompson Sampling (f-dsw TS), for non-stationary environments, that is, when the data streaming is affected by concept drift. The f-dsw TS algorithm is based on Thompson Sampling (TS) and exploits a discount factor on the reward history and an arm-related sliding window to contrast concept drift in non-stationary environments. We investigate how to combine these two sources of information, namely the discount factor and the sliding window, by means of an aggregation function . In particular, we proposed a pessimistic (), an optimistic (), as well as an averaged () version of the f-dsw TS algorithm. A rich set of numerical experiments is performed to evaluate the f-dsw TS algorithm compared to both stationary and non-stationary state-of-the-art TS baselines. We exploited synthetic environments (both randomly-generated and controlled) to test the MAB algorithms under different types of drift, that is, sudden/abrupt, incremental, gradual and increasing/decreasing drift. Furthermore, we adapt four real-world active learning tasks to our framework—a prediction task on crimes in the city of Baltimore, a classification task on insects species, a recommendation task on local web-news, and a time-series analysis on microbial organisms in the tropical air ecosystem. The f-dsw TS approach emerges as the best performing MAB algorithm. At least one of the versions of f-dsw TS performs better than the baselines in synthetic environments, proving the robustness of f-dsw TS under different concept drift types. Moreover, the pessimistic version () results as the most effective in all real-world tasks.

Adaptation Strategies for Automated Machine Learning on Evolving Data.

Automated Machine Learning (AutoML) systems have been shown to efficiently build good models for new datasets. However, it is often not clear how well they can adapt when the data evolves over time. The main goal of this study is to understand the effect of concept drift on the performance of AutoML methods, and which adaptation strategies can be employed to make them more robust to changes in the underlying data. To that end, we propose 6 concept drift adaptation strategies and evaluate their effectiveness on a variety of AutoML approaches for building machine learning pipelines, including Bayesian optimization, genetic programming, and random search with automated stacking. These are evaluated empirically on real-world and synthetic data streams with different types of concept drift. Based on this analysis, we propose ways to develop more sophisticated and robust AutoML techniques.

An Adaptive Heterogeneous Online Learning Ensemble Classifier for Nonstationary Environments.

In recent years, the prevalence of technological advances has led to an enormous and ever-increasing amount of data that are now commonly available in a streaming fashion. In such nonstationary environments, the underlying process generating the data stream is characterized by an intrinsic nonstationary or evolving or drifting phenomenon known as concept drift. Given the increasingly common applications whose data generation mechanisms are susceptible to change, the need for effective and efficient algorithms for learning from and adapting to evolving or drifting environments can hardly be overstated. In dynamic environments associated with concept drift, learning models are frequently updated to adapt to changes in the underlying probability distribution of the data. A lot of work in the area of learning in nonstationary environments focuses on updating the learning predictive model to optimize recovery from concept drift and convergence to new concepts by adjusting parameters and discarding poorly performing models while little effort has been dedicated to investigate what type of learning model is suitable at any given time for different types of concept drift. In this paper, we investigate the impact of heterogeneous online ensemble learning based on online model selection for predictive modeling in dynamic environments. We propose a novel heterogeneous ensemble approach based on online dynamic ensemble selection that accurately interchanges between different types of base models in an ensemble to enhance its predictive performance in nonstationary environments. The approach is known as Heterogeneous Dynamic Ensemble Selection based on Accuracy and Diversity (HDES-AD) and makes use of models generated by different base learners to increase diversity to circumvent problems associated with existing dynamic ensemble classifiers that may experience loss of diversity due to the exclusion of base learners generated by different base algorithms. The algorithm is evaluated on artificial and real-world datasets with well-known online homogeneous online ensemble approaches such as DDD, AFWE, and OAUE. The results show that HDES-AD performed significantly better than the other three homogeneous online ensemble approaches in nonstationary environments.

Hierarchical Reduced-space Drift Detection Framework for Multivariate Supervised Data Streams

In a streaming environment, the characteristics of the data themselves and their relationship with the labels are likely to experience changes as time goes on. Most drift detection methods for supervised data streams are performance-based, that is, they detect changes only after the classication accuracy deteriorates. This may not be sufcient in many application areas where the reason behind a drift is also important. Another category of drift detectors are data distribution-based detectors. Although they can detect some drifts within the input space, changes affecting only the labelling mechanism cannot be identied. Furthermore, little work is available on drift detection for high-dimensional supervised data streams. In this paper we propose an advanced Hierarchical Reduced-space Drift Detection Framework for Supervised Data Streams (HRDS) which captures drifts regardless of their effects on classication performance. This framework suggests monitoring both marginal and class-conditional distributions within a lower-dimensional space specically relevant to the assigned classication task. Experimental comparisons have demonstrated that the proposed HRDS not only achieves high-quality performance on high-dimensional data streams, but also outperforms its competitors in terms of detection recall, precision and F-measure across a wide range of different concept drift types including subtle drifts.

Ensemble Method to classify multi class with concept drift

Real world scenarios is expected to have more number of classes, evolution of new class, change in concept, different distribution of data etc., This paper focus on the joint problem of multi class imbalance problem and different types of concept drift detection. The proposed work Enhanced Concept drift based Resampling ensemble (ECDRE) algorithm addresses the combined issues of Multi Class Imbalance problem with different types of Concept Drifts using the Ensemble Relearning Model. In this framework, EMUOB and EMOOB algorithm helps to re-balance the multi class distribution of the latest block. Window based strategy is used to classify the multi class imbalance dataset with different types of drifts. A group of classifiers form an ensemble and members belonging to the ensembles are updated frequently, where weights are calculated to emphasis misclassification cost. Meanwhile, ECDRE is swiftly adaptable to new conditions. Observed studies proof that ECDRE algorithm is more effective for learning non-stationary environment which suffers from multi class imbalance with different types of concept drift. CDRE algorithm was implemented to deal with multi-class problem and concept drift, but it is not much efficient to handle all types of concept drift. Even in situations of Data Streams which suffer from drifts such as real, virtual and recurring types, ECDRE works well and shows overall better results of above 80% accuracy.

Online Learning With Adaptive Rebalancing in Nonstationary Environments.

An enormous and ever-growing volume of data is nowadays becoming available in a sequential fashion in various real-world applications. Learning in nonstationary environments constitutes a major challenge, and this problem becomes orders of magnitude more complex in the presence of class imbalance. We provide new insights into learning from nonstationary and imbalanced data in online learning, a largely unexplored area. We propose the novel Adaptive REBAlancing (AREBA) algorithm that selectively includes in the training set a subset of the majority and minority examples that appeared so far, while at its heart lies an adaptive mechanism to continually maintain the class balance between the selected examples. We compare AREBA with strong baselines and other state-of-the-art algorithms and perform extensive experimental work in scenarios with various class imbalance rates and different concept drift types on both synthetic and real-world data. AREBA significantly outperforms the rest with respect to both learning speed and learning quality. Our code is made publicly available to the scientific community.

Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams

This work aims to connect two rarely combined research directions, i.e., non-stationary data stream classification and data analysis with skewed class distributions. We propose a novel framework employing stratified bagging for training base classifiers to integrate data preprocessing and dynamic ensemble selection methods for imbalanced data stream classification. The proposed approach has been evaluated based on computer experiments carried out on 135 artificially generated data streams with various imbalance ratios, label noise levels, and types of concept drift as well as on two selected real streams. Four preprocessing techniques and two dynamic selection methods, used on both bagging classifiers and base estimators levels, were considered. Experimentation results showed that, for highly imbalanced data streams, dynamic ensemble selection coupled with data preprocessing could outperform online and chunk-based state-of-art methods.

CONDA-PM—A Systematic Review and Framework for Concept Drift Analysis in Process Mining

Business processes evolve over time to adapt to changing business environments. This requires continuous monitoring of business processes to gain insights into whether they conform to the intended design or deviate from it. The situation when a business process changes while being analysed is denoted as Concept Drift. Its analysis is concerned with studying how a business process changes, in terms of detecting and localising changes and studying the effects of the latter. Concept drift analysis is crucial to enable early detection and management of changes, that is, whether to promote a change to become part of an improved process, or to reject the change and make decisions to mitigate its effects. Despite its importance, there exists no comprehensive framework for analysing concept drift types, affected process perspectives, and granularity levels of a business process. This article proposes the CONcept Drift Analysis in Process Mining (CONDA-PM) framework describing phases and requirements of a concept drift analysis approach. CONDA-PM was derived from a Systematic Literature Review (SLR) of current approaches analysing concept drift. We apply the CONDA-PM framework on current approaches to concept drift analysis and evaluate their maturity. Applying CONDA-PM framework highlights areas where research is needed to complement existing efforts.

Integrating adaptive moving window and just-in-time learning paradigms for soft-sensor design

Most applications of soft sensors in process industries require learning from a stream of data, which may exhibit nonstationary dynamics, or concept drift. In this study, we develop a relevance vector machine (RVM) based novel adaptive learning algorithm called MWAdp-JITL, to meet the demands of continuous processes. The resulting algorithm combines active and passive learning: A moving window (MW) algorithm, which adapts the window size against virtual/real concept drifts, is coupled with a just-in-time learning (JITL) model, constructed using an appropriate region of historical data, and the ensemble weights of the MW and JITL models are adjusted for each query point. Tests on four real industrial datasets and a synthetic data, comprising various concept drift scenarios, show that MWAdp-JITL yields superior prediction accuracy and is generally more robust to changes in algorithm parameters compared to conventional adaptive learning methods and state-of-the-art algorithms from the literature. MWAdp-JITL complies with time limits of online prediction, and is applicable for high dimensional processes under various types of concept drifts. It is seen that MWAdp-JITL can successfully achieve a good balance in bias-variance tradeoff, justifying the use of only two exquisitely selected learners in ensemble learning.

No Free Lunch Theorem for concept drift detection in streaming data classification: A review

AbstractMany real‐world data mining applications have to deal with unlabeled streaming data. They are unlabeled because the sheer volume of the stream makes it impractical to label a significant portion of the data. The data streams can evolve over time and these changes are called concept drifts. Concept drifts have different characteristics, which can be used to categorize them into different types. A trade‐off between performance and cost exists among many concept drift detection approaches. On the one hand, high accuracy detection approach usually requires labeled data, possibly involving high cost for labeling. On the other hand, a variety of methods have been devoted to the topic of concept drift detection with unlabeled data, but these approaches often are most suited for only a subset of the concept drift types. The objective of this survey is to present these methods, categorize them and give recommendations of usage based on their behaviors under different types of concept drift.This article is categorized under: Fundamental Concepts of Data and Knowledge > Data Concepts Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Explainable AI > Classification

A Novel Concept Drift Detection Method for Incremental Learning in Nonstationary Environments.

We present a novel method for concept drift detection, based on: 1) the development and continuous updating of online sequential extreme learning machines (OS-ELMs) and 2) the quantification of how much the updated models are modified by the newly collected data. The proposed method is verified on two synthetic case studies regarding different types of concept drift and is applied to two public real-world data sets and a real problem of predicting energy production from a wind plant. The results show the superiority of the proposed method with respect to alternative state-of-the-art concept drift detection methods. Furthermore, updating the prediction model when the concept drift has been detected is shown to allow improving the overall accuracy of the energy prediction model and, at the same time, minimizing the number of model updatings.