Concept Drifting Data Streams Research Articles

Data stream classification: a review

The tremendous amount of data is generated regularly through areas like networking, telecommunication, stock market, satellite, weather forecasting, etc. So, the classification process becomes important to extract knowledge from such a huge amount of data. The handling of concept drifting data stream and skewed data classification are the major issues and challenges in the data streams mining field. In the presence of concept drift, the performance of the learning algorithm always degrades. On the other side in skewed data problems, majority class accuracy always dominates minority class accuracy which generates the wrong result. This paper discusses the implemented methods which worked for skewed data and concept drifting data as well as their merits and demerits. This paper focused on metrics used in the classification process and issues that arise while learning with skewed and concept drifting data.

STDS: self-training data streams for mining limited labeled data in non-stationary environment

Inthis article, wefocus on the classification problem to semi-supervised learning in non-stationary environment. Semi-supervised learning is a learning task from both labeled and unlabeled data points. There are several approaches to semi-supervised learning in stationary environment which are not applicable directly for data streams. We propose a novel semi-supervised learning algorithm, named STDS. The proposed approach uses labeled and unlabeled data and employs an approach to handle the concept drift in data streams. The main challenge in semi-supervised self-training for data streams is to find a proper selection metric in order to find a set of high-confidence predictions and a proper underlying base learner. We therefore propose an ensemble approach to find a set of high-confidence predictions based on clustering algorithms and classifier predictions. We then employ the Kullback-Leibler (KL) divergence approach to measure the distribution differences between sequential chunks in order to detect the concept drift. When drift is detected, a new classifier is updated from the new set of labeled data in the current chunk; otherwise, a percentage of high-confidence newly labeled data in the current chunk is added to the labeled data in the next chunk for updating the incremental classifier based on the proposed selection metric. The results of our experiments on a number of classification benchmark datasets show that STDS outperforms the supervised and the most of other semi-supervised learning methods.

Novel Class Detection with Concept Drift in Data Stream - AhtNODE

Data stream mining has become an interesting analysis topic and it is a growing interest in data discovery method. There are several applications supporting stream data processing like device network, electronic network, etc. Our approach AhtNODE (Adaptive Hoeffding Tree based NOvel class DEtection) detects novel class in the presence of concept drift in streaming data. It addresses there are three challenges of streaming data: infinite length, concept drift, and concept evolution. This approach automatically detects the novel class whenever it arrives in the data stream. It is a multi-class approach that distinguishes novel class from existing classes. The authors tend to apply the Adaptive Hoeffding Tree as a classification model that is also used to handle the concept drift situation. Previous approaches used the ensemble model to handle concept drift. In AHT, classification is done in the single pass. The experiment result proves the effectiveness of AhtNODE compared to existing ensemble classifier in terms of classification accuracy, speed and use of memory.

Selective prototype-based learning on concept-drifting data streams

Data stream mining has gained increasing attention in recent years due to its wide range of applications. In this paper, we propose a new selective prototype-based learning (SPL) method on evolving data streams, which dynamically maintains representative instances to capture the time-changing concepts, and make predictions in a local fashion. As an instance-based learning model, SPL only maintains some important prototypes (i.e., ISet) via error-driven representativeness learning. The fast condensed nearest neighbor (FCNN) rule, is further introduced to compress these prototypes, making the algorithm also applicable under memory constraints. To better distinguish noises from the instances associated with the new emerging concept, a potential concept instance set (i.e., PSet) is used to store all misclassified instances. Relying on the potential concept instance set, a local-aware distribution-based concept drift detection approach is proposed. SPL has several attractive benefits: (a) it can fit the evolving data streams very well by maintaining a small size of instance set; (b) it is capable of capturing both gradual and sudden concept drifts effectively; (c) it has great capabilities to distinguish noise/outliers from drifting instances. Experimental results show that the SPL has better classification performance than many other state-of-the-art algorithms.

Hybrid short term prediction to address limited timeliness of public transport data streams

In spite of the diversity of solutions developed in the Internet of Things (IoT) domain, some features are shared by numerous IoT deployments and the data they process. These include incompleteness and latency in data transmission from multiple distributed objects. Among others, the systems tracking the location of vehicles are affected by these problems. The primary objective of this work is to address the way the latency in location data acquisition, referred to also as timeliness, can be dealt with. We propose a hybrid method combining machine learning models such as multilayer perceptrons trained in batch mode and online learning methods to perform short-term prediction of vehicle delay data. In this way, stream instances that have not arrived yet from the sensors can be temporarily replaced with predicted values. The method we propose successfully integrates stream mining methods developed for stationary and non-stationary conditions i.e. also the methods developed for concept drifting data streams. For all examined reference data sets and hybridised stream methods, the method reduced prediction error and addressed the risk of using static prediction models not matching or no longer matching the evolving process for which the prediction is performed.

Efficient Ensemble Classification for Multi-Label Data Streams with Concept Drift

Most existing multi-label data streams classification methods focus on extending single-label streams classification approaches to multi-label cases, without considering the special characteristics of multi-label stream data, such as label dependency, concept drift, and recurrent concepts. Motivated by these challenges, we devise an efficient ensemble paradigm for multi-label data streams classification. The algorithm deploys a novel change detection based on Jensen–Shannon divergence to identify different kinds of concept drift in data streams. Moreover, our method tries to consider label dependency by pruning away infrequent label combinations to enhance classification performance. Empirical results on both synthetic and real-world datasets have demonstrated its effectiveness.

Most preferable combination of explicit drift detection approaches with different classifiers for mining concept drifting data streams

Sensors in the real-world applications are the major sources of big data streams with varying underlying data distribution. Continuously generated time varying data streams are commonly referred as concept drifting data streams. Many concept drifting data mining algorithms explicitly utilise the drift detection algorithms for ensuring the forgetting of out-dated concepts and learn new concepts upon occurrence of drifts. In concept drifting data streams, the accuracy of the learner depends on the accuracy of the drift detection algorithm and its promptness towards drifts detection. For maintaining the consistent high accuracy in the classification of concept drifting data streams, it is very important to understand the preferable combinations of drift detection algorithms with the classification algorithms. In order to explore such preferable combinations, this work presents an empirical evaluation of some popular drift detection methods with some state-of-art classification algorithms on some standard benchmark datasets of real world.

Most preferable combination of explicit drift detection approaches with different classifiers for mining concept drifting data streams

Sensors in the real-world applications are the major sources of big data streams with varying underlying data distribution. Continuously generated time varying data streams are commonly referred as concept drifting data streams. Many concept drifting data mining algorithms explicitly utilise the drift detection algorithms for ensuring the forgetting of out-dated concepts and learn new concepts upon occurrence of drifts. In concept drifting data streams, the accuracy of the learner depends on the accuracy of the drift detection algorithm and its promptness towards drifts detection. For maintaining the consistent high accuracy in the classification of concept drifting data streams, it is very important to understand the preferable combinations of drift detection algorithms with the classification algorithms. In order to explore such preferable combinations, this work presents an empirical evaluation of some popular drift detection methods with some state-of-art classification algorithms on some standard benchmark datasets of real world.

Regression Function and Noise Variance Tracking Methods for Data Streams with Concept Drift

Abstract Two types of heuristic estimators based on Parzen kernels are presented. They are able to estimate the regression function in an incremental manner. The estimators apply two techniques commonly used in concept-drifting data streams, i.e., the forgetting factor and the sliding window. The methods are applicable for models in which both the function and the noise variance change over time. Although nonparametric methods based on Parzen kernels were previously successfully applied in the literature to online regression function estimation, the problem of estimating the variance of noise was generally neglected. It is sometimes of profound interest to know the variance of the signal considered, e.g., in economics, but it can also be used for determining confidence intervals in the estimation of the regression function, as well as while evaluating the goodness of fit and in controlling the amount of smoothing. The present paper addresses this issue. Specifically, variance estimators are proposed which are able to deal with concept drifting data by applying a sliding window and a forgetting factor, respectively. A number of conducted numerical experiments proved that the proposed methods perform satisfactorily well in estimating both the regression function and the variance of the noise.

Data Mining Models of High Dimensional Data Streams, and Contemporary Concept Drift Detection Methods: a Comprehensive Review

Concept drift is defined as the distributed data across multiple data streams that change over the time. Concept drift is visible only when the type of collected data changes after some stable period. The emergence of concept drift in data streams leads to increase misclassification and performing degradation of data streams. In order to obtain accurate results, identification of such concept drifts must be visible. This paper focused on a review of the issues related to identifying the changes occurred in the various multivariate high dimensional data streams. The insight of the manuscript is probing the inbuilt difficulties of existing contemporary change-detection methods when they encounter during data dimensions scales.

An Overview of Real World Applications with Concept Drifting Data Streams

In online learning, the concept drifts refers to the situations where the objective variable conforming to the input data changes over time. This change in distribution of data over time can be studied in different forms namely abrupt, gradual, incremental and reoccurring concepts. This paper illustrates the incremental learning with concept drift, different forms of drift, causes of concept drift and present a set of illustrative concept drift applications related with real-life problems. The various dimensions such as learning speed, prediction and classification accuracy, mistakes penalty, adversary activities and true labels are also discussed in relevance to these applications having the problem of concept drift.

Ensemble Classification of Data Streams Based on Attribute Reduction and a Sliding Window

With the current increasing volume and dimensionality of data, traditional data classification algorithms are unable to satisfy the demands of practical classification applications of data streams. To deal with noise and concept drift in data streams, we propose an ensemble classification algorithm based on attribute reduction and a sliding window in this paper. Using mutual information, an approximate attribute reduction algorithm based on rough sets is used to reduce data dimensionality and increase the diversity of reduced results in the algorithm. A double-threshold concept drift detection method and a three-stage sliding window control strategy are introduced to improve the performance of the algorithm when dealing with both noise and concept drift. The classification precision is further improved by updating the base classifiers and their nonlinear weights. Experiments on synthetic datasets and actual datasets demonstrate the performance of the algorithm in terms of classification precision, memory use, and time efficiency.

A two ensemble system to handle concept drifting data streams: recurring dynamic weighted majority

We present an ensemble system, recurring dynamic weighted majority (RDWM) that maintains two ensembles of experts, so as to accurately handle drifting concepts mainly recurrent drifts. The primary online ensemble represents the present concepts and the secondary ensemble represents the old concepts since the beginning of learning. An effective pruning methodology helps to remove redundant and old classifiers, which may have otherwise caused interference in learning the new concepts. Experimental evaluation using datasets proves that RDWM achieves very high generalization accuracy, irrespective of the speed or severity of drift; or presence of noise in the dataset.

Detecting concept drift in data streams using model explanation

Learning from data streams (incremental learning) is increasingly attracting research focus due to many real-world streaming problems and due to many open challenges, among which is the detection of concept drift – a phenomenon when the data distribution changes and makes the current prediction model inaccurate or obsolete. Current state-of-the art detection methods can be roughly split into performance monitoring algorithms and distribution comparing algorithms. In this work we propose a novel concept drift detector that can be combined with an arbitrary classification algorithm. The proposed concept drift detector is based on computing multiple model explanations over time and observing the magnitudes of their changes. The model explanation is computed using a methodology that yields attribute-value contributions for prediction outcomes and thus provides insight into the model’s decision-making process and enables its transparency. The evaluation has revealed that the methods surpass the baseline methods in terms of concept drift detection, accuracy, robustness and sensitivity. To even further augment interpretability, we visualized the detection of concept drift, enabling macro and micro views of the data.

On-line self-adaptive framework for tailoring a neural-agent learning model addressing dynamic real-time scheduling problems

The dynamic nature and time-varying behavior of actual environments provide serious challenges for learning models. Thus, changes may deteriorate the constructed control policy over time, which requires permanent adaptation strategies. Changes usually appear as an evolution in the relationship between instance variables composing stream data, known in machine learning under the term concept drift. Several adaptation strategies have been performed to tackle concept drifting data streams, always assuming that arrived instances are labeled, either completely or partially. However, this assumption is violated in many application areas, especially in the manufacturing field. We propose, in this paper, a new framework called Labeling Extraction from the current Model (LEM). LEM is adapted to retrieve learning labels, relying uniquely on unlabeled received instances and without any external supervision, which has never been previously addressed. Hence, to the best of our knowledge, there has been no effort addressing scheduling manufacturing problems for adaptation to data streams with concept drifts. Experiments are conducted to show the effectiveness of LEM. The obtained results demonstrate the ability of LEM to maintain the stability and efficiency of the control policy approximated by the learning model, by significantly improving its prediction performance, compared to its use without adaptation.

Concept drift detection for data stream learning based on angle optimized global embedding and principal component analysis in sensor networks

As the significant component in Industrial Internet of Things (IIoT), sensor networks have been applied widely in many fields. However, concept drift in data stream produced in sensor networks always brings great difficulty for the robustness of data processing. To solve the problem, we propose a novel concept drift detection method based on angle optimized global embedding (AOGE) and principal component analysis (PCA) for data stream learning in sensors networks. AOGE and PCA analyze the principal components through the projection variance and the projection angle in the subspace, respectively. And then the occurrence of concept drift is determined by observing the change of subspace for each data stream patch. The experiments in synthetic datasets and Intel Lab data demonstrate witness the effectiveness of our method.

Mining multi-dimensional concept-drifting data streams using Bayesian network classifiers

In recent years, a plethora of approaches have been proposed to deal with the increasingly challenging task of mining concept-drifting data streams. However, most of these approaches can only be applied to uni-dimensional classification problems where each input instance has to be assigned to a single output class variable. The problem of mining multi-dimensional data streams, which includes multiple output class variables, is largely unexplored and only few streaming multi-dimensional approaches have been recently introduced. In this paper, we propose a novel adaptive method, named Locally Adaptive-MB-MBC (LA-MB-MBC), for mining streaming multi-dimensional data. To this end, we make use of multi-dimensional Bayesian network classifiers (MBCs) as models. Basically, LA-MB-MBC monitors the concept drift over time using the average log-likelihood score and the Page-Hinkley test. Then, if a concept drift is detected, LA-MB-MBC adapts the current MBC network locally around each changed node. An experimental study carried out using synthetic multi-dimensional data streams shows the merits of the proposed method in terms of concept drift detection as well as classification performance.

Structural XML Classification in Concept Drifting Data Streams

Classification of large, static collections of XML data has been intensively studied in the last several years. Recently however, the data processing paradigm is shifting from static to streaming data, where documents have to be processed online using limited memory and class definitions can change with time in an event called concept drift. As most existing XML classifiers are capable of processing only static data, there is a need to develop new approaches dedicated for streaming environments. In this paper, we propose a new classification algorithm for XML data streams called XSC. The algorithm uses incrementally mined frequent subtrees and a tree-subtree similarity measure to classify new documents in an associative manner. The proposed approach is experimentally evaluated against eight state-of-the-art stream classifiers on real and synthetic data. The results show that XSC performs significantly better than competitive algorithms in terms of accuracy and memory usage.

Empirical Support for Concept Drifting Approaches: Results Based on New Performance Metrics

Various types of online learning algorithms have been developed so far to handle concept drift in data streams. We perform more detailed evaluation of these algorithms through new performance metrics - prequential accuracy, kappa statistic, CPU evaluation time, model cost, and memory usage. Experimental evaluation using various artificial and real-world datasets prove that the various concept drifting algorithms provide highly accurate results in classifying new data instances even in a resource constrained environment, irrespective of size of dataset, type of drift or presence of noise in the dataset. We also present empirically the impact of various features- size of ensemble, period value, threshold value, multiplicative factor and the presence of noise on all the key performance metrics.

An online ensembles approach for handling concept drift in data streams: diversified online ensembles detection

Data Streams are continuous data instances arriving at a very high speed with varying underlying conceptual distribution. We present a novel online ensemble approach, Diversified online ensembles detection (DOED), for handling these drifting concepts in data streams. Our approach maintains two ensembles of weighted experts, an ensemble with low diversity and an ensemble with high diversity, which are updated as per their accuracy in classifying the new data instances. Our approach detects drifts by comparing the two accuracies: an accuracy of an ensemble on the recent examples and its accuracy since the beginning of the learning. The final prediction for an instance is the class predicted by the ensemble which gives better accuracy in classifying the recent examples. When a drift is detected by an ensemble, it is reinitialized still maintaining its diversity levels. Experimental evaluation using various artificial and real-world datasets proves that DOED provides very high accuracy in classifying new data instances, irrespective of the size of dataset, type of drift or presence of noise. We compare DOED with the other learners in terms of new performance metrics such as kappa statistic, model cost, and the evaluation time and memory requirements. Our approach proved to be highly resource effective achieving very high accuracies even in a resource constrained environment.