Evolving Data Streams Research Articles

Selective prototype-based learning on concept-drifting data streams

Data stream mining has gained increasing attention in recent years due to its wide range of applications. In this paper, we propose a new selective prototype-based learning (SPL) method on evolving data streams, which dynamically maintains representative instances to capture the time-changing concepts, and make predictions in a local fashion. As an instance-based learning model, SPL only maintains some important prototypes (i.e., ISet) via error-driven representativeness learning. The fast condensed nearest neighbor (FCNN) rule, is further introduced to compress these prototypes, making the algorithm also applicable under memory constraints. To better distinguish noises from the instances associated with the new emerging concept, a potential concept instance set (i.e., PSet) is used to store all misclassified instances. Relying on the potential concept instance set, a local-aware distribution-based concept drift detection approach is proposed. SPL has several attractive benefits: (a) it can fit the evolving data streams very well by maintaining a small size of instance set; (b) it is capable of capturing both gradual and sudden concept drifts effectively; (c) it has great capabilities to distinguish noise/outliers from drifting instances. Experimental results show that the SPL has better classification performance than many other state-of-the-art algorithms.

General Temporally Biased Sampling Schemes for Online Model Management

To maintain the accuracy of supervised learning models in the presence of evolving data streams, we provide temporally biased sampling schemes that weight recent data most heavily, with inclusion probabilities for a given data item decaying over time according to a specified “decay function.” We then periodically retrain the models on the current sample. This approach speeds up the training process relative to training on all of the data. Moreover, time-biasing lets the models adapt to recent changes in the data while—unlike in a sliding-window approach—still keeping some old data to ensure robustness in the face of temporary fluctuations and periodicities in the data values. In addition, the sampling-based approach allows existing analytic algorithms for static data to be applied to dynamic streaming data essentially without change. We provide and analyze both a simple sampling scheme (Targeted-Size Time-Biased Sampling (T-TBS)) that probabilistically maintains a target sample size and a novel reservoir-based scheme (Reservoir-Based Time-Biased Sampling (R-TBS)) that is the first to provide both control over the decay rate and a guaranteed upper bound on the sample size. If the decay function is exponential, then control over the decay rate is complete, and R-TBS maximizes both expected sample size and sample-size stability. For general decay functions, the actual item inclusion probabilities can be made arbitrarily close to the nominal probabilities, and we provide a scheme that allows a tradeoff between sample footprint and sample-size stability. R-TBS rests on the notion of a “fractional sample” and allows for data arrival rates that are unknown and time varying (unlike T-TBS). The R-TBS and T-TBS schemes are of independent interest, extending the known set of unequal-probability sampling schemes. We discuss distributed implementation strategies; experiments in Spark illuminate the performance and scalability of the algorithms, and show that our approach can increase machine learning robustness in the face of evolving data.

Technical Perspective

Randoms sampling from data streams is a problem with a long history of studies, starting from the famous reservoir sampling algorithm that is at least 50 years old [2]. The reservoir sampling algorithm maintains a random sample over all data items that have ever been received from the stream. This is not suitable for many of today's applications on evolving data streams, where recent data is more important than older ones.

Online Model Management via Temporally Biased Sampling

To maintain the accuracy of supervised learning models in the presence of evolving data streams, we provide temporallybiased sampling schemes that weight recent data most heavily, with inclusion probabilities for a given data item decaying exponentially over time. We then periodically retrain the models on the current sample. We provide and analyze both a simple sampling scheme (T-TBS) that probabilistically maintains a target sample size and a novel reservoirbased scheme (R-TBS) that is the first to provide both control over the decay rate and a guaranteed upper bound on the sample size. The R-TBS and T-TBS schemes are of independent interest, extending the known set of unequalprobability sampling schemes. We discuss distributed implementation strategies; experiments in Spark show that our approach can increase machine learning accuracy and robustness in the face of evolving data.

SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams

The goal of continuous learning is to acquire and fine-tune knowledge incrementally without erasing already existing knowledge. How to mitigate this erasure, known as catastrophic forgetting, is a grand challenge for machine learning, specifically when systems are trained on evolving data streams. Self-organizing incremental neural networks (SOINN) are a class of neural networks that are designed for continuous learning from non-stationary data. Here, we propose a novel method, SOINN+, which is fundamentally different from the previous generations of SOINN with respect to how “forgetting” is modeled. To achieve a more graceful forgetting, we developed three new concepts: idle time, trustworthiness, and unutility of a node. SOINN+ learns a topology-preserving mapping of the input data to a network structure, which reveals clusters of arbitrary shapes in streams of noisy data. Our experiments with synthetic and real-world data sets showed that SOINN+ can maintain discovered structures even under sudden and recurring concept drifts.

Mining Closed Item sets from Tuple-Evolving Data Streams

Frequent Itemset Mining is playing major role in extracting useful knowledge from data streams that are exhibiting high data flow. Studies in data streams shows that every incoming data is considered as new tuple which is considered as revised tuple in some applications called as tuple evolving data streams. Extracting redundant less knowledge from such kind of application helps in better decision making with new challenges.One of the issue is, due to incoming revised tuple, some of the frequent itemsets may turn to infrequent or previously ignore itemsets may become frequent. Other issue is result of FIM may be huge and redundant results.In this paper, we address solution to the problem by finding closed itemsets from tuple revision data streams. We propose an efficient approach MCST that uses compressed SlideTree data structure to maintain stream data,proposeHIS hash tableto maintain itemsets, and CIS tables to maintain closed id sets to improve search performance of HIS.

An outlier detection approach in large-scale data stream using rough set

Outlier detection has become an important research area in the field of stream data mining due to its vast applications. In the literature, many methods have been proposed, but they work well for simple and positive regions of outliers, where boundary regions are not given much importance. Moreover, an algorithm which processes stream data must be effective and able to compute infinite data in one pass or limited number of passes. These problems have motivated us to propose an outlier detection approach for large-scale data stream. The proposed algorithm employs the concept of relative cardinality, entropy outlier factor theory of information-based system, and size-variant sliding window in stream data. In addition, we propose a new methodology for concept drift adaptation on evolving data streams. The proposed method is executed on nine benchmark datasets and compared with six existing methods that are EXPoSE, iForest, OC-SVM, LOF, KDE, and FastAbod. Experimental results show that the proposed method outperforms six existing methods in terms of receiver operating characteristic curve, precision recall, and computational time for positive regions as well as for boundary regions.

Recurring concept meta-learning for evolving data streams

When concept drift is detected during classification in a data stream, a common remedy is to retrain a framework’s classifier. However, this loses useful information if the classifier has learnt the current concept well, and this concept will recur again in the future. Some frameworks retain and reuse classifiers, but it can be time-consuming to select an appropriate classifier to reuse. These frameworks rarely match the accuracy of state-of-the-art ensemble approaches. For many data stream tasks, speed is important: fast, accurate frameworks are needed for time-dependent applications. We propose the Enhanced Concept Profiling Framework (ECPF), which aims to recognise recurring concepts and reuse a classifier trained previously, enabling accurate classification immediately following a drift. The novelty of ECPF is in how it uses similarity of classifications on new data, between a new classifier and existing classifiers, to quickly identify the best classifier to reuse. It always trains both a new classifier and a reused classifier, and retains the more accurate classifier when concept drift occurs. Finally, it creates a copy of reused classifiers, so a classifier well-suited for a recurring concept will not be impacted by being trained on a different concept. In our experiments, ECPF classifies significantly more accurately than a state-of-the-art classifier reuse framework (Diversity Pool) and a state-of-the-art ensemble technique (Adaptive Random Forest) on synthetic datasets with recurring concepts. It classifies real-world datasets five times faster than Diversity Pool, and six times faster than Adaptive Random Forest and is not significantly less accurate than either.

Clustering Over Multiple Evolving Data Streams of the Traffic Cyber-Physical Systems

Extracting and retaining useful information from multiple data streams of the traffic Cyber-Physical Systems (CPS) generated by practical applications have attracted an increasing amount of attentions from related researchers. In this paper, the Incremental Clustering framework is proposed for multiple sensor data streams by low rank approximation Matrix Factorization (IC-MF) monitors the distribution of clusters over multiple sensor data streams based on their correlation. In the IC-MF, both the low-rank matrix approximation and matrix factorization-based clustering are applied, and IC-MF incorporates the historical results and the relationship between the nodes of the current step and previous step. To improve the accuracy of the increments, a low-rank approximation of the adjacency matrix is obtained at each time step, and makes IC-MF work directly in the low-rank subspace. The main idea of IC-MF is to make use of the similarity between two consecutive time steps to quickly update the approximating subspace. The performance and efficiency of the algorithm are demonstrated by traffic CPS experiments on the real and synthetics data sets. The results show that the proposed algorithm surpasses the existing methods for clustering multiple evolving data streams over time.

A buffer-based online clustering for evolving data stream

Data stream clustering plays an important role in data stream mining for knowledge extraction. Numerous researchers have recently studied density-based clustering algorithms due to their capability to generate arbitrarily shaped clusters. However, most of the algorithms are either fully offline, hybrid online/offline, or cannot handle the property of evolving data stream. Recently, a fully online clustering algorithm for evolving data stream called CEDAS was proposed. However, similar to other density-based clustering algorithms, CEDAS requires predefining the global optimal radius of micro-clusters, which is a difficult task; in addition, an erroneous choice deteriorates cluster performance. Moreover, the algorithm ignores the presence of temporarily irrelevant micro-clusters, which may be relevant in the future. In this study, we present a fully online density-based clustering algorithm called buffer-based online clustering for evolving data stream (BOCEDS). This algorithm recursively updates the micro-cluster radius to its local optimal. It also introduces a buffer for storing irrelevant micro-clusters and a fully online pruning method for extracting the temporarily irrelevant micro-cluster from the buffer. In addition, BOCEDS proposes an online micro-cluster energy-updating function based on the spatial information of the data stream. Experimental results are compared with those of CEDAS and other alternative hybrid online/offline density-based clustering algorithms, and BOCEDS proves its superiority over the other clustering algorithms. The sensitivity of clustering parameters is also measured. The proposed algorithm is then applied to real-world weather data streams to demonstrate its capability to detect changes in data stream and discover arbitrarily shaped clusters. The proposed BOCEDS can be available in https://sites.google.com/view/md-manjur-ahmed and https://sites.google.com/view/kamrul-just.

Correction to: Adaptive random forests for evolving data stream classification

The Publisher regrets an error in the spelling of the family name of the sixth author. The correct spelling is Bernhard Pfahringer, as it appears in the author list above.

An Integrated Approach Using Publicly Available Resources for Identifying and Characterizing Chemicals of Potential Toxicity Concern: Proof-of-Concept With Chemicals That Affect Cancer Pathways.

We developed an integrated, modular approach to predicting chemical toxicity relying on in vitro assay data, linkage of molecular targets to disease categories, and software for ranking chemical activity and examining structural features (chemotypes). We evaluate our approach in a proof-of-concept exercise to identify and prioritize chemicals of potential carcinogenicity concern. We identified 137 cancer pathway-related assays from a subset of U.S. EPA's ToxCast platforms. We mapped these assays to key characteristics of carcinogens and found they collectively assess 5 of 10 characteristics. We ranked all 1061 chemicals screened in Phases I and II of ToxCast by their activity in the selected cancer pathway-related assays using Toxicological Prioritization Index software. More chemicals used as biologically active agents (eg, pharmaceuticals) ranked in the upper 50% versus lower 50%. Twenty-three chemotypes are enriched in the top 5% (n = 54) of chemicals; these features may be important for their activity in cancer pathway-related assays. The biological coverage of the ToxCast assays related to cancer pathways is limited and short-term assays may not capture the biology of some key characteristics. Metabolism is also minimal in the assays. The ability of our approach to identify chemicals with cancer hazard is limited with the current input data, but we expect that our approach can be applied with future iterations of ToxCast and other data for improved chemical prioritization and characterization. The novel approach and proof-of-concept exercise described here for ranking chemicals for potential carcinogenicity concern is modular, adaptable, and amenable to evolving data streams.

Detecting concept drift using HEDDM in data stream

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

Detecting concept drift using HEDDM in data stream

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

Incremental Market Behavior Classification in Presence of Recurring Concepts.

In recent years, the problem of concept drift has gained importance in the financial domain. The succession of manias, panics and crashes have stressed the non-stationary nature and the likelihood of drastic structural or concept changes in the markets. Traditional systems are unable or slow to adapt to these changes. Ensemble-based systems are widely known for their good results predicting both cyclic and non-stationary data such as stock prices. In this work, we propose RCARF (Recurring Concepts Adaptive Random Forests), an ensemble tree-based online classifier that handles recurring concepts explicitly. The algorithm extends the capabilities of a version of Random Forest for evolving data streams, adding on top a mechanism to store and handle a shared collection of inactive trees, called concept history, which holds memories of the way market operators reacted in similar circumstances. This works in conjunction with a decision strategy that reacts to drift by replacing active trees with the best available alternative: either a previously stored tree from the concept history or a newly trained background tree. Both mechanisms are designed to provide fast reaction times and are thus applicable to high-frequency data. The experimental validation of the algorithm is based on the prediction of price movement directions one second ahead in the SPDR (Standard & Poor’s Depositary Receipts) S&P 500 Exchange-Traded Fund. RCARF is benchmarked against other popular methods from the incremental online machine learning literature and is able to achieve competitive results.

A Novelty Detector and Extreme Verification Latency Model for Nonstationary Environments

Safe and reliable operation of systems relies on the use of online condition monitoring and diagnostic systems that aim to take immediate actions upon the occurrence of a fault. Model-based solutions are often not practical in nonstationary environments. Thus, the evolving data stream requires the data-driven model to be adaptive. In this paper, we propose a framework for the fault detection and classification that is accomplished on the data stream with both the gradual and abrupt drifts. The framework is only provided with prior information about the possible faults at the initial step; however, despite this, the framework can still detect the novel faults without receiving any update. Furthermore, an efficient fault classification algorithm is presented to maximize the efficiency of the proposed framework. Finally, the proposed framework is applied for diagnosing bearing defects in the induction motors to demonstrate its feasibility for industrial applications.

Dynamic clustering with binary social spider algorithm for streaming dataset

Technical advancement in various fields like social network, health instruments and astronomical devices poses massive capturing and sensing capacity that enables huge data generations. This demands substantial storage space and voluminous data processing capacity. Streaming data clustering imparts an efficient method for handling this dataset by extracting significant information. In this article, dynamic estimation of clusters in evolving data stream is designed by incorporating swarm optimization technique. One of the recently reported algorithms inspired from the social behavior of spiders residing in huge colonies is reformulated in binary domain. The main contribution is to use the binary social spider optimization (BSSO) for dynamic data clustering of evolving dataset (DSC-BSSO). The proposed work is able to prove efficiency and efficacy as compared to the other recent existing algorithms. BSSO is well tested on various benchmark unimodal, multimodal and binary optimization functions. Results are reported in terms of parametric and nonparametric. The testing of DSC-BSSO is also done on various streaming datasets in terms of time and memory complexity. The proposed work is able to obtain compact and well-separated clusters in less than one-fourth of a minute for about 10,000 samples.

Synchronization-based clustering on evolving data stream

Clustering streams of data is of increasing importance in many applications. In this paper, we propose a new synchronization-based clustering approach for evolving data streams, called SyncTree, which maintains all micro-clusters at different levels of granularity depending upon the data recency. Instead of using a sliding window or decay function to focus on recent data, SyncTree summarizes all continuously-arriving objects as synchronized micro-clusters sequentially in a batch fashion. Owing to the powerful concept of synchronization, the derived micro-clusters truly reflect the intrinsic cluster structure rather than summarize statistics of data, and old micro-clusters can be intuitively summarized at a higher level by iterative clustering to fit memory constraints. Building upon the hierarchical micro-clusters, SyncTree allows investigating the cluster structure of the data stream between any two time stamps in the past, and also provides a principled way to analyze the cluster evolution. Empirical results demonstrate that our method has good performance compared to state-of-the-art algorithms.

Activity Recognition with Evolving Data Streams

Activity recognition aims to provide accurate and opportune information on people’s activities by leveraging sensory data available in today’s sensory rich environments. Nowadays, activity recognition has become an emerging field in the areas of pervasive and ubiquitous computing. A typical activity recognition technique processes data streams that evolve from sensing platforms such as mobile sensors, on body sensors, and/or ambient sensors. This article surveys the two overlapped areas of research of activity recognition and data stream mining. The perspective of this article is to review the adaptation capabilities of activity recognition techniques in streaming environment. Categories of techniques are identified based on different features in both data streams and activity recognition. The pros and cons of the algorithms in each category are analysed, and the possible directions of future research are indicated.

Reservoir of diverse adaptive learners and stacking fast hoeffding drift detection methods for evolving data streams

The last decade has seen a surge of interest in adaptive learning algorithms for data stream classification, with applications ranging from predicting ozone level peaks, learning stock market indicators, to detecting computer security violations. In addition, a number of methods have been developed to detect concept drifts in these streams. Consider a scenario where we have a number of classifiers with diverse learning styles and different drift detectors. Intuitively, the current 'best' (classifier, detector) pair is application dependent and may change as a result of the stream evolution. Our research builds on this observation. We introduce the $\mbox{Tornado}$ framework that implements a reservoir of diverse classifiers, together with a variety of drift detection algorithms. In our framework, all (classifier, detector) pairs proceed, in parallel, to construct models against the evolving data streams. At any point in time, we select the pair which currently yields the best performance. We further incorporate two novel stacking-based drift detection methods, namely the $\mbox{FHDDMS}$ and $\mbox{FHDDMS}_{add}$ approaches. The experimental evaluation confirms that the current 'best' (classifier, detector) pair is not only heavily dependent on the characteristics of the stream, but also that this selection evolves as the stream flows. Further, our $\mbox{FHDDMS}$ variants detect concept drifts accurately in a timely fashion while outperforming the state-of-the-art.