Stationary Data Streams Research Articles

Online Learning of Bayesian Classifiers with Nonstationary Data Streams

Article Online Learning of Bayesian Classifiers with Nonstationary Data Streams Peng Wu 1,2,*, and Ning Xiong 1 1 School of Innovation, Design and Engineering Mälardalen University, Västerås 721 23, Sweden 2 Department of Computer Engineering, Taiyuan Institute of Technology, Taiyuan 030008, China * Correspondence: 14112078@bjtu.edu.cn Received: 31 March 2023 Accepted: 13 July 2023 Published: 26 September 2023 Abstract: The advancement in Internet of things and sensor technologies has enabled data to be continuously generated with a high rate, i.e., data streams. It is practically infeasible to store streaming data in a hard disk, and apply a traditional batch learning method to extract a relevant knowledge model from these data. This paper studies online incremental learning with data streams, in which one sample is processed at each time to update the existing model. For the learning target, the Bayesian classifier is adopted which is a computationally economical model of easy deployment for online processing in edges or devices. By using the individual new example, we first present an online learning algorithm to incrementally update classifier parameters in a way equivalent to the offline learning counterpart. In order to adapt to concept drifts in nonstationary environments, the proposed online learning algorithm is improved to enable recent examples to be more impactful during the sequential learning procedure. Preliminary simulation tests reveal that the improved online learning algorithm can lead to faster model adaption than the unimproved online algorithm when the data drift occurs. In case of presumed stationary data streams without drifts, the improved online algorithm is proved to be competent by performing at least as good as (sometimes, even better than) the unimproved algorithm.

Dynamically evolving deep neural networks with continuous online learning

In a big data environment, data streams are sequences of dynamically changing data with unlimited length; they are often associated with concept drift, caused by data distribution transition. The uncertainty of data streams challenges the prediction ability of traditional static algorithms and hampers their practical application. Although many deep neural network models can be tailored to perform in an online manner, they are not suitable for non-stationary data streams that continually change over time. In this work, we propose a deep neural network model that can dynamically adjust to data distribution change by self-growth, that is, by changing its structure from narrow to wide and/or from shallow to deep. Specifically, initialized from a standard multilayer perceptron, the model grows by adding neurons according to a local error and adding layers according to the change in Jenson–Shannon divergence. The growth involves an adaptive memory mechanism, designed to avoid catastrophic forgetting, which may occur when hidden layers are added. In experiments, the proposed model achieved remarkable improvements in adaptability and prediction performance across 12 regression datasets (including non-stationary and stationary data streams), compared with the state-of-the-art dynamic neural network and other online approaches. The influence of the parameters, timeliness, convergence, and size of the pretrained model are explored in the paper.

A Hybrid Sampling Approach for Imbalanced Binary and Multi-Class Data Using Clustering Analysis

Unequal data distribution among different classes usually cause a class imbalance problem. Due to the class imbalance, the classification models become biased toward the majority class and misclassify the minority class. Class imbalance issue becomes more complex when it occurs in multi-class data. The most common method to handle the class imbalance is data resampling that involves either over-sampling minority class instances or under-sampling majority class instances. In the case of under-sampling, there is a chance of losing some crucial information, whereas over-sampling can cause an overfitting problem. Therefore, we propose a novel Cluster-based Hybrid Sampling for Imbalance Data (CBHSID) strategy to address these issues. We calculate the mean of the data observations based on the number of classes. CBHSID uses the calculated mean as a threshold value to segregate majority and minority classes. We apply affinity propagation cluster analysis to each class to create sub-clusters. We calculate the distance of each data item of sub-cluster using centroid mean. We remove data observations that are away from the center of sub-cluster during under-sampling. On the other hand, during the over-sampling, we generate synthetic samples using data observations near to the center of sub-cluster. We compared our proposed approach with a few state-of-the-art data balancing methods on 12 binary and 4 multi-class benchmark datasets. Based on Geometric-Mean (G-Mean), Recall, and F1-score, our method outperformed the other compared methods on 14 datasets out of 16. We identified that CBHSID is suitable for addressing class imbalance issues in both binary and multi-class classifications. In the current state, we have only validated CBHSID on stationary data streams. Consequently, CBHSID can further be tested on non-stationary data streams in online learning environments.

Long short-term memory self-adapting online random forests for evolving data stream regression

Evolving data stream, especially with concept drift is generally accepted as a challenging data type for regression task, because it usually makes machine learning models trained on old data not adapting to new data, and leads to dramatic performance degradation as a result. Moreover, the behavior of a data stream may change in different modes and therefore introduces various concept drifts, e.g., abrupt, incremental, gradual, recurring, even more, complex concept drifts. Although there are some algorithms that can adapt to stationary data streams or a specific type of concept drift in non-stationary data streams, a wide range of practical applications call for machine learning regression models to handle multi-type of data streams. In this work, we propose an online learning strategy called adaptive long and short-term memories online Random Forests regression(ALSM-RFR), where an adaptive memory activation mechanism is designed to make the model switch adaptively between long-term and hybrid (long-term plus short-term) memory modes in the face of stationary data streams or non-stationary data streams with different types of concept drift. In particular, leaf and tree weights in random forests are used to learn information at different timescales, namely, long-term and short-term memories. Moreover, we devise an adaptive memory activation mechanism to formulate the switch decision of memory modes as a classification problem. Numerical experiments show remarkable improvements of the proposed method in the adaptability of stream types and predictive accuracy in data streams across several real datasets and synthetic datasets, compared to the state-of-the-art online approaches. Besides, the convergence and the influence of the parameters involved in our method are evaluated.

Distribution inference from early-stage stationary data streams by transfer learning

Data streams are prevalent in current manufacturing and service systems where real-time data arrive progressively. A quick distribution inference from such data streams at their early stages is extremely useful for prompt decision making in many industrial applications. For example, a quality monitoring scheme can be quickly started if the process data distribution is available and the optimal inventory level can be determined early once the customer demand distribution is estimated. To this end, this article proposes a novel online recursive distribution inference method for stationary data streams that can respond as soon as the streaming data are generated and update as regularly as the data accumulate. A major challenge is that the data size might be too small to produce an accurate estimation at the early stage of data streams. To solve this, we resort to an instance-based transfer learning approach which integrates a sufficient amount of auxiliary data from similar processes or products to aid the distribution inference in our target task. Particularly, the auxiliary data are reweighted automatically by a density ratio fitting model with a prior-belief-guided regularization term to alleviate data scarcity. Our proposed distribution inference method also possesses an efficient online algorithm with recursive formulas to update upon every incoming data point. Extensive numerical simulations and real case studies verify the advantages of the proposed method.

HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints

Online data streaming has become one of the most common data forms in the modern world, which imposes a strong demand for developing hyperparameter optimization techniques for online learning algorithms. In fact, processing online streaming data can be considered as a constraint on the training process. Existing studies do not provide a clear framework for modeling appropriate constraints with hyperparameter optimization in this context. In this paper, we propose a framework, called HyperTube, based on a set of assumptions that clearly define the constraints and objective function for online hyperparameter optimization under limited computing resources. We also introduce a “micro-mini-batch training mechanism” to reuse online data mini-batches in an efficient manner. Numerical experiments compare the performances of different training settings under the constraints of HyperTube. The results on stationary data streams without concept drift indicate that training on incremental data samples with model selection efficiently uses computing power and gives satisfactory validation performance compared with training without model selection. Meanwhile, the results on data streams with significant concept drift indicate that parallel updating could lead to relatively good model performance. In both cases, with the best settings, HyperTube with the micro-mini-batch training mechanism significantly outperforms offline random search with the same amount of computational resources. These settings can be further improved by a modified genetic algorithm. Finally, we develop a systematic method for selecting suitable settings based on a set of criteria.

Incremental one‐class classification on stationary data stream using two‐quarter sphere

AbstractData stream is a sequence of data that has unique features. In many data streams, data from one concept is available, and detection of other types of data in data stream is an essential task. One‐class classification is a famous approach to data classification when data from one class is accessible. The principal task of one‐class classification is separating input data in two different parts: target and outlier data. One of the main challenges concerning data classification on the stationary data streams is the response time, which causes negative effects on effective runtime. Most classifiers, that have been proposed on Rd feature space, solve a quadratic problem for classification that leads to extreme runtime increase. In this paper, an incremental one‐class classification on stationary data streams is proposed using two‐quarter sphere (IOCTQ) in order to achieve lower computation cost of classification time, and a linear optimization problem is solved. IOCTQ divides one data classification problem into two data classifiers, and each data point will be individually classified in one‐quarter sphere. The two‐quarter spheres can be run parallel. The results of the experiments have been compared with state‐of‐the‐art methods and show superiority of the IOCTQ method in classification accuracy and time complexity.

Adaptive kernel density-based anomaly detection for nonlinear systems

This paper presents an unsupervised, density-based approach to anomaly detection. The purpose is to define a smooth yet effective measure of outlierness that can be used to detect anomalies in nonlinear systems. The approach assigns each sample a local outlier score indicating how much one sample deviates from others in its locality. Specifically, the local outlier score is defined as a relative measure of local density between a sample and a set of its neighboring samples. To achieve smoothness in the measure, we adopt the Gaussian kernel function. Further, to enhance its discriminating power, we use adaptive kernel width: in high-density regions, we apply wide kernel widths to smooth out the discrepancy between normal samples; in low-density regions, we use narrow kernel widths to intensify the abnormality of potentially anomalous samples. The approach is extended to an online mode with the purpose of detecting anomalies in stationary data streams. To validate the proposed approach, we compare it with several alternatives using synthetic datasets; the approach is found superior in terms of smoothness, effectiveness and robustness. A further experiment on a real-world dataset demonstrated the applicability of the proposed approach in fault detection tasks.

New Splitting Criteria for Decision Trees in Stationary Data Streams.

The most popular tools for stream data mining are based on decision trees. In previous 15 years, all designed methods, headed by the very fast decision tree algorithm, relayed on Hoeffding's inequality and hundreds of researchers followed this scheme. Recently, we have demonstrated that although the Hoeffding decision trees are an effective tool for dealing with stream data, they are a purely heuristic procedure; for example, classical decision trees such as ID3 or CART cannot be adopted to data stream mining using Hoeffding's inequality. Therefore, there is an urgent need to develop new algorithms, which are both mathematically justified and characterized by good performance. In this paper, we address this problem by developing a family of new splitting criteria for classification in stationary data streams and investigating their probabilistic properties. The new criteria, derived using appropriate statistical tools, are based on the misclassification error and the Gini index impurity measures. The general division of splitting criteria into two types is proposed. Attributes chosen based on type- splitting criteria guarantee, with high probability, the highest expected value of split measure. Type- criteria ensure that the chosen attribute is the same, with high probability, as it would be chosen based on the whole infinite data stream. Moreover, in this paper, two hybrid splitting criteria are proposed, which are the combinations of single criteria based on the misclassification error and Gini index.

Effects of drift and noise on the optimal sliding window size for data stream regression models

ABSTRACTThe analysis of non stationary data streams requires a continuous adaption of the model to the relevant most recent data. This requires that changes in the data stream must be distinguished from noise. Many approaches are based on heuristic adaptation schemes. We analyze simple regression models to understand the joint effects of noise and concept drift and derive the optimal sliding window size for the regression models. Our theoretical analysis and simulations show that a near optimal window size can be crucial. Our models can be used as benchmarks for other models to see how they cope with noise and drift.

Incremental weighted one-class classifier for mining stationary data streams

Big data analytics, especially data stream mining, is among the most popular contemporary machine learning problems. More and more often real-life tasks could generate massive and continuous amounts of data. Standard classifiers cannot cope with a large volume of the training set and/or changing nature of the environment. In this paper, we deal with a problem of continuously arriving objects, that with each time interval may contribute new, useful knowledge to the patter classification system. One-class classification is a very useful tool for stream analysis, as it can be used for tackling outliers, noise, appearance of new classes or imbalanced data to name a few. We propose a novel version of incremental One-Class Support Vector Machine, that assigns weights to each object according to its level of significance. This allows to train more robust one-class classifiers on incremental streams. We present two schemes for estimating weights for new, incoming data and examine their usefulness on a number of benchmark datasets. We also analyze time and memory requirements of our method. Results of experimental investigations prove, that our method can achieve better one-class recognition quality than algorithms used so far.

SOMKE: kernel density estimation over data streams by sequences of self-organizing maps.

In this paper, we propose a novel method SOMKE, for kernel density estimation (KDE) over data streams based on sequences of self-organizing map (SOM). In many stream data mining applications, the traditional KDE methods are infeasible because of the high computational cost, processing time, and memory requirement. To reduce the time and space complexity, we propose a SOM structure in this paper to obtain well-defined data clusters to estimate the underlying probability distributions of incoming data streams. The main idea of this paper is to build a series of SOMs over the data streams via two operations, that is, creating and merging the SOM sequences. The creation phase produces the SOM sequence entries for windows of the data, which obtains clustering information of the incoming data streams. The size of the SOM sequences can be further reduced by combining the consecutive entries in the sequence based on the measure of Kullback-Leibler divergence. Finally, the probability density functions over arbitrary time periods along the data streams can be estimated using such SOM sequences. We compare SOMKE with two other KDE methods for data streams, the M-kernel approach and the cluster kernel approach, in terms of accuracy and processing time for various stationary data streams. Furthermore, we also investigate the use of SOMKE over nonstationary (evolving) data streams, including a synthetic nonstationary data stream, a real-world financial data stream and a group of network traffic data streams. The simulation results illustrate the effectiveness and efficiency of the proposed approach.