Presence Of Concept Drift Research Articles

Susceptibility of AutoML mortality prediction algorithms to model drift caused by the COVID pandemic

BackgroundConcept drift and covariate shift lead to a degradation of machine learning (ML) models. The objective of our study was to characterize sudden data drift as caused by the COVID pandemic. Furthermore, we investigated the suitability of certain methods in model training to prevent model degradation caused by data drift.MethodsWe trained different ML models with the H2O AutoML method on a dataset comprising 102,666 cases of surgical patients collected in the years 2014–2019 to predict postoperative mortality using preoperatively available data. Models applied were Generalized Linear Model with regularization, Default Random Forest, Gradient Boosting Machine, eXtreme Gradient Boosting, Deep Learning and Stacked Ensembles comprising all base models. Further, we modified the original models by applying three different methods when training on the original pre-pandemic dataset: (1) we weighted older data weaker, (2) used only the most recent data for model training and (3) performed a z-transformation of the numerical input parameters. Afterwards, we tested model performance on a pre-pandemic and an in-pandemic data set not used in the training process, and analysed common features.ResultsThe models produced showed excellent areas under receiver-operating characteristic and acceptable precision-recall curves when tested on a dataset from January-March 2020, but significant degradation when tested on a dataset collected in the first wave of the COVID pandemic from April-May 2020. When comparing the probability distributions of the input parameters, significant differences between pre-pandemic and in-pandemic data were found. The endpoint of our models, in-hospital mortality after surgery, did not differ significantly between pre- and in-pandemic data and was about 1% in each case. However, the models varied considerably in the composition of their input parameters. None of our applied modifications prevented a loss of performance, although very different models emerged from it, using a large variety of parameters.ConclusionsOur results show that none of our tested easy-to-implement measures in model training can prevent deterioration in the case of sudden external events. Therefore, we conclude that, in the presence of concept drift and covariate shift, close monitoring and critical review of model predictions are necessary.

Online Machine Learning from Non-stationary Data Streams in the Presence of Concept Drift and Class Imbalance: A Systematic Review

In IoT environment applications generate continuous non-stationary data streams with in-built problems of concept drift and class imbalance which cause classifier performance degradation. The imbalanced data affects the classifier during concept detection and concept adaptation. In general, for concept detection, a separate mechanism is added in parallel with the classifier to detect the concept drift called a drift detector. For concept adaptation, the classifier updates itself or trains a new classifier to replace the older one. In case, the data stream faces a class imbalance issue, the classifier may not properly adapt to the latest concept. In this survey, we study how the existing work addresses the issues of class imbalance and concept drift while learning from nonstationarydata streams. We further highlight the limitation of existing work and challenges caused by other factors of class imbalance alongwith concept drift in data stream classification. Results of our survey found that, out of 1110 studies, by using our inclusion and exclusion criteria, we were able to narrow the pool of articles down to 35 that directly addressed our study objectives. The study found that issues such as multiple concept drift types, dynamic class imbalance ratio, and multi-class imbalance in presence of concept drift are still open for further research. We also observed that, while major research efforts have been dedicated to resolving concept drift and class imbalance, not much attention has been given to with-in-class imbalance, rear examples, and borderline instances when they exist with concept drift in multi-class data. This paper concludes with some suggested future directions.

An efficient and straightforward online vector quantization method for a data stream through remove-birth updating.

The growth of network-connected devices has led to an exponential increase in data generation, creating significant challenges for efficient data analysis. This data is generated continuously, creating a dynamic flow known as a data stream. The characteristics of a data stream may change dynamically, and this change is known as concept drift. Consequently, a method for handling data streams must efficiently reduce their volume while dynamically adapting to these changing characteristics. This article proposes a simple online vector quantization method for concept drift. The proposed method identifies and replaces units with low win probability through remove-birth updating, thus achieving a rapid adaptation to concept drift. Furthermore, the results of this study show that the proposed method can generate minimal dead units even in the presence of concept drift. This study also suggests that some metrics calculated from the proposed method will be helpful for drift detection.

An improved arithmetic optimization algorithm for training feedforward neural networks under dynamic environments

This paper proposes an improved Arithmetic Optimization Algorithm (AOA) to train artificial neural networks (ANNs) under dynamic environments. Despite many successful applications of metaheuristic training of ANNs, these studies assume static environments, which might not be realistic in real-world nonstationary processes. In this study, the training of ANNs is modeled as a dynamic optimization problem, and the proposed AOA is used to optimize connection weights and biases of the ANN under the presence of concept drift. The proposed method is designed to work for classification tasks. The performance of the proposed algorithm has been tested on twelve dynamic classification problems. Comparative analysis with state-of-the-art metaheuristic optimization algorithms has been provided. The superiority of the compared algorithms has been verified using nonparametric statistical tests. The results show that the improved AOA outperforms compared algorithms in training ANNs under dynamic environments. The findings demonstrate the potential of improved AOA for dynamic data-driven applications.

A Survey on Classifying Big Data with Label Noise

Class label noise is a critical component of data quality that directly inhibits the predictive performance of machine learning algorithms. While many data-level and algorithm-level methods exist for treating label noise, the challenges associated with big data call for new and improved methods. This survey addresses these concerns by providing an extensive literature review on treating label noise within big data. We begin with an introduction to the class label noise problem and traditional methods for treating label noise. Next, we present 30 methods for treating class label noise in a range of big data contexts, i.e., high-volume, high-variety, and high-velocity problems. The surveyed works include distributed solutions capable of operating on datasets of arbitrary sizes, deep learning techniques for large-scale datasets with limited clean labels, and streaming techniques for detecting class noise in the presence of concept drift. Common trends and best practices are identified in each of these areas, implementation details are reviewed, empirical results are compared across studies when applicable, and references to 17 open-source projects and programming packages are provided. An emphasis on label noise challenges, solutions, and empirical results as they relate to big data distinguishes this work as a unique contribution that will inspire future research and guide machine learning practitioners.

An effectiveness analysis of transfer learning for the concept drift problem in malware detection

Malware classification is a task that has acquired importance due to the increase in malware distribution. In the literature, the application of machine learning techniques is proposed to tackle this task because machine learning models may be able to detect new malware variants more effectively than traditional signature-based solutions. Nonetheless, there are some difficulties in the application of machine learning in this field, particularly the presence of concept drift, that must be addressed by keeping models up to date in order to detect new threats. In this research, we carry out an evaluation of the performance of transfer learning techniques on the problem of malware detection over different time horizons and on several learning settings. We carry out experiments on unbalanced data with different file types to better reflect additional challenges in malware detection. Our goal is to determine whether transfer learning may be helpful to solve the concept drift problem, and construct models that can detect new malware by using the information obtained from past data.

An adaptive deep learning framework for day-ahead forecasting of photovoltaic power generation

Accurate forecasts of photovoltaic power generation (PVPG) are essential to optimize operations between energy supply and demand. Recently, the propagation of sensors and smart meters has produced an enormous volume of data, which supports the data-driven models for PVPG forecasting. Although emerging deep learning (DL) models, such as the long short-term memory (LSTM), based on historical data, have provided effective solutions for PVPG forecasting with great successes, these models utilize offline learning. As a result, DL models cannot take advantage of the opportunity to learn from newly-arrived data, and are unable to handle concept drift caused by installing extra PV units and unforeseen PV unit failures. Consequently, to improve day-ahead PVPG forecasting accuracy, as well as eliminate the impacts of concept drift, this paper proposes an adaptive LSTM (AD-LSTM), which is a DL framework that can not only acquire general knowledge from historical data, but also dynamically learn specific knowledge from newly-arrived data. A two-phase adaptive learning strategy (TP-ALS) is integrated into AD-LSTM, and a sliding window (SDWIN) algorithm is proposed, to detect concept drift in PV systems. Multiple datasets from PV systems are utilized to assess the feasibility and effectiveness of the proposed approaches. The developed AD-LSTM model demonstrates greater forecasting capability than the conventional offline LSTM model, particularly in the presence of concept drift. The forecasting skill of AD-LSTM can be improved up to 73.11%. Additionally, the proposed AD-LSTM model also achieves superior performance in terms of day-ahead PVPG forecasting for individual days and multiple days to other reference models in the literature.

Machine learning techniques applied to mechanical fault diagnosis and fault prognosis in the context of real industrial manufacturing use-cases: a systematic literature review.

When put into practice in the real world, predictive maintenance presents a set of challenges for fault detection and prognosis that are often overlooked in studies validated with data from controlled experiments, or numeric simulations. For this reason, this study aims to review the recent advancements in mechanical fault diagnosis and fault prognosis in the manufacturing industry using machine learning methods. For this systematic review, we searched Web of Science, ACM Digital Library, Science Direct, Wiley Online Library, and IEEE Xplore between January 2015 and October 2021. Full-length studies that employed machine learning algorithms to perform mechanical fault detection or fault prognosis in manufacturing equipment and presented empirical results obtained from industrial case-studies were included, except for studies not written in English or published in sources other than peer-reviewed journals with JCR Impact Factor, conference proceedings and book chapters/sections. Of 4549 records, 44 primary studies were selected. In 37 of those studies, fault diagnosis and prognosis were performed using artificial neural networks (n = 12), decision tree methods (n = 11), hybrid models (n = 8), or latent variable models (n = 6), with one of the studies employing two different types of techniques independently. The remaining studies employed a variety of machine learning techniques, ranging from rule-based models to partition-based algorithms, and only two studies approached the problem using online learning methods. The main advantages of these algorithms include high performance, the ability to uncover complex nonlinear relationships and computational efficiency, while the most important limitation is the reduction in model performance in the presence of concept drift. This review shows that, although the number of studies performed in the manufacturing industry has been increasing in recent years, additional research is necessary to address the challenges presented by real-world scenarios.

Learning Calibration Functions on the Fly: Hybrid Batch Online Stacking Ensembles for the Calibration of Low-Cost Air Quality Sensor Networks in the Presence of Concept Drift

Deployment of an air quality low-cost sensor network (AQLCSN), with proper calibration of low-cost sensors (LCS), offers the potential to substantially increase the ability to monitor air pollution. However, to leverage this potential, several drawbacks must be ameliorated, thus the calibration of such sensors is becoming an essential component in their use. Commonly, calibration takes place in a laboratory environment using gasses of known composition to measure the response and a linear calibration is often reached. On site calibration is a promising complementary technique where an LCS and a reference instrument are collocated with the former being calibrated to match the measurements of the latter. In a scenario where an AQLCSN is already operational, both calibration approaches are resource and time demanding procedures to be implemented as frequently repeated actions. Furthermore, sensors are sensitive to the local meteorology and adaptation is a slow process making relocation a complex and expensive option. We concentrate our efforts in keeping the LCS positions fixed and propose to blend a genetic algorithm (GA) with a hybrid stacking (HS) ensemble into the GAHS framework. GAHS employs a combination of batch machine learning algorithms and regularly updated online machine learning calibration function(s) for the whole network when a small number of reference instruments are present. Furthermore, we introduce the concept of spatial online learning to achieve better spatial generalization. The frameworks are tested for the case of Thessaloniki where a total of 33 devices are installed. The AQLCSN is calibrated on the basis of on-site matching with high quality observations from three reference station measurements. The O3 LCS are successfully calibrated for 8–10 months and the PM10 LCS calibration is evaluated for 13–24 months showing a strong seasonal dependence on their ability to correctly capture the pollution levels.

An extensive study of C-SMOTE, a Continuous Synthetic Minority Oversampling Technique for Evolving Data Streams

Streaming Machine Learning (SML) studies algorithms that update their models, given an unbounded and often non-stationary flow of data performing a single pass. Online class imbalance learning is a branch of SML that combines the challenges of both class imbalance and concept drift. In this paper, we investigate the binary classification problem by rebalancing an imbalanced stream of data in the presence of concept drift, accessing one sample at a time. We propose an extensive comparative study of Continuous Synthetic Minority Oversampling Technique (C-SMOTE), inspired by the popular sampling technique Smote, as a meta-strategy to pipeline with SML classification algorithms. We benchmark C-SMOTE pipelines on both synthetic and real data streams, containing different types of concept drifts, different imbalance levels, and different class distributions. We bring statistical evidence that models learnt with C-SMOTE pipelines improve the minority class performance concerning both the baseline models and the state-of-the-art methods. We also perform a sensitivity analysis to detect the C-SMOTE impact on the majority class performance for the three types of concept drift and several class distributions. Moreover, we show a computational cost analysis in terms of time and memory consumption.

An Adaptative Active Approach with Monitoring Quality of Stream Modeling in the Presence of Concept Drift

An Adaptative Active Approach with Monitoring Quality of Stream Modeling in the Presence of Concept Drift

Real-Time Concept Drift Detection and Its Application to ECG Data

Prediction of cardiac disease is one the most crucial topics in the sector of medical info evaluation. The stochastic nature and the variation concerning time in electrocardiogram (ECG) signals make it burdensome to investigate its characteristics. Being evolving in nature, it requires a dynamic predictive model. With the presence of concept drift, the model performance will get worse. Thus learning algorithms require an apt adaptive mechanism to accurately handle the drifting data streams. This paper proposes an inceptive approach, Corazon Concept Drift Detection Method (Corazon CDDM), to detect drifts and adapt to them in real-time in electrocardiogram signals. The proposed methodology results in achieving competitive results compared to the methods proposed in the literature for all types of datasets like synthetic, real-world & time-series datasets.

ADAW: Age decay accuracy weighted ensemble method for drifting data stream mining

Dynamic environment data generators are very often in real-world that produce data streams. A data source of a dynamic environment generates data streams in which the underlying data distribution changes very frequently with respect to time and hence results in concept drifts. As compared to the stationary environment, learning in the dynamic environment is very difficult due to the presence of concept drifts. Learning in dynamic environment requires evolutionary and adaptive approaches to be accommodated with the learning algorithms. Ensemble methods are commonly used to build classifiers for learning in a dynamic environment. The ensemble methods of learning are generally described at three very crucial aspects, namely, the learning and testing method employed, result integration method and forgetting mechanism for old concepts. In this paper, we propose a novel approach called Age Decay Accuracy Weighted (ADAW) ensemble architecture for learning in concept drifting data streams. The ADAW method assigned weights to the component classifiers based on its accuracy and its remaining life-time in the ensemble is such a way that ensures maximum accuracy. We empirically evaluated ADAW on benchmark artificial drifting data stream generators and real datasets and compared its performance with ten well-known state-of-the-art existing methods. The experimental results show that ADAW outperforms over the existing methods.

Dynamically Adjusting Diversity in Ensembles for the Classification of Data Streams with Concept Drift

A data stream can be defined as a system that continually generates a lot of data over time. Today, processing data streams requires new demands and challenging tasks in the data mining and machine learning areas. Concept Drift is a problem commonly characterized as changes in the distribution of the data within a data stream. The implementation of new methods for dealing with data streams where concept drifts occur requires algorithms that can adapt to several scenarios to improve its performance in the different experimental situations where they are tested. This research proposes a strategy for dynamic parameter adjustment in the presence of concept drifts. Parameter Estimation Procedure (PEP) is a general method proposed for dynamically adjusting parameters which is applied to the diversity parameter (λ) of several classification ensembles commonly used in the area. To this end, the proposed estimation method (PEP) was used to create Boosting-like Online Learning Ensemble with Parameter Estimation (BOLE-PE), Online AdaBoost-based M1 with Parameter Estimation (OABM1-PE), and Oza and Russell’s Online Bagging with Parameter Estimation (OzaBag-PE), based on the existing ensembles BOLE, OABM1, and OzaBag, respectively. To validate them, experiments were performed with artificial and real-world datasets using Hoeffding Tree (HT) as base classifier. The accuracy results were statistically evaluated using a variation of the Friedman test and the Nemenyi post-hoc test. The experimental results showed that the application of the dynamic estimation in the diversity parameter (λ) produced good results in most scenarios, i.e., the modified methods have improved accuracy in the experiments with both artificial and real-world datasets.

Spam Detection Based on Feature Evolution to Deal with Concept Drift

Electronic messages are still considered the most significant tools in business and personal applications due to their low cost and easy access. However, e-mails have become a major problem owing to the high amount of junk mail, named spam, which fill the e-mail boxes of users. Several approaches have been proposed to detect spam, such as filters implemented in e-mail servers and user-based spam message classification mechanisms. A major problem with these approaches is spam detection in the presence of concept drift, especially as a result of changes in features over time. To overcome this problem, this work proposes a new spam detection system based on analyzing the evolution of features. The proposed method is divided into three steps: 1) spam classification model training; 2) concept drift detection; and 3) knowledge transfer learning. The first step generates classification models, as commonly conducted in machine learning. The second step introduces a new strategy to avoid concept drift: SFS (Similarity-based Features Se- lection) that analyzes the evolution of the features taking into account similarity obtained between the feature vectors extracted from training data and test data. Finally, the third step focuses on the following questions: what, how, and when to transfer acquired knowledge? The proposed method is evaluated using two public datasets. The results of the experiments show that it is possible to infer a threshold to detect changes (drift) in order to ensure that the spam classification model is updated through knowledge transfer. Moreover, our anomaly detection system is able to perform spam classification and concept drift detection as two parallel and independent tasks.

Supervised learning in the presence of concept drift: a modelling framework

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

A clustering and ensemble based classifier for data stream classification

In the era of data mining, the research industry has great attention to data stream mining as well as it has a great impact on a wide range of applications like networking, telecommunication, education, banking, weather forecasting, a stock market, and so on. Because of these data stream mining having more attention from researchers. The handling of concept drifting data streams is one of the major issues and challenges in the data stream mining field. In the presence of the concept drift, the performance of the learning algorithm always degrades. In this paper, a hybrid method has been proposed which are the combination of an ensemble, and grid and density-based clustering methods. The proposed method is tested on both synthetic as well as real data. The proposed method works well in the presence of concept drift and performance is measured in terms of time, accuracy, and memory. As compared with the state-of-art algorithms, the proposed method performed well and gave better accuracy using synthetic datasets like 88.29%, 71.34%, and 75.39% for Hyperplane, RBF, and LED respectively and for real datasets 86.17%, 86.28%, 95.15%, and 99.83% for Adult, Census-Income, KDDCup99%–10%, and Covertype respectively.

A Classification Approach Based on Divergence for Network Traffic in Presence of Concept Drift

A Classification Approach Based on Divergence for Network Traffic in Presence of Concept Drift

Concept learning using one-class classifiers for implicit drift detection in evolving data streams

Data stream mining has become an important research area over the past decade due to the increasing amount of data available today. Sources from various domains generate a near-limitless volume of data in temporal order. Such data are referred to as data streams, and are generally nonstationary as the characteristics of data evolves over time. This phenomenon is called concept drift, and is an issue of great importance in the literature, since it makes models obsolete by decreasing their predictive performance. In the presence of concept drift, it is necessary to adapt to change in data to build more robust and effective classifiers. Drift detectors are designed to run jointly with classification models, updating them when a significant change in data distribution is observed. In this paper, we present an implicit (unsupervised) algorithm called One-Class Drift Detector (OCDD), which uses a one-class learner with a sliding window to detect concept drift. We perform a comprehensive evaluation on mostly recent 17 prevalent concept drift detection methods and an adaptive classifier using 13 datasets. The results show that OCDD outperforms the other methods by producing models with better predictive performance on both real-world and synthetic datasets.

Mining frequent itemsets from streaming transaction data using genetic algorithms

This paper presents a study of mining frequent itemsets from streaming data in the presence of concept drift. Streaming data, being volatile in nature, is particularly challenging to mine. An approach using genetic algorithms is presented, and various relationships between concept drift, sliding window size, and genetic algorithm constraints are explored. Concept drift is identified by changes in frequent itemsets. The novelty of this work lies in determining concept drift using frequent itemsets for mining streaming data, using the genetic algorithm framework. Formulas have been presented for calculating minimum support counts in streaming data using sliding windows. Testing highlighted that the ratio of the window size to transactions per drift was a key to good performance. Getting good results when the sliding window size was too small was a challenge since normal fluctuations in the data could appear to be a concept drift. Window size must be managed in conjunction with support and confidence values in order to achieve reasonable results. This method of detecting concept drift performed well when larger window sizes were used.