Concept Drift Research Articles

Feature-based analyses of concept drift

Feature selection is one of the most relevant preprocessing and analysis techniques in machine learning. It can dramatically increase the performance of learning algorithms and at the same time provide relevant information on the data. In the scenario of online and stream learning, concept drift, i.e., changes in the underlying distribution over time, can cause significant problems for learning models and data analysis. While there do exist feature selection methods for online learning, none of the methods targets feature selection for drift detection, i.e., the challenge to increase the performance of drift detectors by analyzing the drift rather than increasing model accuracy. However, this challenge is particularly relevant for common unsupervised scenarios. In this work, we study feature selection for drift detection and drift monitoring. We develop a formal definition for a feature-wise notion of drift that allows semantic interpretation. Besides, we derive an efficient algorithm by reducing the problem to classical feature selection and analyze the applicability of our approach to feature selection for drift detection on a theoretical level. Finally, we empirically show the relevance of our considerations on several benchmarks.

Scalable concept drift adaptation for stream data mining

Stream data mining aims to handle the continuous and ongoing generation of data flows (e.g. weather, stock and traffic data), which often encounters concept drift as time progresses. Traditional offline algorithms struggle with learning from real-time data, making online algorithms more fitting for mining the stream data with dynamic concepts. Among families of the online learning algorithms, single pass stands out for its efficiency in processing one sample point at a time, and inspecting it only once at most. Currently, there exist online algorithms tailored for single pass over the stream data by converting the problems of classification into minimum enclosing ball. However, these methods mainly focus on expanding the ball to enclose the new data. An excessively large ball might overwrite data of the new concept, creating difficulty in triggering the model updating process. This paper proposes a new online single pass framework for stream data mining, namely Scalable Concept Drift Adaptation (SCDA), and presents three distinct online methods (SCDA-I, SCDA-II and SCDA-III) based on that framework. These methods dynamically adjust the ball by expanding or contracting when new sample points arrive, thereby effectively avoiding the issue of excessively large balls. To evaluate their performance, we conduct the experiments on 7 synthetic and 5 real-world benchmark datasets and compete with the state-of-the-arts. The experiments demonstrate the applicability and flexibility of the SCDA methods in stream data mining by comparing three aspects: predictive performance, memory usage and scalability of the ball. Among them, the SCDA-III method performs best in all these aspects.

One or two things we know about concept drift-a survey on monitoring in evolving environments. Part A: detecting concept drift.

The world surrounding us is subject to constant change. These changes, frequently described as concept drift, influence many industrial and technical processes. As they can lead to malfunctions and other anomalous behavior, which may be safety-critical in many scenarios, detecting and analyzing concept drift is crucial. In this study, we provide a literature review focusing on concept drift in unsupervised data streams. While many surveys focus on supervised data streams, so far, there is no work reviewing the unsupervised setting. However, this setting is of particular relevance for monitoring and anomaly detection which are directly applicable to many tasks and challenges in engineering. This survey provides a taxonomy of existing work on unsupervised drift detection. In addition to providing a comprehensive literature review, it offers precise mathematical definitions of the considered problems and contains standardized experiments on parametric artificial datasets allowing for a direct comparison of different detection strategies. Thus, the suitability of different schemes can be analyzed systematically, and guidelines for their usage in real-world scenarios can be provided.

Learning evolving prototypes for imbalanced data stream classification with limited labels

Real-world data streams often exhibit long-tailed distributions with heavy class imbalance, posing great challenges for data stream classification, especially in the case of label scarcity and concept drift. Several active learning methods have been proposed to address this problem by selecting the most valuable instances for labeling. However, existing methods often struggle to dynamically identify the most valuable instances that truly represent the current concept while still requiring a large label budget. In this work, we propose a new algorithm, LEPID, to combine dynamic micro-cluster concept modeling and local entropy modeling to select current important concepts and prototypes. Specifically, we give greater weight to concept drift prototypes and minority prototypes to focus more on those regions that represent current concepts. We use a local entropy strategy based on micro-clusters to select the most valuable instances for labeling and reduce the label budget. Extensive experiments on real-world and synthetic imbalanced datasets show that, compared to state-of-the-art algorithms, our method can naturally adapt to concept drift and dynamically capture the current and most valuable prototypes to achieve better results even in the case of label scarcity.

ADAPTATION OF THE ALGORITHM FOR DETECTING ANOMALIES IN TIME SERIES FOR NON-STATIONARY STREAMING DATA

Detection of anomalies in streaming time series data has become an important research topic due to the wide range of possible applications, including the detection of extreme weather conditions, malicious attacks on protected facilities, monitoring unauthorised gas and oil leaks, illegal pipeline connections, power cable faults, and water and other environmental pollution. Rapid detection of abnormal conditions and identifying these critical events is essential to protect lives and assets. Therefore, developing appropriate systems and detection methods is an urgent task. Anomaly detection in streaming data is challenging due to its large volume and high speed, presence of noise, and non-stationarity of the signal (or ‘concept drift’). The latter significantly complicates the identification of differences between new ‘typical’ behaviour and abnormal events. Solving this problem requires the algorithm for processing such data to learn and adapt to changing conditions. The paper proposes a modification of the algorithm for detecting anomalies in time series. This algorithm provides early detection of abnormal series in a massive collection of non-stationary streaming time series data. Anomalies are observations that are highly improbable given the previous time series values. The proposed approach is based on the primary detection of the predictive limit for the typical system behaviour using the theory of extreme values, followed by checking for the abnormality of the following series using the sliding window technique. The time series parameters are used as input data and compared by density distribution to detect any significant changes in the distribution of the characteristics. It allows the decision-making model to automatically adapt to the changing environment according to detected changes. Since anomalies are, by definition, exceptions to the typical behaviour of a system, most of the available stored data should reflect that typical behaviour of the system in question. It is unnecessary to have representative samples of all possible types of the standard behaviour of a given system for the algorithm to work well. The basic idea is to have a warm-up data set to obtain initial values for the decision model parameters. It makes it possible to determine if there is any significant difference between the last typical behaviour and the new typical behaviour. The proposed algorithm demonstrates its performance under conditions of noisy non-stationary data in several time series classes. Keywords: concept drift, extreme value theory, multivariate time series, outlier detection.

Temporal Attention for Few-Shot Concept Drift Detection in Streaming Data

Concept drift describes unforeseeable changes in the underlying distribution of streaming data over time. Concept drift is a phenomenon in which the statistical properties of a target domain change over time in an arbitrary way. These changes might be caused by changes in hidden variables that cannot be measured directly. With the onset of the big data era, domains such as social networks, meteorology, and finance are generating copious amounts of streaming data. Embedded within these data, the issue of concept drift can affect the attributes of streaming data in various ways, leading to a decline in the accuracy and performance of models. There is a pressing need for new models to re-adapt to the changes in streaming data. Traditional concept drift detection algorithms struggle to effectively capture and utilize the key feature points of concept drift within complex time series, thereby failing to maintain the accuracy and efficiency of the models. In light of these challenges, this study introduces a novel concept drift detection method that incorporates a temporal attention mechanism within a prototypical network. By integrating a temporal attention mechanism during the feature extraction process, our approach enhances the capability to process complex time series data, preserves temporal locality, strengthens the learning of key features, and reduces the amount of labeled data required. This method significantly improves the detection accuracy and efficiency of small sample streaming data by better capturing the local features of the data. Experiments conducted across multiple datasets demonstrate that this method exhibits comprehensive leading performance in terms of accuracy and F1-score, with excellent recall and precision, thereby validating its effectiveness in enhancing concept drift detection in streaming data.

A novel framework for concept drift detection using autoencoders for classification problems in data streams

A novel framework for concept drift detection using autoencoders for classification problems in data streams

Tiny Machine Learning for Concept Drift.

Tiny machine learning (TML) is a new research area whose goal is to design machine and deep learning (DL) techniques able to operate in embedded systems and the Internet-of-Things (IoT) units, hence satisfying the severe technological constraints on memory, computation, and energy characterizing these pervasive devices. Interestingly, the related literature mainly focused on reducing the computational and memory demand of the inference phase of machine and deep learning models. At the same time, the training is typically assumed to be carried out in cloud or edge computing systems (due to the larger memory and computational requirements). This assumption results in TML solutions that might become obsolete when the process generating the data is affected by concept drift (e.g., due to periodicity or seasonality effect, faults or malfunctioning affecting sensors or actuators, or changes in the users' behavior), a common situation in real-world application scenarios. For the first time in the literature, this article introduces a TML for concept drift (TML-CD) solution based on deep learning feature extractors and a k -nearest neighbors ( k -NNs) classifier integrating a hybrid adaptation module able to deal with concept drift affecting the data-generating process. This adaptation module continuously updates (in a passive way) the knowledge base of TML-CD and, at the same time, employs a change detection test (CDT) to inspect for changes (in an active way) to quickly adapt to concept drift by removing obsolete knowledge. Experimental results on both image and audio benchmarks show the effectiveness of the proposed solution, whilst the porting of TML-CD on three off-the-shelf micro-controller units (MCUs) shows the feasibility of what is proposed in real-world pervasive systems.

A Stacking Framework for Online Locational Marginal Price Prediction Considering Concept Drift

A Stacking Framework for Online Locational Marginal Price Prediction Considering Concept Drift

Application of concept drift detection and adaptive framework for non linear time series data from cardiac surgery

AbstractThe quality of machine learning (ML) models deployed in dynamic environments tends to decline over time due to disparities between the data used for training and the upcoming data available for prediction, which is commonly known as drift. Therefore, it is important for ML models to be capable of detecting any changes or drift in the data distribution and updating the ML model accordingly. This study presents various drift detection techniques to identify drift in the survival outcomes of patients who underwent cardiac surgery. Additionally, this study proposes several drift adaptation strategies, such as adaptive learning, incremental learning, and ensemble learning. Through a detailed analysis of the results, the study confirms the superior performance of ensemble model, achieving a minimum mean absolute error (MAE) of 10.684 and 2.827 for predicting hospital stay and ICU stay, respectively. Furthermore, the models that incorporate a drift adaptive framework exhibit superior performance compared to the models that do not include such a framework.

Online Concept Drift Detector: Optimally Balancing Delay Detection, Runtime, Memory, and Accuracy.

Online learning and real-time data processing are becoming increasingly vital across various domains such as sensor networks, banking, and telecommunications. A significant challenge in this context is concept drift, wherein the statistical properties of the data change over time. Traditional drift detectors often grapple with high memory usage, extended delay in detection, prolonged runtime, and accuracy inconsistencies. This paper introduces a novel Online Drift Detector that meticulously balances these four aspects. By processing data instance-by-instance, our proposed detector optimizes the trade-offs between delay detection, runtime, memory consumption, and accuracy. We incorporate a unique diversity calculation tailored for multi-label problems, ensuring swift drift detection with minimized memory usage and enhanced runtime efficiency. Comparative analyses reveal the dominance of our approach over contemporary drift detection techniques, particularly in the realms of memory efficiency, detection speed, and accuracy. This work substantially augments the field of online data stream processing by offering a refined strategy for timely and efficient concept drift detection across a myriad of applications.

A novel framework for predicting non-stationary production time series of shale gas based on BiLSTM-RF-MPA deep fusion model

Shale gas, as an environmentally friendly fossil energy resource, has gained significant commercial development and shows immense potential. However, accurately predicting shale gas production faces substantial challenges due to the complex law of decline, nonlinear and non-stationary features in production data, which greatly repair the robustness of current models in predicting shale gas production time series. To address these challenges and improve accuracy in production forecasting, this paper introduces a novel and innovative approach: a hybrid proxy model that combines the bi-directional long short-term memory (BiLSTM) neural network and random forest (RF) through deep learning. The BiLSTM neural network is adept at capturing long-term dependencies, making it suitable for understanding the intricate relationships between input and output variables in shale gas production. On the other hand, RF serves a dual purpose: reducing model variance and addressing the concept drift problem that arises in non-stationary time series predictions made by BiLSTM. By integrating these two models, the hybrid approach effectively captures the inherent dependencies present in long and nonstationary production time series, thereby reducing model uncertainty. Furthermore, the combination of BiLSTM and RF is optimized using the recently-proposed marine predators algorithm (MPA) to fine-tune hyperparameters and enhance the overall performance of the proxy model. The results demonstrate that the proposed BiLSTM-RF-MPA model achieves higher prediction accuracy and demonstrates stronger generalization capabilities by effectively handling the complex nonlinear and non-stationary characteristics of shale gas production time series. Compared to other models such as LSTM, BiLSTM, and RF, the proposed model exhibits superior fitting and prediction performance, with an average improvement in performance indicators exceeding 20%. This innovative framework provides valuable insights for forecasting the complex production performance of unconventional oil and gas reservoirs, which sheds light on the development of data-driven proxy models in the field of subsurface energy utilization.

Probabilistic neural networks for incremental learning over time-varying streaming data with application to air pollution monitoring

This paper proposes a novel algorithm for incremental learning over streaming data in a non-stationary environment. The idea refers to the applicability of Probabilistic Neural Networks (PNNs), commonly used as a fast and robust method for solving classification problems in a stationary scenario. It is well known that PNNs have solid mathematical foundations but they fail in the non-stationary scenario and are not able to deal with concept drift. In this paper, the method based on the orthogonal series expansions of unknown probability densities is proposed. This approach significantly extends the applicability of the PNNs. The nonparametric procedures called Incremental Probabilistic Neural Networks (IPNNs) for tracking drifting probability densities or conditional class densities, in the case of classification, are presented. We prove their convergence as the stream data size is growing to infinity. More precisely, we show convergence in probability and with probability one as the sample size tends to infinity. Compared to the existing literature, almost exclusively based on various heuristics, the proposed method is mathematically justified and has solid theoretical foundations. Simulations performed on synthetic and air pollution data confirm the effectiveness of the algorithm.

On the Model Update Strategies for Supervised Learning in AIOps Solutions

AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data produced during the operation of large-scale systems and machine learning models to assist software engineers in their system operations. As operation data produced in the field are constantly evolving due to factors such as the changing operational environment and user base, the models in AIOps solutions need to be constantly maintained after deployment. While prior works focus on innovative modeling techniques to improve the performance of AIOps models before releasing them into the field, when and how to update AIOps models remain an under-investigated topic. In this work, we performed a case study on three large-scale public operation data: two trace datasets from the cloud computing platforms of Google and Alibaba and one disk stats dataset from the BackBlaze cloud storage data center. We empirically assessed five different types of model update strategies for supervised learning regarding their performance, updating cost, and stability. We observed that active model update strategies (e.g., periodical retraining, concept drift guided retraining, time-based model ensembles, and online learning) achieve better and more stable performance than a stationary model. Particularly, applying sophisticated model update strategies (e.g., concept drift detection, time-based ensembles, and online learning) could provide better performance, efficiency, and stability than simply retraining AIOps models periodically. In addition, we observed that, although some update strategies (e.g., time-based ensemble and online learning) can save model training time, they significantly sacrifice model testing time, which could hinder their applications in AIOps solutions where the operation data arrive at high pace and volume and where immediate inferences are required. Our findings highlight that practitioners should consider the evolution of operation data and actively maintain AIOps models over time. Our observations can also guide researchers and practitioners in investigating more efficient and effective model update strategies that fit in the context of AIOps.

Concept drift detection methods based on different weighting strategies

Concept drift detection methods based on different weighting strategies

Ensemble framework for concept drift detection and class imbalance in data streams

Ensemble framework for concept drift detection and class imbalance in data streams

EVALUATING MODEL ROBUSTNESS FOR DEFECT IDENTIFICATION AND CLASSIFICATION IN COMPOSITE AEROSTRUCTURE MATERIAL

Abstract Aircraft structures are required to have a high level of quality to satisfy their need for light weight, efficient flight, and withstanding high loads over their lifespan. These aerostructures are typically made from composite material due to their good tensile strength and resistance to compression. To ensure their structural integrity, the composite material requires inspection for common flaws such as porosity, delaminations, voids, foreign object debris, and other defects. Ultrasonic testing (UT) is a popular non-destructive inspection (NDI) technique used for effectively evaluating composite material. Current inspection methods rely heavily on human experience and are extremely time consuming. Therefore, there is a need for the development of techniques to reduce the manual inspection time. This work compares the performance of different deep learning-based methods in the identification and classification of defects. Deep learning has shown great promise in numerous fields, and we show its effectiveness in the evaluation of composite aerostructure material. The methods developed here are both highly reliable with a top recall value of 98.64% as well as extremely efficient requiring an average of 4 seconds during the inferencing stage to evaluate new composites. Finally, we investigate model robustness to concept drift by measuring its performance over time.

Dynamic Model Interpretation-Guided Online Active Learning Scheme for Real-Time Safety Assessment.

Chunk-level real-time safety assessment of dynamic systems is a critical component of industrial processes, which is essential to prevent hazards and reduce the risk of injury or damage to equipment and facilities, especially in nonstationary environments. In this context, multiple real and complex concept drifts are inevitable in industrial settings, making it crucial to understand their detection and adaptation processes. The incremental learning scheme should also be well considered. However, existing methods have certain limitations in dealing with such issues. In this article, a dynamic model interpretation-guided online active learning scheme, termed a dynamic model interpretation-guided learning scheme (DMI-LS), is proposed. Specifically, the model update strategy with chunk data is designed based on the implementation of the broad learning system. A novel query strategy is then investigated to consider the ranking preference difference, which relies on the interpretation generated by the explainable artificial intelligence method. Several experiments based on the JiaoLong deep-sea manned submersible data are conducted to verify the effects of the proposed DMI-LS. The results show that it outperforms the other advanced existing approaches with different settings in most scenarios.

A multi-agent adaptive deep learning framework for online intrusion detection

The network security analyzers use intrusion detection systems (IDSes) to distinguish malicious traffic from benign ones. The deep learning-based (DL-based) IDSes are proposed to auto-extract high-level features and eliminate the time-consuming and costly signature extraction process. However, this new generation of IDSes still needs to overcome a number of challenges to be employed in practical environments. One of the main issues of an applicable IDS is facing traffic concept drift, which manifests itself as new (i.e. , zero-day) attacks, in addition to the changing behavior of benign users/applications. Furthermore, a practical DL-based IDS needs to be conformed to a distributed (i.e. , multi-sensor) architecture in order to yield more accurate detections, create a collective attack knowledge based on the observations of different sensors, and also handle big data challenges for supporting high throughput networks. This paper proposes a novel multi-agent network intrusion detection framework to address the above shortcomings, considering a more practical scenario (i.e., online adaptable IDSes). This framework employs continual deep anomaly detectors for adapting each agent to the changing attack/benign patterns in its local traffic. In addition, a federated learning approach is proposed for sharing and exchanging local knowledge between different agents. Furthermore, the proposed framework implements sequential packet labeling for each flow, which provides an attack probability score for the flow by gradually observing each flow packet and updating its estimation. We evaluate the proposed framework by employing different deep models (including CNN-based and LSTM-based) over the CIC-IDS2017 and CSE-CIC-IDS2018 datasets. Through extensive evaluations and experiments, we show that the proposed distributed framework is well adapted to the traffic concept drift. More precisely, our results indicate that the CNN-based models are well suited for continually adapting to the traffic concept drift (i.e. , achieving an average detection rate of above 95% while needing just 128 new flows for the updating phase), and the LSTM-based models are a good candidate for sequential packet labeling in practical online IDSes (i.e. , detecting intrusions by just observing their first 15 packets).

Credit Card Fraud Detection Challenges and Solutions: A Review

Credit card fraud has become an increasing problem due to the growing reliance on electronic payment systems and technological advances that have improved fraud techniques. Numerous financial institutions are looking for the best ways to leverage technological advancements to provide better services to their end users, and researchers used various protection methods to provide security and privacy for credit cards. Therefore, it is necessary to identify the challenges and the proposed solutions to address them. This review provides an overview of the most recent research on the detection of fraudulent credit card transactions to protect those transactions from tampering or improper use, which includes imbalance classes, concept drift, and verification latency problems using machine learning and deep learning. It also provides valuable information for academic and industrial researchers and opens new avenues for research aimed at developing robust fraud detection systems.