High-dimensional Data Streams Research Articles

Time dimension feature extraction and classification of high-dimensional large data streams based on unsupervised learning

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

A two-stage monitoring scheme for multiple high-dimensional data streams

With the rapid development of data acquisition and analysis technology, the online monitoring of data streams is particularly important in modern industries. This study proposes an efficient online monitoring scheme for multiple high-dimensional data streams. We first construct a point-wise process monitoring control chart based on statistical extreme value theory. Then we use the clustering method to roughly obtain reliable data streams and suspected abnormal data streams, within the sliding window range, further identify abnormal data streams one by one over time points, and estimate the potential change points of these abnormal streams. Finally, we conduct simulations and case studies on this method, and compare it with existing competitive methods to demonstrate its effectiveness.

An Ensemble Methods of Predicting the New Labels with Concept Drift from a High-Dimensional Data Stream

Multi-Label Learning (MLL) has arisen in data engineering to identify instances based on a specific feature associated with a collection of labels. Adaptive learning necessitates classifying features with New Labels (NLs) if a data stream contains newer perspectives. As a result, an MLL with Emerging Multiple NLs (MuEMNL) and managing High-Dimensional data streams (MuEMNLHD) approaches were developed that divides the NL sets into multiple NLs for efficient classification. However, it did not handle concept drift issues when huge amounts of data arrived at high speeds using limited resources. Hence, this article proposes an adaptive ensemble learning approach to cope with a huge amount of data streams and solve concept drift issues by constructing a MuEMNL-Ensemble Neural Network (ENN) rather than a random forest classifier. It defines the number of NNs in the ensemble, whether or not they use constructive pruning, how many hidden nodes each NN uses, and how many training samples are used to train each NN independently. Also, to solve the concept drifts, pairwise and non-pairwise diversity measures are analyzed while constructing ensemble NN for efficient training using the entire learning examples. Moreover, the tradeoff between the NN’s precision and diversity is maintained simultaneously. At last, the test outcomes reveal that the proposed approach attains a better performance contrasted with the existing MLL approaches.

Control charting methods for monitoring high dimensional data streams: A conceptual classification scheme

There are always challenges in various industrial or non-industrial processes in which the product quality/service is described by a large number of quality characteristics. Thus, statistical process monitoring (SPM) techniques for capturing the quality of high-dimensional processes are becoming increasingly important, and various control charts have been developed to monitor different types of quality characteristics in this area. In general, the construction of conventional control charts to monitor multivariate processes in a high-dimensional setting has some statistical limitations and leads to misleading interactions. As a result, novel control charting techniques have recently been suggested to ameliorate the efficiency of monitoring schemes under high-dimensional data streams. This paper undertakes a conceptual classification structure using content analysis to classify the existing literature in this context for the period 2004–2024. Furthermore, on the basis of 72 selected papers, the research gaps are identified and some directions to stimulate potential for future research are suggested.

Simultaneous optimal control of directional missed discovery rates in data stream diagnosis

High-dimensional data streams are ubiquitous in modern manufacturing, due to their ability to provide valuable information about the industrial system’s performance on a real-time basis. If a shift occurs in a production process, fault diagnosis based on the data streams is of critical importance for identifying the root cause. Existing methods have largely focused on controlling the total missed discovery rate without distinguishing missed signals for positive versus negative components of the shift vector. In practice, however, losses incurred from the two directional shifts can differ substantially, so it is desirable to constrain the proportions of missed signals for positive and negative components at two distinctive levels. In this article, we propose a fault classification procedure that controls the two proportions separately. By formulating the problem as Lagrangian multiplier optimization, we show that the proposed procedure is optimal in the sense that it minimizes the expected number of false discoveries. We also suggest an iterative adjustment algorithm that converges to the optimal Lagrangian parameters. The asymptotic optimality for the data-driven version of our procedure is also established. Theoretical justification and numerical comparison with state-of-the-art methods show that the proposed procedure works well in applications.

Adaptive threshold optimisation for online feature selection using dynamic particle swarm optimisation in determining feature relevancy and redundancy

In the era of data-driven decision-making, managing dynamic data streams characterised by evolving data distributions and high dimensionality presents a formidable challenge for online feature selection. This research addresses the challenge by developing innovative solutions in optimising Online Feature Selection (OFS) to manage feature irrelevancy and redundancy and rigorously validating the proposed method in high-dimensional dynamic data streams. The research employs a structured methodology, introducing a novel method: a Dynamic Particle Swarm Optimisation (PSO)-based Threshold Optimisation method. Dynamic PSO, injected on three benchmark methods of OFS (i.e., Online Streaming Feature Selection (OSFS), Fast-OSFS, and Scalable and Accurate Online Feature Selection for Big Data (SAOLA)), enabled the global best position to be dynamically adjusted to exploit the best solution found in streaming data. This research presents two optimisation variants of the proposed method: RedundantPSO, which optimises redundant features, and IrrelevantPSO, which optimises irrelevant features. Unlike the traditional PSO method on feature selection that uses feature encoding representation, the proposed method is underpinned by two contributions: adaptive threshold particle representation of particle swarm optimisation and enhanced fitness function using minimisation of mean absolute deviation of dependency among feature subsets. Adaptive threshold particle representation combines the feature encoding part with a novel aspect that defines a threshold value of significance level ranging from 0.01 to 0.1. This unique contribution sets the research apart in the field where it enables the adaptive adjustment of the threshold value based on incoming features. Next, the adaptation of Mean Absolute Deviation (MAD) was integrated into the fitness evaluation of PSO to gain a more accurate and reliable measure of fitness for feature selection. During the experiment phase, we analysed various benchmark datasets with highly redundant and relevant behaviour. Our analysis concluded that selecting the appropriate threshold values significantly improved model performance for high-redundancy datasets, highlighting the need for careful threshold selection. The experimental evaluations have revealed that integrating RedundantPSO with OSFS (OSFS+RedundantPSO) resulted in a remarkable enhancement of the OSFS method's accuracy, achieving an impressive average accuracy rate of 76.8%. This substantial improvement includes occasional spikes of up to 3.8% over the baseline OSFS accuracy, showcasing OSFS+RedundantPSO as the top-performing combination. Furthermore, Fast-OSFS + RedundantPSO outperformed Fast-OSFS by a slight margin, reaching an average accuracy of 72.7%, while SAOLA + RedundantPSO exhibited a substantial 3.1% increase in average accuracy over SAOLA, reaching 74.0%. It is noteworthy to highlight that the threshold value searched in the proposed method also significantly impacted the identification of the behaviour of the dataset, either high relevancy or redundancy, even in the absence of prior domain knowledge. A higher threshold signifies the evaluation of a more redundant feature space. In conclusion, the results demonstrated the significant contributions of the method in enhancing model accuracy, adapting to evolving data distributions, and optimising feature subsets with acceptable runtime. The research aims to advance the field of data science, such as cybersecurity, finance, healthcare and more, while empowering end-users to make informed decisions under changing data stream circumstances.

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijds.2022.00011 . Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Adaptive Bernstein change detector for high-dimensional data streams

AbstractChange detection is of fundamental importance when analyzing data streams. Detecting changes both quickly and accurately enables monitoring and prediction systems to react, e.g., by issuing an alarm or by updating a learning algorithm. However, detecting changes is challenging when observations are high-dimensional. In high-dimensional data, change detectors should not only be able to identify when changes happen, but also in which subspace they occur. Ideally, one should also quantify how severe they are. Our approach, ABCD, has these properties. ABCD learns an encoder-decoder model and monitors its accuracy over a window of adaptive size. ABCD derives a change score based on Bernstein’s inequality to detect deviations in terms of accuracy, which indicate changes. Our experiments demonstrate that ABCD outperforms its best competitor by up to 20% in F1-score on average. It can also accurately estimate changes’ subspace, together with a severity measure that correlates with the ground truth.

Temporal Autoregressive Matrix Factorization for High-Dimensional Time Series Prediction of OSS.

Open-source software (OSS) plays an increasingly significant role in modern software development tendency, so accurate prediction of the future development of OSS has become an essential topic. The behavioral data of different open-source software are closely related to their development prospects. However, most of these behavioral data are typical high-dimensional time series data streams with noise and missing values. Hence, accurate prediction on such cluttered data requires the model to be highly scalable, which is not a property of traditional time series prediction models. To this end, we propose a temporal autoregressive matrix factorization (TAMF) framework that supports data-driven temporal learning and prediction. Specifically, we first construct a trend and period autoregressive model to extract trend and period features from OSS behavioral data, and then combine the regression model with a graph-based matrix factorization (MF) to complete the missing values by exploiting the correlations among the time series data. Finally, use the trained regression model to make predictions on the target data. This scheme ensures that TAMF can be applied to different types of high-dimensional time series data and thus has high versatility. We selected ten real developer behavior data from GitHub for case analysis. The experimental results show that TAMF has good scalability and prediction accuracy.

DRSCDM: A Novel Density-Related Clustering for Complex High-Dimensional Data Streams

DRSCDM: A Novel Density-Related Clustering for Complex High-Dimensional Data Streams

Tensor-Based Temporal Control for Partially Observed High-Dimensional Streaming Data

In advanced manufacturing processes, high-dimensional (HD) streaming data (e.g., sequential images or videos) are commonly used to provide online measurements of product quality. Although there exist numerous research studies for monitoring and anomaly detection using HD streaming data, little research is conducted on feedback control based on HD streaming data to improve product quality, especially in the presence of incomplete responses. To address this challenge, this article proposes a novel tensor-based automatic control method for partially observed HD streaming data, which consists of two stages: offline modeling and online control. In the offline modeling stage, we propose a one-step approach integrating parameter estimation of the system model with missing value imputation for the response data. This approach (i) improves the accuracy of parameter estimation, and (ii) maintains a stable and superior imputation performance in a wider range of the rank or missing ratio for the data to be completed, compared to the existing data completion methods. In the online control stage, for each incoming sample, missing observations are imputed by balancing its low-rank information and the one-step-ahead prediction result based on the control action from the last time step. Then, the optimal control action is computed by minimizing a quadratic loss function on the sum of squared deviations from the target. Furthermore, we conduct two sets of simulations and one case study on semiconductor manufacturing to validate the superiority of the proposed framework.

Feature Selection in the Data Stream Based on Incremental Markov Boundary Learning.

Recent years have witnessed the proliferation of techniques for streaming data mining to meet the demands of many real-time systems, where high-dimensional streaming data are generated at high speed, increasing the burden on both hardware and software. Some feature selection algorithms for streaming data are proposed to tackle this issue. However, these algorithms do not consider the distribution shift due to nonstationary scenarios, leading to performance degradation when the underlying distribution changes in the data stream. To solve this problem, this article investigates feature selection in streaming data through incremental Markov boundary (MB) learning and proposes a novel algorithm. Different from existing algorithms focusing on prediction performance on off-line data, the MB is learned by analyzing conditional dependence/independence in data, which uncovers the underlying mechanism and is naturally more robust against the distribution shift. To learn MB in the data stream, the proposal transforms the learned information in previous data blocks to prior knowledge and employs them to assist MB discovery in current data blocks, where the likelihood of distribution shift and reliability of conditional independence test are monitored to avoid the negative impact from invalid prior information. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of the proposed algorithm.

A unified diagnostic framework via symmetrized data aggregation

In statistical process control of high-dimensional data streams, in addition to online monitoring of abnormal changes, fault diagnosis of responsible components has become increasingly important. Existing diagnostic procedures have been designed for some typical models with distribution assumptions. Moreover, there is a lack of systematic approaches to provide a theoretical guarantee of significance in estimating shifted components. In this article, we introduce a new procedure to control the False Discovery Rate (FDR) of fault diagnosis. The proposed method formulates the fault diagnosis as a variable selection problem and utilizes the symmetrized data aggregation technique via sample splitting, data screening, and information pooling to control the FDR. Under some mild conditions, we show that the proposed method can achieve FDR control asymptotically. Extensive numerical studies and two real-data examples demonstrate satisfactory FDR control and remarkable diagnostic power in comparison to existing methods.

Knockoff procedure for false discovery rate control in high-dimensional data streams

Motivated by applications to root-cause identification of faults in high-dimensional data streams that may have very limited samples after faults are detected, we consider multiple testing in models for multivariate statistical process control (SPC). With quick fault detection, only small portion of data streams being out-of-control (OC) can be assumed. It is a long standing problem to identify those OC data streams while controlling the number of false discoveries. It is challenging due to the limited number of OC samples after the termination of the process when faults are detected. Although several false discovery rate (FDR) controlling methods have been proposed, people may prefer other methods for quick detection. With a recently developed method called Knockoff filtering, we propose a knockoff procedure that can combine with other fault detection methods in the sense that the knockoff procedure does not change the stopping time, but may identify another set of faults to control FDR. A theorem for the FDR control of the proposed procedure is provided. Simulation studies show that the proposed procedure can control FDR while maintaining high power. We also illustrate the performance in an application to semiconductor manufacturing processes that motivated this development.

Online directed-structural change-point detection: A segment-wise time-varying dynamic Bayesian network approach

High-dimensional data streams exist in many applications. Generally these high-dimensional streaming data have complex directed conditional dependence relationships evolving over time. However, modeling their directed conditional dependence structure and detecting its change over time in an online way has not been well studied in the current literature. To that end, we propose an ONline Segment-wise tiMe-varying dynAmic Bayesian netwoRk model with exTernal information (ONSMART), together with an online score-based inferring algorithm for directed-structural change-point detection in high-dimensional data. ONSMART adopts a linear vector autoregressive (VAR) model to describe directed inter-slice and intra-slice relations of variables. It further takes additional information about similarities of variables into account and regularizes similar variables to have similar structure positions in the network with graph Laplacian. ONSMART allows the parameters of VAR to change segment-wisely over time to describe the evolution of the conditional dependence structure and adopts a customized pruned exact linear time algorithm framework to identify directed-structural change-point detection. The L-BFGS-B approach is embedded in this framework to obtain the optimal dependence structure for each segment. Numerical studies using synthetic data and real data from a three-phase flow system are performed to verify the effectiveness of ONSMART.

Directional fault classification for correlated High-Dimensional data streams using hidden Markov models

Modern manufacturing systems are often installed with sensor networks which generate high-dimensional data at high velocity. These data streams offer valuable information about the industrial system’s real-time performance. If a shift occurs in the manufacturing process, fault diagnosis based on the data streams becomes a fundamental task as it identifies the affected data streams and provides insights into the root cause. Existing fault diagnostic methods either ignore the correlation between different streams or fail to determine the shift directions. In this paper, we propose a directional fault classification procedure that incorporates the between-stream correlations. We suggest a three-state hidden Markov model that captures the correlation structure and enables inference about the shift direction. We show that our procedure is optimal in the sense that it minimizes the expected number of false discoveries while controlling the proportion of missed signals at a desired level. We also propose a deconvolution-expectation-maximization (DEM) algorithm for estimating the model parameters and establish the asymptotic optimality for the data-driven version of our procedure. Numerical comparisons with an existing approach and an application to a semiconductor production study show that the proposed procedure works well in practice.

Self-Supervised Learning for Online Anomaly Detection in High-Dimensional Data Streams

In this paper, we address the problem of detecting and learning anomalies in high-dimensional data-streams in real-time. Following a data-driven approach, we propose an online and multivariate anomaly detection method that is suitable for the timely and accurate detection of anomalies. We propose our method for both semi-supervised and supervised settings. By combining the semi-supervised and supervised algorithms, we present a self-supervised online learning algorithm in which the semi-supervised algorithm trains the supervised algorithm to improve its detection performance over time. The methods are comprehensively analyzed in terms of computational complexity, asymptotic optimality, and false alarm rate. The performances of the proposed algorithms are also evaluated using real-world cybersecurity datasets, that show a significant improvement over the state-of-the-art results.

HSCFC: High-dimensional streaming data clustering algorithm based on feedback control system

At present, few studies have considered the clustering analysis of high-dimensional data collected by many sensors in a real-time streaming environment. Existing clustering analysis algorithms for high-dimensional data are primarily based on batch processing models, and most of them cannot meet the requirements of incremental high-dimensional data streams, which are extremely common in practical applications. To address the aforementioned problems, this paper focuses on the study of high-dimensional data clustering based on stream processing, and proposes a high-dimensional data stream clustering algorithm based on a feedback control system, which comprises three stages: window principal component analysis, feedback stream clustering, and feedback controller. The classic exponentially weighted attenuation function is used in the window principal component analysis to avoid concept drift in the data stream, and incremental feature extraction executes through the sliding window to improve the iterative efficiency of the data in the window. To minimize the errors caused by variability in projection angles during dimensionality reduction, a feedback stream clustering stage is designed with alternating iterations of window clustering and cluster aggregation. Aiming at the problems caused by manually adjusting the hyperparameters used in high-dimensional data stream clustering, a feedback controller is developed to adjust the hyperparameters in the two other stages by analyzing the clustering results in real time and using a discriminant score to adopt corresponding feedback strategies. The experimental comparisons between the proposed and the traditional algorithms on multiple datasets demonstrate the effectiveness of the former.

Double Sampling Adaptive Thresholding LASSO Variability Chart for Phase II Monitoring of High-Dimensional Data Streams

In many manufacturing or service systems, the product quality and/or process outcome is described in terms of a large number of quality characteristics. Since the use of large sample sizes in such systems imposes significant costs on the manufacturer, the sample size is usually considered smaller than the process dimension. In this condition, which is referred to high-dimensionality, the sample covariance matrix is not a positive semi-definite matrix. Moreover, another issue to establish variability control charts under high-dimensional setting is their speed to detect the process disturbances under both diagonal and off-diagonal shift patterns. To solve these drawbacks, in this study, a double sampling-based adaptive thresholding LASSO control chart called DSATL is developed to monitor the variability matrix of high-dimensional processes under sparsity assumption. The run length distribution of the DSATL chart under seven out-of-control patterns is evaluated taking a different number of quality characteristics into account. The simulation results show that the detectability of the DSATL chart outperforms the classical adaptive thresholding LASSO one under various changes in the process covariance matrix. The results also confirm that the sensitivity of the DSATL chart for real-time detection of covariance matrix anomalies improves as the process dimension increases.

Towards lowering computational power in IoT systems: Clustering algorithm for high-dimensional data stream using entropy window reduction

In a world of connectivity empowered by the advancement of the Internet of Things (IoT), an infinite number of data streams have emerged. Thus, data stream clustering is crucial for extracting hidden knowledge and data mining. Various data stream clustering methods have lately been introduced. Yet, the majority of such algorithms are affected by the curse of high dimensionality. Lately, a fully online buffer-based clustering algorithm for handling evolving data streams (BOCEDS) was developed. Similarly to other existing density-based clustering methods, BOCEDS is not capable of handling high-dimensional data and has high computational power and high memory utilization. This paper introduces an Entropy Window Reduction (EWR) algorithm, which is an improved version of the BOCEDS technique. EWR is a fully online clustering technique for handling high-dimensional data streams using feature ranking and sorting. This process is accomplished by calculating the entropy of specific features with respect to the time window. The findings of the experiments are compared to the outcomes of BOCEDS, CEDAS, and MuDi-Stream algorithms. The outcomes indicate that the EWR algorithm outperformed the baseline clustering algorithms. The results are demonstrated using the KDDCup’99 dataset in terms of quality and complexity evaluation on the average of F-Measures, Jaccard Index, Fowlkes–Mallows index, Purity, and Rand Index as well as the memory usage and computational power with 88%, 66%, 81%, 100%, and 66%, respectively. The results also show low memory usage and computing power in comparison with the baseline algorithms.