High-dimensional Data Streams Research Articles

Online kernel sliced inverse regression

Online dimension reduction techniques are widely utilized for handling high-dimensional streaming data. Extensive research has been conducted on various methods, including Online Principal Component Analysis, Online Sliced Inverse Regression (OSIR), and Online Kernel Principal Component Analysis (OKPCA). However, it is important to note that the exploration of online supervised nonlinear dimension reduction techniques is still limited. This article presents a novel approach called Online Kernel Sliced Inverse Regression (OKSIR), which specifically tackles the challenge of dealing with the increasing dimension of the kernel matrix as the sample size grows. The proposed method incorporates two key components: the approximate linear dependence condition and dictionary variable sets. These components enable a reduced-order approach for online variable updates, improving the efficiency of the process. To solve the OKSIR problem, we formulate it as an online generalized eigen-decomposition problem and employ stochastic optimization techniques to update the dimension reduction directions. Theoretical properties of this online learner are established, providing a solid foundation for its application. Through extensive simulations and real data analysis, we demonstrate that the proposed OKSIR method achieves performance comparable to that of batch processing kernel sliced inverse regression. This research significantly contributes to the advancement of online dimension reduction techniques, enhancing their effectiveness in practical applications.

Temporal Autoregressive Matrix Factorization for High-Dimensional Time Series Prediction of OSS.

Open-source software (OSS) plays an increasingly significant role in modern software development tendency, so accurate prediction of the future development of OSS has become an essential topic. The behavioral data of different open-source software are closely related to their development prospects. However, most of these behavioral data are typical high-dimensional time series data streams with noise and missing values. Hence, accurate prediction on such cluttered data requires the model to be highly scalable, which is not a property of traditional time series prediction models. To this end, we propose a temporal autoregressive matrix factorization (TAMF) framework that supports data-driven temporal learning and prediction. Specifically, we first construct a trend and period autoregressive model to extract trend and period features from OSS behavioral data, and then combine the regression model with a graph-based matrix factorization (MF) to complete the missing values by exploiting the correlations among the time series data. Finally, use the trained regression model to make predictions on the target data. This scheme ensures that TAMF can be applied to different types of high-dimensional time series data and thus has high versatility. We selected ten real developer behavior data from GitHub for case analysis. The experimental results show that TAMF has good scalability and prediction accuracy.

A unified framework for high-dimensional data stream analysis in fault diagnosis

Modern industrial systems routinely generate data in high volume at high velocity. These high-dimensional data streams (HDDS) provide valuable information at granular levels to quality personnel during root cause investigation in cases of a system fault. The goal of fault analysis using HDDS is twofold: (1) identify abnormal data streams and (2) locate the change point when the processes become out of control. Existing research has largely focused on addressing the two issues separately. In this article, we propose a unified framework by formulating the problem as optimal control of hierarchical missed discovery rates in multiple classifications. Theoretically, we establish that our approach minimizes the number of false discoveries while controlling the missed discovery rates at desired levels. Numerically, we develop a computationally efficient algorithm for solving the optimization and demonstrate its superior performance over the existing methods. A data-driven version of the proposed approach is suggested as well. An application to a real data set in semiconductor manufacturing shows that our approach works well in practice.

Learning from high-dimensional cyber-physical data streams: a case of large-scale smart grid

Learning from high-dimensional cyber-physical data streams: a case of large-scale smart grid

Multilinear principal component analysis-based tensor decomposition for fabric weave pattern recognition from high-dimensional streaming data

Multilinear principal component analysis-based tensor decomposition for fabric weave pattern recognition from high-dimensional streaming data

Time dimension feature extraction and classification of high-dimensional large data streams based on unsupervised learning

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

A two-stage monitoring scheme for multiple high-dimensional data streams

With the rapid development of data acquisition and analysis technology, the online monitoring of data streams is particularly important in modern industries. This study proposes an efficient online monitoring scheme for multiple high-dimensional data streams. We first construct a point-wise process monitoring control chart based on statistical extreme value theory. Then we use the clustering method to roughly obtain reliable data streams and suspected abnormal data streams, within the sliding window range, further identify abnormal data streams one by one over time points, and estimate the potential change points of these abnormal streams. Finally, we conduct simulations and case studies on this method, and compare it with existing competitive methods to demonstrate its effectiveness.

An Ensemble Methods of Predicting the New Labels with Concept Drift from a High-Dimensional Data Stream

Multi-Label Learning (MLL) has arisen in data engineering to identify instances based on a specific feature associated with a collection of labels. Adaptive learning necessitates classifying features with New Labels (NLs) if a data stream contains newer perspectives. As a result, an MLL with Emerging Multiple NLs (MuEMNL) and managing High-Dimensional data streams (MuEMNLHD) approaches were developed that divides the NL sets into multiple NLs for efficient classification. However, it did not handle concept drift issues when huge amounts of data arrived at high speeds using limited resources. Hence, this article proposes an adaptive ensemble learning approach to cope with a huge amount of data streams and solve concept drift issues by constructing a MuEMNL-Ensemble Neural Network (ENN) rather than a random forest classifier. It defines the number of NNs in the ensemble, whether or not they use constructive pruning, how many hidden nodes each NN uses, and how many training samples are used to train each NN independently. Also, to solve the concept drifts, pairwise and non-pairwise diversity measures are analyzed while constructing ensemble NN for efficient training using the entire learning examples. Moreover, the tradeoff between the NN’s precision and diversity is maintained simultaneously. At last, the test outcomes reveal that the proposed approach attains a better performance contrasted with the existing MLL approaches.

Control charting methods for monitoring high dimensional data streams: A conceptual classification scheme

There are always challenges in various industrial or non-industrial processes in which the product quality/service is described by a large number of quality characteristics. Thus, statistical process monitoring (SPM) techniques for capturing the quality of high-dimensional processes are becoming increasingly important, and various control charts have been developed to monitor different types of quality characteristics in this area. In general, the construction of conventional control charts to monitor multivariate processes in a high-dimensional setting has some statistical limitations and leads to misleading interactions. As a result, novel control charting techniques have recently been suggested to ameliorate the efficiency of monitoring schemes under high-dimensional data streams. This paper undertakes a conceptual classification structure using content analysis to classify the existing literature in this context for the period 2004–2024. Furthermore, on the basis of 72 selected papers, the research gaps are identified and some directions to stimulate potential for future research are suggested.

Simultaneous optimal control of directional missed discovery rates in data stream diagnosis

High-dimensional data streams are ubiquitous in modern manufacturing, due to their ability to provide valuable information about the industrial system’s performance on a real-time basis. If a shift occurs in a production process, fault diagnosis based on the data streams is of critical importance for identifying the root cause. Existing methods have largely focused on controlling the total missed discovery rate without distinguishing missed signals for positive versus negative components of the shift vector. In practice, however, losses incurred from the two directional shifts can differ substantially, so it is desirable to constrain the proportions of missed signals for positive and negative components at two distinctive levels. In this article, we propose a fault classification procedure that controls the two proportions separately. By formulating the problem as Lagrangian multiplier optimization, we show that the proposed procedure is optimal in the sense that it minimizes the expected number of false discoveries. We also suggest an iterative adjustment algorithm that converges to the optimal Lagrangian parameters. The asymptotic optimality for the data-driven version of our procedure is also established. Theoretical justification and numerical comparison with state-of-the-art methods show that the proposed procedure works well in applications.

Adaptive threshold optimisation for online feature selection using dynamic particle swarm optimisation in determining feature relevancy and redundancy

In the era of data-driven decision-making, managing dynamic data streams characterised by evolving data distributions and high dimensionality presents a formidable challenge for online feature selection. This research addresses the challenge by developing innovative solutions in optimising Online Feature Selection (OFS) to manage feature irrelevancy and redundancy and rigorously validating the proposed method in high-dimensional dynamic data streams. The research employs a structured methodology, introducing a novel method: a Dynamic Particle Swarm Optimisation (PSO)-based Threshold Optimisation method. Dynamic PSO, injected on three benchmark methods of OFS (i.e., Online Streaming Feature Selection (OSFS), Fast-OSFS, and Scalable and Accurate Online Feature Selection for Big Data (SAOLA)), enabled the global best position to be dynamically adjusted to exploit the best solution found in streaming data. This research presents two optimisation variants of the proposed method: RedundantPSO, which optimises redundant features, and IrrelevantPSO, which optimises irrelevant features. Unlike the traditional PSO method on feature selection that uses feature encoding representation, the proposed method is underpinned by two contributions: adaptive threshold particle representation of particle swarm optimisation and enhanced fitness function using minimisation of mean absolute deviation of dependency among feature subsets. Adaptive threshold particle representation combines the feature encoding part with a novel aspect that defines a threshold value of significance level ranging from 0.01 to 0.1. This unique contribution sets the research apart in the field where it enables the adaptive adjustment of the threshold value based on incoming features. Next, the adaptation of Mean Absolute Deviation (MAD) was integrated into the fitness evaluation of PSO to gain a more accurate and reliable measure of fitness for feature selection. During the experiment phase, we analysed various benchmark datasets with highly redundant and relevant behaviour. Our analysis concluded that selecting the appropriate threshold values significantly improved model performance for high-redundancy datasets, highlighting the need for careful threshold selection. The experimental evaluations have revealed that integrating RedundantPSO with OSFS (OSFS+RedundantPSO) resulted in a remarkable enhancement of the OSFS method's accuracy, achieving an impressive average accuracy rate of 76.8%. This substantial improvement includes occasional spikes of up to 3.8% over the baseline OSFS accuracy, showcasing OSFS+RedundantPSO as the top-performing combination. Furthermore, Fast-OSFS + RedundantPSO outperformed Fast-OSFS by a slight margin, reaching an average accuracy of 72.7%, while SAOLA + RedundantPSO exhibited a substantial 3.1% increase in average accuracy over SAOLA, reaching 74.0%. It is noteworthy to highlight that the threshold value searched in the proposed method also significantly impacted the identification of the behaviour of the dataset, either high relevancy or redundancy, even in the absence of prior domain knowledge. A higher threshold signifies the evaluation of a more redundant feature space. In conclusion, the results demonstrated the significant contributions of the method in enhancing model accuracy, adapting to evolving data distributions, and optimising feature subsets with acceptable runtime. The research aims to advance the field of data science, such as cybersecurity, finance, healthcare and more, while empowering end-users to make informed decisions under changing data stream circumstances.

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijds.2022.00011 . Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Adaptive Bernstein change detector for high-dimensional data streams

AbstractChange detection is of fundamental importance when analyzing data streams. Detecting changes both quickly and accurately enables monitoring and prediction systems to react, e.g., by issuing an alarm or by updating a learning algorithm. However, detecting changes is challenging when observations are high-dimensional. In high-dimensional data, change detectors should not only be able to identify when changes happen, but also in which subspace they occur. Ideally, one should also quantify how severe they are. Our approach, ABCD, has these properties. ABCD learns an encoder-decoder model and monitors its accuracy over a window of adaptive size. ABCD derives a change score based on Bernstein’s inequality to detect deviations in terms of accuracy, which indicate changes. Our experiments demonstrate that ABCD outperforms its best competitor by up to 20% in F1-score on average. It can also accurately estimate changes’ subspace, together with a severity measure that correlates with the ground truth.

DRSCDM: A Novel Density-Related Clustering for Complex High-Dimensional Data Streams

DRSCDM: A Novel Density-Related Clustering for Complex High-Dimensional Data Streams

Self-Starting Monitoring and Dynamic Sampling of High-Dimensional Data Streams

Self-Starting Monitoring and Dynamic Sampling of High-Dimensional Data Streams

Tensor-Based Temporal Control for Partially Observed High-Dimensional Streaming Data

In advanced manufacturing processes, high-dimensional (HD) streaming data (e.g., sequential images or videos) are commonly used to provide online measurements of product quality. Although there exist numerous research studies for monitoring and anomaly detection using HD streaming data, little research is conducted on feedback control based on HD streaming data to improve product quality, especially in the presence of incomplete responses. To address this challenge, this article proposes a novel tensor-based automatic control method for partially observed HD streaming data, which consists of two stages: offline modeling and online control. In the offline modeling stage, we propose a one-step approach integrating parameter estimation of the system model with missing value imputation for the response data. This approach (i) improves the accuracy of parameter estimation, and (ii) maintains a stable and superior imputation performance in a wider range of the rank or missing ratio for the data to be completed, compared to the existing data completion methods. In the online control stage, for each incoming sample, missing observations are imputed by balancing its low-rank information and the one-step-ahead prediction result based on the control action from the last time step. Then, the optimal control action is computed by minimizing a quadratic loss function on the sum of squared deviations from the target. Furthermore, we conduct two sets of simulations and one case study on semiconductor manufacturing to validate the superiority of the proposed framework.

Feature Selection in the Data Stream Based on Incremental Markov Boundary Learning.

Recent years have witnessed the proliferation of techniques for streaming data mining to meet the demands of many real-time systems, where high-dimensional streaming data are generated at high speed, increasing the burden on both hardware and software. Some feature selection algorithms for streaming data are proposed to tackle this issue. However, these algorithms do not consider the distribution shift due to nonstationary scenarios, leading to performance degradation when the underlying distribution changes in the data stream. To solve this problem, this article investigates feature selection in streaming data through incremental Markov boundary (MB) learning and proposes a novel algorithm. Different from existing algorithms focusing on prediction performance on off-line data, the MB is learned by analyzing conditional dependence/independence in data, which uncovers the underlying mechanism and is naturally more robust against the distribution shift. To learn MB in the data stream, the proposal transforms the learned information in previous data blocks to prior knowledge and employs them to assist MB discovery in current data blocks, where the likelihood of distribution shift and reliability of conditional independence test are monitored to avoid the negative impact from invalid prior information. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of the proposed algorithm.

A unified diagnostic framework via symmetrized data aggregation

In statistical process control of high-dimensional data streams, in addition to online monitoring of abnormal changes, fault diagnosis of responsible components has become increasingly important. Existing diagnostic procedures have been designed for some typical models with distribution assumptions. Moreover, there is a lack of systematic approaches to provide a theoretical guarantee of significance in estimating shifted components. In this article, we introduce a new procedure to control the False Discovery Rate (FDR) of fault diagnosis. The proposed method formulates the fault diagnosis as a variable selection problem and utilizes the symmetrized data aggregation technique via sample splitting, data screening, and information pooling to control the FDR. Under some mild conditions, we show that the proposed method can achieve FDR control asymptotically. Extensive numerical studies and two real-data examples demonstrate satisfactory FDR control and remarkable diagnostic power in comparison to existing methods.

Knockoff procedure for false discovery rate control in high-dimensional data streams

Motivated by applications to root-cause identification of faults in high-dimensional data streams that may have very limited samples after faults are detected, we consider multiple testing in models for multivariate statistical process control (SPC). With quick fault detection, only small portion of data streams being out-of-control (OC) can be assumed. It is a long standing problem to identify those OC data streams while controlling the number of false discoveries. It is challenging due to the limited number of OC samples after the termination of the process when faults are detected. Although several false discovery rate (FDR) controlling methods have been proposed, people may prefer other methods for quick detection. With a recently developed method called Knockoff filtering, we propose a knockoff procedure that can combine with other fault detection methods in the sense that the knockoff procedure does not change the stopping time, but may identify another set of faults to control FDR. A theorem for the FDR control of the proposed procedure is provided. Simulation studies show that the proposed procedure can control FDR while maintaining high power. We also illustrate the performance in an application to semiconductor manufacturing processes that motivated this development.

Online directed-structural change-point detection: A segment-wise time-varying dynamic Bayesian network approach

High-dimensional data streams exist in many applications. Generally these high-dimensional streaming data have complex directed conditional dependence relationships evolving over time. However, modeling their directed conditional dependence structure and detecting its change over time in an online way has not been well studied in the current literature. To that end, we propose an ONline Segment-wise tiMe-varying dynAmic Bayesian netwoRk model with exTernal information (ONSMART), together with an online score-based inferring algorithm for directed-structural change-point detection in high-dimensional data. ONSMART adopts a linear vector autoregressive (VAR) model to describe directed inter-slice and intra-slice relations of variables. It further takes additional information about similarities of variables into account and regularizes similar variables to have similar structure positions in the network with graph Laplacian. ONSMART allows the parameters of VAR to change segment-wisely over time to describe the evolution of the conditional dependence structure and adopts a customized pruned exact linear time algorithm framework to identify directed-structural change-point detection. The L-BFGS-B approach is embedded in this framework to obtain the optimal dependence structure for each segment. Numerical studies using synthetic data and real data from a three-phase flow system are performed to verify the effectiveness of ONSMART.