High-dimensional Data Streams Research Articles

Self-Supervised Learning for Online Anomaly Detection in High-Dimensional Data Streams

In this paper, we address the problem of detecting and learning anomalies in high-dimensional data-streams in real-time. Following a data-driven approach, we propose an online and multivariate anomaly detection method that is suitable for the timely and accurate detection of anomalies. We propose our method for both semi-supervised and supervised settings. By combining the semi-supervised and supervised algorithms, we present a self-supervised online learning algorithm in which the semi-supervised algorithm trains the supervised algorithm to improve its detection performance over time. The methods are comprehensively analyzed in terms of computational complexity, asymptotic optimality, and false alarm rate. The performances of the proposed algorithms are also evaluated using real-world cybersecurity datasets, that show a significant improvement over the state-of-the-art results.

HSCFC: High-dimensional streaming data clustering algorithm based on feedback control system

At present, few studies have considered the clustering analysis of high-dimensional data collected by many sensors in a real-time streaming environment. Existing clustering analysis algorithms for high-dimensional data are primarily based on batch processing models, and most of them cannot meet the requirements of incremental high-dimensional data streams, which are extremely common in practical applications. To address the aforementioned problems, this paper focuses on the study of high-dimensional data clustering based on stream processing, and proposes a high-dimensional data stream clustering algorithm based on a feedback control system, which comprises three stages: window principal component analysis, feedback stream clustering, and feedback controller. The classic exponentially weighted attenuation function is used in the window principal component analysis to avoid concept drift in the data stream, and incremental feature extraction executes through the sliding window to improve the iterative efficiency of the data in the window. To minimize the errors caused by variability in projection angles during dimensionality reduction, a feedback stream clustering stage is designed with alternating iterations of window clustering and cluster aggregation. Aiming at the problems caused by manually adjusting the hyperparameters used in high-dimensional data stream clustering, a feedback controller is developed to adjust the hyperparameters in the two other stages by analyzing the clustering results in real time and using a discriminant score to adopt corresponding feedback strategies. The experimental comparisons between the proposed and the traditional algorithms on multiple datasets demonstrate the effectiveness of the former.

Double Sampling Adaptive Thresholding LASSO Variability Chart for Phase II Monitoring of High-Dimensional Data Streams

In many manufacturing or service systems, the product quality and/or process outcome is described in terms of a large number of quality characteristics. Since the use of large sample sizes in such systems imposes significant costs on the manufacturer, the sample size is usually considered smaller than the process dimension. In this condition, which is referred to high-dimensionality, the sample covariance matrix is not a positive semi-definite matrix. Moreover, another issue to establish variability control charts under high-dimensional setting is their speed to detect the process disturbances under both diagonal and off-diagonal shift patterns. To solve these drawbacks, in this study, a double sampling-based adaptive thresholding LASSO control chart called DSATL is developed to monitor the variability matrix of high-dimensional processes under sparsity assumption. The run length distribution of the DSATL chart under seven out-of-control patterns is evaluated taking a different number of quality characteristics into account. The simulation results show that the detectability of the DSATL chart outperforms the classical adaptive thresholding LASSO one under various changes in the process covariance matrix. The results also confirm that the sensitivity of the DSATL chart for real-time detection of covariance matrix anomalies improves as the process dimension increases.

Towards lowering computational power in IoT systems: Clustering algorithm for high-dimensional data stream using entropy window reduction

In a world of connectivity empowered by the advancement of the Internet of Things (IoT), an infinite number of data streams have emerged. Thus, data stream clustering is crucial for extracting hidden knowledge and data mining. Various data stream clustering methods have lately been introduced. Yet, the majority of such algorithms are affected by the curse of high dimensionality. Lately, a fully online buffer-based clustering algorithm for handling evolving data streams (BOCEDS) was developed. Similarly to other existing density-based clustering methods, BOCEDS is not capable of handling high-dimensional data and has high computational power and high memory utilization. This paper introduces an Entropy Window Reduction (EWR) algorithm, which is an improved version of the BOCEDS technique. EWR is a fully online clustering technique for handling high-dimensional data streams using feature ranking and sorting. This process is accomplished by calculating the entropy of specific features with respect to the time window. The findings of the experiments are compared to the outcomes of BOCEDS, CEDAS, and MuDi-Stream algorithms. The outcomes indicate that the EWR algorithm outperformed the baseline clustering algorithms. The results are demonstrated using the KDDCup’99 dataset in terms of quality and complexity evaluation on the average of F-Measures, Jaccard Index, Fowlkes–Mallows index, Purity, and Rand Index as well as the memory usage and computational power with 88%, 66%, 81%, 100%, and 66%, respectively. The results also show low memory usage and computing power in comparison with the baseline algorithms.

Methods for Predicting the Rise of the New Labels from a High-Dimensional Data Stream

Methods for Predicting the Rise of the New Labels from a High-Dimensional Data Stream

Online monitoring of high-dimensional asynchronous and heterogeneous data streams for shifts in location and scale

Recent advancement of sensor technology has made it possible to monitor high-dimensional data streams in various manufacturing systems for quality improvement. However, existing monitoring schemes commonly assume that all data streams have the same sampling interval. This assumption does not always hold in practice, which poses new and unique challenges for multivariate statistical process control. In this paper, we propose a generic nonparametric monitoring framework to online monitor high-dimensional asynchronous and heterogeneous data streams, where sampling intervals of data streams are different from each other, and measurements of each data stream follow arbitrary distributions. In particular, we first propose a quantile-based nonparametric framework to monitor each data stream locally for possible shifts in both location and scale. Then, for unsampled measurements due to different sampling intervals, a compensation strategy based on the Bayesian approach is introduced. Furthermore, we develop a global monitoring scheme using the sum of top- r local statistics, which can quickly detect a wide range of possible shifts in all directions. Simulations and case studies are conducted to evaluate the performance and demonstrate the superiority of the proposed method.

Subspace Clustering in High-Dimensional Data Streams: A Systematic Literature Review

Clustering high dimensional data is challenging as data dimensionality increases the distance between data points, resulting in sparse regions that degrade clustering performance. Subspace clustering is a common approach for processing high-dimensional data by finding relevant features for each cluster in the data space. Subspace clustering methods extend traditional clustering to account for the constraints imposed by data streams. Data streams are not only high-dimensional, but also unbounded and evolving. This necessitates the development of subspace clustering algorithms that can handle high dimensionality and adapt to the unique characteristics of data streams. Although many articles have contributed to the literature review on data stream clustering, there is currently no specific review on subspace clustering algorithms in high-dimensional data streams. Therefore, this article aims to systematically review the existing literature on subspace clustering of data streams in high-dimensional streaming environments. The review follows a systematic methodological approach and includes 18 articles for the final analysis. The analysis focused on two research questions related to the general clustering process and dealing with the unbounded and evolving characteristics of data streams. The main findings relate to six elements: clustering process, cluster search, subspace search, synopsis structure, cluster maintenance, and evaluation measures. Most algorithms use a two-phase clustering approach consisting of an initialization stage, a refinement stage, a cluster maintenance stage, and a final clustering stage. The density-based top-down subspace clustering approach is more widely used than the others because it is able to distinguish true clusters and outliers using projected micro-clusters. Most algorithms implicitly adapt to the evolving nature of the data stream by using a time fading function that is sensitive to outliers. Future work can focus on the clustering framework, parameter optimization, subspace search techniques, memory-efficient synopsis structures, explicit cluster change detection, and intrinsic performance metrics. This article can serve as a guide for researchers interested in high-dimensional subspace clustering methods for data streams.

Estimation of high-dimensional change-points under a group sparsity structure

Change-points are a routine feature of ‘big data’ observed in the form of high-dimensional data streams. In many such data streams, the component series possess group structures and it is natural to assume that changes only occur in a small number of all groups. We propose a new change point procedure, called groupInspect, that exploits the group sparsity structure to estimate a projection direction so as to aggregate information across the component series to successfully estimate the change-point in the mean structure of the series. We prove that the estimated projection direction is minimax optimal, up to logarithmic factors, when all group sizes are of comparable order. Moreover, our theory provide strong guarantees on the rate of convergence of the change-point location estimator. Numerical studies demonstrates the competitive performance of groupInspect in a wide range of settings and a real data example confirms the practical usefulness of our procedure.

Online sparse sliced inverse regression for high-dimensional streaming data

Due to the demand for tackling the problem of streaming data with high-dimensional covarites, we propose an online sparse sliced inverse regression (OSSIR) method for online sufficient dimension reduction (SDR). The existing online SDR methods focus on the case when [Formula: see text] (dimension of covariates) is small. In this paper, we adapt the sparse sliced inverse regression to cope with high-dimensional streaming data where the dimension [Formula: see text] is large. There are two important steps in our method, one is to extend the online principal component analysis to iteratively obtain the eigenvalues and eigenvectors of the kernel matrix, the other is to use the truncated gradient to perform online [Formula: see text] regularization. Theoretical properties of the proposed online learner are established. By comparing with several existing methods in simulations and real data applications, we demonstrate the effectiveness and efficiency of our method.

PERCEPT: A New Online Change-Point Detection Method using Topological Data Analysis

Topological data analysis (TDA) provides a set of data analysis tools for extracting embedded topological structures from complex high-dimensional datasets. In recent years, TDA has been a rapidly growing field which has found success in a wide range of applications, including signal processing, neuroscience and network analysis. In these applications, the online detection of changes is of crucial importance, but this can be highly challenging since such changes often occur in low-dimensional embeddings within high-dimensional data streams. We thus propose a new method, called PERsistence diagram-based ChangE-PoinT detection (PERCEPT), which leverages the learned topological structure from TDA to sequentially detect changes. PERCEPT follows two key steps: it first learns the embedded topology as a point cloud via persistence diagrams, then applies a nonparametric monitoring approach for detecting changes in the resulting point cloud distributions. This yields a nonparametric, topology-aware framework which can efficiently detect online geometric changes. We investigate the effectiveness of PERCEPT over existing methods in a suite of numerical experiments where the data streams have an embedded topological structure. We then demonstrate the usefulness of PERCEPT in two applications on solar flare monitoring and human gesture detection.

A Bayesian Partially Observable Online Change Detection Approach with Thompson Sampling

This article proposes a Bayesian learning framework for online change detection of high-dimensional data streams where only a subset of variables can be observed at each time point due to limited sensing capacities. On the one hand, we need to build a change detection scheme based on partial observations. On the other, the scheme should be able to adaptively and actively select the most critical sensing variables to observe to maximize the detection power. To address these two points, in this article, first, a novel Bayesian Spike-Slab Composite Decomposition (BSSCD) is proposed to decompose the high-dimensional signals onto normal and abnormal bases, where the projection coefficients are efficiently estimated via variational Bayesian inference. Built upon it, the posterior Bayes factor is constructed as the detection statistic. Second, by further formulating the detection statistic as the reward function of combinatorial multi-armed bandit (CMAB), a Thompson sampling strategy is proposed for selecting the potential changed variables with the balance of exploration and exploitation. The efficacy and applicability of our method are demonstrated in practice with numerical studies and a real case study.

Online Multivariate Anomaly Detection and Localization for High-Dimensional Settings.

This paper considers the real-time detection of abrupt and persistent anomalies in high-dimensional data streams. The goal is to detect anomalies quickly and accurately so that the appropriate countermeasures could be taken in time before the system possibly gets harmed. We propose a sequential and multivariate anomaly detection method that scales well to high-dimensional datasets. The proposed method follows a nonparametric, i.e., data-driven, and semi-supervised approach, i.e., trains only on nominal data. Thus, it is applicable to a wide range of applications and data types. Thanks to its multivariate nature, it can quickly and accurately detect challenging anomalies, such as changes in the correlation structure. Its asymptotic optimality and computational complexity are comprehensively analyzed. In conjunction with the detection method, an effective technique for localizing the anomalous data dimensions is also proposed. The practical use of proposed algorithms are demonstrated using synthetic and real data, and in variety of applications including seizure detection, DDoS attack detection, and video surveillance.

A Generic Online Nonparametric Monitoring and Sampling Strategy for High-Dimensional Heterogeneous Processes

With the rapid advancement of in-process measurements and sensor technology driven by zero-defect manufacturing applications, high-dimensional heterogeneous processes that continuously collect distinct physical characteristics frequently appear in modern industries. Such large-volume high-dimensional data place a heavy demand on data collection, transmission, and analysis in practice. Thus, practitioners often need to decide which informative data streams to observe given the resource constraints at each data acquisition time, which poses significant challenges for multivariate statistical process control and quality improvement. In this article, we propose a generic online nonparametric monitoring and sampling scheme to quickly detect mean shifts occurring in heterogeneous processes when only partial observations are available at each acquisition time. Our innovative idea is to seamlessly integrate the Thompson sampling (TS) algorithm with a quantile-based nonparametric cumulative sum (CUSUM) procedure to construct local statistics of all data streams based on the partially observed data. Furthermore, we develop a global monitoring scheme of using the sum of top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${r}$ </tex-math></inline-formula> local statistics, which can quickly detect a wide range of possible mean shifts. Tailored to monitoring the heterogeneous data streams, the proposed method balances between exploration that searches unobserved data streams for possible mean shifts and exploitation that focuses on highly suspicious data streams for quick shift detection. Both simulations and a case study are comprehensively conducted to evaluate the performance and demonstrate the superiority of the proposed method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the critical challenge of online process monitoring by considering the cost-effectiveness and resource constraints in practice (e.g., limited number of sensors, limited transmission bandwidth or energy constraint, and limited processing time). Unlike the existing methodologies which rely on the restrictive assumptions (e.g., normally distributed, exchangeable data streams) or require historical full observations of all data streams to be available offline for training, this paper proposes a novel monitoring and sampling strategy that allows the practitioners to cost-effectively monitor high-dimensional heterogeneous data streams that contain distinct physical characteristics and follow different distributions. To implement the proposed methodology, it is necessary: (i) to identify sample quantiles for each data stream based on historical in-control data offline; (ii) to determine which data streams to observe at each acquisition time based on the resource constraints; and (iii) to automatically screen out the suspicious data streams to form the global monitoring statistic. Experimental results through simulations and a case study have shown that the proposed method has much better performance than the existing methods in reducing detection delay and effectively dealing with heterogeneous data streams.

Learn-to-adapt: Concept drift adaptation for hybrid multiple streams

Existing concept drift adaptation (CDA) methods aim to continually update outdated classifiers in a single-labeled stream scenario. However, real-world data streams are massive, with hybrids of labeled and unlabeled streams. In this paper, we discuss CDA in multiple data streams that may contain unlabeled drifting streams. To address this realistic and complex problem, we rethink the concept drift problem by adopting a meta-learning approach and introduce a Learn-to-Adapt framework (L2A). The L2A framework simultaneously 1) makes adaptations for drifting labeled streams, and 2) leverages knowledge from labeled drifting streams to make adaptations for unlabeled stream prediction. In L2A, a meta-representor with an adapter in the meta-training stage is designed to learn the invariant representations for drifting streams, enabling the model to quickly produce a good generalization of new concepts with limited training samples. In the online stage, the meta-representor will be adapted continually under the control of the adapter and will contribute to adapting the classifiers for unlabeled drifting stream prediction. Compared to existing CDA methods which mostly only adapt the classifiers, L2A adapts the feature extractor and classifier in a feedback process, which is advanced in dealing with more complex and high-dimensional data streams.

Bandit Change-Point Detection for Real-Time Monitoring High-Dimensional Data Under Sampling Control

In many real-world problems of real-time monitoring high-dimensional streaming data, one wants to detect an undesired event or change quickly once it occurs, but under the sampling control constraint in the sense that one might be able to only observe or use selected components data for decision-making per time step in the resource-constrained environments. In this article, we propose to incorporate multi-armed bandit approaches into sequential change-point detection to develop an efficient bandit change-point detection algorithm based on the limiting Bayesian approach to incorporate a prior knowledge of potential changes. Our proposed algorithm, termed Thompson-Sampling-Shiryaev-Roberts-Pollak (TSSRP), consists of two policies per time step: the adaptive sampling policy applies the Thompson Sampling algorithm to balance between exploration for acquiring long-term knowledge and exploitation for immediate reward gain, and the statistical decision policy fuses the local Shiryaev–Roberts–Pollak statistics to determine whether to raise a global alarm by sum shrinkage techniques. Extensive numerical simulations and case studies demonstrate the statistical and computational efficiency of our proposed TSSRP algorithm.

Online Structural Change-Point Detection of High-dimensional Streaming Data via Dynamic Sparse Subspace Learning

igh-dimensional streaming data are becoming increasingly ubiquitous in many fields. They often lie in multiple low-dimensional subspaces, and the manifold structures may change abruptly on the time scale due to pattern shift or occurrence of anomalies. However, the problem of detecting the structural changes in a real-time manner has not been well studied. To fill this gap, we propose a dynamic sparse subspace learning approach for online structural change-point detection of high-dimensional streaming data. A novel multiple structural change-point model is proposed and the asymptotic properties of the estimators are investigated. A tuning method based on Bayesian information criterion and change-point detection accuracy is proposed for penalty coefficients selection. An efficient Pruned Exact Linear Time based algorithm is proposed for online optimization and change-point detection. The effectiveness of the proposed method is demonstrated through several simulation studies and a real case study on gesture data for motion tracking. Supplementary materials for this article are available online.

Online Privacy-Preserving Data-Driven Network Anomaly Detection

We study online privacy-preserving anomaly detection in a setting in which the data are distributed over a network and locally sensitive to each node, and a probabilistic data model is unknown. We design and analyze a data-driven solution scheme where each node observes a high-dimensional data stream for which it computes a local outlierness score. This score is then perturbed, encrypted, and sent to a network operator. The network operator then decrypts an aggregate statistic over the network and performs online network anomaly detection via the proposed generalized cumulative sum (CUSUM) algorithm. We derive an asymptotic lower bound and an asymptotic approximation for the average false alarm period of the proposed algorithm. Additionally, we derive an asymptotic upper bound and asymptotic approximation for the average detection delay of the proposed algorithm under a certain anomaly. We show the analytical tradeoff between the anomaly detection performance and the differential privacy level, controlled via the local perturbation noise. Experiments illustrate that the proposed algorithm offers a good tradeoff between privacy and quick anomaly detection against the UDP flooding and spam attacks in a real Internet of Things (IoT) network.

Discriminative Similarity-Balanced Online Hashing for Supervised Image Retrieval

When virtualizing large-scale images of the real world, online hashing provides an efficient scheme for fast retrieval and compact storage. It converts high-dimensional streaming data into compact binary hash codes while saving the structural characteristics between samples into the Hamming space. Existing works usually update the hashing function based on the similarity between input data, or design a codebook to assign code words for each single input sample. However, assigning code words to multiple samples while retaining the balanced similarity of the image instances is still challenging. To address this issue, we propose a novel discriminative similarity-balanced online hashing (DSBOH) framework in this work. In particular, we first obtain the Hadamard codebook that guides the generation of discriminative binary codes according to label information. Then, we maintain the correlation between the new data and the previously arrived data by the balanced similarity matrix, which is also generated by semantic information. Finally, we joined the Hadamard codebook and the balanced similarity matrix into a unified hashing function to simultaneously maintain discrimination and balanced similarity. The proposed method is optimized by an alternating optimization technique. Extensive experiments on the CIFAR-10, MNIST, and Places205 datasets demonstrate that our proposed DSBOH performs better than several state-of-the-art online hashing methods in terms of effectiveness and efficiency.

Detection and Identification of Cyber and Physical Attacks on Distribution Power Grids with PVs: An Online High-Dimensional Data-driven Approach.

Cyber and physical attacks threaten the security of distribution power grids. The emerging renewable energy sources such as photovoltaics (PVs) introduce new potential vulnerabilities. Based on the electric waveform data measured by waveform sensors in the distribution power networks, in this paper, we propose a novel high-dimensional data-driven cyber physical attack detection and identification approach (HCADI). Firstly, we analyze the cyber and physical attack impacts (including cyber attacks on the solar inverter causing unusual harmonics) on electric waveforms in distribution power grids. Then, we construct a high dimensional streaming data feature matrix based on signal analysis of multiple sensors in the network. Next, we propose a novel mechanism including leverage score based attack detection and binary matrix factorization based attack diagnosis. By leveraging the data structure and binary coding, our HCADI approach does not need the training stage for both detection and the root cause diagnosis, which is needed for machine learning/deep learning-based methods. To the best of our knowledge, it is the first attempt to use raw electrical waveform data to detect and identify the power electronics cyber/physical attacks in distribution power grids with PVs.

IPMOD: An efficient outlier detection model for high-dimensional medical data streams

Outlier detection in high-dimensional medical data streams in real-time is critical and challenging research, which is of great help to disease prevention and source analysis. Although academia has done a lot of research on outlier detection of time series data streams, these methods have the following two shortcomings: (1) Insufficient detection accuracy on high-dimensional data streams; (2) Insufficient accuracy in dynamic data streams scenarios low. To this end, we propose a sliding window model based on efficient pruning and information entropy, namely IPMOD(Information Entropy-Pruning Multi-dimensional Outlier Detection). In IPMOD, we first designed a new index weight measurement method combined with information entropy to quantify the weight of different indexes in multi-dimensional data, to determine the influence of different attributes on the prediction results. Then we designed a new sliding window and sub-sequence measurement mechanism to judge whether the data is abnormal based on the distance between the target sequence and the non-self-match. After that, we designed a pruning strategy to further reduce the computational complexity of the algorithm. The final comprehensive experiment shows that our proposed scheme not only has higher detection accuracy than many current schemes on multiple sets of real data-sets but also can quickly detect outliers in different medical data streams in real-time.