On-line Principal Component Analysis Research Articles

ROIPCA: an online memory-restricted PCA algorithm based on rank-one updates

Abstract Principal components analysis (PCA) is a fundamental algorithm in data analysis. Its memory-restricted online versions are useful in many modern applications, where the data are too large to fit in memory, or when data arrive as a stream of items. In this paper, we propose ROIPCA and fROIPCA, two online PCA algorithms that are based on rank-one updates. While ROIPCA is typically more accurate, fROIPCA is faster and has comparable accuracy. We show the relation between fROIPCA and an existing popular gradient algorithm for online PCA, and in particular, prove that fROIPCA is in fact a gradient algorithm with an optimal learning rate. We demonstrate numerically the advantages of our algorithms over existing state-of-the-art algorithms in terms of accuracy and runtime.

Stochastic Gauss–Newton Algorithms for Online PCA

Stochastic Gauss–Newton Algorithms for Online PCA

On the optimality of the Oja’s algorithm for online PCA

In this paper we analyze the behavior of the Oja’s algorithm for online/streaming principal component subspace estimation. It is proved that with high probability it performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution. Moreover, it is the first time to show that the convergence rate, namely the upper bound of the approximation, exactly matches the lower bound of an approximation obtained by the offline/classical PCA up to a constant factor.

Online sparse sliced inverse regression for high-dimensional streaming data

Due to the demand for tackling the problem of streaming data with high-dimensional covarites, we propose an online sparse sliced inverse regression (OSSIR) method for online sufficient dimension reduction (SDR). The existing online SDR methods focus on the case when [Formula: see text] (dimension of covariates) is small. In this paper, we adapt the sparse sliced inverse regression to cope with high-dimensional streaming data where the dimension [Formula: see text] is large. There are two important steps in our method, one is to extend the online principal component analysis to iteratively obtain the eigenvalues and eigenvectors of the kernel matrix, the other is to use the truncated gradient to perform online [Formula: see text] regularization. Theoretical properties of the proposed online learner are established. By comparing with several existing methods in simulations and real data applications, we demonstrate the effectiveness and efficiency of our method.

Minimum cost‐compression risk in principal component analysis

SummaryPrincipal Component Analysis (PCA) is a popular multivariate analytic tool which can be used for dimension reduction without losing much information. Data vectors containing a large number of features arriving sequentially may be correlated with each other. An effective algorithm for such situations is online PCA. Existing Online PCA research works revolve around proposing efficient scalable updating algorithms focusing on compression loss only. They do not take into account the size of the dataset at which further arrival of data vectors can be terminated and dimension reduction can be applied. It is well known that the dataset size contributes to reducing the compression loss – the smaller the dataset size, the larger the compression loss while larger the dataset size, the lesser the compression loss. However, the reduction in compression loss by increasing dataset size will increase the total data collection cost. In this paper, we move beyond the scalability and updation problems related to Online PCA and focus on optimising a cost‐compression loss which considers the compression loss and data collection cost. We minimise the corresponding risk using a two‐stage PCA algorithm. The resulting two‐stage algorithm is a fast and an efficient alternative to Online PCA and is shown to exhibit attractive convergence properties with no assumption on specific data distributions. Experimental studies demonstrate similar results and further illustrations are provided using real data. As an extension, a multi‐stage PCA algorithm is discussed as well. Given the time complexity, the two‐stage PCA algorithm is emphasised over the multi‐stage PCA algorithm for online data.

Nearly optimal stochastic approximation for online principal subspace estimation

Principal component analysis (PCA) has been widely used in analyzing high-dimensional data. It converts a set of observed data points of possibly correlated variables into a set of linearly uncorrelated variables via an orthogonal transformation. To handle streaming data and reduce the complexities of PCA, (subspace) online PCA iterations were proposed to iteratively update the orthogonal transformation by taking one observed data point at a time. Existing works on the convergence of (subspace) online PCA iterations mostly focus on the case where the samples are almost surely uniformly bounded. In this paper, we analyze the convergence of a subspace online PCA iteration under more practical assumption and obtain a nearly optimal finite-sample error bound. Our convergence rate almost matches the minimax information lower bound. We prove that the convergence is nearly global in the sense that the subspace online PCA iteration is convergent with high probability for random initial guesses. This work also leads to a simpler proof of the recent work on analyzing online PCA for the first principal component only.

Process monitoring of abnormal working conditions in the zinc roasting process with an ALD-based LOF-PCA method

Timely and accurate detection of abnormal working conditions can ensure stability, improve production efficiency and reduce pollution of an industrial process. However, the production data of an industrial process has non-Gaussian and time-varying characteristics due to the diverse feed composition and complex reaction mechanisms. To address the above issue, an improved online principal component analysis (PCA) algorithm based on the selective model update is proposed in this study. First, considering the non-Gaussian nature of the process data, a local outlier factor-based (LOF) abnormality detection logic is used to replace the T2 and squared prediction error (SPE) statistics in traditional PCA algorithms. Then, to adapt to the time-varying characteristics of the process data, an approximate linear dependence (ALD) algorithm is used to evaluate the independent degree between the new sample and training samples. Only those samples containing new information are used to update the monitoring model, which can improve model performance and reduce the frequency of online updates. The zinc roasting process (ZRP) is used as an example to illustrate the proposed approach. Industrial data collected from a ZRP is used to demonstrate the performance of the ALD-based LOF-PCA method in the early detection of two typical abnormal working conditions in the ZRP.

A hybrid partial least squares regression-based real time pore pressure estimation method for complex geological drilling process

Accurate real-time estimation of pore pressure is essential for the geomechanical analysis of wellbore stability. Conventional empirical methods may find it difficult to capture pore pressure trends, especially in the complex geological environments. In this study, a data-driven pore pressure estimation method is developed on the basis of hybrid partial least squares regression. This method, which combines empirical methods, comprised three stages: data preprocessing, depth series segmentation, and model establishment and switching. First, concerning the existence of outliers and noises, an outlier detection and wavelet filtering algorithm are introduced to obtain reliable model parameters. Additionally, Pearson correlation-analysis is employed to determine strongly correlated attributes with pore pressure in the data preprocessing stage. Afterward, an online principal component analysis similarity method is proposed for depth series segmentation, considering the varying drilling depth. Finally, a real-time data-driven pore pressure estimation model that integrates conventional empirical methods is established on the basis of partial least squares regression, and a model switching strategy is further developed and will be activated when performance deteriorates. The proposed method can be applied to a wide range of formations, and a real case study is conducted using actual data from a drilling site in Utah. The mean absolute error and root mean square error of the proposed method achieve 0.5128 and 0.8056 in the online condition, and achieved 1.4592 and 2.0100 in the offline condition, which are at least 45% less than those of other nine well-known methods. The results indicate the superior performance of our method on this well.

Online Stochastic Principal Component Analysis

This paper studied Principal Component Analysis (PCA) in an online. The problem is posed as a subspace optimization problem and solved using gradient based algorithms. One such algorithm is the Variance-Reduced PCA (VR-PCA). The VR-PCA was designed as an improvement to the classical online PCA algorithm known as the Oja’s method where it only handled one sample at a time. The paper developed Block VR-PCA as an improved version of VR-PCA. Unlike prominent VR-PCA, the Block VR-PCA was designed to handle more than one dimension in subspace optimization at a time and it showed good performance. The Block VR-PCA and Block Oja method were compared experimentally in MATLAB using synthetic and real data sets, their convergence results showed Block VR-PCA method appeared to achieve a minimum steady state error than Block Oja method.
 Keywords: Online Stochastic; Principal Component Analysis; Block Variance-Reduced; Block Oja

Dimensionality reduction in the context of dynamic social media data streams

In recent years social media became an important part of everyday life for many people. A big challenge of social media is, to find posts, that are interesting for the user. Many social networks like Twitter handle this problem with so-called hashtags. A user can label his own Tweet (post) with a hashtag, while other users can search for posts containing a specified hashtag. But what about finding posts which are not labeled by the creator? We provide a way of completing hashtags for unlabeled posts using classification on a novel real-world Twitter data stream. New posts will be created every second, thus this context fits perfectly for non-stationary data analysis. Our goal is to show, how labels (hashtags) of social media posts can be predicted by stream classifiers. In particular, we employ random projection (RP) as a preprocessing step in calculating streaming models. Also, we provide a novel real-world data set for streaming analysis called NSDQ with a comprehensive data description. We show that this dataset is a real challenge for state-of-the-art stream classifiers. While RP has been widely used and evaluated in stationary data analysis scenarios, non-stationary environments are not well analyzed. In this paper, we provide a use case of RP on real-world streaming data, especially on NSDQ dataset. We discuss why RP can be used in this scenario and how it can handle stream-specific situations like concept drift. We also provide experiments with RP on streaming data, using state-of-the-art stream classifiers like adaptive random forest and concept drift detectors. Additionally, we experimentally evaluate an online principal component analysis (PCA) approach in the same fashion as we do for RP. To obtain higher dimensional synthetic streams, we use random Fourier features (RFF) in an online manner which allows us, to increase the number of dimensions of low dimensional streams.

Robust Subspace Tracking Algorithms in Signal Processing: A Brief Survey

Principal component analysis (PCA) and subspace estimation (SE) are popular data analysis tools and used in a wide range of applications. The main interest in PCA/SE is for dimensionality reduction and low-rank approximation purposes. The emergence of big data streams have led to several essential issues for performing PCA/SE. Among them are (i) the size of such data streams increases over time, (ii) the underlying models may be time-dependent, and (iii) problem of dealing with the uncertainty and incompleteness in data. A robust variant of PCA/SE for such data streams, namely robust online PCA or robust subspace tracking (RST), has been introduced as a good alternative. The main goal of this paper is to provide a brief survey on recent RST algorithms in signal processing. Particularly, we begin this survey by introducing the basic ideas of the RST problem. Then, different aspects of RST are reviewed with respect to different kinds of non-Gaussian noises and sparse constraints. Our own contributions on this topic are also highlighted.

Adaptive dimensionality reduction for neural network-based online principal component analysis.

“Principal Component Analysis” (PCA) is an established linear technique for dimensionality reduction. It performs an orthonormal transformation to replace possibly correlated variables with a smaller set of linearly independent variables, the so-called principal components, which capture a large portion of the data variance. The problem of finding the optimal number of principal components has been widely studied for offline PCA. However, when working with streaming data, the optimal number changes continuously. This requires to update both the principal components and the dimensionality in every timestep. While the continuous update of the principal components is widely studied, the available algorithms for dimensionality adjustment are limited to an increment of one in neural network-based and incremental PCA. Therefore, existing approaches cannot account for abrupt changes in the presented data. The contribution of this work is to enable in neural network-based PCA the continuous dimensionality adjustment by an arbitrary number without the necessity to learn all principal components. A novel algorithm is presented that utilizes several PCA characteristics to adaptivly update the optimal number of principal components for neural network-based PCA. A precise estimation of the required dimensionality reduces the computational effort while ensuring that the desired amount of variance is kept. The computational complexity of the proposed algorithm is investigated and it is benchmarked in an experimental study against other neural network-based and incremental PCA approaches where it produces highly competitive results.

Widening the scope of an eigenvector stochastic approximation process and application to streaming PCA and related methods

We prove the almost sure convergence of Oja-type processes to eigenvectors of the expectation B of a random matrix while relaxing the i.i.d.assumption on the observed random matrices (Bn) and assuming either (Bn) converges to B or (E[Bn|Tn]) converges to B where Tn is the sigma-field generated by the events before time n. As an application of this generalization, the online PCA of a random vector Z can be performed when there is a data stream of i.i.d.observations of Z, even when both the metric M used and the expectation of Z are unknown and estimated online. Moreover, in order to update the stochastic approximation process at each step, we are no longer bound to using only a mini-batch of observations of Z, but all previous observations up to the current step can be used without having to store them. This is useful not only when dealing with streaming data but also with Big Data as one can process the latter sequentially as a data stream. In addition the general framework of this process, unlike other algorithms in the literature, also covers the case of factorial methods related to PCA.

Neurons as Canonical Correlation Analyzers.

Normative models of neural computation offer simplified yet lucid mathematical descriptions of murky biological phenomena. Previously, online Principal Component Analysis (PCA) was used to model a network of single-compartment neurons accounting for weighted summation of upstream neural activity in the soma and Hebbian/anti-Hebbian synaptic learning rules. However, synaptic plasticity in biological neurons often depends on the integration of synaptic currents over a dendritic compartment rather than total current in the soma. Motivated by this observation, we model a pyramidal neuronal network using online Canonical Correlation Analysis (CCA). Given two related datasets represented by distal and proximal dendritic inputs, CCA projects them onto the subspace which maximizes the correlation between their projections. First, adopting a normative approach and starting from a single-channel CCA objective function, we derive an online gradient-based optimization algorithm whose steps can be interpreted as the operation of a pyramidal neuron. To model networks of pyramidal neurons, we introduce a novel multi-channel CCA objective function, and derive from it an online gradient-based optimization algorithm whose steps can be interpreted as the operation of a pyramidal neuron network including its architecture, dynamics, and synaptic learning rules. Next, we model a neuron with more than two dendritic compartments by deriving its operation from a known objective function for multi-view CCA. Finally, we confirm the functionality of our networks via numerical simulations. Overall, our work presents a simplified but informative abstraction of learning in a pyramidal neuron network, and demonstrates how such networks can integrate multiple sources of inputs.

Convergence analysis of Oja’s iteration for solving online PCA with nonzero-mean samples

Principal component analysis (PCA) is one of the most popular multivariate data analysis techniques for dimension reduction and data mining, and is widely used in many fields ranging from industry and biology to finance and social development. When working on big data, it is of great necessity to consider the online version of PCA, in which only a small subset of samples could be stored. To handle the online PCA problem, Oja (1982) presented the stochastic power method under the assumption of zero-mean samples, and there have been lots of theoretical analysis and modified versions of this method in recent years. However, a common circumstance where the samples have nonzero mean is seldom studied. In this paper, we derive the convergence rate of a nonzero-mean version of Oja’s algorithm with diminishing stepsizes. In the analysis, we succeed in handling the dependency between each iteration, which is caused by the updated mean term for data centering. Furthermore, we verify the theoretical results by several numerical tests on both artificial and real datasets. Our work offers a way to deal with the top-1 online PCA when the mean of the given data is unknown.

Online eigenvector transformation reflecting concept drift for improving network intrusion detection

AbstractCurrently, large data streams are constantly being generated in diverse environments, and continuous storage of the data and periodic batch‐type principal component analysis (PCA) are becoming increasingly difficult. Various online PCA algorithms have been proposed to solve this problem. In this study, we propose an online PCA methodology based on online eigenvector transformation with the moving average of the data stream that can reflect concept drift. We compared the network intrusion detection performance based on online transformation of eigenvectors with that of offline methods by applying three machine learning algorithms. Both online and offline methods demonstrated excellent performance in terms of precision. However, in terms of the recall ratio, the performance of the proposed methodology with integrated online eigenvector transformation was better; thus, the F1‐measure also indicated better performance. The visualization of the principal component score shows the effectiveness of our method.

ADAPTIVE HUMAN MACHINE INTERACTION APPROACH FOR FEATURE SELECTION-EXTRACTION TASK IN MEDICAL DATA MINING

Feature Selection task is one of the most complicated and actual in the areas of Data Mining and Human Machine Interaction. Many approaches to its solving are based on non-mathematical and presentative hypothesis. New approach to evaluation of medical features information quantity, based on optimized combination of feature selection and feature extraction methods is proposed. This approach allows us to produce optimal reduced number of features with linguistic interpreting of each of them. Hybrid system of feature selection/extraction based on Neural Network-Physician interaction is investigated. This system is numerically simple, can produce feature selection/extraction with any number of factors in online mode using neural network-physician interaction based on Oja’s neurons for online principal component analysis and calculating distance between first principal component and all input features. A series of experiments confirms efficiency of proposed approaches in Medical Data Mining area and allows physicians to have the most informative features without losing their linguistic interpreting.

Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations

Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations

Near-optimal stochastic approximation for online principal component estimation

Principal component analysis (PCA) has been a prominent tool for high-dimensional data analysis. Online algorithms that estimate the principal component by processing streaming data are of tremendous practical and theoretical interests. Despite its rich applications, theoretical convergence analysis remains largely open. In this paper, we cast online PCA into a stochastic nonconvex optimization problem, and we analyze the online PCA algorithm as a stochastic approximation iteration. The stochastic approximation iteration processes data points incrementally and maintains a running estimate of the principal component. We prove for the first time a nearly optimal finite-sample error bound for the online PCA algorithm. Under the subgaussian assumption, we show that the finite-sample error bound closely matches the minimax information lower bound.

Online Principal Component Analysis in High Dimension: Which Algorithm to Choose?

SummaryPrincipal component analysis (PCA) is a method of choice for dimension reduction. In the current context of data explosion, online techniques that do not require storing all data in memory are indispensable to perform the PCA of streaming data and/or massive data. Despite the wide availability of recursive algorithms that can efficiently update the PCA when new data are observed, the literature offers little guidance on how to select a suitable algorithm for a given application. This paper reviews the main approaches to online PCA, namely, perturbation techniques, incremental methods and stochastic optimisation, and compares the most widely employed techniques in terms statistical accuracy, computation time and memory requirements using artificial and real data. Extensions of online PCA to missing data and to functional data are detailed. All studied algorithms are available in the package onlinePCA on CRAN.