Principal Subspace Research Articles

Self-consistency and a generalized principal subspace theorem

Principal subspace theorems deal with the problem of finding subspaces supporting optimal approximations of multivariate distributions. The optimality criterion considered in this paper is the minimization of the mean squared distance between the given distribution and an approximating distribution, subject to some constraints. Statistical applications include, but are not limited to, cluster analysis, principal components analysis and projection pursuit. Most principal subspace theorems deal with elliptical distributions or with mixtures of spherical distributions. We generalize these results using the notion of self-consistency. We also show their connections with the skew-normal distribution and projection pursuit techniques. We also discuss their implications, with special focus on principal points and self-consistent points. Finally, we access the practical relevance of the theoretical results by means of several simulation studies.

Modeling and Optimization for Big Data Analytics: (Statistical) learning tools for our era of data deluge

With pervasive sensors continuously collecting and storing massive amounts of information, there is no doubt this is an era of data deluge. Learning from these large volumes of data is expected to bring significant science and engineering advances along with improvements in quality of life. However, with such a big blessing come big challenges. Running analytics on voluminous data sets by central processors and storage units seems infeasible, and with the advent of streaming data sources, learning must often be performed in real time, typically without a chance to revisit past entries. Workhorse signal processing (SP) and statistical learning tools have to be re-examined in todays high-dimensional data regimes. This article contributes to the ongoing cross-disciplinary efforts in data science by putting forth encompassing models capturing a wide range of SP-relevant data analytic tasks, such as principal component analysis (PCA), dictionary learning (DL), compressive sampling (CS), and subspace clustering. It offers scalable architectures and optimization algorithms for decentralized and online learning problems, while revealing fundamental insights into the various analytic and implementation tradeoffs involved. Extensions of the encompassing models to timely data-sketching, tensor- and kernel-based learning tasks are also provided. Finally, the close connections of the presented framework with several big data tasks, such as network visualization, decentralized and dynamic estimation, prediction, and imputation of network link load traffic, as well as imputation in tensor-based medical imaging are highlighted.

An Effective Neural Learning Algorithm for Extracting Cross-Correlation Feature Between Two High-Dimensional Data Streams

A novel information criterion for principal singular subspace tracking is proposed and a corresponding principal singular subspace gradient flow is derived based on the information criterion in this paper. The information criterion exhibits a unique global minimum attained if and only if the state matrices of the left and right neural networks span the left and right principal singular subspace of a cross-correlation matrix between two high-dimensional vector sequences, respectively. The proposed gradient flow can efficiently track an orthonormal basis of the principal singular subspace, and it has fast convergence speed, good suitability for data matrix close to singular matrix and excellent self-stabilizing property. The global asymptotic stability and self-stabilizing property of the proposed algorithm are analyzed. The simulation experiments validate the excellent performance of the proposed algorithm.

Vertex-algebraic structure of principal subspaces of standard $A^{(2)}_2$-modules, I

Extending earlier work of the authors, this is the first in a series of papers devoted to the vertex-algebraic structure of principal subspaces of standard modules for twisted affine Kac–Moody algebras. In this part, we develop the necessary theory of principal subspaces for the affine Lie algebra [Formula: see text], which we expect can be extended to higher rank algebras. As a "test case", we consider the principal subspace of the basic [Formula: see text]-module and explore its structure in depth.

Impact Damage Detection and Identification Using Eddy Current Pulsed Thermography Through Integration of PCA and ICA

Eddy current pulsed thermography (ECPT) is implemented for detection and separation of impact damage and resulting damages in carbon fiber reinforced plastic (CFRP) samples. Complexity and nonhomogeneity of fiber texture as well as multiple defects limit detection identification and characterization from transient images of the ECPT. In this paper, an integration of principal component analysis (PCA) and independent component analysis (ICA) on transient thermal videos has been proposed. This method enables spatial and temporal patterns to be extracted according to the transient response behavior without any training knowledge. In the first step, using the PCA, the data is transformed to orthogonal principal component subspace and the dimension is reduced. Multichannel morphological component analysis, as an ICA method, is then implemented to deal with the sparse and independence property for detecting and separating the influences of different layers, defects, and their combination information in the CFRP. Because different transient behaviors exist, multiple types of defects can be identified and separated by calculating the cross-correlation of the estimated mixing vectors between impact the ECPT sequences and nondefect ECPT sequences.

Optimal Estimation and Rank Detection for Sparse Spiked Covariance Matrices.

This paper considers a sparse spiked covariancematrix model in the high-dimensional setting and studies the minimax estimation of the covariance matrix and the principal subspace as well as the minimax rank detection. The optimal rate of convergence for estimating the spiked covariance matrix under the spectral norm is established, which requires significantly different techniques from those for estimating other structured covariance matrices such as bandable or sparse covariance matrices. We also establish the minimax rate under the spectral norm for estimating the principal subspace, the primary object of interest in principal component analysis. In addition, the optimal rate for the rank detection boundary is obtained. This result also resolves the gap in a recent paper by Berthet and Rigollet [2] where the special case of rank one is considered.

Principal subspaces for quantum affine algebra [formula omitted

We consider principal subspace W(Λ) of integrable highest weight module L(Λ) for quantum affine algebra Uq(slˆn+1). We introduce quantum analogues of the quasi-particles associated with the principal subspaces for slˆn+1 and discover certain relations among them. By using these relations we find, for certain highest weight Λ, combinatorial bases of principal subspace W(Λ) in terms of monomials of quantum quasi-particles.

Quality‐relevant fault diagnosis with concurrent phase partition and analysis of relative changes for multiphase batch processes

Multiplicity of phases as indicated by changes of process characteristics is an inherent nature of many batch processes for both normal and fault cases. To more efficiently perform online fault diagnosis via reconstruction for multiphase batch processes, the phase nature and the relationship between normal and fault cases within each phase should be deeply addressed. This article proposes a quality‐relevant fault diagnosis strategy with concurrent phase partition and analysis of relative changes for multiphase batch processes. First, a concurrent phase partition algorithm is developed. The basic idea is to track the changes of process characteristics at normal and fault statuses jointly so that multiple sequential modeling phases are identified simultaneously for both normal and fault cases. Then, the relative changes from the normal status to each fault case are analyzed in each phase to reveal the specific fault effects more efficiently. The fault effects are decomposed in two different monitoring subspaces, principal subspace, and residual subspace, by capturing their different roles in removing out‐of‐control signals. The significant increases relative to the normal case are judged to be responsible for the concerned alarm monitoring statistics in each phase. The others are composed of general variations that are deemed to still follow normal rules and thus insignificant to remove alarm monitoring statistics. Those alarm‐responsible fault deviations are then used to develop reconstruction models which can more efficiently recover the fault‐free part for online fault diagnosis. The proposed algorithm is illustrated with a typical multiphase batch process with one normal case and three fault cases. © 2014 American Institute of Chemical Engineers AIChE J, 60: 2048–2062, 2014

Dual subspace learning via geodesic search on Stiefel manifold

Oja’s principal subspace algorithm is a well-known and powerful technique for learning and tracking principal information of time series. However, Oja’s algorithm is divergent when performing the task of minor subspace analysis. In the present paper, we transform Oja’s algorithm into a dual learning algorithm in the sense of fulfilling principal subspace analysis as well as minor subspace analysis via geodesic search on Stiefel manifold. Also inherent stability is guaranteed for the proposed geodesic based algorithm due to the fact the weight matrix rigourously evolves on the compact Stiefel manifold. The effectiveness of the proposed algorithm is further verified in the section of numerical simulation.

Sparse PCA: Optimal rates and adaptive estimation

Principal component analysis (PCA) is one of the most commonly used statistical procedures with a wide range of applications. This paper considers both minimax and adaptive estimation of the principal subspace in the high dimensional setting. Under mild technical conditions, we first establish the optimal rates of convergence for estimating the principal subspace which are sharp with respect to all the parameters, thus providing a complete characterization of the difficulty of the estimation problem in term of the convergence rate. The lower bound is obtained by calculating the local metric entropy and an application of Fano’s lemma. The rate optimal estimator is constructed using aggregation, which, however, might not be computationally feasible. We then introduce an adaptive procedure for estimating the principal subspace which is fully data driven and can be computed efficiently. It is shown that the estimator attains the optimal rates of convergence simultaneously over a large collection of the parameter spaces. A key idea in our construction is a reduction scheme which reduces the sparse PCA problem to a high-dimensional multivariate regression problem. This method is potentially also useful for other related problems.

A Compressed PCA Subspace Method for Anomaly Detection in High-Dimensional Data

Random projection is widely used as a method of dimension reduction. In recent years, its combination with standard techniques of regression and classification has been explored. Here we examine its use with principal component analysis (PCA) and subspace detection methods. Specifically, we show that, under appropriate conditions, with high probability the magnitude of the residuals of a PCA analysis of randomly projected data behaves comparably to that of the residuals of a similar PCA analysis of the original data. Our results indicate the feasibility of applying subspace-based anomaly detection algorithms to randomly projected data, when the data are high-dimensional but have a covariance of an appropriately compressed nature. We illustrate in the context of computer network traffic anomaly detection.

Subspace Decomposition-Based Reconstruction Modeling for Fault Diagnosis in Multiphase Batch Processes

In the present work, a fault diagnosis strategy is developed for multiphase batch processes based on comprehensive subspace decomposition and reconstruction modeling method. Phase nature of fault processes is analyzed and a principal component of fault deviations (PCFD) modeling algorithm is proposed in each phase to characterize the fault effects responsible to out-of-control monitoring statistics in two different monitoring subspaces, principal component subspace (PCS), and residual subspace (RS), respectively. Phase-based reconstruction models are then developed in each monitoring subspace based on those significant fault deviations, which can better explain fault characteristics. The action of fault diagnosis is then taken by finding the specific reconstruction model that can well eliminate alarming signals. Critical-to-diagnosis phases are identified to provide more-reliable fault isolation results. The proposed in-line fault diagnosis method can give an enhanced understanding of different fault characteristics across different phases and it is easier to identify the fault root cause in each local phase. Its performance on fault diagnosis is illustrated using data from a multiphase batch process: injection molding.

Subspace decomposition approach of fault deviations and its application to fault reconstruction

In the present work, a new subspace decomposition approach of fault deviations is developed in the context of principal component analysis (PCA) based monitoring system for fault diagnosis via reconstruction. The fault effects are decomposed in different monitoring subspaces, principal subspace (PCS) and residual subspace (RS), and the significant fault deviations that are responsible for the concerned alarming monitoring statistic are calculated. This is achieved by designing a two-step feature decomposition procedure in each monitoring subspace. In the first step, the relative fault deviations are sorted by comparing the fault variations with the normal variations. All possible fault deviations that may contribute to the out-of-control monitoring statistics are collected. In the second step, PCA is performed on the chosen fault information where the largest fault deviation directions are decomposed in order. By the two-step decomposition, in each monitoring subspace, two different parts are separated for the purpose of fault reconstruction. One is composed of the concerned fault deviations that contribute to alarming monitoring statistics which are thus significant to remove the out-of-control signals. The other is composed of general variations that are deemed to follow normal rules and thus insignificant to remove alarming monitoring statistics. Theoretical support is framed and the related statistical characteristics are analyzed. Its feasibility and performance are illustrated with data from the three-tank system and the Tennessee Eastman (TE) benchmark process.

Incipient fault detection and diagnosis based on Kullback–Leibler divergence using Principal Component Analysis: Part I

Detection of faults under the Principal Component Analysis (PCA) framework can be made into either the principal or the residual subspace. Because of the large amount of variabilities naturally present in the principal subspace, there are usually ambiguities to detect small variations caused by incipient faults with the use of the first principal components. Distance-based detection and diagnosis methodology is usually used and the Hotelling's T2 is the most common statistical distance defined in the principal subspace. However, because the T2 often fails in detecting small shifts, the residual subspace has become the privileged space for fault detection with the SPE criterion. Therefore, there is a challenge to detect incipient faults within the principal subspace. We propose a fault detection approach based on a probability distribution measure. Residuals are generated by comparing the probability density of each of the latent scores to a reference one, using the Kullback–Leibler Divergence. From simulations it is shown that the proposed criterion successfully detects incipient faults which are undetectable by the distance discriminants. Also, it allows to isolate the fault and gives insights to the severity level of the detected abnormality thanks to its global character. A theoretical analysis is conducted to support the approach and the simulation results.

Sparse principal component analysis and iterative thresholding

Principal component analysis (PCA) is a classical dimension reduction method which projects data onto the principal subspace spanned by the leading eigenvectors of the covariance matrix. However, it behaves poorly when the number of features p is comparable to, or even much larger than, the sample size n. In this paper, we propose a new iterative thresholding approach for estimating principal subspaces in the setting where the leading eigenvectors are sparse. Under a spiked covariance model, we find that the new approach recovers the principal subspace and leading eigenvectors consistently, and even optimally, in a range of high-dimensional sparse settings. Simulated examples also demonstrate its competitive performance.

Differentiation of big-six non-O157 Shiga-toxin producing Escherichia coli (STEC) on spread plates of mixed cultures using hyperspectral imaging

There have been considerable recent advances in the technology for rapidly detecting foodborne pathogens. However, a traditional culture method is still the “gold standard” for presumptive-positive pathogen screening although it is labor-intensive, ineffective in testing large amount of food samples, and cannot completely prevent unwanted background microflora from growing together with target microorganisms on agar media. We have developed multivariate classification models based on visible and near-infrared hyperspectral imaging for rapid presumptive-positive screening of six representative non-O157 Shiga-toxin producing Escherichia coli (STEC) serogroups (O26, O45, O103, O111, O121, and O145) on agar plates of pure and mixed cultures. The classification models were developed with spread plates of pure cultures. In this study, we evaluated the performance of the classification models with independent validation samples of mixed cultures that were not used during training and found the best classification model for differentiating non-O157 STEC colonies on spread plates of mixed cultures. A validation protocol appropriate to hyperspectral imaging of mixed cultures was developed. An additional independent validation set of 12 spread plates with pure cultures was used as positive controls to help the validation process with the mixed cultures and to affirm the model performance. One imaging experiment with colonies obtained from two serial dilutions was performed. A total of six agar plates of mixed cultures were prepared, where O45, O111 and O121 serogroups that were relatively easy to differentiate were inoculated into all six plates and then each of O26, O103 and O145 serogroups was added into the mixture of the three common bacterial cultures. The number of mixed colonies grown after 24-h incubation was 331 and the number of pixels associated with the grown colonies was 16,379. The best model found from this validation study was based on pre-processing with standard normal variate and detrending, first derivative, spectral smoothing, and k-nearest neighbor classification (kNN, k = 3) of scores in the principal component subspace spanned by 12 principal components. The results showed 95 % overall detection accuracy at pixel level and 97 % at colony level. The developed model was proven to be still valid even for the independent validation samples although the size of a validation set was small and only one experiment was performed. This study was an important first step in validating and updating multivariate classification models for rapid screening of ground beef samples contaminated by non-O157 STEC pathogens using hyperspectral imaging.

A fault magnitude based strategy for effective fault classification

A common approach in fault diagnosis is monitoring the deviations of measured variables from the values at normal operations to identify the root causes of faults. When the number of conceivable faults is larger than that of predictive variables, conventional approaches can yield ambiguous diagnosis results including multiple fault candidates. To address the issue, this work proposes a fault magnitude based strategy. Signed digraph is first used to identify qualitative relationships between process variables and faults. Empirical models for predicting process variables under assumed faults are then constructed with support vector regression (SVR). Fault magnitude data are projected onto principal components subspace, and the mapping from scores to fault magnitudes is learned via SVR. This model can estimate fault magnitudes and discriminate a true fault among multiple candidates when different fault magnitudes yield distinguishable responses in the monitored variables. The efficacy of the proposed approach is illustrated on an actuator benchmark problem.

A Principal Component Analysis Based Fault Detection Method in Etch Process of Semiconductor Manufacturing

A Principal Component Analysis based Fault Detection method is proposed here to detect faults in etch process of semiconductor manufacturing. The main idea of this method is to calculate the loading vector and build the fault detection model according to training data. Using this model, the main information of fault data can be obtained immediately and easily. Also the principal component subspace and residual subspace can be constructed. Then, faults are detected by calculating Squared Prediction Error. Finally, an industrial example of Lam 9600 TCP metal etcher at Texas Instruments is used to demonstrate the performance of the proposed PCA-based method in fault detection, and the results show that it has such advantages as simple algorithm and low time cost, thus especially adapts to the real time fault detection of semiconductor manufacturing.

CS Decomposition Based Bayesian Subspace Estimation

In numerous applications, it is required to estimate the principal subspace of the data, possibly from a very limited number of samples. Additionally, it often occurs that some rough knowledge about this subspace is available and could be used to improve subspace estimation accuracy in this case. This is the problem we address herein and, in order to solve it, a Bayesian approach is proposed. The main idea consists of using the CS decomposition of the semi-orthogonal matrix whose columns span the subspace of interest. This parametrization is intuitively appealing and allows for non informative prior distributions of the matrices involved in the CS decomposition and very mild assumptions about the angles between the actual subspace and the prior subspace. The posterior distributions are derived and a Gibbs sampling scheme is presented to obtain the minimum mean-square distance estimator of the subspace of interest. Numerical simulations and an application to real hyperspectral data assess the validity and the performances of the estimator.

Stable Subspace Tracking Algorithm Based on a Signed URV Decomposition

Subspace estimation and tracking are of fundamental importance in many signal processing algorithms. The class of “Schur subspace estimators” provides a complete parametrization of all “principal subspace estimates,” defined as the column spans of corresponding low-rank matrix approximants that lie within a specified 2-norm distance of a given matrix. The parametrization is found in terms of a two-sided hyperbolic decomposition (Hyperbolic URV, or HURV), which can be computed using hyperbolic rotations. Unfortunately, such rotations are commonly associated with numerical instabilities.In this paper, we present a numerically stable, non-iterative algorithm to compute the HURV, called the Signed URV (SURV) algorithm. We show that this algorithm implicitly imposes certain constraints on the HURV such that important norm bounds that guarantee stability are satisfied. The constraints also restrict the parametrization of the subspace estimate such that it becomes close to the principal subspace provided by the SVD (which is a special case within this class). The complexity of the algorithm is of the same order as that of a QR update. Updating and downdating are of the same complexity and are both numerically stable. SURV is proven to provide rank estimates consistent with the SVD with the same rank threshold. It can replace an SVD where only subspace estimation is needed. Typical applications would e.g. be the detection of the number of signals in array signal processing, and subspace estimation for source separation and interference mitigation, such as the first step in MUSIC and ESPRIT-type algorithms. Simulation results demonstrate the numerical stability and confirm that this algorithm provides exact rank estimates and good principal subspace estimates as compared to the SVD.