Principal Component Analysis Problem Research Articles

Solving the missing value problem in PCA by Orthogonalized-Alternating Least Squares (O-ALS)

Dealing with missing data poses a challenge in Principal Component Analysis (PCA) since the most common algorithms are not designed to handle them. Several approaches have been proposed to solve the missing value problem in PCA, such as Imputation based on SVD (I-SVD), where missing entries are filled by imputation and updated in every iteration until convergence of the PCA model, and the adaptation of the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm, able to work skipping the missing entries during the least-squares estimation of scores and loadings. However, some limitations have been reported for both approaches. On the one hand, convergence of the I-SVD algorithm can be very slow for data sets with a high percentage of missing data. On the other hand, the orthogonality properties among scores and loadings might be lost when using NIPALS.To solve these issues and perform PCA of data sets with missing values without the need of imputation steps, a novel algorithm called Orthogonalized-Alternating Least Squares (O-ALS) is proposed. The O-ALS algorithm is an alternating least-squares algorithm that estimates the scores and loadings subject to the Gram-Schmidt orthogonalization constraint. The way to estimate scores and loadings is adapted to work only with the available information.In this study, the performance of O-ALS is tested and compared with NIPALS and I-SVD in simulated data sets and in a real case study. The results show that O-ALS is an accurate and fast algorithm to analyze data with any percentage and distribution pattern of missing entries, being able to provide correct scores and loadings in cases where I-SVD and NIPALS do not perform satisfactorily.

Free Energy Wells and Overlap Gap Property in Sparse PCA

AbstractWe study a variant of the sparse PCA (principal component analysis) problem in the “hard” regime, where the inference task is possible yet no polynomial‐time algorithm is known to exist. Prior work, based on the low‐degree likelihood ratio, has conjectured a precise expression for the best possible (subexponential) runtime throughout the hard regime. Following instead a statistical physics‐inspired point of view, we show bounds on the depth of free energy wells for various Gibbs measures naturally associated to the problem. These free energy wells imply hitting time lower bounds that corroborate the low‐degree conjecture: we show that a class of natural MCMC (Markov chain Monte Carlo) methods (with worst‐case initialization) cannot solve sparse PCA with less than the conjectured runtime. These lower bounds apply to a wide range of values for two tuning parameters: temperature and sparsity misparametrization. Finally, we prove that the overlap gap property (OGP), a structural property that implies failure of certain local search algorithms, holds in a significant part of the hard regime. © 2022 Wiley Periodicals, Inc.

FAST-PCA: A Fast and Exact Algorithm for Distributed Principal Component Analysis

Principal Component Analysis (PCA) is a fundamental data preprocessing tool in the world of machine learning. While PCA is often thought of as a dimensionality reduction method, the purpose of PCA is actually two-fold: dimension reduction and uncorrelated feature learning. Furthermore, the enormity of the dimensions and sample size in the modern day datasets have rendered the centralized PCA solutions unusable. In that vein, this paper reconsiders the problem of PCA when data samples are distributed across nodes in an arbitrarily connected network. While a few solutions for distributed PCA exist, those either overlook the uncorrelated feature learning aspect of the PCA, tend to have high communication overhead that makes them inefficient and/or lack ‘exact’ or ‘global’ convergence guarantees. To overcome these aforementioned issues, this paper proposes a distributed PCA algorithm termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FAST-PCA (Fast and exAct diSTributed PCA)</i> . The proposed algorithm is efficient in terms of communication and is proven to converge linearly and exactly to the principal components, leading to dimension reduction as well as uncorrelated features. The claims are further supported by experimental results.

Graph Convolution RPCA With Adaptive Graph.

Principal component analysis (PCA) is warmly welcomed in dimensionality reduction and its applications. Due to the high sensitivity of PCA to outliers, a series of PCA methods are proposed to enhance the robustness of PCA. Besides, the representation ability of the existing PCA methods has limitations as well. To enhance the robustness and representation ability of robust PCA, we elaborate a novel Graph Convolution Robust PCA method (GRPCA) to incorporate the manifold structure into PCA. It constructs a sparse graph based on the local connectivity structure of samples. Graph auto-encoder is utilized to solve the robust PCA problem under the low-rank and sparse constraints. With the dual-decoder, GRPCA learns the low-dimensional embeddings that reconstruct the manifold structure and low-rank approximation simultaneously. Furthermore, since the graph suffers from misconnection triggered by occlusions, the local connectivity structure of low-dimensional embeddings is utilized to modify the graph. Our proposed method excels in both the clustering of low-dimensional embeddings and the low-rank recovery. Lastly, extensive experiments conducted on six real-world datasets demonstrated the efficiency and superiority of the proposed GRPCA.

A linearly convergent algorithm for distributed principal component analysis

Principal Component Analysis (PCA) is the workhorse tool for dimensionality reduction in this era of big data. While often overlooked, the purpose of PCA is not only to reduce data dimensionality, but also to yield features that are uncorrelated. Furthermore, the ever-increasing volume of data in the modern world often requires storage of data samples across multiple machines, which precludes the use of centralized PCA algorithms. This paper focuses on the dual objective of PCA, namely, dimensionality reduction and decorrelation of features, but in a distributed setting. This requires estimating the eigenvectors of the data covariance matrix, as opposed to only estimating the subspace spanned by the eigenvectors, when data is distributed across a network of machines. Although a few distributed solutions to the PCA problem have been proposed recently, convergence guarantees and/or communications overhead of these solutions remain a concern. With an eye towards communications efficiency, this paper introduces a feedforward neural network-based one time-scale distributed PCA algorithm termed Distributed Sanger’s Algorithm (DSA) that estimates the eigenvectors of the data covariance matrix when data is distributed across an undirected and arbitrarily connected network of machines. Furthermore, the proposed algorithm is shown to converge linearly to a neighborhood of the true solution. Numerical results are also provided to demonstrate the efficacy of the proposed solution.

Constrained principal component analysis with stochastically ordered scores for high-dimensional mass spectrometry data

In this paper, we consider a constrained principal component analysis (PCA) for the projection of high-dimensional samples from different groups to a lower-dimensional space for which the principal scores are stochastically ordered over the groups. We express the problem as the minimization of a constrained biconvex problem and develop an iterative algorithm to solve it. We numerically show that the solution to our constrained PCA problem approximately rotates the principal coordinates of the ordinary PCA to achieve ordered scores. Consequently, our approach significantly improves the scores and the corresponding loading matrix compared to the original PCA if their true values are ordered over groups. We finally apply our method to two data examples: (i) the direct infusion multiple reaction monitoring mass spectrometry (DI-MRM-MS) data of white rice to verify the authenticity of adulterated Japonica rice and (ii) high-dimensional lipidomics data from ultra-performance liquid chromatography-mass spectrometry/mass spectrometry (UPLC-MS/MS) analysis of patients with liver diseases.

A Topological Approach of Principal Component Analysis

Large datasets are increasingly widespread in many disciplines. The exponential growth of data requires the development of more data analysis methods in order to process information more efficiently. In order to better visualize the data, many methods such as Principal Component Analysis (PCA) and MultiDimensional Scaling (MDS) allow to extract a low-dimensional structure from high-dimensional data set. The proposed approach, called Topological Principal Component Analysis (TPCA), is a multidimensional descriptive method witch studies a homogeneous set of continuous variables defined on the same set of individuals. It is a topological method of data analysis that consists of comparing and classifying proximity measures from among some of the most widely used proximity measures for continuous data. Proximity measures play an important role in many areas of data analysis, the results strongly depend on the proximity measure chosen. So, among the many existing measures, which one is most useful? Are they all equivalent? How to identify the one that is most appropriate to analyze the correlation structure of a set of quantitative variables. TPCA proposes an appropriate adjacency matrix associated to an unknown proximity measure according to the data under consideration, then analyzes and visualizes, with graphic representations, the relationship structure of the variables relating to, the well known PCA problem. Its uses the concept of neighborhood graphs and compares a set of proximity measures for continuous data which can be more-or-less equivalent a topological equivalence criterion between two proximity measures is defined and statistically tested according to the topological correlation between the variables considered. An example on real data illustrates the proposed approach.

Real-Time Principal Component Analysis

We propose a variant of Principal Component Analysis (PCA) that is suited for real-time applications. In the real-time version of the PCA problem, we maintain a window over the most recent data and project every incoming row of data into a lower-dimensional subspace, which we generate as the output of the model. The goal is to reduce the reconstruction error of the output from the input and to retain major components pertaining to previous distributions of the data. We use the reconstruction error as the termination criteria to update the eigenspace as new data arrives. We then propose two variants of this algorithm that are progressively more time efficient. To verify whether our proposed model can capture the essence of the changing distribution of large datasets in real time, we have implemented the algorithms and compared performance against carefully designed simulations that change distributions of data sources over time in a controllable manner. Furthermore, we have demonstrated that proposed algorithms can capture the changing distributions of real-life datasets by running simulations on datasets from a variety of real-time applications, e.g., localization, activity recognition, customer expenditure, and so forth. Results show that straightforward modifications to convert PCA to use a sliding window of datasets do not work because of the difficulties associated with determination of optimal window size. Instead, we propose algorithmic enhancements that rely on spectral analysis to improve dimensionality reduction. Results show that our methods can successfully capture the changing distribution of data in a real-time scenario, thus enabling real-time PCA.

Skull similarity comparison based on SPCA

As an important biological feature of the human, the skull plays an active role in assisting criminal investigation, victim identification, etc. This paper proposes a method based on Sparse Principal Component Analysis (SPCA) for comparison of skull similarity. Compared with Principal Component Analysis (PCA), SPCA can not only effectively reduce the data dimension,but also produce sparse principal components which are easy to explain. Each principal component of PCA is a linear combination of all original variables. It’s difficulty in explaining the corresponding relationship between principal components and features. SPCA makes the loadings sparse, and thus highlights the main part of the principal component, which can solve the problem of PCA that has difficulty in explaining the result. The experimental results show that the dimensionality reduction data by SPCA is superior to PCA in the aspects of complexity, discrimination, stability, interpretability, and similarity evaluation. These indicate that the comparison of skull similarity based on SPCA is accurate and stable, which can provide an effective direction for improving the accuracy of craniofacial reconstruction and obtaining accurate reconstruction results.

Convergence analysis of Oja’s iteration for solving online PCA with nonzero-mean samples

Principal component analysis (PCA) is one of the most popular multivariate data analysis techniques for dimension reduction and data mining, and is widely used in many fields ranging from industry and biology to finance and social development. When working on big data, it is of great necessity to consider the online version of PCA, in which only a small subset of samples could be stored. To handle the online PCA problem, Oja (1982) presented the stochastic power method under the assumption of zero-mean samples, and there have been lots of theoretical analysis and modified versions of this method in recent years. However, a common circumstance where the samples have nonzero mean is seldom studied. In this paper, we derive the convergence rate of a nonzero-mean version of Oja’s algorithm with diminishing stepsizes. In the analysis, we succeed in handling the dependency between each iteration, which is caused by the updated mean term for data centering. Furthermore, we verify the theoretical results by several numerical tests on both artificial and real datasets. Our work offers a way to deal with the top-1 online PCA when the mean of the given data is unknown.

Nonconvex Regularized Robust PCA Using the Proximal Block Coordinate Descent Algorithm

This work addresses the robust principal component analysis (PCA) problem using generalized nonoconvex regularization for low-rank and sparsity promotion. While the popular nuclear and $\ell _1$ -norm penalties have a bias problem, nonconvex regularization can alleviate the bias problem and can be expected to achieve better performance. In this paper, a proximal block coordinate descent (BCD) algorithm is used to efficiently solve the nonconvex regularized robust PCA problem. It is globally convergent under weak conditions. Further, for a popular class of penalties having discontinuous threshoding functions, we establish the convergence to a restricted strictly local minimizer and, also, a local linear convergence rate for the proximal BCD algorithm. Moreover, convergence to a local minimizer has been derived for hard-thresholding. Our result is the first on nonconvex robust PCA with established convergence to strictly local minimizer with local linear convergence rate. Numerical experiments have been provided to demonstrate the performance of the new algorithm.

Robust PCA Using Generalized Nonconvex Regularization

Recently, the robustification of principal component analysis (PCA) has attracted much research attention in numerous areas of science and engineering. The most popular and successful approach is to model the robust PCA problem as a low-rank matrix recovery problem in the presence of sparse corruption. With this model, the nuclear norm and $\ell _{1}$ -norm penalties are usually used for low-rank and sparsity promotion. Although the nuclear norm and $\ell _{1}$ -norm are favorable due to their convexity, they have a bias problem. In comparison, nonconvex penalties can be expected to yield better recovery performance. In this paper, we consider a formulation for robust PCA using generalized nonconvex penalties for low-rank and sparsity inducing. This nonconvex formulation is efficiently solved by a multi-block alternative direction method of multipliers (ADMM) algorithm. A sufficient condition for the convergence of this new ADMM algorithm has been derived. Furthermore, to address the important issue of nonconvex penalty selection, we have evaluated the new algorithm via numerical experiments in various low-rank and sparsity conditions. The results indicate that, “exact” recovery of the low-rank principle component can be achieved only by nonconvex regularization. MATLAB code is available at https://github.com/FWen/RPCA.git .

Penalized Orthogonal Iteration for Sparse Estimation of Generalized Eigenvalue Problem

ABSTRACTWe propose a new algorithm for sparse estimation of eigenvectors in generalized eigenvalue problems (GEPs). The GEP arises in a number of modern data-analytic situations and statistical methods, including principal component analysis (PCA), multiclass linear discriminant analysis (LDA), canonical correlation analysis (CCA), sufficient dimension reduction (SDR), and invariant co-ordinate selection. We propose to modify the standard generalized orthogonal iteration with a sparsity-inducing penalty for the eigenvectors. To achieve this goal, we generalize the equation-solving step of orthogonal iteration to a penalized convex optimization problem. The resulting algorithm, called penalized orthogonal iteration, provides accurate estimation of the true eigenspace, when it is sparse. Also proposed is a computationally more efficient alternative, which works well for PCA and LDA problems. Numerical studies reveal that the proposed algorithms are competitive, and that our tuning procedure works well. We demonstrate applications of the proposed algorithm to obtain sparse estimates for PCA, multiclass LDA, CCA, and SDR. Supplementary materials for this article are available online.

Three iteratively reweighted least squares algorithms for $$L_1$$ L 1 -norm principal component analysis

Principal component analysis (PCA) is often used to reduce the dimension of data by selecting a few orthonormal vectors that explain most of the variance structure of the data. $$L_1$$ PCA uses the $$L_1$$ norm to measure error, whereas the conventional PCA uses the $$L_2$$ norm. For the $$L_1$$ PCA problem minimizing the fitting error of the reconstructed data, we propose three algorithms based on iteratively reweighted least squares. We first develop an exact reweighted algorithm. Next, an approximate version is developed based on eigenpair approximation when the algorithm is near convergent. Finally, the approximate version is extended based on stochastic singular value decomposition. We provide convergence analyses, and compare their performance against benchmark algorithms in the literature. The computational experiment shows that the proposed algorithms consistently perform the best and the scalability is improved as we use eigenpair approximation and stochastic singular value decomposition.

Characterizing Variability of Modular Brain Connectivity with Constrained Principal Component Analysis.

Characterizing the variability of resting-state functional brain connectivity across subjects and/or over time has recently attracted much attention. Principal component analysis (PCA) serves as a fundamental statistical technique for such analyses. However, performing PCA on high-dimensional connectivity matrices yields complicated “eigenconnectivity” patterns, for which systematic interpretation is a challenging issue. Here, we overcome this issue with a novel constrained PCA method for connectivity matrices by extending the idea of the previously proposed orthogonal connectivity factorization method. Our new method, modular connectivity factorization (MCF), explicitly introduces the modularity of brain networks as a parametric constraint on eigenconnectivity matrices. In particular, MCF analyzes the variability in both intra- and inter-module connectivities, simultaneously finding network modules in a principled, data-driven manner. The parametric constraint provides a compact module-based visualization scheme with which the result can be intuitively interpreted. We develop an optimization algorithm to solve the constrained PCA problem and validate our method in simulation studies and with a resting-state functional connectivity MRI dataset of 986 subjects. The results show that the proposed MCF method successfully reveals the underlying modular eigenconnectivity patterns in more general situations and is a promising alternative to existing methods.

Satellite image fusion using fuzzy logic

Abstract Image fusion is a method of combining the Multispectral (MS) and Panchromatic (PAN) images into one image contains more information than any of the input. Image fusion aim is to decrease unknown and weaken common data in the fused output image at the same time improving necessary information. Fused images are helpful in various applications like, remote sensing, computer vision, biometrics, change detection, image analysis and image classification. Conventional fusion methods are having some side effects like assertive spatial information and uncertain color information is an usually the problem in PCA and wavelet transform based fusion is a computationally in depth process. In order to overcome these side effects and to propose alternative soft computing fusion approach for conventional fusion methods we exploit image fusion using fuzzy logic technique to fuse two source images obtained from different sensors to enhance both spectral and spatial information. The proposed work here further compared with two common fusion methods like, principal component analysis (PCA) and wavelet transform along with quality assessment metrics. Exploratory outputs demonstrated in order that fuzzy based image fusion technique can actively retains more information compared to PCA and wavelet transform approaches while enhancing the spatial and spectral resolution of the satellite images.

Rapid multivariate analysis of 3D ToF‐SIMS data: graphical processor units (GPUs) and low‐discrepancy subsampling for large‐scale principal component analysis

Principal component analysis (PCA) and other multivariate analysis methods have been used increasingly to analyse and understand depth‐profiles in XPS, AES and SIMS. For large images or three‐dimensional (3D) imaging depth‐profiles, PCA has been difficult to apply until now simply because of the size of the matrices of data involved. In a recent paper, we described two algorithms, random vector 1 (RV1) and random vector 2 (RV2), that improve the speed of PCA and allow datasets of unlimited size, respectively. In this paper, we now apply the RV2 algorithm to perform PCA on full 3D time‐of‐flight SIMS data for the first time without subsampling. The dataset we process in this way is a 128 × 128 pixel depth‐profile of 120 layers, each voxel having a 70 439 value mass spectrum associated with it. This forms over a terabyte of data when uncompressed and took 27 h to process using the RV2 algorithm using a conventional windows desktop personal computer (PC). While full PCA (e.g. using RV2) is to be preferred for final reports or publications, a much more rapid method is needed during analysis sessions to inform decisions on the next analytical step. We have therefore implemented the RV1 algorithm on a PC having a graphical processor unit (GPU) card containing 2880 individual processor cores. This increases the speed of calculation by a factor of around 4.1 compared with what is possible using a fast commercially available desktop PC having central processing units alone, and full PCA is performed in less than 7 s. The size of the dataset that can be processed in this way is limited by the size of the memory on the GPU card. This is typically sufficient for two‐dimensional images but not 3D depth‐profiles without sampling. We have therefore examined efficient sampling schemes that allow a good approximate solution to the PCA problem for large 3D datasets. We find that low‐discrepancy series such as Sobol series sampling gives more rapid convergence than random sampling, and we recommend such methods for routine use. Using the GPU and low‐discrepancy series together, we anticipate that any time‐of‐flight SIMS dataset, of whatever size, can be efficiently and accurately processed into PCA components in a maximum of around 10 s using a commercial PC with a widely available GPU card, although the longer RV2 approach is still to be preferred for the presentation of final results, such as in published papers. Copyright © 2016 The Authors Surface and Interface Analysis Published by John Wiley &amp; Sons Ltd

Memory Efficient PCA Methods for Large Group ICA.

Principal component analysis (PCA) is widely used for data reduction in group independent component analysis (ICA) of fMRI data. Commonly, group-level PCA of temporally concatenated datasets is computed prior to ICA of the group principal components. This work focuses on reducing very high dimensional temporally concatenated datasets into its group PCA space. Existing randomized PCA methods can determine the PCA subspace with minimal memory requirements and, thus, are ideal for solving large PCA problems. Since the number of dataloads is not typically optimized, we extend one of these methods to compute PCA of very large datasets with a minimal number of dataloads. This method is coined multi power iteration (MPOWIT). The key idea behind MPOWIT is to estimate a subspace larger than the desired one, while checking for convergence of only the smaller subset of interest. The number of iterations is reduced considerably (as well as the number of dataloads), accelerating convergence without loss of accuracy. More importantly, in the proposed implementation of MPOWIT, the memory required for successful recovery of the group principal components becomes independent of the number of subjects analyzed. Highly efficient subsampled eigenvalue decomposition techniques are also introduced, furnishing excellent PCA subspace approximations that can be used for intelligent initialization of randomized methods such as MPOWIT. Together, these developments enable efficient estimation of accurate principal components, as we illustrate by solving a 1600-subject group-level PCA of fMRI with standard acquisition parameters, on a regular desktop computer with only 4 GB RAM, in just a few hours. MPOWIT is also highly scalable and could realistically solve group-level PCA of fMRI on thousands of subjects, or more, using standard hardware, limited only by time, not memory. Also, the MPOWIT algorithm is highly parallelizable, which would enable fast, distributed implementations ideal for big data analysis. Implications to other methods such as expectation maximization PCA (EM PCA) are also presented. Based on our results, general recommendations for efficient application of PCA methods are given according to problem size and available computational resources. MPOWIT and all other methods discussed here are implemented and readily available in the open source GIFT software.

Mapping of Fe mineral potential by spatially weighted principal component analysis in the eastern Tianshan mineral district, China

Principal component analysis (PCA) is one of the commonly used methods to integrate multi-source geological data to enhance understanding of geo-information. Each principal component (PC) obtained by PCA reflects a different aspect of the geo-information contained in a dataset. Confined by statistical significance, some PCs are not acceptable for interpretation. In this paper, the same problem in PCA occurs in mapping potential areas of Fe mineralization in eastern Tianshan mineral district, China. By spatially weighting correlation coefficient matrixes between variables, a spatially weighted principal component analysis (SWPCA) can deal with the shortcoming of PCA, thus improving the statistical acceptability of eigenvectors and eigenvalues derived by ordinary PCA. Based on the geological model in the study area, a current weighting factor is defined to enhance the geo-information possessed by the ordinary PC1. Compared with the loading of input layers on ordinary PCA, SWPC1 shows more significant physical meaning than PC1. Meanwhile, remarkable increases on the eigenvalues of SWPC2 and SWPC3 are demonstrated to exist making these spatially weighted principal components more acceptable in a statistical sense. In comparison with both loadings and scores on ordinary PCs, the improved geo-information carried by SWPCs can help with better interpretations of the geological phenomena.

The Analysis of Multivariate Data Using Semi-Definite Programming

A model is presented for analyzing general multivariate data. The model puts as its prime objective the dimensionality reduction of the multivariate problem. The only requirement of the model is that the input data to the statistical analysis be a covariance matrix, a correlation matrix, or more generally a positive semi-definite matrix. The model is parameterized by a scale parameter and a shape parameter both of which take on non-negative values smaller than unity. We first prove a wellknown heuristic for minimizing rank and establish the conditions under which rank can be replaced with trace. This result allows us to solve our rank minimization problem as a Semi-Definite Programming (SDP) problem by a number of available solvers. We then apply the model to four case studies dealing with four well-known problems in multivariate analysis. The first problem is to determine the number of underlying factors in factor analysis (FA) or the number of retained components in principal component analysis (PCA). It is shown that our model determines the number of factors or components more efficiently than the commonly used methods. The second example deals with a problem that has received much attention in recent years due to its wide applications, and it concerns sparse principal components and variable selection in PCA. When applied to a data set known in the literature as the pitprop data, we see that our approach yields PCs with larger variances than PCs derived from other approaches. The third problem concerns sensitivity analysis of the multivariate models, a topic not widely researched in the sequel due to its difficulty. Finally, we apply the model to a difficult problem in PCA known as lack of scale invariance in the solutions of PCA. This is the problem that the solutions derived from analyzing the covariance matrix in PCA are generally different (and not linearly related to) the solutions derived from analyzing the correlation matrix. Using our model, we obtain the same solution whether we analyze the correlation matrix or the covariance matrix since the analysis utilizes only the signs of the correlations/covariances but not their values. This is where we introduce a new type of PCA, called Sign PCA, which we speculate on its applications in social sciences and other fields of science.