Sparse Principal Component Analysis Method Research Articles

A New Basis for Sparse Principal Component Analysis

Previous versions of sparse principal component analysis (PCA) have presumed that the eigen-basis (a p × k matrix) is approximately sparse. We propose a method that presumes the p × k matrix becomes approximately sparse after a k × k rotation. The simplest version of the algorithm initializes with the leading k principal components. Then, the principal components are rotated with an k × k orthogonal rotation to make them approximately sparse. Finally, soft-thresholding is applied to the rotated principal components. This approach differs from prior approaches because it uses an orthogonal rotation to approximate a sparse basis. One consequence is that a sparse component need not to be a leading eigenvector, but rather a mixture of them. In this way, we propose a new (rotated) basis for sparse PCA. In addition, our approach avoids “deflation” and multiple tuning parameters required for that. Our sparse PCA framework is versatile; for example, it extends naturally to a two-way analysis of a data matrix for simultaneous dimensionality reduction of rows and columns. We provide evidence showing that for the same level of sparsity, the proposed sparse PCA method is more stable and can explain more variance compared to alternative methods. Through three applications—sparse coding of images, analysis of transcriptome sequencing data, and large-scale clustering of social networks, we demonstrate the modern usefulness of sparse PCA in exploring multivariate data. An R package, epca, and the supplementary materials for this article are available online.

Exactly Uncorrelated Sparse Principal Component Analysis

Sparse principal component analysis (PCA) aims to find principal components as linear combinations of a subset of the original input variables without sacrificing the fidelity of the classical PCA. Most existing sparse PCA methods produce correlated sparse principal components. We argue that many applications of PCA prefer uncorrelated principal components. However, handling sparsity and uncorrelatedness properties in a sparse PCA method is nontrivial. This article proposes an exactly uncorrelated sparse PCA method named EUSPCA, whose formulation is motivated by original views and motivations of PCA as advocated by Pearson and Hotelling. EUSPCA is a non-smooth constrained non-convex manifold optimization problem. We solve it by combining augmented Lagrangian and non-monotone proximal gradient methods. We observe that EUSPCA produces uncorrelated components and maintains a similar or better level of fidelity based on adjusted total variance through simulated and real data examples. In contrast, existing sparse PCA methods produce significantly correlated components. Supplemental materials for this article are available online.

A critical assessment of sparse PCA (research): why (one should acknowledge that) weights are not loadings

Principal component analysis (PCA) is an important tool for analyzing large collections of variables. It functions both as a pre-processing tool to summarize many variables into components and as a method to reveal structure in data. Different coefficients play a central role in these two uses. One focuses on the weights when the goal is summarization, while one inspects the loadings if the goal is to reveal structure. It is well known that the solutions to the two approaches can be found by singular value decomposition; weights, loadings, and right singular vectors are mathematically equivalent. What is often overlooked, is that they are no longer equivalent in the setting of sparse PCA methods which induce zeros either in the weights or the loadings. The lack of awareness for this difference has led to questionable research practices in sparse PCA. First, in simulation studies data is generated mostly based only on structures with sparse singular vectors or sparse loadings, neglecting the structure with sparse weights. Second, reported results represent local optima as the iterative routines are often initiated with the right singular vectors. In this paper we critically re-assess sparse PCA methods by also including data generating schemes characterized by sparse weights and different initialization strategies. The results show that relying on commonly used data generating models can lead to over-optimistic conclusions. They also highlight the impact of choice between sparse weights versus sparse loadings methods and the initialization strategies. The practical consequences of this choice are illustrated with empirical datasets.

True sparse PCA for reducing the number of essential sensors in virtual metrology

In the semiconductor industry, virtual metrology (VM) is a cost-effective and efficient technique for monitoring the processes from one wafer to another. This technique is implemented by generating a predictive model that uses real-time data from equipment sensors in conjunction with measured wafer quality characteristics. Before establishing a prediction model for the VM system, appropriate selection of relevant input variables should be performed to maintain the efficiency of subsequent analyses considering the large dimensionality of the sensor data inputs. However, wafer production processes usually employ multiple sensors, which leads to cost escalations. Herein, we propose a variant of the sparse principal component analysis (PCA) called true sparse PCA (TSPCA). The proposed method uses a small number of input variables in the first few principal components. The main contribution of the proposed TSPCA is reducing the number of essential sensors. Our experimental results demonstrate that compared to the existing sparse PCA methods, the proposed approach can reduce the number of sensors required while explaining an approximately equivalent amount of variance.

Speckle-noise filtering based on non-local mean sparse principal component analysis method

Speckle-noise filtering is an extensive and key process in coherent interferometric techniques to obtain important and accurate information from the recorded interferogram or fringe pattern. The speckle-noise inherent to these interferograms, recorded by digital image sensors in these optical techniques, is eliminated by an appropriate image processing method. Since the beginning and further development of these techniques, a wide variety of speckle noise filtering methods and techniques have been proposed. The present research work aims to exploit the performance of the non-local sparse principal component analysis (NLS-PCA) method for speckle noise reduction as applied to the interferometric fringe patterns obtained from digital holographic and digital speckle pattern interferometric techniques. The performance of the NLS-PCA is evaluated on numerical simulations using different types of speckle fringe patterns resulting in the method being highly effective in filtering the speckle noise and providing superior results evaluated in terms of peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), Edge preservation Index (EPI), and Equivalent Number of look (ENL), in comparison to widely known methods such as windowed Fourier transform method, Lee filter, Weiner filter. Furthermore, the proposed method maintains the finer details and preserves the fringe edges effectively. The performance of the NLS-PCA method is also demonstrated on experimental speckle fringe patterns and digital holograms.

Interpretable Dimension Reduction for MRI Channel Suppression.

Channel suppression can reduce the redundant information in multiple channel receiver coils and accelerate reconstruction speed to meet real-time imaging requirements. The principal component analysis has been used for channel suppression, but it is difficult to be interpreted because all channels contribute to principal components. Furthermore, the importance of interpretability in machine learning has recently attracted increasing attention in radiology. To improve the interpretability of PCA-based channel suppression, a sparse PCA method is proposed to reduce the most coils' loadings to be zero. Channel suppression is formulated as solving a nonlinear eigenvalue problem using the inverse power method instead of the direct matrix decomposition. Experimental results of in vivo data show that the sparse PCA-based channel suppression not only improves the interpretability with sparse channels, but also improves reconstruction quality compared to the standard PCA-based reconstruction with the similar reconstruction time.

Using ℓ1-Relaxation and Integer Programming to Obtain Dual Bounds for Sparse PCA

Dual Bounds of Sparse Principal Component Analysis Sparse principal component analysis (PCA) is a widely used dimensionality reduction tool in machine learning and statistics. Compared with PCA, sparse PCA enhances the interpretability by incorporating a sparsity constraint. However, unlike PCA, conventional heuristics for sparse PCA cannot guarantee the qualities of obtained primal feasible solutions via associated dual bounds in a tractable fashion without underlying statistical assumptions. In “Using L1-Relaxation and Integer Programming to Obtain Dual Bounds for Sparse PCA,” Santanu S. Dey, Rahul Mazumder, and Guanyi Wang present a convex integer programming (IP) framework of sparse PCA to derive dual bounds. They show the worst-case results on the quality of the dual bounds provided by the convex IP. Moreover, the authors empirically illustrate that the proposed convex IP framework outperforms existing sparse PCA methods of finding dual bounds.

Sparse principal component analysis using bootstrap method

Sparse principal component analysis (SPCA) gives a sparse description of the loading matrix. In order to determine the “0” values in the loading matrix, often artificial thresholds are set on the loading values. To resolve these issues, two methods are proposed in this article to calculate the confidence intervals of the loading values. The first method is based on a resampling technique, while the second method estimates the error variance of data to calculate confidence intervals of the loadings. The position and number of non-zero loadings (NNZL) for the PCs are chosen based on a hypothesis test for “0”. Both methods lead to sparse structures of PCs. The fault detection and diagnosis performance of the proposed SPCA techniques are compared with the traditional PCA and three widely used SPCA methods for the benchmark continuous stirred tank heater (CSTH) process. The outcomes indicate that the proposed approaches perform better than the traditional PCA and benchmark SPCA (i.e., AV) method in fault detection and diagnosis.

Tensor sparse PCA and face recognition: a novel approach

Face recognition is the important field in machine learning and pattern recognition research area. It has a lot of applications in military, finance, public security, to name a few. In this paper, the combination of the tensor sparse PCA with the nearest-neighbor method (and with the kernel ridge regression method) will be proposed and applied to the face dataset. Experimental results show that the combination of the tensor sparse PCA with any classification system does not always reach the best accuracy performance measures. However, the accuracy of the combination of the sparse PCA method and one specific classification system is always better than the accuracy of the combination of the PCA method and one specific classification system and is always better than the accuracy of the classification system itself.

Lymphocyte-Monocyte-Neutrophil Index: A Predictor of Severity of Coronavirus Disease 2019 Patients Produced by Sparse Principal Component Analysis

Background: It is important to recognize severe ill coronavirus disease 2019 (COVID-19) patients from moderate ones in order to save more lives. We attempted to present an predictor for disease severity from clinical laboratory markers using sparse principal component analysis method. Methods: Forty-four clinical characteristics and laboratory markers of 82 COVID-19 patients (Hefei cohort) from the First Affiliated Hospital of University of Science and Technology of China (USTC) were analyzed retrospectively and sparse principal component analysis (SPCA) was performed to examine the correlation between the markers and extract relevant features. The controlling parameter alpha of SPCA was adjusted for better variable selection. Then the produced principal components (PCs) by SPCA were subjected to multivariate logistic regression for disease severity prediction, and the significant PCs were selected. Then, a Lymphocyte-Monocyte-Neutrophil index (LMN index) was deduced from the significant PCs and was used for disease severity prediction. Furthermore, an independent cohort including 169 COVID-19 patients (Nanchang Cohort) from the First Affiliated Hospital of NanChang University was used as a validation dataset and prediction efficiency of LMN index and classical clinical markers were also evaluated. Findings: Using SPCA, the first to thirteenth PCs accounted for 81·7% of the cumulative proportion variance of the original 44 clinical characteristics and laboratory markers. Multivariate logistic regression revealed the PC1 was significantly associated with disease severity with odds ratio of 74272·28 (623·83 - 178483250). When the controlling parameter alpha was adjusted to 0·001, the PC1 is only dependent on five laboratory markers: lymphocyte count (LYM), lymphocyte percentage (LYM%), neutrophil count (NEU), monocyte count (MONO) ,and serum phosphorus. LMN index determined by LYM, LYM%, NEU ,and MONO was deduced from the PC1 and significant relationships were investigated between LMN indices with age, comorbidity status and CD4+ ,and CD8 T lymphocyte counts. More important, during hospitalization, LMN indices decreased obviously as treatment takes effect, and they declined more sharply for mild ill COVID-19 patients compared with those of severe ill ones. When used to predict disease progression, the LMN index gave the accuracy of 0·780 and 0·760 in the training data (Hefei cohort) and the independent validation data (Nanchang Cohort) respectively, which was more efficient than classical clinical markers. Interpretation: Using SPCA method, the LMN index determined by four blood routine test markers was deduced. It showed robust disease severity prediction efficiency of COVID-19 patients and have the potential for clinical applications. Funding Statement: Fundamental Research Funds for the Central Universities of China(No. YD9110002001). Declaration of Interests: XLM reports grants from Fundamental Research Funds for the Central Universities of China. All other authors declare no competing interests. Ethics Approval Statement: This study was approved by the Ethics Committee of the First Affiliated Hospital of USTC and the Ethics Committee of the First Affiliated Hospital of NanChang University.

Weighted sparse principal component analysis

Sparse principal component analysis (SPCA) has been shown to be a fruitful method for the analysis of high-dimensional data. So far, however, no method has been proposed that allows to assign elementwise weights to the matrix of residuals, although this may have several useful applications. We propose a novel SPCA method that includes the flexibility to weight at the level of the elements of the data matrix. The superior performance of the weighted SPCA approach compared to unweighted SPCA is shown for data simulated according to the prevailing multiplicative-additive error model. In addition, applying weighted SPCA to genomewide transcription rates obtained soon after vaccination, resulted in a biologically meaningful selection of variables with components that are associated to the measured vaccine efficacy. The MATLAB implementation of the weighted sparse PCA method is freely available from https://github.com/katrijnvandeun/WSPCA.

Sparse principal component based high-dimensional mediation analysis

Causal mediation analysis aims to quantify the intermediate effect of a mediator on the causal pathway from treatment to outcome. When dealing with multiple mediators, which are potentially causally dependent, the possible decomposition of pathway effects grows exponentially with the number of mediators. An existing approach incorporated the principal component analysis (PCA) to address this challenge based on the fact that the transformed mediators are conditionally independent given the orthogonality of the principal components (PCs). However, the transformed mediator PCs, which are linear combinations of original mediators, can be difficult to interpret. A sparse high-dimensional mediation analysis approach is proposed which adopts the sparse PCA method to the mediation setting. The proposed approach is applied to a task-based functional magnetic resonance imaging study, illustrating its ability to detect biologically meaningful results related to an identified mediator.

Robust High-Order Manifold Constrained Sparse Principal Component Analysis for Image Representation

In order to efficiently utilize the information in the data and eliminate the negative effects of outliers in the principal component analysis (PCA) method, in this paper, we propose a novel robust sparse PCA method based on maximum correntropy criterion (MCC) with high-order manifold constraints called the RHSPCA. Compared with the traditional PCA methods, the proposed RHSPCA has the following benefits: 1) the MCC regression term is more robust to outliers than the MSE-based regression term; 2) thanks to the high-order manifold constraints, the low-dimensional representations can preserve the local relations of the data and greatly improve the clustering and classification performance for image processing tasks; and 3) in order to further counteract the adverse effects of outliers, the MCC-based samples’ mean is proposed to better centralize the data. We also propose a new solver based on the half-quadratic technique and accelerated block coordinate update strategy to solve the RHSPCA model. Extensive experimental results show that the proposed method can outperform the state-of-the-art robust PCA methods on a variety of image processing tasks, including reconstruction, clustering, and classification, on outliers contaminated datasets.

Kernel Principal Component Analysis Allowing Sparse Representation and Sample Selection

Kernel principal component analysis (KPCA) is a kernelized version of principal component analysis (PCA). A kernel principal component is a superposition of kernel functions. Due to the number of kernel functions equals the number of samples, each component is not a sparse representation. Our purpose is to sparsify coefficients expressing in linear combination of kernel functions, two types of sparse kernel principal component are proposed in this paper. The method for solving sparse problem comprises two steps: (a) we start with the Pythagorean theorem and derive an explicit regression expression of KPCA and (b) two types of regularization $l_1$-norm or $l_{2,1}$-norm are added into the regression expression in order to obtain two different sparsity form, respectively. As the proposed objective function is different from elastic net-based sparse PCA (SPCA), the SPCA method cannot be directly applied to the proposed cost function. We show that the sparse representations are obtained in its iterative optimization by conducting an alternating direction method of multipliers. Experiments on toy examples and real data confirm the performance and effectiveness of the proposed method.

Sparse principal component analysis with missing observations

Principal component analysis (PCA) is a commonly used statistical method in a wide range of applications. However, it does not work well when the number of features is larger than the sample size. We consider the estimation of the sparse principal subspace in the high dimensional setting with missing data motivated by the analysis of single-cell RNA sequence data. We propose a two step estimation procedure, and establish the rates of convergence for estimating the principal subspace. Simulated examples with various missing mechanisms show its competitive performance compared to existing sparse PCA methods. We apply the method to single-cell data and show that the proposed method can better distinguish cell types than other PCA methods.

Projection sparse principal component analysis: An efficient least squares method

We propose a new sparse principal component analysis (SPCA) method in which the solutions are obtained by projecting the full cardinality principal components onto subsets of variables. The resulting components are guaranteed to explain a given proportion of variance. The computation of these solutions is very efficient. The proposed method compares well with the optimal least squares sparse components. We show that other SPCA methods fail to identify the best sparse approximations of the principal components and explain less variance than our solutions. We illustrate and compare our method with others with extensive simulations and with the analysis of the computational results for nine datasets of increasing dimensions up to 16,000 variables.

Sparse Principal Component Analysis With Preserved Sparsity Pattern.

Principal component analysis (PCA) is widely used for feature extraction and dimension reduction in pattern recognition and data analysis. Despite its popularity, the reduced dimension obtained from the PCA is difficult to interpret due to the dense structure of principal loading vectors. To address this issue, several methods have been proposed for sparse PCA, all of which estimate loading vectors with few non-zero elements. However, when more than one principal component is estimated, the associated loading vectors do not possess the same sparsity pattern. Therefore, it becomes difficult to determine a small subset of variables from the original feature space that have the highest contribution in the principal components. To address this issue, an adaptive block sparse PCA method is proposed. The proposed method is guaranteed to obtain the same sparsity pattern across all principal components. Experiments show that applying the proposed sparse PCA method can help improve the performance of feature selection for image processing applications. We further demonstrate that our proposed sparse PCA method can be used to improve the performance of blind source separation for functional magnetic resonance imaging data.

Sparse supervised principal component analysis (SSPCA) for dimension reduction and variable selection

Principal component analysis (PCA)11PCA:principal component analysis, SPCA: sparse PCA, SSPCA: sparse supervised PCA, SPLS: sparse partial least squares, PMD: penalized matrix decomposition, SVD: singular value decomposition, HSIC: Hilbert Schmidt independence criterion, RKHS: reproducing kernel Hilbert space, SIMPLS: statistically inspired modification of PLS, SVM: support vector machine, CV: cross validation, RBF: radial basis function, RMSE: root mean square error, ROI:region of interest, NIR: near infrared, SSC: solvable solid content, KNN: K nearest neighbour.is one of the main unsupervised pre-processing methods for dimension reduction. When the training labels are available, it is worth using a supervised PCA strategy. In cases that both dimension reduction and variable selection are required, sparse PCA (SPCA) methods are preferred. In this paper, a sparse supervised PCA (SSPCA) method is proposed for pre-processing. This method is appropriate especially in problems where, a high dimensional input necessitates the use of a sparse method and a target label is also available to guide the variable selection strategy. Such a method is valuable in many Engineering and scientific problems, when the number of training samples is also limited. The Hilbert Schmidt Independence Criteria (HSIC) is used to form an objective based on minimization of a loss function and an L1 norm is used for regularization of the Eigen vectors. While the proposed objective function allows a sparse low rank solution for both linear and non-linear relationships between the input and response matrices, other similar methods in this case are only based on a linear model. The objective is solved based on penalized matrix decomposition (PMD) algorithm. We compare the proposed method with PCA, PMD-based SPCA and supervised PCA. In addition, SSPCA is also compared with sparse partial least squares (SPLS), due to the similarity between the two objective functions. Experimental results from the simulated as well as real data sets show that, SSPCA provides an appropriate trade-off between accuracy and sparsity. Comparisons show that, in terms of sparsity, SSPCA performs the highest level of variable reduction and also, in terms of accuracy it is one of the most successful methods. Therefore, the Eigen vectors found by SSPCA can be used for feature selection in various high dimensional problems.

Incorporating biological information in sparse principal component analysis with application to genomic data

BackgroundSparse principal component analysis (PCA) is a popular tool for dimensionality reduction, pattern recognition, and visualization of high dimensional data. It has been recognized that complex biological mechanisms occur through concerted relationships of multiple genes working in networks that are often represented by graphs. Recent work has shown that incorporating such biological information improves feature selection and prediction performance in regression analysis, but there has been limited work on extending this approach to PCA. In this article, we propose two new sparse PCA methods called Fused and Grouped sparse PCA that enable incorporation of prior biological information in variable selection.ResultsOur simulation studies suggest that, compared to existing sparse PCA methods, the proposed methods achieve higher sensitivity and specificity when the graph structure is correctly specified, and are fairly robust to misspecified graph structures. Application to a glioblastoma gene expression dataset identified pathways that are suggested in the literature to be related with glioblastoma.ConclusionsThe proposed sparse PCA methods Fused and Grouped sparse PCA can effectively incorporate prior biological information in variable selection, leading to improved feature selection and more interpretable principal component loadings and potentially providing insights on molecular underpinnings of complex diseases.

Compressive sparse principal component analysis for process supervisory monitoring and fault detection

This paper presents a novel sparse principal component analysis method, which is named the compressive sparse principal component analysis (CSPCA). CSPCA ensures that the effects of principal components (PCs) with small scores (eigenvalues/variances) on monitoring performance are taken into account during deriving the first PCs, and measurements are adaptively compressed and partially reconstructed without prior knowledge of data sparsity. The proposed method employs the strategy of screening, reconstructing, and detecting for process supervisory monitoring. Data-screening algorithm is employed to sift out data with essential characteristics of abnormal situations at the screening stage. Data selected are adaptively compressed, and abnormal features are highlighted by the partial reconstruction algorithm at the reconstructing stage. A new SPCA is developed by introducing L2,1-norm to replace the usual norm in the traditional SPCA, and is employed to analyse data reconstructed at the detecting stage. The effectiveness of the compressive sparse principal component analysis is evaluated on the Pitprops data set and the Tennessee-Eastman process with promising results.